CN102880832B - A kind of implementation method of the system of the data magnanimity management under cluster - Google Patents
A kind of implementation method of the system of the data magnanimity management under cluster Download PDFInfo
- Publication number
- CN102880832B CN102880832B CN201210309450.XA CN201210309450A CN102880832B CN 102880832 B CN102880832 B CN 102880832B CN 201210309450 A CN201210309450 A CN 201210309450A CN 102880832 B CN102880832 B CN 102880832B
- Authority
- CN
- China
- Prior art keywords
- data
- implementation method
- node
- database
- management
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention belongs to computer realm, disclose the implementation method of the system that the data magnanimity under a kind of cluster manages, it is after calculating application program calculating operation completes, the data that calculating node has calculated are copied to management node by the script handling procedure of operation, stagger the time when copying out, carry out secondary copy by object-line thread management process, and revise data mode.The present invention ensure that the real-time of data, security, correctness, high efficiency.The present invention processed calculate isolated island, not all should process, design cycle leakage and the universal in-problem corresponding solution of result that postpones.And the present invention improves the stability of system, reliability.
Description
Technical field
The present invention relates to computer realm, the implementation method of the system of the data magnanimity management being specifically related under a kind of cluster.
Background technology
Management is the important leverage of IT system Recycle mechanism, and different information technoloy equipments has the management system of oneself.Particularly
Large-scale calculations data center, it is necessary to carry out the equipment such as operational management calculating, storage, network by the management system concentrated,
Can quickly respond business change with process data center, anomalous event, Continuous optimization.Under cloud computing environment
Data center, people are more and more higher to the requirement of computing power, in engineering design, aviation, bioscience, medical science, army
The calculating of the numerous areas such as thing also becomes increasingly complex, and scale is also becoming the growth of progression, and the hugest calculating task is right
Have no idea in single computer, therefore much all used ultra-large cluster to calculate, carrying out
In Large-scale parallel computing, the scheduling of calculating task is the topic that a comparison is popular, equally, has calculated and how to have managed
A data also critically important topic, the most just can make to calculate the data produced, present in real time in front of the user,
And data do not occur confusion, the data produced in a lot of scientific algorithm are all the data that confidentiality is the highest, how to manage this
The security of a little data guarantee data is a problem made people think deeply.
Under cloud computing environment, the operational mode of respective independent separate can not support that the expansion of cloud service, new IT run
Pattern proposes challenge to traditional management framework, to virtualization, dynamic, relevance, automates, real-time, efficiently
Property, the requirement of security etc. improves constantly, the problem of existing system:
Real-time is not strong, and user has calculated and can not just get calculating data after calculating task really completes, and always has
Certain time-delay；
Security is the highest, and it is nonsensical that a lot of data are put in storage in fact, and also has secure data area a lot
Problem；
Correctness is the highest, if there being bigger error to produce in scientific algorithm, to such an extent as to may affect the safety produced
Property；
High efficiency is strong, and the utilization rate of resource entirety is the highest.
Summary of the invention
For the deficiencies in the prior art, the present invention provides the implementation method of the system that the data magnanimity under a kind of cluster manages,
Ensure that the real-time of data, security, correctness, high efficiency.
The implementation method of the system of the data magnanimity management under a kind of cluster that the present invention provides, it thes improvement is that, counts
After calculation application program calculating operation completes, the data that calculating node has calculated are copied by the script handling procedure of operation
To management node, stagger the time when copying out, carry out secondary copy by object-line thread management process, and revise data mode.
Wherein, described script handling procedure is just to be automatically copied to after job run completes share storage.
Wherein, the step that described object-line thread management process carries out secondary copy includes:
(1) PBS inquiry calculates node；
(2) java process inquiry database passes through the data record that the script handling procedure of operation is failed；
(3) data failed for the script handling procedure by operation are copied to the shared mesh of shared disk by script packing
Record；
(4) peripheral thread updates database pass flag；
(5) thread sleep.If necessary, set the length of one's sleep and carry out the secondary copy process of next round.
Wherein, step (1) PBS inquiry calculates node, and the running status of operation generates a book keeping operation file.
Wherein, by the conversion of file of keeping accounts described in java process monitoring, and the conversion of described book keeping operation file is write in real time
In mysql database, form the history information to operation in database；The history information structure of at least one
Become data sheet.
Wherein, data sheet is provided with index function, and uses database connection pool to deposit the connection number of n quantity.
Wherein, it is encrypted setting during data copy.
Compared with the prior art, the invention have the benefit that
The present invention ensure that the real-time of data, security, correctness, high efficiency, it is to avoid the data of user are stolen,
Reveal.
The present invention processed calculate isolated island, not all should problem and devise the cycle leakage and postpone result generally exist
The corresponding solution of problem.
The present invention improves the experience of the cloud computing service of user.
The present invention improves the stability of system, reliability.
Accompanying drawing explanation
Communication scheme between multiple blades or PC node that Fig. 1 provides for the present invention.
Fig. 2 carries out the flow chart of secondary copy for the object-line thread management process that the present invention provides.
Detailed description of the invention
Below in conjunction with the accompanying drawings the detailed description of the invention of the present invention is described in further detail.
Communication scheme such as Fig. 1 between multiple blades or PC node during the present embodiment Distributed Calculation based on cluster
Shown in, it is coupled together by high-speed local area network, and is equipped with and certain supports software parallel, forms a loose couplings
Concurrent computational system, use PBS carry out cluster management, job scheduling.In figure, distributed computing fabric comprises three classes
Node, submits node to, manages node, calculates node.
Submitting to node to be responsible for submitting a job to PBS and manage node, the resource of group system is unified by management node
Monitoring and rational management, make the resources such as the CPU of the cluster abundant Sharing computer of each node, internal memory, disk, calculates node
It is responsible for the evaluation work of task, returns to checkout result after having calculated manage in the storage of node, and at cluster
The management node carry NFS of system, all calculating nodes can access by carry, and management node is the core calculated, but
The calculating of its not supplemental characteristic, and simply group system is managed, it is respectively as follows: task management, node administration,
Telecommunication management, data base administration.
The present embodiment, after calculating application program calculating operation completes, carries out data record, will calculate node and calculate
Data copy back management node, the present embodiment ensures the real-time of data, high efficiency, correctness by two ways；
1) processed by the script of operation, i.e. job run completes the most automatically to copy back and shares storage, and the present embodiment is
Share storage, i.e. the copy of local disk, the problem that there is not disk I/O, calculating and the data record of therefore operation are several
It is same process, its delay time negligible；
2) being managed by peripheral thread, it is secondary copy process, and flow chart is as shown in Figure 2.PBS inquires about calculating
Node, is managed operation, and the running status of each operation is carried out record, can generate a book keeping operation file and carry out record
The details of operation.The present invention uses the process of a java to monitor the change of book keeping operation file, and is written in real time
In mysql database, to form in database the history information to operation, data sheet can be generated in the future and use,
Being continually changing over time, the data in database can be continuously increased, and the pressure of inquiry also the most constantly increases, therefore
The present embodiment is newly-built index in historical data table, also using database connection pool to deposit n quantity (n simultaneously
For positive integer) connection number so that the search efficiency of database is higher, as first kind of way make a mistake will calculate
Result copies back storage, and the second way has a java thread and constantly monitored whether that data reclaim failure automatically,
If unsuccessfully will pack with script (i.e. may be considered calculation procedure itself oneself complete), will be processed by the script of operation
The data of procedure failure copy the share directory of shared disk to, the most again by the status modifier of record.After this process completes
Thread enters sleep, if necessary, sets the length of one's sleep and carries out the secondary copy process of next round.
When user data is transferred to cloud computing service business by network, do not allowed the data quilt of user by strict cipher mode
Stealing, the data produced in cloud computing ensure the safety of storage data, and the data in storage are also encryptions.Give equally
The result data of the transmission of user is also encryption.
The security of data, high efficiency, correctness, and real-time can be well ensured by the way of above.Allow user
The management of imperceptible calculating and data is separate.The Consumer's Experience promoted greatly, also provide enhanced architecture simultaneously.
Finally should be noted that: above example only in order to illustrate that technical scheme is not intended to limit, although
Being described in detail the present invention with reference to above-described embodiment, those of ordinary skill in the field are it is understood that still
The detailed description of the invention of the present invention can be modified or equivalent, and appointing without departing from spirit and scope of the invention
What amendment or equivalent, it all should be contained in the middle of scope of the presently claimed invention.
Claims (6)
1. the implementation method of the system of the management of the data magnanimity under a cluster, it is characterised in that calculate application program meter
Can be regarded as after industry completes, copy management node by calculating the data that node calculated to by the script handling procedure of operation,
Stagger the time when copying out, carry out secondary copy by object-line thread management process, and revise data mode；
The step that described object-line thread management process carries out secondary copy includes:
(1) PBS inquiry calculates node；
(2) java process inquiry database passes through the data record that the script handling procedure of operation is failed；
(3) script packing, copies data failed for the script handling procedure by operation to share directory；
(4) peripheral thread updates database pass flag；
(5) thread sleep.
2. implementation method as claimed in claim 1, it is characterised in that described script handling procedure is that job run is complete
Share directory just it is automatically copied to after one-tenth.
3. implementation method as claimed in claim 1, it is characterised in that step (1) PBS inquiry calculates node, and
The running status of operation is generated a book keeping operation file.
4. implementation method as claimed in claim 3, it is characterised in that by file of keeping accounts described in java process monitoring
Conversion, and the conversion of described book keeping operation file is written in mysql database in real time, is formed in database operation
History information；The history information of at least one constitutes data sheet.
5. implementation method as claimed in claim 4, it is characterised in that be provided with index function in data sheet, and
Database connection pool is used to deposit the connection number of n quantity.
6. implementation method as claimed in claim 1, it is characterised in that be encrypted setting during data copy.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210309450.XA CN102880832B (en) | 2012-08-28 | 2012-08-28 | A kind of implementation method of the system of the data magnanimity management under cluster |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210309450.XA CN102880832B (en) | 2012-08-28 | 2012-08-28 | A kind of implementation method of the system of the data magnanimity management under cluster |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102880832A CN102880832A (en) | 2013-01-16 |
| CN102880832B true CN102880832B (en) | 2016-08-31 |
Family
ID=47482153
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210309450.XA Active CN102880832B (en) | 2012-08-28 | 2012-08-28 | A kind of implementation method of the system of the data magnanimity management under cluster |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102880832B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103198097B (en) * | 2013-03-11 | 2016-02-10 | 中国科学院计算机网络信息中心 | A kind of magnanimity earth science data method for parallel processing based on distributed file system |
| CN109086134A (en) * | 2018-07-19 | 2018-12-25 | 郑州云海信息技术有限公司 | A kind of operation method and device of deep learning operation |
| CN108958892A (en) * | 2018-08-14 | 2018-12-07 | 郑州云海信息技术有限公司 | A kind of method and apparatus creating the container for deep learning operation |
| CN112528456B (en) * | 2019-09-18 | 2024-05-07 | 曙光信息产业(北京)有限公司 | Heterogeneous node computing system and method |
| CN110781189B (en) * | 2019-10-25 | 2022-08-26 | 北京达佳互联信息技术有限公司 | Document platform construction method and device, electronic equipment and storage medium |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101951411A (en) * | 2010-10-13 | 2011-01-19 | 戴元顺 | Cloud scheduling system and method and multistage cloud scheduling system |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100153482A1 (en) * | 2008-12-10 | 2010-06-17 | Full Armor Corporation | Cloud-Based Automation of Resources |
| US20110126197A1 (en) * | 2009-11-25 | 2011-05-26 | Novell, Inc. | System and method for controlling cloud and virtualized data centers in an intelligent workload management system |
-
2012
- 2012-08-28 CN CN201210309450.XA patent/CN102880832B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101951411A (en) * | 2010-10-13 | 2011-01-19 | 戴元顺 | Cloud scheduling system and method and multistage cloud scheduling system |
Non-Patent Citations (1)
| Title |
|---|
| 《集群资源管理***PBS及其应用》;李全枝等;《微机发展》;20050430;第15卷(第4期);第4-7页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102880832A (en) | 2013-01-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102103518B (en) | System for managing resources in virtual environment and implementation method thereof | |
| CN109643312B (en) | Hosted query service | |
| US10394611B2 (en) | Scaling computing clusters in a distributed computing system | |
| Abourezq et al. | Database-as-a-service for big data: An overview | |
| CN102880832B (en) | A kind of implementation method of the system of the data magnanimity management under cluster | |
| Sun et al. | Using ethereum blockchain in Internet of Things: A solution for electric vehicle battery refueling | |
| CN104160381A (en) | Managing tenant-specific data sets in a multi-tenant environment | |
| Jeong et al. | Anomaly teletraffic intrusion detection systems on hadoop-based platforms: A survey of some problems and solutions | |
| CN103049482B (en) | The implementation method that in a kind of distributed heterogeneous system, data fusion stores | |
| CN106354833A (en) | Platform for achieving data management and sharing exchange on basis of B/S framework | |
| CN103399894A (en) | Distributed transaction processing method on basis of shared storage pool | |
| CN105335143A (en) | Business processing method and apparatus | |
| CN107070709A (en) | A kind of NFV implementation methods based on bottom NUMA aware | |
| CN104008100B (en) | Cluster environment concurrent processing method | |
| Taniar et al. | Concurrency control issues in Grid databases | |
| Feng et al. | Review of hadoop performance optimization | |
| Liu et al. | The application of block chain technology in spot exchange | |
| JP2017533518A (en) | Database data distribution management | |
| Ke et al. | Research and application of enterprise big data governance | |
| CN107147733A (en) | Service recovery method based on SOA | |
| Lakhe et al. | Introducing Hadoop | |
| Zhang et al. | Data Distributed Storage Scheme in Internet of Things Based on Blockchain | |
| CN107885834A (en) | A kind of Hadoop big datas component uniformly verifies system | |
| Bhattacharya et al. | Distributed data recovery architecture based on schema segregation | |
| JP4462504B1 (en) | A multi-transaction control system with update access counters owned by the transaction process that is the starting point for maintaining consistency |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20211022 Address after: 100089 zone A-1, floor 2, building 36, yard 8, Dongbeiwang West Road, Haidian District, Beijing Patentee after: Shuguang zhisuan Information Technology Co.,Ltd. Address before: 100193 No.36 Zhongguancun Software Park, No.8 Dongbeiwang West Road, Haidian District, Beijing Patentee before: Dawning Information Industry (Beijing) Co.,Ltd. |