CN103188306A - Distributed preprocessing method and distributed preprocessing system - Google Patents
Distributed preprocessing method and distributed preprocessing system Download PDFInfo
- Publication number
- CN103188306A CN103188306A CN2011104556253A CN201110455625A CN103188306A CN 103188306 A CN103188306 A CN 103188306A CN 2011104556253 A CN2011104556253 A CN 2011104556253A CN 201110455625 A CN201110455625 A CN 201110455625A CN 103188306 A CN103188306 A CN 103188306A
- Authority
- CN
- China
- Prior art keywords
- node
- processing
- module
- data
- executive
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Hardware Redundancy (AREA)
Abstract
The invention discloses a distributed preprocessing method and a distributed preprocessing system. The distributed preprocessing method includes the following steps: a process node receives data sent from a source node; a management module configures types of multiple executive modules and an executive order according to different demands of the process node; and the process node invokes an executive module collection which is deployed by the process node and configured by the management module for carrying out preprocessing operation on the received data, wherein the executive module collection comprises the multiple executive modules. Through the distributed preprocessing method and the distributed preprocessing system, the preprocessing process of the process node can be managed dynamically and flexibly, the executive module collection is managed, and consistency and atomicity of the preprocessing process of the process node are guaranteed; and the distributed preprocessing system guarantees that preprocessing intermediate results of the multiple executive modules of the process node are processed in an internal memory and do not need to be stored temporarily, and therefore preprocessing performance is improved.
Description
Technical field
The present invention relates to distributed system in the computer realm, particularly, relate to distributed pretreated method and system.
Background technology
Existing distributed data loads, the logical architecture of pretreatment system comprises:
Source node (being source node): be deployed in source side, encapsulate data source to be loaded, the data in the data source are sent to the Dest node;
Dest node (being destination node): be deployed in the storage side, receive the data from the Source node, import in the storage of rear end;
Process node: before the Dest node is with the data storage, can introduce a plurality of Process nodes, data be carried out some preliminary treatment, as filtration, conversion etc.The Process node can share a node with the Dest node physically.
The preliminary treatment work of distributed data loading at present, pretreatment system, main mode by manual intervention configuration Process node.If desired data are carried out the multistep preliminary treatment, then need to introduce a plurality of Process nodes, meeting be landed data and be deposited interim memory space in after each Process node processing, and next Process node reads data from interim memory space again and handles.
In realizing process of the present invention, the inventor finds that there are the following problems at least in the prior art:
At first, adopt mode static, hard coded, can not on-the-fly modify in case just set, the preprocessing process that upgrades, upgrades is more loaded down with trivial details, needs to increase or upgrade Process node and configuration thereof;
Secondly, a plurality of preprocessing process intermediate object programs also need interim storage, and waste system read-write IO causes performance to reduce.
Summary of the invention
The objective of the invention is to propose a kind of distributed pretreated method and system at the lower defective of distributed system preliminary treatment efficient in the prior art.
For achieving the above object, according to an aspect of the present invention, provide a kind of distributed pretreated method.
Distributed preprocess method according to the embodiment of the invention comprises:
Processing node is received from the data that source node sends;
Administration module is according to kind and the execution sequence of the different demand configuration Executive Modules of processing node;
Processing node calls the Executive Module set of self disposing that includes a plurality of Executive Modules by the administration module configuration data that receive is carried out pretreatment operation.
In technique scheme, also comprise after the step of pretreatment operation: the Executive Module set places internal memory to store preliminary treatment intermediate object program.
In technique scheme, administration module with the corresponding a plurality of processing nodes of identical destination node in the identical Executive Module of configuration carry out pretreatment operation.
In technique scheme, method also comprises:
Administration module is collected the process status of each processing node when data are carried out preliminary treatment;
If there is the inconsistent state of process in processing node when data are carried out preliminary treatment, administration module then rolls back to the state of being untreated with all processing nodes, after confirming that all processing nodes are finished renewal, start all processing nodes more simultaneously and carry out pretreatment operation;
If processing node is when carrying out preliminary treatment to data, exist the section processes node to start, the state that other node does not start, the processing node that administration module will not start suspends, the processing node that has started is rolled back to the state of being untreated, after confirming that all processing nodes are finished renewal, start all processing nodes more simultaneously and carry out pretreatment operation.In technique scheme, method also comprises:
Initialization, module interface inspection, module loading and the unloading of Executive Module execution environment, the operation of execution are finished in the Executive Module set.
In technique scheme, method also comprises:
Executive Module is gathered according to the configuration storage allocation, and pending data are kept in the internal memory;
The Executive Module set passes to Executive Module with the internal memory pointer as parameter, and Executive Module distributes required memory and preliminary treatment intermediate object program is kept in the internal memory;
After Executive Module is integrated into preliminary treatment intermediate object program in the described internal memory and is called, discharge internal memory.
For achieving the above object, according to another aspect of the present invention, provide a kind of distributed pretreatment system.
Distributed pretreatment system according to the embodiment of the invention comprises:
Processing node is used for being received from the data that source node sends, and calls the Executive Module set of self disposing that includes a plurality of Executive Modules by the administration module configuration data that receive are carried out pretreatment operation;
Administration module is used for kind and execution sequence according to the different demand configuration Executive Modules of processing node.
In technique scheme, the Executive Module set is used for placing internal memory to store preliminary treatment intermediate object program.
In technique scheme, administration module also is used for carrying out pretreatment operation at the Executive Module identical with the corresponding a plurality of processing nodes configurations of identical destination node.
In technique scheme, administration module is used for collecting the process status of each processing node when data are carried out preliminary treatment, according to process status, processing node is carried out state processing;
Administration module comprises:
State is collected submodule, is used for collecting the process status of each processing node when data are carried out preliminary treatment;
The state processing submodule is used for according to the process status of processing node when data are carried out preliminary treatment, and processing node is carried out state processing:
If there is the inconsistent state of process in processing node when data are carried out preliminary treatment, all processing nodes are rolled back to the state of being untreated, after confirming that all processing nodes are finished renewal, start all processing nodes more simultaneously and carry out pretreatment operation;
If processing node is when carrying out preliminary treatment to data, exist the section processes node to start, the state that other node does not start, the processing node that does not start is suspended, the processing node that has started is rolled back to the state of being untreated, after confirming that all processing nodes are finished renewal, start all processing nodes more simultaneously and carry out pretreatment operation.
In technique scheme, the Executive Module set comprises: the initialization submodule is used for finishing initialization, module interface inspection, module loading and the unloading of Executive Module execution environment, the operation of execution.
In technique scheme, the Executive Module set comprises: the memory management submodule is used for according to the configuration storage allocation, and pending data are kept in the internal memory, the internal memory pointer is passed to Executive Module as parameter, after the preliminary treatment intermediate object program in internal memory is called, discharge internal memory;
Executive Module is used for finishing set processing logic, and preliminary treatment intermediate object program is kept at from the internal memory of Executive Module set application.
Distributed preprocess method and the system of various embodiments of the present invention can manage the preprocessing process of Process node dynamically, neatly, and the set of management Executive Module guarantees consistency and the atomicity of Process node preprocessing process; System has guaranteed that the preliminary treatment intermediate object program of a plurality of Executive Modules of Process node all handles in internal memory, do not need interim storage, has improved the preliminary treatment performance.
Other features and advantages of the present invention will be set forth in the following description, and, partly from specification, become apparent, perhaps understand by implementing the present invention.Purpose of the present invention and other advantages can realize and obtain by specifically noted structure in the specification of writing, claims and accompanying drawing.
Below by drawings and Examples, technical scheme of the present invention is described in further detail.
Description of drawings
Accompanying drawing is used to provide further understanding of the present invention, and constitutes the part of specification, is used from explanation the present invention with embodiments of the invention one, is not construed as limiting the invention.In the accompanying drawings:
Fig. 1 is the framework schematic diagram of the distributed pretreatment system according to the present invention;
Fig. 2 is the structural representation of the Executive Module set according to the present invention;
Fig. 3 is the structural representation that Process node preprocessing process upgrades according to the present invention;
Fig. 4 is that the Process node guarantees the conforming structural representation of preprocessing process according to the present invention;
Fig. 5 is the structural representation that the Process node administration upgrades according to the present invention;
Fig. 6 is the schematic flow sheet of the distributed preprocess method according to the present invention;
Fig. 7 is the structural representation of the distributed pretreatment system according to the present invention.
Embodiment
Below in conjunction with accompanying drawing the preferred embodiments of the present invention are described, should be appreciated that preferred embodiment described herein only is used for description and interpretation the present invention, and be not used in restriction the present invention.
Distributed pretreatment system of the present invention has designed main controlled node PMaster, can dynamically, organically manage the preprocessing process of Process node (being processing node), guarantees consistency and the atomicity of Process node preprocessing process; And system has guaranteed that the intermediate object program that the Process node calls a plurality of Executive Modules all handles in internal memory, do not need interim storage.
As shown in Figure 1, distributed pretreatment system of the present invention comprises administration module (being PManager) and Executive Module set (being PModule), and administration module is deployed on the main controlled node PMaster usually, and the Executive Module set is deployed on the processing node Process.
As shown in Figure 2, dispose a plurality of Executive Modules among the PModule, i.e. Module1, Module2, Module3......ModuleK, PModule mainly finishes the operation of Executive Module, comprise and carry out engine, memory management, management and monitoring etc., PModule is mainly by constituting as lower member:
Carry out engine (being Execute Engine): finish initialization, module interface inspection, module loading and the work such as unloading, execution of Executive Module execution environment;
Memory management: allocate internal memory as requested in advance, and in time reclaim internal memory.In order to ensure the execution efficient of Executive Module, memory management is one of Core Feature, mainly comprises following operation:
Carry out engine according to the configuration storage allocation, and pending data are kept in the internal memory;
The internal memory pointer is passed to Executive Module as parameter, and Executive Module distributes required memory and the preliminary treatment intermediate object program of data is kept in the internal memory;
The preliminary treatment intermediate object program of each Executive Module all is temporarily stored in the internal memory, can be used by other modules;
After all modules were carried out and finished, the internal memory that this lot data takies was released, and data result is persistence as requested;
Management and monitoring: according to the requirement of PManager, finish the management work of Executive Module, and policer operation is provided.
PManager realizes the management to Executive Module, comprises functions such as installation, deployment, upgrading, degradation, status tracking, mainly by constituting as lower member:
The module management parts: main every bookkeeping of implementing Executive Module owing to be distributed system, therefore needs to support the atomicity of bookkeeping, namely one operates on all nodes or all successes, perhaps all fail, and partly success, otherwise can cause data inconsistent;
Module and slip condition database: mainly implement the management of Executive Module and state thereof;
Interpersonal interface: offer management system and show.
The major function of PManager comprises:
1) manages all Executive Modules: the Executive Module interface of definition standard, need guaranteeing provides as required preprocessing process to different Process nodes: the preprocessing process of configuration Process node, for example, used which Executive Module in the preprocessing process, what order to move Executive Module etc. successively according to, upgrade the preprocessing process of Process node;
As shown in Figure 3, if there is Process node preprocessing process to upgrade, then revises configuration and notify corresponding Process node;
Configuration information is as follows:
Source1 process1 (Executive Module 1, Executive Module 2, Executive Module 3) dest1
Source2 process2 (Executive Module 1, Executive Module 3, Executive Module 4) dest2
SourceN processN (Executive Module 1, Executive Module 5) destN
2) can guarantee the consistency of the preprocessing process of a plurality of Process nodes that same Source node is handled: when adopting load balancing or highly reliable mode to load data, a Source node may send the data to a plurality of Process nodes and a plurality of Dest node, needs the consistency of the preprocessing process of these a plurality of Process nodes of assurance.
For example, under highly reliable load mode situation, the corresponding Dest node of each Process node is handled the data that this Dest node receives.System need guarantee the consistency of the preprocessing process of a plurality of Dest nodes that same Source node sends to, this is by the processing for same Source node, and one or more identical Executive Modules realize for this Source node disposes in different Process nodes.Being consistent as the Executive Module configuration at the Source2 node in the Process2 among Fig. 4 and the ProcessM node, all is Executive Module 1, Executive Module 3, Executive Module 4.
Simultaneously, also need to guarantee different Source node datas that the Process node is received to provide preprocessing process as required.As providing different Executive Module configurations at Source1 node and Source2 node in the Process2 node among Fig. 4.
The configuration information of Fig. 4 is as follows:
Source1 process1 (module 1, module 2) dest1, process2 (module 1, module 2, module 3) dest2
Source2 process2 (module 1, module 3, module 4) dest2, processM (module 1, module 3, module 4) destM
SourceN processM (module 1, module 5) destM, process1 (module 1, module 5) dest1
3) management update comprises: the renewal of (1) Process node comprises its additions and deletions, upgrading, rollback; (2) at the renewal of pretreated each Process node preprocessing process of specific Source node.
Notice Process node updates information: comprise module upgrade, rollback information, and/or node preprocessing process lastest imformation, the operation that increases, deletes Executive Module as the preliminary treatment at specific Source node;
Collect the preprocessing process state of Process node, in case the problem of Discovery Status or configuration, system can in time intervene.
As shown in Figure 5, the preprocessing process of being responsible for the pretreated Process1 of Source1 node and Process2 node is upgraded newly-increased Executive Module 4 after Executive Module 3.
Consistency and the atomicity of preprocessing tasks when guaranteeing to upgrade, when namely guaranteeing to upgrade, the preprocessing process of Process node to be updated is not all also carried out, or rolls back to not executing state.System at first need to judge whether rollback by collecting the Process node state, and the preprocessing process with Process node to be updated suspends then, waits to upgrade and finishes, and restarts preprocessing process.As shown in Figure 5, system obtains the node state of Process1 and Process2, also do not have treatment S ource1 data as Process1, and Process2 has begun treatment S ource1 data; Suspend the Process1 node, notice Process2 rolls back to the state of being untreated and suspends; Notice Process1 and Process2 node updates preprocessing process, and confirm to upgrade; Start the preprocessing process of Process1 and Process2 node simultaneously.
Those skilled in the art can understand, in Fig. 5, if Process1 node and Process2 node have all begun the data for the treatment of S ource1, but both processes are inconsistent, to notify Process1 node and Process2 node to roll back to the state of being untreated, suspend Process1 node and Process2 node, start the preprocessing process of Process1 and Process2 node simultaneously.
4) for fear of the Single Point of Faliure of Pmaster node, also flow bottleneck for the control that answering system node increase in size causes, system supports a plurality of Pmaster nodes, and keeps between the Pmaster node copying synchronously and data consistency.
Method embodiment
According to the embodiment of the invention, a kind of distributed preprocess method is provided, as shown in Figure 6, present embodiment comprises:
Step 102: processing node Process is received from the data that source node S ource sends, and calls Executive Module set PModule the data that receive are carried out pretreatment operation;
Step 104: Executive Module set PModule places internal memory to store preliminary treatment intermediate object program;
Step 106: the described data that processing node Process will receive place destination node Dest.
Distributed preprocess method of the present invention, system manages the preprocessing process of Process node dynamically, neatly, the set of management Executive Module, consistency and the atomicity of assurance Process node preprocessing process; System has guaranteed that the preliminary treatment intermediate object program of a plurality of Executive Modules of Process node all handles in internal memory, do not need interim storage, has improved the preliminary treatment performance.
Distributed preprocess method of the present invention, compare with existing pattern and to have following advantage:
Existing pattern: lot of data, as the data of 1000GB, be loaded into (as oracle) in the database, the user finishes preliminary treatment by sending sql command at database then, finishes repeatedly preliminary treatment, then needs to send a plurality of sql commands; But when each SQL operated, database all needed:
From disk scanning 1000GB data, the data decimation that satisfies customer requirements is come out, be put in the another one database
The IO operation amounts to:
When (1) putting in storage, the data of having preserved 1000GB;
(2) N SQL preliminary treatment needs to read for N time the data of 1000GB;
(3) accumulative total: write for 1 time, N time the 1000GB data are read;
Pattern of the present invention: mass data, as the 1000GB data, be transferred on the PModule, PModule finishes N preliminary treatment (internal memory) successively, the disk operating that the data of 1000GB are taken place:
(1) warehouse-in, the data of preserving 1000GB;
(2) N preliminary treatment do not need the disk I operation;
(3) accumulative total: write for 1 time;
Eliminated reading the time of N 1000GB data.
System embodiment
According to the embodiment of the invention, provide a kind of distributed pretreatment system.As shown in Figure 7, present embodiment comprises:
Processing node 10 is used for being received from the data that source node sends, and calls the Executive Module set of self disposing that includes a plurality of Executive Modules by the administration module configuration data that receive are carried out pretreatment operation;
Administration module 20 is used for kind and execution sequence according to the different demand configuration Executive Modules of processing node 10.
Executive Module set 30 is used for placing internal memory to store preliminary treatment intermediate object program.
Wherein:
Administration module 20 also is used for carrying out pretreatment operation at the Executive Module identical with the corresponding a plurality of processing nodes configurations of identical destination node.
Wherein:
Administration module 20 is used for collecting the process status of each processing node 10 when data are carried out preliminary treatment, according to process status, processing node 10 is carried out state processing;
Administration module 20 comprises:
State is collected submodule 21, is used for collecting the process status of each processing node when data are carried out preliminary treatment;
State processing submodule 22 is used for according to the process status of processing node when data are carried out preliminary treatment, and processing node is carried out state processing:
If there is the inconsistent state of process in processing node when data are carried out preliminary treatment, all processing nodes are rolled back to the state of being untreated, after confirming that all processing nodes are finished renewal, start all processing nodes more simultaneously and carry out pretreatment operation;
If processing node is when carrying out preliminary treatment to data, exist the section processes node to start, the state that other node does not start, the processing node that does not start is suspended, the processing node that has started is rolled back to the state of being untreated, after confirming that all processing nodes are finished renewal, start all processing nodes more simultaneously and carry out pretreatment operation.
Wherein:
Executive Module set 30 comprises: initialization submodule 31 is used for finishing initialization, module interface inspection, module loading and the unloading of Executive Module execution environment, the operation of execution.
Memory management submodule 32 is used for according to the configuration storage allocation, and pending data are kept in the internal memory, and the internal memory pointer is passed to Executive Module as parameter, after the preliminary treatment intermediate object program in internal memory is called, discharges internal memory;
Executive Module is used for finishing set processing logic, and preliminary treatment intermediate object program is kept at from the internal memory of Executive Module set application.
Distributed pretreatment system of the present invention is managed the preprocessing process of Process node dynamically, neatly, and the set of management Executive Module guarantees consistency and the atomicity of Process node preprocessing process; System has guaranteed that the preliminary treatment intermediate object program of a plurality of Executive Modules of Process node all handles in internal memory, do not need interim storage, has improved the preliminary treatment performance.
One of ordinary skill in the art will appreciate that: all or part of step that realizes said method embodiment can be finished by the relevant hardware of program command, aforesaid program can be stored in the computer read/write memory medium, this program is carried out the step that comprises said method embodiment when carrying out; And aforesaid storage medium comprises: various media that can be program code stored such as ROM, RAM, magnetic disc or CD.
It should be noted that at last: the above only is the preferred embodiments of the present invention, be not limited to the present invention, although with reference to previous embodiment the present invention is had been described in detail, for a person skilled in the art, it still can be made amendment to the technical scheme that aforementioned each embodiment puts down in writing, and perhaps part technical characterictic wherein is equal to replacement.Within the spirit and principles in the present invention all, any modification of doing, be equal to replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (12)
1. a distributed preprocess method is characterized in that, comprising:
Processing node is received from the data that source node sends;
Administration module is according to kind and the execution sequence of the different demand configuration Executive Modules of described processing node;
Described processing node calls the Executive Module set of self disposing that includes a plurality of Executive Modules by described administration module configuration the described data that receive is carried out pretreatment operation.
2. method according to claim 1 is characterized in that, also comprises after the step of pretreatment operation: described Executive Module set places internal memory to store preliminary treatment intermediate object program.
3. method according to claim 1 is characterized in that, described administration module with the corresponding a plurality of processing nodes of identical destination node in the identical Executive Module of configuration carry out pretreatment operation.
4. method according to claim 1 is characterized in that, also comprises:
Described administration module is collected the process status of each processing node when data are carried out preliminary treatment;
If there is the inconsistent state of process in processing node when data are carried out preliminary treatment, described administration module then rolls back to the state of being untreated with all processing nodes, after confirming that all processing nodes are finished renewal, start all processing nodes more simultaneously and carry out pretreatment operation;
If processing node is when carrying out preliminary treatment to data, exist the section processes node to start, the state that other node does not start, the processing node that described administration module will not start suspends, the processing node that has started is rolled back to the state of being untreated, after confirming that all processing nodes are finished renewal, start all processing nodes more simultaneously and carry out pretreatment operation.
5. method according to claim 1 is characterized in that, also comprises:
Initialization, module interface inspection, module loading and the unloading of Executive Module execution environment, the operation of execution are finished in described Executive Module set.
6. method according to claim 1 is characterized in that, also comprises:
Described Executive Module is gathered according to the configuration storage allocation, and pending data are kept in the internal memory;
Described Executive Module set passes to Executive Module with the internal memory pointer as parameter, and Executive Module is finished set processing logic, and preliminary treatment intermediate object program is kept at from the internal memory of Executive Module set application;
After described Executive Module is integrated into preliminary treatment intermediate object program in the described internal memory and is called, discharge internal memory.
7. a distributed pretreatment system is characterized in that, comprising:
Processing node is used for being received from the data that source node sends, and calls the Executive Module set of self disposing that includes a plurality of Executive Modules by the administration module configuration the described data that receive are carried out pretreatment operation;
Administration module is used for kind and execution sequence according to the different demand configuration Executive Modules of described processing node.
8. system according to claim 7 is characterized in that, described Executive Module set is used for placing internal memory to store preliminary treatment intermediate object program.
9. system according to claim 7 is characterized in that, described administration module also is used for carrying out pretreatment operation at the Executive Module identical with the corresponding a plurality of processing nodes configurations of identical destination node.
10. system according to claim 7 is characterized in that,
Described administration module is used for collecting the process status of each processing node when data are carried out preliminary treatment, according to described process status, described processing node is carried out state processing;
Described administration module comprises:
State is collected submodule, is used for collecting the process status of each processing node when data are carried out preliminary treatment;
The state processing submodule is used for according to the process status of processing node when data are carried out preliminary treatment, and described processing node is carried out state processing:
If there is the inconsistent state of process in processing node when data are carried out preliminary treatment, all processing nodes are rolled back to the state of being untreated, after confirming that all processing nodes are finished renewal, start all processing nodes more simultaneously and carry out pretreatment operation;
If processing node is when carrying out preliminary treatment to data, exist the section processes node to start, the state that other node does not start, the processing node that does not start is suspended, the processing node that has started is rolled back to the state of being untreated, after confirming that all processing nodes are finished renewal, start all processing nodes more simultaneously and carry out pretreatment operation.
11. system according to claim 7 is characterized in that, described Executive Module set comprises: the initialization submodule is used for finishing initialization, module interface inspection, module loading and the unloading of Executive Module execution environment, the operation of execution.
12. system according to claim 7, it is characterized in that, described Executive Module set comprises: the memory management submodule, be used for according to the configuration storage allocation, and pending data are kept in the internal memory, the internal memory pointer is passed to Executive Module as parameter, after the preliminary treatment intermediate object program in described internal memory is called, discharge internal memory;
Described Executive Module is used for finishing set processing logic, and preliminary treatment intermediate object program is kept at from the internal memory of Executive Module set application.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110455625.3A CN103188306B (en) | 2011-12-30 | 2011-12-30 | Distributed preprocess method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110455625.3A CN103188306B (en) | 2011-12-30 | 2011-12-30 | Distributed preprocess method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103188306A true CN103188306A (en) | 2013-07-03 |
| CN103188306B CN103188306B (en) | 2016-04-27 |
Family
ID=48679271
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110455625.3A Active CN103188306B (en) | 2011-12-30 | 2011-12-30 | Distributed preprocess method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103188306B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105426493A (en) * | 2015-11-24 | 2016-03-23 | 北京中电普华信息技术有限公司 | Data processing system and method applied to distributed storage system |
| CN106021543A (en) * | 2016-05-26 | 2016-10-12 | 中国农业银行股份有限公司 | Data pre-processing method and device |
| CN107037989A (en) * | 2017-05-17 | 2017-08-11 | 北京小米移动软件有限公司 | Data processing method and device in distributed computing system |
| CN108733691A (en) * | 2017-04-18 | 2018-11-02 | 北京京东尚科信息技术有限公司 | Data preprocessing method and device |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1176426A (en) * | 1996-05-15 | 1998-03-18 | 摩托罗拉公司 | Method and apparatus for sequencing computer instruction execution in data processing system |
| US20050198313A1 (en) * | 2004-01-12 | 2005-09-08 | International Business Machines Corp. | Method, system and computer program product for dynamically optimized handling of messages received from a message transmission backbone by a data processing system node |
| CN1890658A (en) * | 2003-12-02 | 2007-01-03 | 交互式内容引擎有限责任公司 | Synchronized data transfer system |
| CN1969258A (en) * | 2004-06-16 | 2007-05-23 | 先进微装置公司 | Multipurpose media access data processing system |
| US20070169113A1 (en) * | 2005-11-03 | 2007-07-19 | International Business Machines Corporation | Method and apparatus for provisioning software on a network of computers |
| CN101526982A (en) * | 2001-02-16 | 2009-09-09 | 索尼株式会社 | Data processing method and the device thereof |
| CN101609544A (en) * | 2009-07-21 | 2009-12-23 | 华中科技大学 | A kind of image processor data load and method of testing and device |
| CN102063330A (en) * | 2011-01-05 | 2011-05-18 | 北京航空航天大学 | Performance data acquisition method for large-scale parallel program |
| CN102279888A (en) * | 2011-08-24 | 2011-12-14 | 北京新媒传信科技有限公司 | Method and system for scheduling tasks |
| CN102279730A (en) * | 2010-06-10 | 2011-12-14 | 阿里巴巴集团控股有限公司 | Parallel data processing method, device and system |
-
2011
- 2011-12-30 CN CN201110455625.3A patent/CN103188306B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1176426A (en) * | 1996-05-15 | 1998-03-18 | 摩托罗拉公司 | Method and apparatus for sequencing computer instruction execution in data processing system |
| CN101526982A (en) * | 2001-02-16 | 2009-09-09 | 索尼株式会社 | Data processing method and the device thereof |
| CN1890658A (en) * | 2003-12-02 | 2007-01-03 | 交互式内容引擎有限责任公司 | Synchronized data transfer system |
| US20050198313A1 (en) * | 2004-01-12 | 2005-09-08 | International Business Machines Corp. | Method, system and computer program product for dynamically optimized handling of messages received from a message transmission backbone by a data processing system node |
| CN1969258A (en) * | 2004-06-16 | 2007-05-23 | 先进微装置公司 | Multipurpose media access data processing system |
| US20070169113A1 (en) * | 2005-11-03 | 2007-07-19 | International Business Machines Corporation | Method and apparatus for provisioning software on a network of computers |
| CN101609544A (en) * | 2009-07-21 | 2009-12-23 | 华中科技大学 | A kind of image processor data load and method of testing and device |
| CN102279730A (en) * | 2010-06-10 | 2011-12-14 | 阿里巴巴集团控股有限公司 | Parallel data processing method, device and system |
| CN102063330A (en) * | 2011-01-05 | 2011-05-18 | 北京航空航天大学 | Performance data acquisition method for large-scale parallel program |
| CN102279888A (en) * | 2011-08-24 | 2011-12-14 | 北京新媒传信科技有限公司 | Method and system for scheduling tasks |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105426493A (en) * | 2015-11-24 | 2016-03-23 | 北京中电普华信息技术有限公司 | Data processing system and method applied to distributed storage system |
| CN105426493B (en) * | 2015-11-24 | 2019-11-19 | 北京中电普华信息技术有限公司 | A kind of data processing system and method applied to distributed memory system |
| CN106021543A (en) * | 2016-05-26 | 2016-10-12 | 中国农业银行股份有限公司 | Data pre-processing method and device |
| CN108733691A (en) * | 2017-04-18 | 2018-11-02 | 北京京东尚科信息技术有限公司 | Data preprocessing method and device |
| CN107037989A (en) * | 2017-05-17 | 2017-08-11 | 北京小米移动软件有限公司 | Data processing method and device in distributed computing system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103188306B (en) | 2016-04-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8286154B2 (en) | Apparatus and method for live loading of version upgrades in a process control environment | |
| US20170134526A1 (en) | Seamless cluster servicing | |
| CN108427571A (en) | A kind of dynamic link library update method and server | |
| CN102023881A (en) | Method and device for upgrading software as well as embedded device | |
| CN101964725B (en) | Method and system for realizing upgrading without interrupting service | |
| CN111831289B (en) | Micro-service automatic deployment management system and method | |
| CN103188306A (en) | Distributed preprocessing method and distributed preprocessing system | |
| CN102053831A (en) | Linux operating system customizing method and system | |
| CN101004681A (en) | Method for implementing long jumping dynamic patch in embedded system | |
| CN106155759A (en) | virtual machine batch creation method and device | |
| CN109885642B (en) | Hierarchical storage method and device for full-text retrieval | |
| CN104102535A (en) | Process migration method and migratable operating system | |
| CN110471774A (en) | A kind of data processing method and device based on unified task schedule | |
| CN109960517A (en) | Production method, device, system and the storage medium of image file | |
| CN102946413A (en) | Method and system for resource preprocessing in dispatching and deployment performing process of virtual machine | |
| CN111158743B (en) | Big data operation and maintenance management platform | |
| CN113918281A (en) | Method for improving cloud resource expansion efficiency of container | |
| CN104182295A (en) | Data backup method and data backup device | |
| CN104077266A (en) | Multi-core operating system realization method, realization device and realization system | |
| CN108958745B (en) | Device and method for deploying Spark cluster on cloud platform | |
| CN103677879A (en) | Software installation method and corresponding software installation device | |
| CN114579364A (en) | Cloud native database backup method based on hybrid cloud | |
| EP2605127B1 (en) | Processing Framework For Generating Pre-Configuration Packages | |
| CN104503811B (en) | Upgrade method for communication device and system based on single memory block | |
| CN112350837B (en) | Cloud platform-based power application cluster management method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |