CN110134533A - It is a kind of can lot size scheduling data system and method - Google Patents
It is a kind of can lot size scheduling data system and method Download PDFInfo
- Publication number
- CN110134533A CN110134533A CN201910399131.4A CN201910399131A CN110134533A CN 110134533 A CN110134533 A CN 110134533A CN 201910399131 A CN201910399131 A CN 201910399131A CN 110134533 A CN110134533 A CN 110134533A
- Authority
- CN
- China
- Prior art keywords
- node
- layer
- scheduling
- lot size
- establishment
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/541—Client-server
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention belongs to big data processing technology fields, and in particular to it is a kind of can lot size scheduling data system and method, comprising: framework builds unit, is used to build the three-tier architecture of the system；Establishment of item unit is used to obtain the establishment of item information of user's secondary development, and disposes multi-stage scheduling node based on the establishment of item information；Traffic control unit is used to carry out the batch task scheduling of load balancing by the multi-stage scheduling node.The present invention can not only lot size scheduling data, and can carry out manually being arranged and intervene, load balancing when scheduling has perfect scheduling controlling strategy.
Description
Technical field
The invention belongs to big data processing technology fields, and in particular to it is a kind of can lot size scheduling data system and method.
Background technique
In big data era, data are gold, and data are the critical assets of group, Nai Ge enterprise, entire society, run number well
According to, make good use of the important proposition that data are entire societies.Want to make good use of data, should just run data well first.And lot size scheduling is certainly
Dynamicization technology exactly runs the important guarantee of data well.In numerous big and small data warehouses, Data Mart and various
It is that lot size scheduling automatic technology allows disengaging, storage, cleaning, filtering, roughing, the fining-off of mass data in data pond
Orderly, efficiently it is unfolded etc. various work.
Currently, existing azkaban scheduling tool can solve based on timed task, it is based on time interval, is based on relational dependence
Relative complex scheduler task.But Azkaban dispatch scale limitation, have manually participate in not flexible, scheduling load is unbalanced,
The disadvantages of incomplete scheduling controlling strategy.
Summary of the invention
For the defects in the prior art, the present invention provides and it is a kind of can lot size scheduling data system and method, no
Only can lot size scheduling data, and can carry out manually being arranged and intervene, load balancing when scheduling has perfect scheduling controlling plan
Slightly.
In a first aspect, the present invention provides it is a kind of can lot size scheduling data system characterized by comprising
Framework builds unit, is used to build the three-tier architecture of the system；
Establishment of item unit is used to obtain the establishment of item information of user's secondary development, and is based on the establishment of item
Information disposes multi-stage scheduling node；
Traffic control unit is used to carry out the batch task scheduling of load balancing by the multi-stage scheduling node.
Wherein, the three-tier architecture includes application layer, control layer and destination layer.
Wherein, the three-tier architecture of the system is built using typical C/S model.
Wherein, the establishment of item information that user's secondary development is obtained by the application layer is believed according to the establishment of item
Breath disposes the multi-stage scheduling node of the control layer.
Wherein, in the operational process of project, the control layer bears the destination layer by multi-stage scheduling node
Balanced batch task scheduling is carried, the destination layer executes corresponding task program according to the batch task scheduling of control layer.
Wherein, the application layer is client, and the control layer is server-side, and the destination layer is to be deployed in ETL service
Task program on device.
Wherein, the control layer is made of, the control layer using multistage pyramid structure a variety of different types of nodes
Including EM node, Server node and Agent node, the Agent node includes MAgent node and SAgent node；
The EM node is controlled for the access authority to application layer, is also used to for being communicated with application layer
Manage and control effective operation of all nodes；
The Server node saves Agent for completing for being communicated respectively at EM node and Agent node
The scheduling controlling of point；
The Agent node, using master-salve proxy cascade system, for being communicated with destination layer, for according to target
The ETL server resource use state of layer, carries out load balancing deployment, and the ETL server for assigning the task to relative free is held
Row task program.
Wherein, the establishment of item information includes between project name, each node and each node in project assignment process
Connection relationship.
Wherein, the application layer includes Admin module, Designer module and Monitor module；
The Admin module is used for, for managing and being arranged project name；
The Designer module is used for, between each node and each node for being arranged in project work flow
Connection relationship；
The Monitor module is used for, for running project and monitored item purpose operational process.
Wherein, each node is made of the different multiple component processes of function, completes to communicate by Socket between node,
It completes to communicate by message queue mode between component processes.
Wherein, the component processes include FDC process, DRR process, DAR process, STR process, KIM process, NLS process,
SPS process, CPG process, UCD process, EMR process, JMM process, DSY process and FIM process.
Second aspect, the invention also provides it is a kind of can lot size scheduling data automation implementation method, be suitable for right
It is required that 1-7 it is described in any item can lot size scheduling data system, which comprises the following steps:
The three-tier architecture of the system is built using typical C/S model, the three-tier architecture includes application layer, control layer
And destination layer；
The establishment of item information that user's secondary development is obtained by the application layer, according to the deployment of establishment of item information
The multi-stage scheduling node of control layer；
In the operational process of project, the control layer carries out load balancing to the destination layer by multi-stage scheduling node
Batch task scheduling, the destination layer executes corresponding task program according to the batch task scheduling of the control layer.
Wherein, the control layer is made of, the control layer using multistage pyramid structure a variety of different types of nodes
Including EM node, Server node and Agent node, the Agent node includes MAgent node and SAgent node；
The EM node is controlled for the access authority to application layer, is also used to for being communicated with application layer
Manage and control effective operation of all nodes；
The Server node saves Agent for completing for being communicated respectively at EM node and Agent node
The scheduling controlling of point；
The Agent node, using master-salve proxy cascade system, for being communicated with destination layer, for according to target
The ETL server resource use state of layer, carries out load balancing deployment, and the ETL server for assigning the task to relative free is held
Row task program.
Wherein, each node is made of the different multiple component processes of function, completes to communicate by Socket between node,
It completes to communicate by message queue mode between component processes；
The component processes include FDC process, DRR process, DAR process, STR process, KIM process, NLS process, SPS into
Journey, CPG process, UCD process, EMR process, JMM process, DSY process and FIM process.
The embodiment of the present invention, can not only lot size scheduling data, and can carry out manually being arranged and intervene, load is equal when scheduling
Weighing apparatus has perfect scheduling controlling strategy.
Detailed description of the invention
It, below will be to specific in order to illustrate more clearly of the specific embodiment of the invention or technical solution in the prior art
Embodiment or attached drawing needed to be used in the description of the prior art are briefly described.In all the appended drawings, similar element
Or part is generally identified by similar appended drawing reference.In attached drawing, each element or part might not be drawn according to actual ratio.
Fig. 1 be the embodiment of the present invention in can lot size scheduling data system structure chart；
Fig. 2 is the three-tier architecture schematic diagram of system described in the embodiment of the present invention；
Fig. 3 be the embodiment of the present invention in can lot size scheduling data automation implementation method flow chart.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation description, it is clear that described embodiments are some of the embodiments of the present invention, instead of all the embodiments.Based on this hair
Embodiment in bright, every other implementation obtained by those of ordinary skill in the art without making creative efforts
Example, shall fall within the protection scope of the present invention.
It should be appreciated that ought use in this specification and in the appended claims, term " includes " and "comprising" instruction
Described feature, entirety, step, operation, the presence of element and/or component, but one or more of the other feature, whole is not precluded
Body, step, operation, the presence or addition of element, component and/or its set.
It is also understood that mesh of the term used in this description of the invention merely for the sake of description specific embodiment
And be not intended to limit the present invention.As description of the invention and it is used in the attached claims, unless on
Other situations are hereafter clearly indicated, otherwise " one " of singular, "one" and "the" are intended to include plural form.
It will be further appreciated that the term "and/or" used in description of the invention and the appended claims is
Refer to any combination and all possible combinations of one or more of associated item listed, and including these combinations.
Embodiment one:
Present embodiments provide it is a kind of can lot size scheduling data system, as shown in Figure 1, comprising:
Framework builds unit, is used to build the three-tier architecture of the system；
Establishment of item unit is used to obtain the establishment of item information of user's secondary development, and is based on the establishment of item
Information disposes multi-stage scheduling node；
Traffic control unit is used to carry out the batch task scheduling of load balancing by the multi-stage scheduling node.
The three-tier architecture for the system that the present embodiment is built is as shown in Fig. 2, the application layer is client, the control
Layer is server-side, and the destination layer is the various task programs being deployed on ETL server.Patent document
201520554128.2 disclosing a kind of big data processing platform network architecture, including core layer switch, application virtualization clothes
Business device, data-base cluster, storage array, backup server and at least an interchanger；The application virtualization server, data
Library cluster, storage array and backup server are separately connected the core layer switch, and the storage array connects the friendship
It changes planes, the interchanger is separately connected the application virtualization server and data-base cluster.This technical solution meets processing
Hardware environment required for big data；And there is open, scalability.Current mass data is mainly stored in traditional SQL
In database, the NoSQL database used with big data technology is very different, simultaneously because the diversity feature of data,
Before handling data using big data platform, need data to import the storage system of big data platform oneself, and when importing one
As need into ETL (data warehouse technology) processing, complete the extraction of Various types of data, clean, load etc. processes.
ETL is the abbreviation of English Extract-Transform-Load, for describing data from source terminal by extracting
(extract), the process of (transform), load (load) to destination are converted.Traditional ETL tool is setting one special
Some transform engines are placed between data source and target data warehouse, it is used to use all conversion programs, this method solution
It has determined the problem of using different programming languages in different system platform, it is only necessary to grasp the programming language of ETL tool itself
?.
Application layer mainly divides admin, designer, monitor from the angle of function in the present invention.Control layer is multistage
Pyramid architecture, top layer are service control node, complete various dispatch service controls and provide various operations for client to answer
With service.And the completion of Agent layer Agent layer is interacted with the control of target stratum server.In addition, Agent layer passes through master-salve proxy cascade side
Formula is, it can be achieved that be scheduled control, realization load balancing etc. to the server of clustered deploy(ment).Destination layer is that entire product is controlled
The target of system, such as our ET server, application station etc..
In the present embodiment, after the three-tier architecture for putting up basis, some projects, establishment of item can be created by application layer
After good, in the operational process of project, according to the demand of operation task, the batch that control layer carries out load balancing to destination layer is appointed
Business scheduling, destination layer execute corresponding task program according to scheduling.
The application layer in the present embodiment includes Admin module, Designer module, Monitor module etc.；
The Admin module is used for, for managing and being arranged project name；
The Designer module is used for, between each node and each node for being arranged in project work flow
Connection relationship；
The Monitor module is used for, for running project and monitored item purpose operational process.
Establishment of item information described in the present embodiment includes project name, each node and Ge Jie in project assignment process
Connection relationship between point.User creates specific project assignment process by Admin and Designer, can pass through after having created
Monitor carries out simulation monitoring project operation process.
The embodiment of the present invention control layer is a Pyramid, adopts and is made of a variety of different types of nodes, described
Control layer includes EM node (i.e. core node), Server node (i.e. control node) and Agent node (i.e. agent node), institute
Stating Agent node includes MAgent (i.e. home agent node) node and SAgent node (i.e. from agent node).It is these types of different
The node of type has the function of different and function.
The EM node is controlled for the access authority to application layer, is also used to for being communicated with application layer
Manage and control effective operation of all nodes；
The Server node saves Agent for completing for being communicated respectively at EM node and Agent node
The scheduling controlling of point；
The Agent node, using master-salve proxy cascade system, for being communicated with destination layer, for according to target
The ETL server resource use state of layer, carries out load balancing deployment, and the ETL server for assigning the task to relative free is held
Row task program.
In the present embodiment, complete to communicate by Socket between node.In actual operation, every transaction all generates once
Connecting moves, this is not only embodied in the communication between client and core node, meanwhile, it is also embodied in logical between all nodes
Letter.Be between core node it is reciprocity, each node can to other nodes initiate service request, each node is both client
End and server-side.The control layer of the present embodiment is a multilayer logic system, by disposing different nodes in Different Logic layer
It shows, this multilayered structure is not fixed and invariable, and user can carry out control layer according to the scale and demand of project flexible
Deployment, whole system can be simple, can also be complicated.
Each node of the present embodiment is made of the different multiple component processes of function, passes through message team between component processes
Column mode is completed to communicate.The component processes include FDC (Flow Dispatch Core) process, DRR (Dispatch
Request Router) process, DAR (Dispatch Answer Router) process, STR (Send Message To
Remoto) process, KIM (Kernel Integrate Manange) process, NLS (Net Listten) process, SPS
(Search Plugin State) process, CPG (Call Plugin) process, UCD (User Command Deal) journey, EMR
(Kernel Event Manange And Release) process, JMM (Job Mutex Manange) process, DSY (Data
Synchronous) process and FIM (Flow Instance Manange) process.
In the present embodiment, different component processes have the function of different, and in practical applications, user can be according to project
The component processes for selecting to need are needed, in order to effectively realize synchronous communication and asynchronous communication between component processes, in physical message team
On the basis of column, assignment of logical request queue and response queue are distinguished for each component processes.
Request queue: receiving the queue of other component processes request messages, at this point, current process is the service for the service that provides
End.
Response queue: receiving the message queue of the response message of other service processes, at this point, current process is request service
Client.
Since each process has request queue and response queue, illustrates each process i.e. and can provide service, can also request
Service, when the service of offer, component is server-side；When a service is requested, component is client.This feature and product core section
Communication mechanism is similar between point, and communication is reciprocity between core node, and equally, communication is also equity between node inner assembly.
In the present embodiment when being scheduled to ETL server, using load-balancing mechanism, load balancing deployment be in order to
Physical resource is efficiently used, and improves ETL treatment effeciency.The cascade system that it mainly passes through agency is realized.Load balancing is phase
For a cluster, i.e., is executed at one in the execution domain that agency's cascade is constituted and realize load balancing.For in a cluster,
It is required that the task deployment on each ETL server is the same.Control layer will appoint according to ETL server resource service condition in cluster
Business is automatically assigned to the ETL host of relative free and executes task program.
In conclusion the system of this implementation can not only lot size scheduling data, and can carry out manually be arranged intervene, scheduling
When load balancing, have perfect scheduling controlling strategy.
Embodiment two:
Present embodiments provide it is a kind of can lot size scheduling data automation implementation method, suitable for described in embodiment one
Can lot size scheduling data system, comprising the following steps:
S1, the three-tier architecture of the system is built using typical C/S model, and the three-tier architecture includes application layer, control
Preparative layer and destination layer；
S2 obtains the establishment of item information of user's secondary development by application layer, is disposed and is controlled according to establishment of item information
The multi-stage scheduling node of layer；
S3, in the operational process of project, control layer carries out load balancing to destination layer by multi-stage scheduling node and criticizes
Task schedule is measured, destination layer executes corresponding task program according to the batch task scheduling of control layer.
The three-tier architecture for the system that the present embodiment is built is as shown in Fig. 2, the application layer is client, the control
Layer is server-side, and the destination layer is the various task programs being deployed on ETL server.In the present embodiment.
Application layer mainly divides admin, designer, monitor from the angle of function.Control layer is multistage pyramid frame
Structure, top layer are service control node, complete various dispatch service controls and provide various operation application services for client.And
The completion of Agent layer Agent layer is interacted with the control of target stratum server.In addition, Agent layer pass through master-salve proxy cascade system, it can be achieved that
Control is scheduled to the server of clustered deploy(ment), realizes load balancing etc..Destination layer is the target that entire product is controlled,
Such as our ET server, application station etc..
In the present embodiment, after the three-tier architecture for putting up basis, some projects, establishment of item can be created by application layer
After good, in the operational process of project, according to the demand of operation task, the batch that control layer carries out load balancing to destination layer is appointed
Business scheduling, destination layer execute corresponding task program according to scheduling.
The application layer in the present embodiment includes Admin module, Designer module, Monitor module etc.；
The Admin module is used for, for managing and being arranged project name；
The Designer module is used for, between each node and each node for being arranged in project work flow
Connection relationship；
The Monitor module is used for, for running project and monitored item purpose operational process.
Establishment of item information described in the present embodiment includes project name, each node and Ge Jie in project assignment process
Connection relationship between point.User creates specific project assignment process by Admin and Designer, can pass through after having created
Monitor carries out simulation monitoring project operation process.
The embodiment of the present invention control layer is a Pyramid, adopts and is made of a variety of different types of nodes, described
Control layer includes EM node (i.e. core node), Server node (i.e. control node) and Agent node (i.e. agent node), institute
Stating Agent node includes MAgent (i.e. home agent node) node and SAgent node (i.e. from agent node).It is these types of different
The node of type has the function of different and function.
The EM node is controlled for the access authority to application layer, is also used to for being communicated with application layer
Manage and control effective operation of all nodes；
The Server node saves Agent for completing for being communicated respectively at EM node and Agent node
The scheduling controlling of point；
The Agent node, using master-salve proxy cascade system, for being communicated with destination layer, for according to target
The ETL server resource use state of layer, carries out load balancing deployment, and the ETL server for assigning the task to relative free is held
Row task program.
In the present embodiment, complete to communicate by Socket between node.In actual operation, every transaction all generates once
Connecting moves, this is not only embodied in the communication between client and core node, meanwhile, it is also embodied in logical between all nodes
Letter.Be between core node it is reciprocity, each node can to other nodes initiate service request, each node is both client
End and server-side.The control layer of the present embodiment is a multilayer logic system, by disposing different nodes in Different Logic layer
It shows, this multilayered structure is not fixed and invariable, and user can carry out control layer according to the scale and demand of project flexible
Deployment, whole system can be simple, can also be complicated.
Each node of the present embodiment is made of the different multiple component processes of function, passes through message team between component processes
Column mode is completed to communicate.The component processes include FDC (Flow Dispatch Core) process, DRR (Dispatch
Request Router) process, DAR (Dispatch Answer Router) process, STR (Send Message To
Remoto) process, KIM (Kernel Integrate Manange) process, NLS (Net Listten) process, SPS (Search
Plugin State) process, CPG (Call Plugin) process, UCD (User Command Deal) journey, EMR (Kernel
Event Manange And Release) process, JMM (Job Mutex Manange) process, DSY (Data
Synchronous) process and FIM (Flow Instance Manange) process.
In the present embodiment, different component processes have the function of different, and in practical applications, user can be according to project
The component processes for selecting to need are needed, in order to effectively realize synchronous communication and asynchronous communication between component processes, in physical message team
On the basis of column, assignment of logical request queue and response queue are distinguished for each component processes.
Request queue: receiving the queue of other component processes request messages, at this point, current process is the service for the service that provides
End.
Response queue: receiving the message queue of the response message of other service processes, at this point, current process is request service
Client.
Since each process has request queue and response queue, illustrates each process i.e. and can provide service, can also request
Service, when the service of offer, component is server-side；When a service is requested, component is client.This feature and product core section
Communication mechanism is similar between point, and communication is reciprocity between core node, and equally, communication is also equity between node inner assembly.
In the present embodiment when being scheduled to ETL server, using load-balancing mechanism, load balancing deployment be in order to
Physical resource is efficiently used, and improves ETL treatment effeciency.The cascade system that it mainly passes through agency is realized.Load balancing is phase
For a cluster, i.e., is executed at one in the execution domain that agency's cascade is constituted and realize load balancing.For in a cluster,
It is required that the task deployment on each ETL server is the same.Control layer will appoint according to ETL server resource service condition in cluster
Business is automatically assigned to the ETL host of relative free and executes task program.
In conclusion the method for this implementation can not only lot size scheduling data, and can carry out manually be arranged intervene, scheduling
When load balancing, have perfect scheduling controlling strategy.
Those of ordinary skill in the art may be aware that system and method described in conjunction with the examples disclosed in this document
Step can be realized with electronic hardware, computer software, or a combination of the two, in order to clearly demonstrate hardware and software
Interchangeability generally describes each exemplary composition and step according to function in the above description.These functions are studied carefully
Unexpectedly it is implemented in hardware or software, the specific application and design constraint depending on technical solution.Professional technique people
Member can use different methods to achieve the described function each specific application, but this realization is it is not considered that super
The scope of the present invention out.
In several embodiments provided herein, it should be understood that disclosed method and system can pass through it
Its mode is realized.For example, in addition the division of above step, only a kind of logical function partition can have in actual implementation
Division mode, such as multiple steps can be combined into a step, and a step step can also be split as multiple steps.It can also
The purpose of the embodiment of the present invention is realized to select some or all of step therein according to the actual needs.
Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention., rather than its limitations；To the greatest extent
Pipe present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that: its according to
So be possible to modify the technical solutions described in the foregoing embodiments, or to some or all of the technical features into
Row equivalent replacement；And these are modified or replaceed, various embodiments of the present invention technology that it does not separate the essence of the corresponding technical solution
The range of scheme should all cover within the scope of the claims and the description of the invention.
Claims (10)
1. one kind can lot size scheduling data system characterized by comprising
Framework builds unit, is used to build the three-tier architecture of the system；
Establishment of item unit is used to obtain the establishment of item information of user's secondary development, and is based on the establishment of item information
Dispose multi-stage scheduling node；
Traffic control unit is used to carry out the batch task scheduling of load balancing by the multi-stage scheduling node.
2. one kind according to claim 1 can lot size scheduling data system, which is characterized in that the three-tier architecture includes
Application layer, control layer and destination layer.
3. one kind according to claim 1 can lot size scheduling data system, which is characterized in that use typical C/S mould
Formula builds the three-tier architecture of the system.
4. one kind according to claim 2 can lot size scheduling data system, it is characterised in that obtained by the application layer
The establishment of item information for taking family secondary development disposes the multi-stage scheduling section of the control layer according to the establishment of item information
Point.
5. one kind according to claim 2 can lot size scheduling data system, which is characterized in that in the operational process of project
In, the control layer carries out the batch task scheduling of load balancing, the target by multi-stage scheduling node to the destination layer
Layer executes corresponding task program according to the batch task scheduling of control layer.
6. one kind according to claim 2 can lot size scheduling data system, which is characterized in that the application layer be client
End, the control layer are server-side, and the destination layer is the task program being deployed on ETL server.
7. one kind according to claim 2 can lot size scheduling data system, which is characterized in that
The control layer is made of using multistage pyramid structure a variety of different types of nodes, and the control layer includes EM section
Point, Server node and Agent node, the Agent node include MAgent node and SAgent node；
The EM node is controlled for being communicated with application layer for the access authority to application layer, is also used to manage
With the effective operation for controlling all nodes；
The Server node, for being communicated respectively at EM node and Agent node, for completing to Agent node
Scheduling controlling；
The Agent node, using master-salve proxy cascade system, for being communicated with destination layer, for according to destination layer
ETL server resource use state carries out load balancing deployment, and the ETL server for assigning the task to relative free, which executes, appoints
Business program.
8. one kind can lot size scheduling data automation implementation method, be suitable for claim 1-7 it is described in any item can batch
The system for dispatching data, which comprises the following steps:
The three-tier architecture of the system is built using typical C/S model, the three-tier architecture includes application layer, control layer and mesh
Mark layer；
The establishment of item information that user's secondary development is obtained by the application layer disposes the control according to establishment of item information
The multi-stage scheduling node of layer；
In the operational process of project, the control layer carries out load balancing to the destination layer by multi-stage scheduling node and criticizes
Task schedule is measured, the destination layer executes corresponding task program according to the batch task scheduling of the control layer.
9. one kind according to claim 8 can lot size scheduling data automation implementation method, which is characterized in that
The control layer is made of using multistage pyramid structure a variety of different types of nodes, and the control layer includes EM section
Point, Server node and Agent node, the Agent node include MAgent node and SAgent node；
The EM node is controlled for being communicated with application layer for the access authority to application layer, is also used to manage
With the effective operation for controlling all nodes；
The Server node, for being communicated respectively at EM node and Agent node, for completing to Agent node
Scheduling controlling；
The Agent node, using master-salve proxy cascade system, for being communicated with destination layer, for according to destination layer
ETL server resource use state carries out load balancing deployment, and the ETL server for assigning the task to relative free, which executes, appoints
Business program.
10. one kind according to claim 9 can lot size scheduling data automation implementation method, which is characterized in that it is each
Node is made of the different multiple component processes of function, completes to communicate by Socket between node, is passed through between component processes
Message queue mode is completed to communicate；
The component processes include FDC process, DRR process, DAR process, STR process, KIM process, NLS process, SPS process,
CPG process, UCD process, EMR process, JMM process, DSY process and FIM process.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910399131.4A CN110134533B (en) | 2019-05-14 | 2019-05-14 | System and method capable of scheduling data in batches |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910399131.4A CN110134533B (en) | 2019-05-14 | 2019-05-14 | System and method capable of scheduling data in batches |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110134533A true CN110134533A (en) | 2019-08-16 |
| CN110134533B CN110134533B (en) | 2020-04-28 |
Family
ID=67573989
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910399131.4A Active CN110134533B (en) | 2019-05-14 | 2019-05-14 | System and method capable of scheduling data in batches |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110134533B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113094192A (en) * | 2021-04-23 | 2021-07-09 | 杭州网易云音乐科技有限公司 | Data processing method, device, medium and equipment |
| CN113761046A (en) * | 2021-09-13 | 2021-12-07 | 中远海运科技股份有限公司 | Workflow ETL-based processing method and system |
| CN114553956A (en) * | 2022-01-04 | 2022-05-27 | 北京国电通网络技术有限公司 | Data transmission method and system based on UEP middleware |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080104264A1 (en) * | 2006-11-01 | 2008-05-01 | Duerk Vicky P | Load balancing in a storage system |
| CN101951411A (en) * | 2010-10-13 | 2011-01-19 | 戴元顺 | Cloud scheduling system and method and multistage cloud scheduling system |
| CN101957780A (en) * | 2010-08-17 | 2011-01-26 | 中国电子科技集团公司第二十八研究所 | Resource state information-based grid task scheduling processor and grid task scheduling processing method |
| CN104239144A (en) * | 2014-09-22 | 2014-12-24 | 珠海许继芝电网自动化有限公司 | Multilevel distributed task processing system |
| CN105703940A (en) * | 2015-12-10 | 2016-06-22 | 中国电力科学研究院 | Multistage dispatching distributed parallel computing-oriented monitoring system and monitoring method |
| US20180210694A1 (en) * | 2017-01-26 | 2018-07-26 | Gibson Brands, Inc. | Plug-in load balancing |
| CN109254846A (en) * | 2018-08-01 | 2019-01-22 | 国电南瑞科技股份有限公司 | The dynamic dispatching method and system of CPU and GPU cooperated computing based on two-level scheduler |
| CN109743390A (en) * | 2019-01-04 | 2019-05-10 | 深圳壹账通智能科技有限公司 | Method for scheduling task, device, computer equipment and storage medium |
-
2019
- 2019-05-14 CN CN201910399131.4A patent/CN110134533B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080104264A1 (en) * | 2006-11-01 | 2008-05-01 | Duerk Vicky P | Load balancing in a storage system |
| CN101957780A (en) * | 2010-08-17 | 2011-01-26 | 中国电子科技集团公司第二十八研究所 | Resource state information-based grid task scheduling processor and grid task scheduling processing method |
| CN101951411A (en) * | 2010-10-13 | 2011-01-19 | 戴元顺 | Cloud scheduling system and method and multistage cloud scheduling system |
| CN104239144A (en) * | 2014-09-22 | 2014-12-24 | 珠海许继芝电网自动化有限公司 | Multilevel distributed task processing system |
| CN105703940A (en) * | 2015-12-10 | 2016-06-22 | 中国电力科学研究院 | Multistage dispatching distributed parallel computing-oriented monitoring system and monitoring method |
| US20180210694A1 (en) * | 2017-01-26 | 2018-07-26 | Gibson Brands, Inc. | Plug-in load balancing |
| CN109254846A (en) * | 2018-08-01 | 2019-01-22 | 国电南瑞科技股份有限公司 | The dynamic dispatching method and system of CPU and GPU cooperated computing based on two-level scheduler |
| CN109743390A (en) * | 2019-01-04 | 2019-05-10 | 深圳壹账通智能科技有限公司 | Method for scheduling task, device, computer equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 方勇杰: "适应多级调度安全稳定分析资源共享的分布式计算管理平台", 《电力***自动化》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113094192A (en) * | 2021-04-23 | 2021-07-09 | 杭州网易云音乐科技有限公司 | Data processing method, device, medium and equipment |
| CN113094192B (en) * | 2021-04-23 | 2024-05-28 | 杭州网易云音乐科技有限公司 | Data processing method, device, medium and equipment |
| CN113761046A (en) * | 2021-09-13 | 2021-12-07 | 中远海运科技股份有限公司 | Workflow ETL-based processing method and system |
| CN114553956A (en) * | 2022-01-04 | 2022-05-27 | 北京国电通网络技术有限公司 | Data transmission method and system based on UEP middleware |
| CN114553956B (en) * | 2022-01-04 | 2024-01-09 | 北京国电通网络技术有限公司 | Data transmission method and system based on unified extensible firmware protocol (UEP) middleware |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110134533B (en) | 2020-04-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110134533A (en) | It is a kind of can lot size scheduling data system and method | |
| CN109286653A (en) | Intelligent cloud engineering platform | |
| CN108762900A (en) | High frequency method for scheduling task, system, computer equipment and storage medium | |
| CN104391918B (en) | The implementation method of distributed networks database query priority management based on equity deployment | |
| CN106126332A (en) | Distributed timing task scheduling system and method | |
| CN109302321A (en) | Server, service demand processing system, method and monitoring system | |
| CN108243012B (en) | Charging application processing system, method and device in OCS (online charging System) | |
| CN110209646A (en) | A kind of data platform system calculated based on real-time streaming | |
| CN105930417B (en) | A kind of big data ETL interactive process platform based on cloud computing | |
| CN107943963A (en) | Mass data distributed rule engine operation system based on cloud platform | |
| CN113672240A (en) | Container-based multi-machine-room batch automatic deployment application method and system | |
| CN110245029A (en) | A kind of data processing method, device, storage medium and server | |
| CN102780583B (en) | Method for service description, service combination and service quality assessment of Internet of Things | |
| CN103442038A (en) | Master-slave distributed cooperation type HLA simulation management and control system | |
| CN103646134A (en) | Service-oriented networked simulation system dynamic generation method | |
| CN109800226A (en) | A kind of data administer in task management method and device | |
| CN105094984A (en) | Resource scheduling method and system | |
| CN106528169A (en) | Web system development reusable method based on AnGo dynamic evolution model | |
| CN103024791A (en) | Multi-service processing model and multi-service processing method in communication test system | |
| US20080021955A1 (en) | Message oriented middleware server pipeline architecture | |
| CN100547967C (en) | Mesh operation supervisor and grid job management method | |
| CN107944743A (en) | The operation and maintenance method, apparatus and system of a kind of large computer system | |
| CN106354612A (en) | Service-oriented packaging method and system of cloud diagnosis resource | |
| US20020156611A1 (en) | Performance simulation process, and multiprocessor application production process, and devices for implementing said processes | |
| CN106330556B (en) | A kind of method and apparatus calling related information for generating service module |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |