CN110780869A - Distributed batch scheduling - Google Patents

Distributed batch scheduling Download PDF

Info

Publication number
CN110780869A
CN110780869A CN201911054368.5A CN201911054368A CN110780869A CN 110780869 A CN110780869 A CN 110780869A CN 201911054368 A CN201911054368 A CN 201911054368A CN 110780869 A CN110780869 A CN 110780869A
Authority
CN
China
Prior art keywords
task
management
scheduling
batch scheduling
batch
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201911054368.5A
Other languages
Chinese (zh)
Inventor
赵爱政
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Liaoning Zhenxing Bank Co Ltd
Original Assignee
Liaoning Zhenxing Bank Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Liaoning Zhenxing Bank Co Ltd filed Critical Liaoning Zhenxing Bank Co Ltd
Priority to CN201911054368.5A priority Critical patent/CN110780869A/en
Publication of CN110780869A publication Critical patent/CN110780869A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/50Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5083Techniques for rebalancing the load in a distributed system
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00Arrangements for software engineering
    • G06F8/30Creation or generation of source code
    • G06F8/34Graphical or visual programming
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/48Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806Task transfer initiation or dispatching
    • G06F9/4843Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The invention relates to the technical field of software system architecture, and discloses a distributed batch scheduling, which comprises a batch scheduling processing system, a batch scheduling data storage device, a service registration center, a batch scheduling management and control system, a meta-service center, a unified management and control center and a unified management and control center data storage device, and further comprises the following steps: the batch scheduling processing system is used for implementing and scheduling processing tasks; the batch scheduling data storage device is used for storing the batch scheduling processing task related persistent data; the service registration center is used for providing a service registration function, the cost can be saved through the design, the overall requirement on hardware is reduced by adopting a distributed deployment architecture, the operation and maintenance cost is saved, the characteristics of decentralized deployment and centralized maintenance are realized due to the fact that only one set of unified management and control center is needed, the expansibility is high, and the performance can be linearly increased along with the increase of batch scheduling equipment.

Description

Distributed batch scheduling
Technical Field
The invention relates to the technical field of software system architecture, in particular to distributed batch scheduling.
Background
Batch processing is an indispensable part of an enterprise-level business system and is used for developing vital data batch processing business in an enterprise information system, a batch scheduling platform refers to a scheduling platform which uniformly arranges batch tasks of all systems, uniformly preprocesses the batch tasks, uniformly schedules the batch tasks at regular time or manually schedules the batch tasks, and uniformly governs batch jobs through remote system scheduling and the like, and a distributed batch scheduling platform refers to a distributed architecture adopted by the batch scheduling platform, embodies high expandability, and highlights high cohesion and low coupling architecture.
However, the existing Spring Batch only provides a set of development framework for Batch operation, and does not consider the deployment architecture and the operation and maintenance cost; quartz also simply provides a set of development framework for timing scheduling; Control-M is too high in cost, is not a distributed architecture, is deployed in a centralized manner and is not strong in expansibility. Accordingly, one skilled in the art provides a distributed batch scheduling to solve the problems set forth in the background above.
Disclosure of Invention
The present invention is directed to a distributed batch scheduling method, which solves the problems set forth in the background art.
In order to achieve the purpose, the invention provides the following technical scheme:
a distributed batch scheduling comprises a batch scheduling processing system, a batch scheduling data storage device, a service registration center, a batch scheduling management and control system, a meta-service center, a unified management and control center and a unified management and control center data storage device, and further comprises the following steps:
the batch scheduling processing system is used for implementing and scheduling processing tasks; the batch scheduling data storage device is used for storing the batch scheduling processing task related persistent data; the service registration center is used for providing a service registration function; the batch scheduling control system is used for relevant definition and control of batch scheduling of users; the meta-service center is used for providing scheduling service related to query for the unified management and control center; the unified control center is used for persisting the received related data to a unified control center data storage device and the like; the unified control center data storage device is used for storing relevant data of the unified control center. The batch scheduling processing system comprises a resource receiving module, a resource judging module, a resource inquiring module, a resource processing module and a resource scheduling module, wherein the resource receiving module is used for receiving task information sent by the unified control center; the resource judging module is used for judging whether the application task meets preset filtering conditions, if not, the application task is immediately terminated, and if so, the next step is executed; the resource query module is used for querying the execution state of the task in the database, waiting for the processing result of the previous task if the task is being executed, and executing the next step if the task is not being executed; the resource processing module is used for sequencing the tasks according to the required resources from large to small, sequencing the resource occupancy rates on the task nodes according to the sequence from small to large, and finally distributing the sequenced tasks on each sequenced task node; and the resource scheduling module is used for scheduling the tasks according to the resources required by the tasks and the resource occupancy rates of the task nodes.
As a still further scheme of the invention: the batch scheduling processing system includes but is not limited to: the method comprises the steps of manual task scheduling, timing task scheduling, task exception handling, a scheduling engine, dynamic job fragmentation, task concurrent calling, function packaging to a service registration center, persistent batch scheduling of relevant data on a data storage device and the like.
As a still further scheme of the invention: relevant content of the batch scheduling data storage persisted includes, but is not limited to: job definition data, task definition data, job arrangement data, task arrangement data, job running data, task running data, abnormal situation data, and the like.
As a still further scheme of the invention: the batch scheduling management and control system comprises but is not limited to: parameter management, job definition, job compiling into tasks, task definition, task arrangement, remote calling node management, distributed service calling management, function encapsulation to a service registration center, persistent batch scheduling of relevant data to a data storage device and the like.
As a still further scheme of the invention: the meta-service centers include, but are not limited to: definition management of services, authentication management of services, service routing management, etc.
As a still further scheme of the invention: the unified management and control center includes but is not limited to: the system comprises a visual user management system, a visual role management system, a visual environment management system, a visual business management system, a visual parameter management system, a visual menu management system, a visual job compiling interface, a visual job definition interface, a visual task compiling interface, a visual remote calling node management interface, a visual job monitoring interface, a visual task operation interface, a visual abnormal monitoring interface, a remote calling batch scheduling management and control system, a meta-service center for inquiring related scheduling services, a remote calling batch scheduling processing system, a unified management and control center data storage device and the like.
As a still further scheme of the invention: the method for sequentially distributing the sequenced tasks to the sequenced task nodes comprises the following steps: selecting task nodes with the resource occupancy rate smaller than a preset first numerical value from the sequenced task nodes; and allocating the a-th task to the a-th task node, wherein the initial values of a and A are both 1, judging whether the resource occupancy rate of the a-th task node is greater than or equal to a preset second numerical value after the a-th task is allocated, if so, enabling a = a +1 and A = A +1, returning to the step of allocating the a-th task to the a-th task node, and if not, enabling a = a +1 and returning to the step of allocating the a-th task to the a-th task node.
Compared with the prior art, the invention has the beneficial effects that:
1: the cost is saved, and the overall requirement on hardware is reduced by adopting a distributed deployment architecture;
2: the operation and maintenance cost is saved, and the characteristics of decentralized deployment and centralized maintenance are realized due to the fact that only one set of unified control center is needed;
3: the system has high expansibility, and the performance can linearly increase along with the increase of batch scheduling equipment;
4: the design can also carry out corresponding task scheduling according to the required resources and the resource occupancy rate of the task nodes, and the problem caused by load balancing based on the number of tasks is avoided.
Drawings
FIG. 1 is a schematic diagram of a distributed batch scheduling architecture;
FIG. 2 is a functional block diagram of a batch scheduling processing system in distributed batch scheduling.
Detailed Description
Referring to fig. 1-2, in an embodiment of the present invention, a distributed batch scheduling includes a batch scheduling processing system 100, a batch scheduling data storage device 200, a service registry 300, a batch scheduling management and control system 400, a meta service center 500, a unified management and control center 600, and a unified management and control center data storage device 700, and further includes the following steps:
the batch scheduling processing system 100 is used for implementing scheduling of processing tasks; the batch scheduling data storage device 200 is used for storing batch scheduling processing task related persistent data; the service registry 300 is used for providing a service registration function; the batch scheduling management and control system 400 is used for defining and managing and controlling batch scheduling of users; the meta-service center 500 is used for providing scheduling services related to query for the unified management and control center 600; the unified management and control center 600 is used for persisting the received related data to the unified management and control center data storage device 700, and the like; the unified management and control center data storage device 700 is used for storing relevant data of the unified management and control center 600. The batch scheduling processing system 100 comprises a resource receiving module 101, a resource judging module 102, a resource inquiring module 103, a resource processing module 104 and a resource scheduling module 105, wherein the resource receiving module 101 is used for receiving task information sent by the unified control center 600; the resource judging module 102 is configured to judge whether the application task satisfies a preset filtering condition, if not, terminate immediately, and if so, execute the next step; the resource query module 103 is configured to query an execution state of the task in the database, wait for a processing result of a previous task if the task is being executed, and execute a next step if the task is not being executed; the resource processing module 104 is configured to sort the tasks according to the required resources in a descending order, sort the resource occupancy rates on the task nodes in a descending order, and finally distribute the sorted tasks to the sorted task nodes respectively; the resource scheduling module 105 is configured to perform task scheduling according to resources required by the task and the resource occupancy rate of the task node;
preferably: the batch scheduling processing system 100 includes, but is not limited to: manual task scheduling, timing task scheduling, task exception handling, scheduling engine, job dynamic fragmentation, task concurrent calling, function encapsulation to the service registry 300, persistent batch scheduling of relevant data to the data storage device 200, and the like.
Preferably: relevant content of the batch scheduling data store 200 persisted includes, but is not limited to: job definition data, task definition data, job arrangement data, task arrangement data, job running data, task running data, abnormal situation data, and the like.
Preferably: the batch scheduling management and control system 400 includes, but is not limited to: parameter management, job definition, job compilation into tasks, task definition, task orchestration, remote call node management, distributed service call management, encapsulating functions into a service registry 300, persistently scheduling data storage 200 for relevant data in batches, and the like.
Preferably: the meta-service center 500 includes, but is not limited to: definition management of services, authentication management of services, service routing management, etc.
Preferably: the unified governing center 600 includes but is not limited to: visual user management, visual role management, visual environment management, visual business management, visual parameter management, visual menu management, visual job making interface, visual job definition interface, visual task making interface, visual remote calling node management interface, visual job monitoring interface, visual task operation interface, visual abnormal monitoring interface, remote calling batch scheduling management and control system 400, inquiring relevant scheduling service from meta-service center 500, remote calling batch scheduling processing system 100, and persisting relevant data to unified management and control center data storage device 700 and the like.
Preferably: the method for sequentially distributing the sequenced tasks to the sequenced task nodes comprises the following steps: selecting task nodes with the resource occupancy rate smaller than a preset first numerical value from the sequenced task nodes; and allocating the a-th task to the A-th task node, wherein the initial values of a and A are both 1, judging whether the resource occupancy rate of the A-th task node is greater than or equal to a preset second numerical value after the a-th task is allocated, if so, enabling a = a +1 and A = A +1, returning to the step of allocating the a-th task to the A-th task node, and if not, enabling a = a +1 and returning to the step of allocating the a-th task to the A-th task node.
The distributed batch scheduling platform is designed by adopting a micro-service architecture, the distributed batch scheduling architecture can visually and centrally manage a plurality of environments (testing/developing/pressure testing/production) and various service unit batch services only by one set of unified control center, and the distributed batch scheduling architecture is the main characteristic of the architecture, and the batch scheduling control system and the batch scheduling processing system can access new batch scheduling services along with the paired deployment of the service units.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art should be considered to be within the technical scope of the present invention, and the technical solutions and the inventive concepts thereof according to the present invention are equivalent to or changed within the technical scope of the present invention.

Claims (7)

1. A distributed batch scheduling comprises a batch scheduling processing system 100, a batch scheduling data storage device 200, a service registration center 300, a batch scheduling management and control system 400, a meta-service center 500, a unified management and control center 600 and a unified management and control center data storage device 700, and further comprises the following steps:
the batch scheduling processing system 100 is used for implementing scheduling of processing tasks; the batch scheduling data storage device 200 is used for storing batch scheduling processing task related persistent data; the service registration center 300 is used for providing a service registration function; the batch scheduling management and control system 400 is used for defining and managing and controlling batch scheduling of users; the meta-service center 500 is configured to provide scheduling services related to query for the unified management and control center 600; the unified management and control center 600 is configured to persist the received related data to the unified management and control center data storage device 700, and the like; the unified management and control center data storage device 700 is used for storing relevant data of the unified management and control center 600;
the batch scheduling processing system 100 comprises a resource receiving module 101, a resource judging module 102, a resource inquiring module 103, a resource processing module 104 and a resource scheduling module 105, wherein the resource receiving module 101 is used for receiving task information sent by the unified control center 600; the resource judging module 102 is configured to judge whether the application task meets a preset filtering condition, if not, terminate immediately, and if so, execute the next step; the resource query module 103 is configured to query an execution state of the task in a database, wait for a processing result of a previous task if the task is being executed, and execute a next step if the task is not being executed; the resource processing module 104 is configured to sort the tasks according to the required resources in a descending order, sort the resource occupancy rates on the task nodes in a descending order, and finally distribute the sorted tasks to the sorted task nodes respectively; the resource scheduling module 105 is configured to perform task scheduling according to the resources required by the task and the resource occupancy rate of the task node.
2. The distributed batch scheduling of claim 1 wherein said batch scheduling processing system 100 includes but is not limited to: manual task scheduling, timing task scheduling, task exception handling, scheduling engine, job dynamic fragmentation, task concurrent calling, function encapsulation to the service registry 300, persistent batch scheduling of relevant data to the data storage device 200, and the like.
3. A distributed batch schedule as defined in claim 1 in which the relevant content persisted by the batch schedule data store 200 includes, but is not limited to: job definition data, task definition data, job arrangement data, task arrangement data, job running data, task running data, abnormal situation data, and the like.
4. The distributed batch scheduling of claim 1 wherein said batch scheduling management and control system 400 includes but is not limited to: parameter management, job definition, job compilation into tasks, task definition, task orchestration, remote call node management, distributed service call management, encapsulating functions into a service registry 300, persistently scheduling data storage 200 for relevant data in batches, and the like.
5. The distributed batch scheduling of claim 1 wherein said meta-service center 500 includes but is not limited to: definition management of services, authentication management of services, service routing management, etc.
6. The distributed batch scheduling of claim 1 wherein said unified management and control center 600 includes but is not limited to: visual user management, visual role management, visual environment management, visual business management, visual parameter management, visual menu management, visual job making interface, visual job definition interface, visual task making interface, visual remote calling node management interface, visual job monitoring interface, visual task operation interface, visual abnormal monitoring interface, remote calling batch scheduling management and control system 400, inquiring relevant scheduling service from meta-service center 500, remote calling batch scheduling processing system 100, and persisting relevant data to unified management and control center data storage device 700 and the like.
7. The distributed batch scheduling of claim 1 wherein said sequentially assigning the sequenced tasks to the sequenced task nodes comprises the steps of: selecting task nodes with the resource occupancy rate smaller than a preset first numerical value from the sequenced task nodes; and allocating the a-th task to the a-th task node, wherein the initial values of a and a are both 1, judging whether the resource occupancy rate of the a-th task node is greater than or equal to a preset second value after the a-th task is allocated, if so, making a be a +1 and a be a +1, returning to the step of allocating the a-th task to the a-th task node, and if not, making a be a +1 and returning to the step of allocating the a-th task to the a-th task node.
CN201911054368.5A 2019-10-31 2019-10-31 Distributed batch scheduling Pending CN110780869A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911054368.5A CN110780869A (en) 2019-10-31 2019-10-31 Distributed batch scheduling

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911054368.5A CN110780869A (en) 2019-10-31 2019-10-31 Distributed batch scheduling

Publications (1)

Publication Number Publication Date
CN110780869A true CN110780869A (en) 2020-02-11

Family

ID=69388246

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911054368.5A Pending CN110780869A (en) 2019-10-31 2019-10-31 Distributed batch scheduling

Country Status (1)

Country Link
CN (1) CN110780869A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111459639A (en) * 2020-04-03 2020-07-28 杭州趣维科技有限公司 Distributed task management platform and method supporting global multi-machine-room deployment
CN112818065A (en) * 2020-11-25 2021-05-18 辽宁振兴银行股份有限公司 Distributed architecture based on springclosed + mysql + eureka

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1787000A (en) * 2005-11-15 2006-06-14 中国工商银行股份有限公司 System and method for centralized managing and dispatching of batch work
CN103679392A (en) * 2013-12-26 2014-03-26 拉卡拉支付有限公司 Task scheduling processing method and system
WO2016101638A1 (en) * 2014-12-23 2016-06-30 国家电网公司 Operation management method for electric power system cloud simulation platform
WO2016101423A1 (en) * 2014-12-23 2016-06-30 中兴通讯股份有限公司 Operation scheduling method and cloud scheduling server
CN105975334A (en) * 2016-04-25 2016-09-28 深圳市永兴元科技有限公司 Distributed scheduling method and system of task
CN109002348A (en) * 2018-07-26 2018-12-14 郑州云海信息技术有限公司 Load-balancing method and device in a kind of virtualization system

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1787000A (en) * 2005-11-15 2006-06-14 中国工商银行股份有限公司 System and method for centralized managing and dispatching of batch work
CN103679392A (en) * 2013-12-26 2014-03-26 拉卡拉支付有限公司 Task scheduling processing method and system
WO2016101638A1 (en) * 2014-12-23 2016-06-30 国家电网公司 Operation management method for electric power system cloud simulation platform
WO2016101423A1 (en) * 2014-12-23 2016-06-30 中兴通讯股份有限公司 Operation scheduling method and cloud scheduling server
CN105975334A (en) * 2016-04-25 2016-09-28 深圳市永兴元科技有限公司 Distributed scheduling method and system of task
CN109002348A (en) * 2018-07-26 2018-12-14 郑州云海信息技术有限公司 Load-balancing method and device in a kind of virtualization system

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111459639A (en) * 2020-04-03 2020-07-28 杭州趣维科技有限公司 Distributed task management platform and method supporting global multi-machine-room deployment
CN111459639B (en) * 2020-04-03 2023-10-20 杭州小影创新科技股份有限公司 Distributed task management platform and method supporting global multi-machine room deployment
CN112818065A (en) * 2020-11-25 2021-05-18 辽宁振兴银行股份有限公司 Distributed architecture based on springclosed + mysql + eureka

Similar Documents

Publication Publication Date Title
CN108762896B (en) Hadoop cluster-based task scheduling method and computer equipment
CN115328663B (en) Method, device, equipment and storage medium for scheduling resources based on PaaS platform
CN102567086B (en) Task scheduling method, equipment and system
KR100509794B1 (en) Method of scheduling jobs using database management system for real-time processing
CN104657212A (en) Task scheduling method and system
CN109766172B (en) Asynchronous task scheduling method and device
CN110780869A (en) Distributed batch scheduling
CN114168302A (en) Task scheduling method, device, equipment and storage medium
CN103595654A (en) HQoS implementation method, device and network equipment based on multi-core CPUs
CN113886069A (en) Resource allocation method and device, electronic equipment and storage medium
CN102243655B (en) Method and device for managing database connections
CN115048206A (en) Resource scheduling method and server
CN114168297A (en) Method, device, equipment and medium for scheduling collection tasks
CN110912967A (en) Service node scheduling method, device, equipment and storage medium
CN102243653A (en) Method and device for managing database connections
CN111913784A (en) Task scheduling method and device, network element and storage medium
CN111752675B (en) Internet of things platform based on containerization technology
CN117873666A (en) Task scheduling method, chip, electronic device and readable storage medium
CN114896050B (en) Task scheduling method and system based on cluster resources
CN113051064A (en) Task scheduling method, device, equipment and storage medium
CN112035579A (en) Graph management method, data storage method, data query method, device and storage medium
CN115712572A (en) Task testing method and device, storage medium and electronic device
CN111770179B (en) High-performance high-availability cloud networking gateway implementation method, medium and terminal
CN113204434B (en) Planned task execution method and device based on k8s and computer equipment
US20050114858A1 (en) Cancelled task management in a computer system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20200211