CN112685161B - Unified task scheduling system - Google Patents
Unified task scheduling system Download PDFInfo
- Publication number
- CN112685161B CN112685161B CN202011623543.0A CN202011623543A CN112685161B CN 112685161 B CN112685161 B CN 112685161B CN 202011623543 A CN202011623543 A CN 202011623543A CN 112685161 B CN112685161 B CN 112685161B
- Authority
- CN
- China
- Prior art keywords
- task
- module
- machine
- scheduling
- tasks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000015654 memory Effects 0.000 claims description 11
- 230000001960 triggered effect Effects 0.000 claims description 6
- 230000001174 ascending effect Effects 0.000 claims description 3
- 210000001503 joint Anatomy 0.000 abstract description 3
- 230000009286 beneficial effect Effects 0.000 abstract description 2
- 238000000034 method Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
Landscapes
- Multi Processors (AREA)
Abstract
The invention discloses a unified task scheduling system. The system comprises a task starting module, a task stopping module, a group module, a scheduling module and a machine module; the task starting module is used for sending various task starting requests and is connected with the group module; the task stopping module is used for carrying a task id to send a task stopping request and is connected with the group module; the group module is used for defining the scheduling rules of various tasks and is connected with the scheduling module; the scheduling module is used for forwarding tasks according to rules and is connected with the machine module; the machine module is used for executing and recording tasks. The beneficial effects of the invention are as follows: the unified specification processing can be carried out on various tasks, so that a calling end does not need to be concerned with the service of really executing the tasks, only needs to be in butt joint with the task scheduling service, and meanwhile, dynamic lateral expansion is supported to increase the upper limit of task operation.
Description
Technical Field
The invention relates to the technical field related to task gateways, in particular to a unified task scheduling system.
Background
In actual service, if a task, such as live transcoding or offline transcoding, needs to be issued, a task service interface is generally directly docked in the current service, which increases the configuration complexity of the current service: addresses of different tasks need to be configured, the maximum running number of tasks cannot be dynamically controlled, and meanwhile, the lateral expansion is limited (task addresses need to be newly added in the configuration of services).
Disclosure of Invention
The invention aims to overcome the defects in the prior art and provides a unified task scheduling system supporting dynamic transverse expansion.
In order to achieve the above purpose, the present invention adopts the following technical scheme:
a unified task scheduling system comprises a task starting module, a task stopping module, a group module, a scheduling module and a machine module;
the task starting module is used for sending various task starting requests and is connected with the group module;
the task stopping module is used for carrying a task id to send a task stopping request and is connected with the group module;
the group module is used for defining the scheduling rules of various tasks and is connected with the scheduling module;
the scheduling module is used for forwarding tasks according to rules and is connected with the machine module;
The machine module is used for executing and recording tasks.
The invention discloses a unified task scheduling system which comprises a task starting module, a task stopping module, a group module, a scheduling module and a machine module.
Preferably, the task starting module is used for starting task information, is triggered by a calling end service, and is attached with a task type, a task required performance value and task content to send an http request to the group module; the task stopping module is used for stopping the current task, and is triggered by the calling end service, and the task id recorded by the attached task starting module sends an http request to the group module.
Preferably, the group module is an entry of the whole task scheduling system, and before the whole system operates, the following information needs to be defined:
first: task types, namely task types supported by the whole task scheduling system;
Second,: scheduling metrics fall into two categories: task number, machine performance value;
Third,: scheduling algorithms fall into two categories: load balancing, most preferred.
Preferably, the task number refers to the maximum number of tasks operated by each machine module; the machine performance value represents a quantized value that the machine module is subject to, calculated from the configuration of the machine, the formula: for the core number of the cpu and the memory size, the performance value required by the current task needs to be calculated in the task starting module to issue the task, and the machine is used up as soon as one of the two indexes of the cpu and the memory arrives first.
Preferably, the load balancing means that tasks are evenly distributed to the machine module; the most priority refers to that one machine occupies the full and then goes to the other machine for delivery.
Preferably, after the above definition is completed, the system starts to receive the requests of the task start module and the task stop module, and the different request processing logics are different:
(1) After receiving a request of a task starting module, firstly acquiring information defined in a group module according to the transmitted type parameter, wherein the information comprises a scheduling index and a scheduling algorithm, and then combining the information and the parameter transmitted by the task starting module and transmitting the information to the scheduling module;
(2) And after receiving the request of the task stopping module, directly carrying the task id transferred by the task stopping module and forwarding the task id to the scheduling module.
Preferably, the scheduling module receives the start and stop requests forwarded by the group module, and the processing logic is different:
(a) After receiving a task starting request forwarded by a group module, inquiring whether a machine module has matched task machines or not according to task types, inquiring whether the machine module has spare machines or not according to scheduling indexes, finally obtaining an optimal machine according to a scheduling algorithm, calling a starting task in the machine module to issue, generating a unique identifier of the task by the scheduling module after the task is issued successfully, and storing in a related manner with a task id and machine information which are actually operated in the machine module and responding to the task starting module by adopting a random UUID mode;
(b) After receiving the task stopping request forwarded by the group module, directly inquiring the machine information of the task according to the information stored by the scheduling module and the transmitted task id, and then calling the stopping task in the machine module.
Preferably, the machine module is a machine which really runs tasks and is a module supporting dynamic infinite lateral expansion, each machine module needs to be associated with a group module, so that the machine module directly has task types, maximum task numbers and maximum performance values, and in addition, the machine module needs to additionally store dynamic information which is used for inquiring and using by a scheduling module.
Preferably, the specific query is as follows:
(i) Inquiring whether a judging standard of an idle machine exists or not: whether the number of the current running tasks or the performance value occupied by the current tasks reaches the upper limit defined by the group module or not;
(ii) Algorithm for querying the optimal machine: and after the idle machine set is obtained, the first machine is arranged and fetched according to the number of the residual tasks or the residual performance value in ascending order or descending order.
Preferably, the machine module also differs in processing logic when receiving start and stop tasks:
(A) After receiving the task starting request, executing real task starting, and then accumulating data according to a scheduling algorithm configured in the current group module;
(B) After receiving the task stopping request, executing real task stopping, and then performing data reduction according to a scheduling algorithm configured in the current group module.
The beneficial effects of the invention are as follows: the unified specification processing can be carried out on various tasks, so that a calling end does not need to be concerned with the service of really executing the tasks, only needs to be in butt joint with the task scheduling service, and meanwhile, dynamic lateral expansion is supported to increase the upper limit of task operation.
Drawings
Fig. 1 is a system block diagram of the present invention.
Detailed Description
The invention is further described below with reference to the drawings and detailed description.
In the embodiment shown in fig. 1, a unified task scheduling system includes a task start module, a task stop module, a group module, a scheduling module, and a machine module;
The task starting module is used for sending various task starting requests and is connected with the group module; the task starting module is used for starting task information, is triggered by the calling end service, and is attached with a task type, a performance value required by a task and task content to send an http request to the group module, for example: starting a live broadcast transcoding task example when the rainbow cloud:
Request for
POST http:// unified task service address/task/starttype =live & performanceVal =1-2
<xml>
<task>
<....>
</task>
</xml>
Response to
{taskId:xxx}
Wherein:
type is task type
PerformanceVal is the performance value required by the current task, 1-2 represents the resource that requires 1 core 2GB of memory
Xml is the content of the transcoding task and needs to run in the machine module
The task id is the current task id responded by the task id (the task stopping module) and needs to be recorded at the calling end, and the task id needs to carry the parameter
The task stopping module is used for carrying a task id to send a task stopping request and is connected with the group module; the task stopping module is used for stopping the current task, and is triggered by the calling end service, and the task id recorded by the attached task starting module sends an http request to the group module, for example: stopping when the iris live transcoding task paradigm:
Request for
POST http:// unified task service address/task/stop
Wherein:
xxx is task identification recorded by task starting module
The group module is used for defining the scheduling rules of various tasks and is connected with the scheduling module; the group module is an entrance of the whole task scheduling system, and before the whole system operates, the following information needs to be defined:
First: task types, i.e., task types supported by the overall task scheduling system, such as: live transcoding, on-demand transcoding, timed tasks, fast-coded tasks, etc. (the task type is not limited, depending on the specific service);
Second,: scheduling metrics fall into two categories: task number, machine performance value; the task number refers to the maximum number of tasks operated by each machine module; the machine performance value represents a quantized value that the machine module is subject to, calculated from the configuration of the machine, the formula: for the number of cpu cores and the memory size, for example, the performance value of a machine with 4 cores and 8GB memories is 4-8, the performance value required by the current task needs to be calculated in a task starting module to issue the task, for example, the resource required by one high-definition live transcoding task is 2 cores and 2GB memories, the performance value of the task is 2-2, and the machine is used up by which of the two indexes of the cpu and the memory arrives first, so that the machine with 4 cores and 8GB memories can only run 2 high-definition live transcoding tasks.
Third,: scheduling algorithms fall into two categories: load balancing and most priority; load balancing means that tasks are averagely issued to a machine module, for example, 2 machines and 10 tasks are provided, and each machine runs 5 tasks (provided that the scheduling index is the number of tasks and the number of the tasks is set to be more than 5); the most preferred is that one machine occupies the full and then issues to another machine, for example, 2 machines, the scheduling index is the task number, and the task number is set to 10, if 10 tasks are issued, one of the task numbers is 10, and the other is 0.
The definition examples are as follows:
Wherein:
groupId is a [ group module ] identifier
DISPATCHTYPE is a scheduling index, taskNum represents the task number, and pv represents the performance value
MaxVal is the maximum value under different scheduling criteria, such as 10 or 3-9
DispatchAlgorithm is a scheduling algorithm, lb represents load balancing, mf represents maximum priority
After the definition is completed, the system starts to receive the requests of the task starting module and the task stopping module, and different request processing logics are different:
(1) After receiving a request of a task starting module, firstly acquiring information defined in a group module according to the transmitted type parameter, wherein the information comprises a scheduling index and a scheduling algorithm, and then combining the information and the parameter transmitted by the task starting module and transmitting the information to the scheduling module;
(2) And after receiving the request of the task stopping module, directly carrying the task id transferred by the task stopping module and forwarding the task id to the scheduling module.
The scheduling module is used for forwarding tasks according to rules and is connected with the machine module; the scheduling module has different processing logic when receiving the start and stop requests forwarded by the group module:
(a) After receiving a task starting request forwarded by a group module, the machine module needs to inquire whether a matched task machine exists or not according to a task type, then inquires whether a spare machine exists or not according to a scheduling index, finally obtains an optimal machine according to a scheduling algorithm, calls a starting task in the machine module to issue, and after the task is issued successfully, the scheduling module needs to generate a unique identifier of the task, and stores and responds to the task starting module by associating with a task id and machine information which are actually operated in the machine module in a random UUID mode, wherein the stored information is as follows:
Wherein:
the taskId is the task id that responds to [ task Start Module ]
HostId machine identification of [ machine module ] for subsequent querying of the associated machine according to task id
HostTaskId is the true running task id in [ machine Module ], which is used when stopping tasks
(B) After receiving the task stopping request forwarded by the group module, directly inquiring the machine information of the task according to the information stored by the scheduling module and the transmitted task id, and then calling the stopping task in the machine module.
The machine module is used for executing and recording tasks; the machine module refers to a machine which really runs tasks and is a module supporting dynamic infinite lateral expansion, each machine module needs to be associated with a group module, so that the machine module directly has task types, maximum task numbers (if a scheduling index is the task number) and maximum performance values (if the scheduling index is the performance value), and additionally needs to store dynamic information, such as the number of currently running tasks and the total performance value occupied by all the currently running tasks, and the stored information is exemplified as follows:
Wherein:
hostId is a machine identification for scheduling module queries for machine usage
The groupId is a (group module) identifier, and is used for obtaining the scheduling index, the scheduling algorithm and the like of the machine
CurrentTaskNum is the total number of tasks running on the current machine, which is used as a requirement for inquiring the idle machine and the optimal machine
CurrentPerformanceVal is the total performance value of the task running on the current machine, 3-7 represents the resources of the 3-core 7GB memory that the machine has used as a necessary condition for inquiring the idle machine and the optimal machine [ scheduling module ]
The dynamic information is used for inquiring and using by the scheduling module; the specific query is as follows:
(i) Inquiring whether a judging standard of an idle machine exists or not: whether the number of the current running tasks or the performance value occupied by the current tasks reaches the upper limit defined by the group module or not;
(ii) Algorithm for querying the optimal machine: after the idle machine set is obtained, the first is arranged according to the number of the remaining tasks or the remaining performance value in ascending (load balancing) order or descending (most priority) order.
The processing logic of the machine module in receiving start and stop tasks is also different:
(A) After receiving a task starting request, executing real task starting, and then accumulating data according to a scheduling algorithm configured in a group module to which the task belongs, such as adding 1 to the number of tasks and adding 1 to the current total performance value to the task performance value;
(B) After receiving the task stopping request, executing real task stopping, and then performing data reduction according to a scheduling algorithm configured in the current group module, such as subtracting 1 from the number of tasks and subtracting the task performance value from the current total performance value.
The invention discloses a unified task scheduling system which comprises a task starting module, a task stopping module, a group module, a scheduling module and a machine module. The method is applied to rainbow cloud products, saves a great deal of butt joint work for calling end service in the process of integrating offline transcoding tasks and online transcoding tasks, and can adjust resources as required to adapt to business scenes of different orders.
Claims (7)
1. The unified task scheduling system is characterized by comprising a task starting module, a task stopping module, a group module, a scheduling module and a machine module;
the task starting module is used for sending various task starting requests and is connected with the group module;
the task stopping module is used for carrying a task id to send a task stopping request and is connected with the group module;
The group module is used for defining the scheduling rules of various tasks and is connected with the scheduling module; the group module is an entrance of the whole task scheduling system, and before the whole system operates, the following information needs to be defined:
first: task types, namely task types supported by the whole task scheduling system;
Second,: scheduling metrics fall into two categories: task number, machine performance value;
third,: scheduling algorithms fall into two categories: load balancing and most priority;
After the definition is completed, the system starts to receive the requests of the task starting module and the task stopping module, and different request processing logics are different:
(1) After receiving a request of a task starting module, firstly acquiring information defined in a group module according to the transmitted type parameter, wherein the information comprises a scheduling index and a scheduling algorithm, and then combining the information and the parameter transmitted by the task starting module and transmitting the information to the scheduling module;
(2) The task id which is directly carried with the task is transmitted to the scheduling module after the task stopping module request is received;
The scheduling module is used for forwarding tasks according to rules and is connected with the machine module; the scheduling module has different processing logic when receiving the start and stop requests forwarded by the group module:
(a) After receiving a task starting request forwarded by a group module, inquiring whether a machine module has matched task machines or not according to task types, inquiring whether the machine module has spare machines or not according to scheduling indexes, finally obtaining an optimal machine according to a scheduling algorithm, calling a starting task in the machine module to issue, generating a unique identifier of the task by the scheduling module after the task is issued successfully, and storing in a related manner with a task id and machine information which are actually operated in the machine module and responding to the task starting module by adopting a random UUID mode;
(b) After receiving a task stopping request forwarded by the group module, directly inquiring the machine information of the task according to the information stored by the scheduling module and the transmitted task id, and then calling the stopping task in the machine module;
The machine module is used for executing and recording tasks.
2. The unified task scheduling system according to claim 1, wherein the task starting module is configured to start task information, triggered by a calling end service, with a task type, a task required performance value, and a task content, and send an http request to the group module; the task stopping module is used for stopping the current task, and is triggered by the calling end service, and the task id recorded by the attached task starting module sends an http request to the group module.
3. A unified task scheduling system according to claim 1 wherein the number of tasks is the maximum number of tasks that each machine module runs; the machine performance value represents a quantized value that the machine module is subject to, calculated from the configuration of the machine, the formula: for the core number of the cpu and the memory size, the performance value required by the current task needs to be calculated in the task starting module to issue the task, and the machine is used up as soon as one of the two indexes of the cpu and the memory arrives first.
4. A unified task scheduling system according to claim 1, wherein the load balancing means that tasks are evenly distributed to the machine modules; the most priority refers to that one machine occupies the full and then goes to the other machine for delivery.
5. A unified task scheduling system according to claim 1, wherein the machine modules are machines for actually running tasks, and are modules for supporting dynamic infinite lateral expansion, each machine module needs to be associated with a group module, so that the machine modules directly have task types, maximum task numbers and maximum performance values, and in addition, the machine modules need to additionally store dynamic information for the scheduling module to query.
6. A unified task scheduling system according to claim 5, wherein the specific query is as follows:
(i) Inquiring whether a judging standard of an idle machine exists or not: whether the number of the current running tasks or the performance value occupied by the current tasks reaches the upper limit defined by the group module or not;
(ii) Algorithm for querying the optimal machine: and after the idle machine set is obtained, the first machine is arranged and fetched according to the number of the residual tasks or the residual performance value in ascending order or descending order.
7. A unified task scheduling system according to claim 6 wherein the processing logic of the machine module in receiving start and stop tasks is also different:
(A) After receiving the task starting request, executing real task starting, and then accumulating data according to a scheduling algorithm configured in the current group module;
(B) After receiving the task stopping request, executing real task stopping, and then performing data reduction according to a scheduling algorithm configured in the current group module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011623543.0A CN112685161B (en) | 2020-12-31 | 2020-12-31 | Unified task scheduling system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011623543.0A CN112685161B (en) | 2020-12-31 | 2020-12-31 | Unified task scheduling system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112685161A CN112685161A (en) | 2021-04-20 |
| CN112685161B true CN112685161B (en) | 2024-04-23 |
Family
ID=75454033
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011623543.0A Active CN112685161B (en) | 2020-12-31 | 2020-12-31 | Unified task scheduling system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112685161B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102831494A (en) * | 2012-07-23 | 2012-12-19 | 北京星网锐捷网络技术有限公司 | Scheduling method, scheduling device and scheduling system |
| CN105260248A (en) * | 2014-07-15 | 2016-01-20 | ***股份有限公司 | Task scheduling system with dynamic regulation function and scheduling method |
| CN110647398A (en) * | 2019-09-18 | 2020-01-03 | 北京航空航天大学 | Intersection control task scheduling method facing edge calculation and based on task criticality and timeliness |
| CN110895487A (en) * | 2018-09-12 | 2020-03-20 | 北京奇虎科技有限公司 | Distributed task scheduling system |
| CN111026519A (en) * | 2018-10-10 | 2020-04-17 | 上海宝信软件股份有限公司 | Distributed task priority scheduling method and system and storage medium |
| CN111429033A (en) * | 2020-04-21 | 2020-07-17 | 贵州新致普惠信息技术有限公司 | Workflow task scheduling system and method and service terminal |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030214657A1 (en) * | 2002-05-14 | 2003-11-20 | Stringham Gary Glen | Task scheduling and automated task performance in printers |
| US8185422B2 (en) * | 2006-07-31 | 2012-05-22 | Accenture Global Services Limited | Work allocation model |
-
2020
- 2020-12-31 CN CN202011623543.0A patent/CN112685161B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102831494A (en) * | 2012-07-23 | 2012-12-19 | 北京星网锐捷网络技术有限公司 | Scheduling method, scheduling device and scheduling system |
| CN105260248A (en) * | 2014-07-15 | 2016-01-20 | ***股份有限公司 | Task scheduling system with dynamic regulation function and scheduling method |
| CN110895487A (en) * | 2018-09-12 | 2020-03-20 | 北京奇虎科技有限公司 | Distributed task scheduling system |
| CN111026519A (en) * | 2018-10-10 | 2020-04-17 | 上海宝信软件股份有限公司 | Distributed task priority scheduling method and system and storage medium |
| CN110647398A (en) * | 2019-09-18 | 2020-01-03 | 北京航空航天大学 | Intersection control task scheduling method facing edge calculation and based on task criticality and timeliness |
| CN111429033A (en) * | 2020-04-21 | 2020-07-17 | 贵州新致普惠信息技术有限公司 | Workflow task scheduling system and method and service terminal |
Non-Patent Citations (2)
| Title |
|---|
| Scheduling Different Types of Bag-of-Tasks Jobs in Distributed Systems;Georgios L. Stavrinides;2019 10th International Conference on Information and Communication Systems (ICICS);20190822;全文 * |
| 基于多核平台的入侵检测***的设计与实现;庄卓俊;中国优秀硕士学位论文全文数据库-信息科技辑;20111215;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112685161A (en) | 2021-04-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111580995B (en) | Synchronous communication method and system of distributed cloud platform and Internet of things intelligent terminal based on MQTT asynchronous communication scene | |
| CN101694665B (en) | Method and device for data query of heterogeneous data source | |
| US20120136946A1 (en) | Cluster server in instant messaging system and method for communicating between clusters | |
| US20130297801A1 (en) | Managing multiple proxy servers in a multi-tenant application system environment | |
| CN105978968A (en) | Real-time transmission processing method, server and system of mass data | |
| CN107093138A (en) | Auction Ask-Bid System and its operation method based on distributed clog-free asynchronous message tupe | |
| CN101383839A (en) | Data distribution system based on data server and implementation method | |
| CN102025650A (en) | Message processing system and message processing method of enterprise service bus | |
| CN104243610B (en) | A kind of distributed document transmits method of servicing | |
| CN108494826A (en) | A kind of distribution cloud storage method and system | |
| CN107065813A (en) | A kind of workshop appliance data monitoring system and method | |
| CN112866421B (en) | Intelligent contract operation method and device based on distributed cache and NSQ | |
| CN103297453A (en) | Method, browser end and system for achieving instant communication | |
| CN114710571B (en) | Data packet processing system | |
| CN103546528B (en) | Distributed message method for pushing and system | |
| CN102750368A (en) | High-speed importing method of cluster data in data base | |
| CN1972276B (en) | A management method and system for protocol access | |
| CN112685161B (en) | Unified task scheduling system | |
| CN104253702B (en) | A kind of terminal management method and system | |
| CN101938383A (en) | On-line monitoring method and system of user account with multiple services as well as monitoring service system | |
| CN104468520A (en) | Identity authentication method and device | |
| CN117032987A (en) | Distributed task scheduling method, system, equipment and computer readable medium | |
| CN104050284B (en) | A kind of data asynchronous query method and system based on one process | |
| CN107819831B (en) | Metaq and mns-based universal message system | |
| CN113727138A (en) | HLS intranet source returning method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |