CN112685161A - Unified task scheduling system - Google Patents

Unified task scheduling system Download PDF

Info

Publication number
CN112685161A
CN112685161A CN202011623543.0A CN202011623543A CN112685161A CN 112685161 A CN112685161 A CN 112685161A CN 202011623543 A CN202011623543 A CN 202011623543A CN 112685161 A CN112685161 A CN 112685161A
Authority
CN
China
Prior art keywords
task
module
machine
scheduling
tasks
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202011623543.0A
Other languages
Chinese (zh)
Other versions
CN112685161B (en
Inventor
代存折
李妃军
刘国强
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Arcvideo Technology Co ltd
Original Assignee
Hangzhou Arcvideo Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Arcvideo Technology Co ltd filed Critical Hangzhou Arcvideo Technology Co ltd
Priority to CN202011623543.0A priority Critical patent/CN112685161B/en
Publication of CN112685161A publication Critical patent/CN112685161A/en
Application granted granted Critical
Publication of CN112685161B publication Critical patent/CN112685161B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Landscapes

  • Multi Processors (AREA)

Abstract

The invention discloses a unified task scheduling system. The system comprises a task starting module, a task stopping module, a group module, a scheduling module and a machine module; the task starting module is used for sending various task starting requests and is connected with the group module; the task stopping module is used for carrying a task id to send a task stopping request and is connected with the group module; the group module is used for defining the scheduling rules of various tasks and is connected with the scheduling module; the scheduling module is used for forwarding tasks according to rules and is connected with the machine module; the machine module is used for executing and recording tasks. The invention has the beneficial effects that: the method can perform unified and standard processing on various tasks, so that a calling end does not need to care about the service of really executing the tasks, only needs to be connected with task scheduling service, and simultaneously supports dynamic transverse expansion to increase the upper limit of task operation.

Description

Unified task scheduling system
Technical Field
The invention relates to the technical field related to task gateways, in particular to a unified task scheduling system.
Background
In actual services, if a task needs to be issued, such as a live transcoding or offline transcoding task, a task service interface is generally directly connected to a current service, and in this case, the configuration complexity of the current service is increased: the addresses of different tasks need to be configured, the maximum running number of the tasks cannot be dynamically controlled, and meanwhile, the horizontal expansion is limited (the task addresses need to be newly added in the configuration of the service).
Disclosure of Invention
The invention provides a unified task scheduling system supporting dynamic lateral extension to overcome the defects in the prior art.
In order to achieve the purpose, the invention adopts the following technical scheme:
a unified task scheduling system comprises a task starting module, a task stopping module, a group module, a scheduling module and a machine module;
the task starting module is used for sending various task starting requests and is connected with the group module;
the task stopping module is used for carrying a task id to send a task stopping request and is connected with the group module;
the group module is used for defining the scheduling rules of various tasks and is connected with the scheduling module;
the scheduling module is used for forwarding tasks according to rules and is connected with the machine module;
the machine module is used for executing and recording tasks.
The invention discloses a unified task scheduling system, which comprises a task starting module, a task stopping module, a group module, a scheduling module and a machine module, and can carry out unified and standardized processing on various tasks through the scheme, so that a calling end does not need to care about the service of really executing the tasks, only needs to be connected with task scheduling service, and simultaneously supports dynamic transverse expansion to increase the upper limit of task operation.
Preferably, the task starting module is used for starting task information, is triggered by a calling end service, and sends an http request to the group module together with a task type, a performance value required by the task and task content; and the task stopping module is used for stopping the current task, is triggered by the calling end service, and sends an http request to the group module by the task id recorded by the attached task starting module.
Preferably, the group module is an entry of the whole task scheduling system, and before the whole system runs, the following information needs to be defined:
firstly, the method comprises the following steps: the task type is the task type supported by the whole task scheduling system;
secondly, the method comprises the following steps: scheduling indexes are divided into two types: number of tasks, machine performance values;
thirdly, the method comprises the following steps: the scheduling algorithm is divided into two types: load balancing, at most priority.
Preferably, the task number refers to the maximum number of tasks run by each machine module; the machine performance value represents a quantified value that the machine module is subjected to, the value being calculated from the configuration of the machine, and the formula: for the cpu core number-memory size, a performance value required by the current task needs to be calculated in the task starting module to issue the task, and the cpu and the memory which arrive first represent that the machine is used up.
Preferably, the load balancing means that the tasks are issued to the machine modules evenly; the most priority means that one machine is occupied fully and then is sent to another machine.
Preferably, after the above definition is completed, the system starts to receive the requests of the task starting module and the task stopping module, and different request processing logics are different:
(1) after receiving a task starting module request, acquiring information defined in a group module according to a transmitted type parameter, wherein the information comprises a scheduling index and a scheduling algorithm, and then merging the information and the parameter transmitted by a task starting module and forwarding the information to a scheduling module;
(2) and after receiving the request of the task stopping module, directly carrying the task id transmitted by the task stopping module and forwarding the task id to the scheduling module.
Preferably, the scheduling module, when receiving the start and stop requests forwarded by the group module, has different processing logic:
(a) after receiving a task starting request forwarded by a group module, inquiring whether a machine module has a matched task machine according to a task type, inquiring whether an idle machine exists according to a scheduling index, obtaining an optimal machine according to a scheduling algorithm, calling a starting task in the machine module for issuing, after the task is successfully issued, generating a unique identifier of the task by the scheduling module, and performing associated storage with a task id and machine information which are actually operated in the machine module and responding to the task starting module by adopting a random UUID mode;
(b) after receiving a task stop request forwarded by the group module, directly inquiring machine information of the task according to the transmitted task id and the information stored by the scheduling module, and then calling the stop task in the machine module.
Preferably, the machine module is a machine that really runs tasks, and is a module that supports dynamic infinite horizontal expansion, each machine module needs to be associated with a group module, so that the machine module directly has a task type, a maximum task number, and a maximum performance value, and in addition, the machine module needs to additionally store dynamic information, and the dynamic information is used for the scheduling module to perform query.
Preferably, the specific query is as follows:
(i) inquiring whether a judgment standard of an idle machine exists: whether the number of the current running tasks or the performance value occupied by the current tasks reaches the upper limit defined by the group module or not;
(ii) the algorithm for querying the optimal machine: and after the idle machine set is obtained, arranging and taking the first one in an ascending or descending order according to the residual task quantity or the residual performance value.
Preferably, the machine module has different processing logic when receiving start and stop tasks:
(A) after receiving a task starting request, executing real task starting, and then performing data accumulation according to a scheduling algorithm configured in a current group module to which the real task starting belongs;
(B) and after receiving the task stopping request, executing real task stopping, and then reducing data according to a scheduling algorithm configured in the current group module to which the real task stopping request belongs.
The invention has the beneficial effects that: the method can perform unified and standard processing on various tasks, so that a calling end does not need to care about the service of really executing the tasks, only needs to be connected with task scheduling service, and simultaneously supports dynamic transverse expansion to increase the upper limit of task operation.
Drawings
FIG. 1 is a system block diagram of the present invention.
Detailed Description
The invention is further described with reference to the following figures and detailed description.
In the embodiment shown in fig. 1, a unified task scheduling system includes a task starting module, a task stopping module, a group module, a scheduling module, and a machine module;
the task starting module is used for sending various task starting requests and is connected with the group module; the task starting module is used for starting task information, is triggered by a calling end service, and sends an http request to the group module together with a task type, a performance value required by the task and task content, for example: when the rainbow cloud live broadcast transcoding task paradigm is started:
request for
POST http:// unified task service address/task/starttype ═ live & performanceVal ═ 1-2
<xml>
<task>
<....>
</task>
</xml>
Response to
{taskId:xxx}
Wherein:
type is task type
The performanceVal is a performance value required by the current task, and 1-2 represents that resources of 1 kernel and 2GB of memory are required
xml is the content of the transcoding task, and needs to run in the machine module
the task id is the task id of the [ group module ] response, which needs to be recorded at the calling end, and the [ task stopping module ] needs to carry the parameter
The task stopping module is used for carrying a task id to send a task stopping request and is connected with the group module; the task stopping module is used for stopping the current task, is triggered by a calling end service, and sends an http request to the group module by the task id recorded by the attached task starting module, for example: stop when the rainbow cloud live transcoding task paradigm:
request for
POST http:// unified task service address/task/stop
Wherein:
xxx is the task identification recorded by the (task starting module)
The group module is used for defining the scheduling rules of various tasks and is connected with the scheduling module; the group module is an inlet of the whole task scheduling system, and before the whole system runs, the following information needs to be defined:
firstly, the method comprises the following steps: the task type, that is, the task type supported by the whole task scheduling system, includes: live transcoding, on-demand transcoding, timed tasks, fast-coded tasks, and the like (the task type is not limited, and is determined according to specific services);
secondly, the method comprises the following steps: scheduling indexes are divided into two types: number of tasks, machine performance values; the task number refers to the maximum task number of each machine module; the machine performance value represents a quantified value that the machine module is subjected to, the value being calculated from the configuration of the machine, and the formula: for the cpu core number-memory size, for example, the performance value of a 4-core 8-GB memory machine is 4-8, a performance value required by a current task needs to be calculated in a task starting module to issue the task, if a resource required by a high-definition live broadcast transcoding task is a 2-core 2-GB memory, the task performance value is 2-2, and if the cpu and the memory arrive first, the machine is used up, and the 4-core 8-GB memory machine can only run 2 high-definition live broadcast transcoding tasks of this type.
Thirdly, the method comprises the following steps: the scheduling algorithm is divided into two types: load balancing, most preferred; load balancing means that tasks are evenly distributed to a machine module, for example, 2 machines with 10 tasks, and each machine runs 5 tasks (if the scheduling index is the number of tasks and the number of tasks is greater than 5); the most priority is that one machine is occupied fully and then is issued to another machine, for example, 2 machines, the scheduling index is the number of tasks, and the number of tasks is set to 10, if 10 tasks are issued, one of the tasks is 10, and the other is 0.
Examples of definitions are as follows:
Figure BDA0002876816990000061
wherein:
the groupId is [ group module ] mark
dispatch type is a scheduling index, taskNum represents the number of tasks, and pv represents a performance value
maxVal is the maximum value under different scheduling indexes, such as 10 or 3-9
dispatch algorithm is scheduling algorithm, lb represents load balancing, mf represents maximum priority
After the above definition is completed, the system starts to receive the requests of the task starting module and the task stopping module, and the processing logics of different requests are different:
(1) after receiving a task starting module request, acquiring information defined in a group module according to a transmitted type parameter, wherein the information comprises a scheduling index and a scheduling algorithm, and then merging the information and the parameter transmitted by a task starting module and forwarding the information to a scheduling module;
(2) and after receiving the request of the task stopping module, directly carrying the task id transmitted by the task stopping module and forwarding the task id to the scheduling module.
The scheduling module is used for forwarding tasks according to rules and is connected with the machine module; the scheduling module has different processing logics when receiving the starting and stopping requests forwarded by the group module:
(a) after receiving a task starting request forwarded by a group module, inquiring whether a machine module has a matched task machine according to a task type, then inquiring whether an idle machine exists according to a scheduling index, finally obtaining an optimal machine according to a scheduling algorithm, calling a starting task in the machine module to issue, after the task is successfully issued, generating a unique identifier of the task by the scheduling module, performing associated storage with a task id and machine information which are actually operated in the machine module by adopting a random UUID mode, and responding to the task starting module, wherein the stored information example is as follows:
Figure BDA0002876816990000071
wherein:
taskId is the task id in response to [ task Start Module ]
The hostId is a machine identifier of a machine module, and is used for subsequently inquiring the machine to which the hostId belongs according to the task id
The hostTaskId is the task id actually running in the machine module, and is used when stopping the task
(b) After receiving a task stop request forwarded by the group module, directly inquiring machine information of the task according to the transmitted task id and the information stored by the scheduling module, and then calling the stop task in the machine module.
The machine module is used for executing and recording tasks; the machine module is a machine which really runs tasks, is a module which supports dynamic infinite horizontal expansion, each machine module needs to be associated with a group module, so that the machine module directly has a task type, a maximum task number (if a scheduling index is the task number) and a maximum performance value (if the scheduling index is the performance value), in addition, the machine module needs to additionally store dynamic information, such as the current running task number and the total performance value occupied by all the tasks, and the stored information example is as follows:
Figure BDA0002876816990000081
wherein:
hostId is machine identification used by the scheduling module to query machine usage
The group id is a group module identifier, and is used for acquiring the scheduling index, scheduling algorithm and the like of the machine
currentTaskNum is the total number of tasks operated by the current machine, and is used as a necessary condition for inquiring an idle machine and an optimal machine (a scheduling module)
currentPerformanceVal is a task total performance value of current machine operation, 3-7 represents that the machine has used resources of 3-core 7GB memory, and is used as a necessary condition for inquiring an idle machine and an optimal machine (a scheduling module)
The dynamic information is used for the scheduling module to inquire and use; the specific query is as follows:
(i) inquiring whether a judgment standard of an idle machine exists: whether the number of the current running tasks or the performance value occupied by the current tasks reaches the upper limit defined by the group module or not;
(ii) the algorithm for querying the optimal machine: and after the idle machine set is obtained, arranging the first one in an ascending order (load balancing) or descending order (most priority) according to the residual task number or residual performance value.
The processing logic of the machine module in receiving the start and stop tasks is also different:
(A) after receiving a task starting request, executing real task starting, and then performing data accumulation according to a scheduling algorithm configured in a group module to which the current task belongs, such as adding 1 to the number of tasks, and adding a current total performance value to the task performance value;
(B) after receiving a task stopping request, executing real task stopping, and then performing data reduction according to a scheduling algorithm configured in a currently-belonging group module, for example, reducing the number of tasks by 1, and reducing the current total performance value by the task performance value.
The invention discloses a unified task scheduling system, which comprises a task starting module, a task stopping module, a group module, a scheduling module and a machine module, and can carry out unified and standardized processing on various tasks through the scheme, so that a calling end does not need to care about the service of really executing the tasks, only needs to be connected with task scheduling service, and simultaneously supports dynamic transverse expansion to increase the upper limit of task operation. The method is applied to the rainbow cloud product, saves a large amount of butt joint work for calling end service in integrating an offline transcoding task and an online transcoding task, and can adjust resources as required to adapt to service scenes with different magnitudes.

Claims (10)

1. A unified task scheduling system is characterized by comprising a task starting module, a task stopping module, a group module, a scheduling module and a machine module;
the task starting module is used for sending various task starting requests and is connected with the group module;
the task stopping module is used for carrying a task id to send a task stopping request and is connected with the group module;
the group module is used for defining the scheduling rules of various tasks and is connected with the scheduling module;
the scheduling module is used for forwarding tasks according to rules and is connected with the machine module;
the machine module is used for executing and recording tasks.
2. The system according to claim 1, wherein the task start module is configured to start task information, is triggered by a calling end service, and sends an http request to the group module with a task type, a performance value required by a task, and a task content; and the task stopping module is used for stopping the current task, is triggered by the calling end service, and sends an http request to the group module by the task id recorded by the attached task starting module.
3. The system as claimed in claim 2, wherein the group module is an entry of the whole task scheduling system, and the following information needs to be defined before the whole task scheduling system is operated:
firstly, the method comprises the following steps: the task type is the task type supported by the whole task scheduling system;
secondly, the method comprises the following steps: scheduling indexes are divided into two types: number of tasks, machine performance values;
thirdly, the method comprises the following steps: the scheduling algorithm is divided into two types: load balancing, at most priority.
4. A unified task scheduler according to claim 3, wherein said task number is the maximum number of tasks run by each machine module; the machine performance value represents a quantified value that the machine module is subjected to, the value being calculated from the configuration of the machine, and the formula: for the cpu core number-memory size, a performance value required by the current task needs to be calculated in the task starting module to issue the task, and the cpu and the memory which arrive first represent that the machine is used up.
5. The system of claim 3, wherein the load balancing means that tasks are distributed to the machine modules on an average; the most priority means that one machine is occupied fully and then is sent to another machine.
6. A unified task scheduler according to claim 3, 4 or 5 wherein after the above definition is completed, the system starts to receive the requests of the task start module and the task stop module, and the different request processing logics are different:
(1) after receiving a task starting module request, acquiring information defined in a group module according to a transmitted type parameter, wherein the information comprises a scheduling index and a scheduling algorithm, and then merging the information and the parameter transmitted by a task starting module and forwarding the information to a scheduling module;
(2) and after receiving the request of the task stopping module, directly carrying the task id transmitted by the task stopping module and forwarding the task id to the scheduling module.
7. The system of claim 6, wherein the scheduling module, upon receiving the start and stop requests forwarded by the group module, has different processing logic:
(a) after receiving a task starting request forwarded by a group module, inquiring whether a machine module has a matched task machine according to a task type, inquiring whether an idle machine exists according to a scheduling index, obtaining an optimal machine according to a scheduling algorithm, calling a starting task in the machine module for issuing, after the task is successfully issued, generating a unique identifier of the task by the scheduling module, and performing associated storage with a task id and machine information which are actually operated in the machine module and responding to the task starting module by adopting a random UUID mode;
(b) after receiving a task stop request forwarded by the group module, directly inquiring machine information of the task according to the transmitted task id and the information stored by the scheduling module, and then calling the stop task in the machine module.
8. The system as claimed in claim 7, wherein the machine module is a real task-running machine, and is a module supporting dynamic infinite horizontal extension, each machine module needs to be associated with a group module, so as to directly possess task type, maximum task number, and maximum performance value, and in addition, the machine module needs to additionally store dynamic information, and the dynamic information is used for the scheduling module to perform query.
9. The system of claim 8, wherein the specific queries are as follows:
(i) inquiring whether a judgment standard of an idle machine exists: whether the number of the current running tasks or the performance value occupied by the current tasks reaches the upper limit defined by the group module or not;
(ii) the algorithm for querying the optimal machine: and after the idle machine set is obtained, arranging and taking the first one in an ascending or descending order according to the residual task quantity or the residual performance value.
10. The unified task scheduler of claim 9, wherein the processing logic of the machine module upon receiving start and stop tasks is also different:
(A) after receiving a task starting request, executing real task starting, and then performing data accumulation according to a scheduling algorithm configured in a current group module to which the real task starting belongs;
(B) and after receiving the task stopping request, executing real task stopping, and then reducing data according to a scheduling algorithm configured in the current group module to which the real task stopping request belongs.
CN202011623543.0A 2020-12-31 2020-12-31 Unified task scheduling system Active CN112685161B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011623543.0A CN112685161B (en) 2020-12-31 2020-12-31 Unified task scheduling system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011623543.0A CN112685161B (en) 2020-12-31 2020-12-31 Unified task scheduling system

Publications (2)

Publication Number Publication Date
CN112685161A true CN112685161A (en) 2021-04-20
CN112685161B CN112685161B (en) 2024-04-23

Family

ID=75454033

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011623543.0A Active CN112685161B (en) 2020-12-31 2020-12-31 Unified task scheduling system

Country Status (1)

Country Link
CN (1) CN112685161B (en)

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030214657A1 (en) * 2002-05-14 2003-11-20 Stringham Gary Glen Task scheduling and automated task performance in printers
US20080066072A1 (en) * 2006-07-31 2008-03-13 Accenture Global Services Gmbh Work Allocation Model
CN102831494A (en) * 2012-07-23 2012-12-19 北京星网锐捷网络技术有限公司 Scheduling method, scheduling device and scheduling system
CN105260248A (en) * 2014-07-15 2016-01-20 ***股份有限公司 Task scheduling system with dynamic regulation function and scheduling method
CN110647398A (en) * 2019-09-18 2020-01-03 北京航空航天大学 Intersection control task scheduling method facing edge calculation and based on task criticality and timeliness
CN110895487A (en) * 2018-09-12 2020-03-20 北京奇虎科技有限公司 Distributed task scheduling system
CN111026519A (en) * 2018-10-10 2020-04-17 上海宝信软件股份有限公司 Distributed task priority scheduling method and system and storage medium
CN111429033A (en) * 2020-04-21 2020-07-17 贵州新致普惠信息技术有限公司 Workflow task scheduling system and method and service terminal

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030214657A1 (en) * 2002-05-14 2003-11-20 Stringham Gary Glen Task scheduling and automated task performance in printers
US20080066072A1 (en) * 2006-07-31 2008-03-13 Accenture Global Services Gmbh Work Allocation Model
CN102831494A (en) * 2012-07-23 2012-12-19 北京星网锐捷网络技术有限公司 Scheduling method, scheduling device and scheduling system
CN105260248A (en) * 2014-07-15 2016-01-20 ***股份有限公司 Task scheduling system with dynamic regulation function and scheduling method
CN110895487A (en) * 2018-09-12 2020-03-20 北京奇虎科技有限公司 Distributed task scheduling system
CN111026519A (en) * 2018-10-10 2020-04-17 上海宝信软件股份有限公司 Distributed task priority scheduling method and system and storage medium
CN110647398A (en) * 2019-09-18 2020-01-03 北京航空航天大学 Intersection control task scheduling method facing edge calculation and based on task criticality and timeliness
CN111429033A (en) * 2020-04-21 2020-07-17 贵州新致普惠信息技术有限公司 Workflow task scheduling system and method and service terminal

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
GEORGIOS L. STAVRINIDES: "Scheduling Different Types of Bag-of-Tasks Jobs in Distributed Systems", 2019 10TH INTERNATIONAL CONFERENCE ON INFORMATION AND COMMUNICATION SYSTEMS (ICICS), 22 August 2019 (2019-08-22) *
庄卓俊: "基于多核平台的入侵检测***的设计与实现", 中国优秀硕士学位论文全文数据库-信息科技辑, 15 December 2011 (2011-12-15) *

Also Published As

Publication number Publication date
CN112685161B (en) 2024-04-23

Similar Documents

Publication Publication Date Title
CN111225069B (en) Distributed market data processing system and method
CN106101178A (en) A kind of subscription message distribution, method of reseptance, device and related system
CN101383839A (en) Data distribution system based on data server and implementation method
CN106294472A (en) The querying method of a kind of Hadoop data base HBase and device
CN104243610B (en) A kind of distributed document transmits method of servicing
CN102025650A (en) Message processing system and message processing method of enterprise service bus
CN102467411A (en) Workflow processing and workflow agent method, device and system
CN106130960B (en) Judgement system, load dispatching method and the device of steal-number behavior
CN107065813A (en) A kind of workshop appliance data monitoring system and method
CN104378221A (en) Asynchronous calling method applied to integrated circuit production line monitoring system
CN112866421B (en) Intelligent contract operation method and device based on distributed cache and NSQ
CN102333130A (en) Method and system for accessing cache server and intelligent cache scheduler
CN102750368A (en) High-speed importing method of cluster data in data base
CN111913784B (en) Task scheduling method and device, network element and storage medium
CN101431475B (en) Settings of high-performance streaming media server and method for reading high-performance program
CN112685161B (en) Unified task scheduling system
CN108259605B (en) Data calling system and method based on multiple data centers
CN110932393B (en) Substation information protection master station system and data initialization method thereof
CN104468520A (en) Identity authentication method and device
CN113297218B (en) Multi-system data interaction method, device and system
CN107819831B (en) Metaq and mns-based universal message system
CN108243348B (en) A kind of stream process request distribution server
CN110647575A (en) Distributed heterogeneous processing framework construction method and system
CN111724262B (en) Subsequent package query system of application server and working method thereof
CN114238481A (en) Distributed real-time data importing device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant