CN114661432A - Task scheduling method, device, equipment and storage medium - Google Patents
Task scheduling method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN114661432A CN114661432A CN202011548515.7A CN202011548515A CN114661432A CN 114661432 A CN114661432 A CN 114661432A CN 202011548515 A CN202011548515 A CN 202011548515A CN 114661432 A CN114661432 A CN 114661432A
- Authority
- CN
- China
- Prior art keywords
- task
- target
- normal
- scheduling
- abnormal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Retry When Errors Occur (AREA)
Abstract
The invention discloses a task scheduling method, a device, equipment and a storage medium, wherein a scheduling node of a task scheduling system stores at least one task in a workflow to be processed into a normal task set; according to the scheduling priority corresponding to the tasks in the normal task set, taking out the target tasks to be scheduled from the normal task set; according to the information of the task execution node configured by the target task, executing the distribution processing of the target task; and if the target task fails to be distributed, adding the target task into the abnormal task set, and returning to execute the operation of taking out the target task to be scheduled from the normal task set until the undistributed task does not exist in the normal task set. The invention can reduce the situation of task scheduling blockage caused by the abnormity of the task execution node.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method, an apparatus, a device, and a storage medium for task scheduling.
Background
Workflow is the abstraction and general description of the workflow and the business rules between the various operational steps in the workflow. A workflow typically includes one or more tasks (tasks), each task representing some work that needs to be performed. The task scheduling system can perform task scheduling on the tasks in the workflow so as to distribute the tasks in the workflow to the task execution programs in the task execution nodes for running.
In the task scheduling process, if a task execution program on a task execution node specified by a certain task has an exception, the task distribution can fail. However, the task scheduling system needs to schedule each task in turn according to the sequence of each task in the workflow, and therefore, if a certain task fails to be distributed, scheduling of other tasks after the task in the workflow is blocked, thereby affecting the processing efficiency of the workflow.
Disclosure of Invention
In view of the above, the present invention provides a task scheduling method, apparatus, device and storage medium that overcome the above problems or at least partially solve the above problems.
In one aspect, the present application provides a task scheduling method, which is applied to a scheduling node of a task scheduling system, where the task scheduling system includes the scheduling node and a plurality of task execution nodes, and the method includes:
storing at least one task in a workflow to be processed into a normal task set, wherein the at least one task has respective scheduling priority, and each task is configured with information of a task execution node for executing the task;
according to the scheduling priority corresponding to the tasks in the normal task set, taking out the target task to be scheduled from the normal task set;
according to the information of the task execution node configured by the target task, executing the distribution processing of the target task;
and if the target task fails to be distributed, adding the target task into an abnormal task set, and returning to execute the operation of taking out the target task to be scheduled from the normal task set until the undistributed task does not exist in the normal task set.
Preferably, after the adding the target task to the abnormal task set, the method further includes:
setting a delay retry duration for the target task;
and after the time length of the target task added into the abnormal task set reaches the delayed retry time length of the target task, transferring the target task from the abnormal task set to the normal task set.
Preferably, after the target task is added to the abnormal task set, the method further includes:
updating the entering times of the target task entering the abnormal task set;
the setting of the delayed retry duration for the target task includes:
and setting a delayed retry time length for the target task according to the updated entry times, wherein the more the entry times, the longer the delayed retry time length.
Preferably, if the target task fails to be distributed, adding the target task to an abnormal task set includes:
if the target task fails to be distributed, updating the distribution failure times of the target task;
if the distribution failure times do not reach the set times, returning the information for executing the task execution node configured according to the target task, and executing the distribution processing operation of the target task;
and if the distribution failure times reach the set times, adding the target task into an abnormal task set.
Preferably, the storing at least one task in the workflow to be processed into the normal task set includes:
determining at least one task to be scheduled and processed in a workflow to be processed;
adding the at least one task into a normal task queue according to the respective scheduling priority of the at least one task, wherein the task with higher scheduling priority is ranked in the normal task queue more ahead;
the taking out the target task to be scheduled from the normal task set according to the scheduling priority corresponding to the task in the normal task set comprises:
and taking out the target task to be scheduled from the normal task queue according to the sequence of the tasks in the normal task queue.
Preferably, the setting of the delayed retry duration for the target task includes:
setting a delayed retry duration for the target task through an exception monitoring thread;
after the time length for adding the target task into the abnormal task set at this time reaches the delayed retry time length of the target task, adding the target task into the normal task set, including:
monitoring the time length of each task in the abnormal task set currently added into the abnormal task set through the abnormal monitoring thread;
and after the abnormal monitoring thread confirms that the time length of the target task added to the abnormal task set at this time reaches the delayed retry time length of the target task, transferring the target task from the abnormal task set to the normal task set.
In another aspect, the present application further provides a task scheduling apparatus, applied to a scheduling node of a task scheduling system, where the task scheduling system includes the scheduling node and a plurality of task execution nodes, and the apparatus includes:
the task storage unit is used for storing at least one task in a workflow to be processed into a normal task set, the at least one task has respective scheduling priority, and each task is provided with information of a task execution node used for executing the task;
the task scheduling unit is used for taking out a target task to be scheduled from the normal task set according to the scheduling priority corresponding to the tasks in the normal task set;
the task distribution unit is used for executing distribution processing of the target task according to the information of the task execution node configured by the target task;
and the exception handling unit is used for adding the target task into an exception task set if the target task fails to be distributed, and returning to execute the operation of taking out the target task to be scheduled from the normal task set until undistributed tasks do not exist in the normal task set.
Preferably, the method further comprises the following steps:
the delay setting unit is used for setting delay retry time length for the target task after the target task is added into an abnormal task set by the abnormal processing unit;
and the task retry unit is used for transferring the target task from the abnormal task set to the normal task set after the time length of adding the target task into the abnormal task set at this time reaches the delayed retry time length of the target task.
In yet another aspect, the present application further provides a storage medium including a stored program, wherein the program performs the task scheduling method as described in any one of the above.
In yet another aspect, the present application further provides an electronic device comprising at least one processor, at least one memory connected to the processor, and a bus;
the processor and the memory complete mutual communication through a bus;
the processor is configured to call program instructions in the memory to perform a task scheduling method as described in any one of the above.
By means of the technical scheme, in the process of sequentially scheduling each task in the normal task set, if a certain task fails to be distributed, the task is added to the abnormal task set outside the normal task set, so that the scheduling node can continuously take out other tasks from the normal task set to continue scheduling, the condition that the scheduling of subsequent tasks is influenced due to the failure of task distribution is reduced, and the scheduling efficiency of the tasks in the workflow is improved.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
FIG. 1 is a block diagram illustrating an architecture of an embodiment of a task scheduling system according to the present application;
FIG. 2 is a flowchart illustrating an embodiment of a task scheduling method according to the present application;
FIG. 3 is a flowchart illustrating a task scheduling method according to another embodiment of the present application;
FIG. 4 is a flowchart illustrating a task scheduling method according to another embodiment of the present application;
FIG. 5 is a schematic diagram illustrating an exemplary embodiment of a task scheduler;
fig. 6 shows a schematic structural diagram of an electronic device according to the present application.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
For ease of understanding, a task scheduling system to which the present application is applicable will be described below.
Fig. 1 is a schematic diagram illustrating a component architecture of an embodiment of a task scheduling system according to the present application.
As can be seen from fig. 1, the task scheduling system may include a scheduling node 101 and a plurality of task execution nodes 102.
The scheduling node and the task execution node can be connected through a network.
The scheduling node may obtain a workflow to be scheduled, which may include one or more tasks (tasks) to be performed. Each task represents a certain work to be performed, for example, a task may be a program to be performed to complete a certain work.
The scheduling node may distribute tasks in the workflow to different task execution nodes to perform execution of the tasks by the task execution nodes.
The task execution node can run a task execution program (such as worker), and the task execution program can process the task distributed to the task execution node by the scheduling node.
Of course, fig. 1 is only an example of one constituent structure of the task scheduling system, and in practical applications, the task scheduling system may also have other forms, such as may further include other device nodes, and the present application does not limit the specific structural form of the task scheduling system.
The task scheduling method of the present application is described below with reference to a flowchart.
Fig. 2 is a schematic flowchart illustrating a task scheduling method according to an embodiment of the present invention, where the method is applied to a scheduling node of a task scheduling system, and the task scheduling system includes the scheduling node and a plurality of task execution nodes. The method of the embodiment comprises the following steps:
s201, storing at least one task in the workflow to be processed to a normal task set.
At least one task has respective scheduling priority, and each task is configured with information of a task execution node for executing the task.
The workflow includes one or more tasks to be executed, and generally, the workflow includes a plurality of tasks to be executed.
The scheduling node can split each task included in the workflow and obtain the scheduling priority of each task.
For example, the workflow may be represented in the form of a directed acyclic graph, and by analyzing each branch in the directed acyclic graph, each task involved in the workflow and the scheduling priority corresponding to each task may be obtained.
The scheduling priority of the task is the priority of the task to be scheduled and executed, and the task with the high scheduling priority needs to be scheduled and executed preferentially.
Different tasks may have different requirements for hardware and software operating environments, and therefore, in the present application, each task in a workflow is configured with information of a task execution node corresponding to the task, and on this basis, the task needs to be scheduled to the task execution node configured by the task for processing.
The information of the task execution node configured by the task may be information that can uniquely identify the task execution node, such as an address or a code of the task execution node.
The normal task set may be a queue or other forms, which is not limited in this application.
It is understood that in the case of a workflow having a plurality of tasks, there may be dependencies between different tasks in the workflow, for example, a task needs to be executed after another task is completed, based on the execution result of another task, and the like. In this case, the scheduling node may store at least one task that needs to be preferentially executed based on the dependency relationship to the normal task set according to the dependency relationship before each task in the workflow. And any other task which is not stored in the normal task set in the workflow needs to be added into the normal task execution set after the task which the task depends on is distributed and successfully executed.
For example, task 1, task 2, task 3, task 4, task 5, and task 6 exist in the workflow, wherein task 2 needs to depend on task 1, task 5 needs to depend on task 3 and task 4, and task 6 does not need to depend on other tasks, and on the basis of this, task 1, task 3, task 4, and task 6 can be stored into the normal task set. Correspondingly, after the task 1 is scheduled and successfully executed, the task 2 is added into the normal task set; task 5 needs to be re-added to the normal set of tasks after tasks 3 and 4 are successfully executed.
S202, according to the scheduling priority corresponding to the tasks in the normal task set, taking out the target tasks to be scheduled from the normal task set.
In the application, target tasks which are not scheduled and distributed can be stored in the constructed normal task set, and simultaneously, after one task is taken out of the normal task set, one less task is taken out of the normal task set
It will be appreciated that the target task taken from the normal task set schedules the highest priority task of the remaining tasks in the normal task set.
And S203, executing the distribution processing of the target task according to the information of the task execution node configured by the target task.
It can be understood that, according to the information of the task execution node configured by the target task, the scheduling node may determine the task execution node to which the target task needs to be distributed, and perform an operation of allocating the task to the task execution node.
For example, according to the information of the task execution node configured by the target task, whether the task execution node exists or not and whether the task execution node has an abnormality (e.g., a network abnormality with a communication node or a task execution program failure in the task execution node or the like) may be first queried, and in a case where the task execution node exists and the operation state of the task execution node is normal, the target task may be distributed to the task execution node.
And S204, if the target task fails to be distributed, adding the target task into the abnormal task set, and returning to the operation of the S202 until the undistributed task does not exist in the normal task set.
The target task distribution failure means that the target task cannot be distributed to the task execution node configured by the target task due to the abnormality of the task execution node configured by the target task.
Distribution failure caused by task execution node exception can be divided into a plurality of cases.
For example, the task execution node may be classified into two types, one type is a fault of the task execution node, such as a network fault, an equipment fault of the task execution node, or a task execution program exception.
One type of task execution node configured for the target task does not exist, for example, a task execution node which does not exist in a scheduling system is configured for the target task in error; or the task execution node configured for the target task is a node in two node groups, and no intersected node exists in the two node groups, so that the condition that the task execution node configured for the task does not exist exists occurs.
In the application, in order to add the target task to another task set except the normal task set, namely the abnormal task set, when the target task fails to be distributed due to the abnormal task execution node, the task failed to be distributed is transferred from the normal task set to the abnormal task set. On this basis, the scheduling node can continuously perform scheduling distribution on the task with the scheduling priority behind the target task in the normal task set, so as to reduce the possibility that the scheduling of other tasks behind the target task in the normal task set is blocked due to the failure of target task distribution.
It will be appreciated that if the target task distribution is successful, the scheduling node will also return to perform step S202 to continue scheduling other tasks remaining in the normal task set.
The exception task set may be a queue or a file, etc., without limitation.
According to the method and the device, each task in the workflow is added into the normal task set, and in the process of sequentially scheduling each task in the normal task set, if a certain task fails to be distributed, the task is added into the abnormal task set except the normal task set, so that the scheduling node can continuously take out other tasks from the normal task set to continue scheduling, the condition that the subsequent tasks are affected due to the failure of task distribution is reduced, and the scheduling efficiency of the tasks in the workflow is improved.
It can be understood that after the task with failed distribution is added to the abnormal task set, if the task configuration task execution node is recovered to be normal, the task can be distributed normally. Based on this, after the task execution node configured by the task returns to normal, the task in the abnormal task set may be redistributed, and after the task is added to the abnormal task set, the task is added from the abnormal task set to the normal task set again at an interval of time.
For example, after adding the target task to the abnormal task set, the method further includes: a delayed retry duration is set for the target task. On the basis, if the time length for adding the target task into the abnormal task set reaches the delayed retry time length of the target task, the target task is added into the normal task set.
The delay retry duration may be a set duration, for example, the set duration may be 5 minutes.
The delay retry duration may be task dependent, e.g., the greater the number of times a task joins the exception set, the longer the delay retry duration is set for the task.
This is explained below with reference to fig. 3. Referring to fig. 3, which shows a flowchart of another embodiment of the task scheduling method according to the present application, the method of the present embodiment is also applied to the aforementioned scheduling node, and the method of the present embodiment may include:
s301, storing at least one task in the workflow to be processed into a normal task set.
At least one task has respective scheduling priority, and each task is configured with information of a task execution node for executing the task.
S302, according to the scheduling priority corresponding to the tasks in the normal task set, the target tasks to be scheduled are taken out from the normal task set.
And S303, executing the distribution processing of the target task according to the information of the task execution node configured by the target task.
S304, if the target task fails to be distributed, adding the target task into the abnormal task set, and returning to the operation of S202 until the undistributed task does not exist in the normal task set.
The above steps can be referred to the related descriptions of the previous embodiments, and are not described herein again.
As an optional mode, in order to reduce the misjudgment of the task distribution failure, the application may further update the distribution failure times of the target task after determining that the target task is failed to be distributed. If the recorded distribution failure times of the target task is added with one, for example, if the target task is the first distribution failure, the distribution failure times are one; and if the distribution failure times of the target task is 1, updating the distribution failure times of the target task to be 2 after the distribution failure of the target task is currently determined.
Accordingly, if the number of times of failed distribution of the target task does not reach the set number of times, the process returns to execution S303 to retry distribution of the target task.
If the distribution failure times of the target task reach the set times, it can be shown that the task execution node configured by the target task has a fault, in which case, the target task can be added to the abnormal task set.
S305, updating the entering times of the target task entering the abnormal task set.
And the entering times are the total times that the target task is added into the abnormal task set.
For example, when the target task is added to the abnormal task set, the number of times of entry of the target task may be increased by one from the original value before or after the target task enters the abnormal task set. For example, assuming that the target task is added to the abnormal task set for the first time, the updated entry number is 1; assuming that the target task has been added to the abnormal task set once before, after the target task is transferred from the abnormal task set to the normal task set, the target task is rescheduled and distributed again and then fails to be distributed, so that the target task is added to the abnormal task set again, and the updated entry times are 2 times.
S306, setting a delay retry time length for the target task according to the updated entry times.
Wherein, the more the number of the task entering is, the longer the delay retry time is. That is to say, the more times a task enters the abnormal task set, the longer the time the task stays in the abnormal task set after entering the abnormal task set this time.
For example, the correspondence between the different entry times and the delayed retry time period may be set in advance. For example, if the number of entries is 1, the delay retry duration is 5 minutes; on the basis, the delay retry time length is increased by 2 minutes every time the entry times are increased by 1 time, namely the entry times are 2 times, and the delay retry time length is 7 minutes; the number of entries is 3, the delay retry duration is 9 minutes, and so on.
Of course, there may be other ways to set the delay retry duration of the task according to the number of times the task enters, which is not limited to this.
S307, if the time length of the target task added into the abnormal task set reaches the delayed retry time length of the target task, the target task is transferred from the abnormal task set to the normal task set.
The purpose of transferring the target task from the abnormal task set to the normal task set is to dispatch and distribute the target task again, so that after the abnormality of the task execution node configured by the target task is repaired, the target task can be distributed to the task execution node to be executed, and the reliable execution of the target task is ensured.
It can be understood that, the more the number of times of entering the target task is, the longer the time required for repairing the abnormality of the task execution node configured by the target task is, and therefore, by prolonging the delay retry duration of the target task, the target task can be prevented from being frequently transferred to the normal task set, which can be beneficial to reducing the data resource consumption caused by frequently retrying the scheduling distribution of the target task.
In order to ensure the reliable scheduling of each task in the normal task set and simultaneously monitor each task in the abnormal task set, the abnormal monitoring thread for monitoring the abnormal task set by one thread can be independently started. For example, tasks may be added to and removed from the normal set of tasks for scheduling by a task scheduling thread, while the set of tasks may be exception by an exception monitoring thread.
As an optional manner, in order to facilitate scheduling the task according to the scheduling priority of the task, in the present application, the normal task set may be a queue, and for convenience of differentiation, it is referred to as a normal task queue. Similarly, the exception task set may be an exception task queue.
The following description is made with reference to an embodiment, and as shown in fig. 4, it shows a schematic flow chart of another embodiment of a task scheduling method according to the present application, where the method of this embodiment may include:
s401, at least one task to be scheduled and processed in the workflow to be processed is determined.
Each task is configured with information of a task execution node for executing the task, and each task has a scheduling priority.
For example, the workflow may be analyzed, the tasks included in the workflow and the dependency relationship between the tasks may be determined, and the scheduling priority corresponding to each task may be obtained, on this basis, the task to be currently scheduled may be determined according to the dependency relationship between the tasks, and step S402 may be executed.
It is understood that, as tasks in the normal task queue are scheduled and executed, other tasks in the workflow that are not added to the subsequently mentioned normal task queue still need to be added to the normal task queue continuously, and therefore, the task determined to be scheduled may need to be executed multiple times. The application is not limited to the specific process of this step.
S402, adding at least one task into a normal task queue according to the respective scheduling priority of the at least one task.
Wherein, the task with higher scheduling priority is ranked more ahead in the normal task queue.
And S403, taking out the target task to be scheduled from the normal task queue according to the sequence of the tasks in the normal task queue.
S404, distributing and processing the target task according to the information of the task execution node configured by the target task.
S405, detecting whether the target task fails to be distributed, and if so, executing the step S406; if not, go to step S408;
s406, updating the distribution failure times of the target task, detecting whether the distribution failure times of the target task reaches the set times, and if so, executing the step S407; if not, returning to the step S404;
it is understood that, after the distribution failure number of the target task is updated, detecting whether the distribution failure number of the target task reaches the set number means detecting whether the updated distribution failure number reaches the set number.
S407, adding the target task into the abnormal task queue.
S408, detecting whether the scheduled task is still not completed in the normal task queue, if so, returning to execute the step S403; if not, the task scheduling is ended.
And S409, updating the number of times of the target task entering the abnormal task queue through the abnormal monitoring thread.
And S410, setting a delayed retry time length for the target task through the exception monitoring thread according to the entering times.
It is understood that S409 and S410 are only one way for the exception monitoring thread to set the delay retry duration for the target task, and in practical applications, if the delay retry duration set by each entry of the target task into the exception task set (e.g., the exception task queue) is a fixed duration, the exception monitoring thread may also set the delay retry duration for the target task directly, regardless of the number of entries.
S411, the time length of each task in the abnormal task queue currently added to the abnormal task set is monitored through the abnormal monitoring thread.
S412, after the abnormal monitoring thread confirms that the time length of the target task added into the abnormal task queue reaches the delayed retry time length of the target task, the target task is transferred from the abnormal task queue to the normal task queue.
Here, steps S409 to S412 executed by the abnormality monitoring thread are executed after step S407, but the steps may be executed in synchronization with step S408.
It should be noted that, the embodiment is described by taking the form of taking the abnormal task set as the abnormal task queue as an example, but it is understood that the abnormal task set is also applicable to the embodiment if the abnormal task queue is another form of abnormal task set. Similarly, the replacement of the normal task queue with other forms of normal task sets is also applicable to the present embodiment.
The application also provides a task scheduling device corresponding to the task scheduling method. As shown in fig. 5, which illustrates a schematic structural diagram of an embodiment of a task scheduling apparatus according to the present application, the apparatus of this embodiment may be applied to a scheduling node of a task scheduling system, where the task scheduling system includes the scheduling node and a plurality of task execution nodes, and the apparatus includes:
a task storage unit 501, configured to store at least one task in a to-be-processed workflow into a normal task set, where the at least one task has a respective scheduling priority, and each task is configured with information of a task execution node for executing the task;
a task scheduling unit 502, configured to take out a target task to be scheduled from the normal task set according to a scheduling priority corresponding to a task in the normal task set;
a task distributing unit 503, configured to execute distribution processing of the target task according to information of the task execution node configured by the target task;
and the exception processing unit 504 is configured to, if the target task fails to be distributed, add the target task to the exception task set, and return to execute the operation of taking out the target task to be scheduled from the normal task set until no undistributed task exists in the normal task set.
In one possible implementation, the apparatus may further include:
the delay setting unit is used for setting a delay retry time length for the target task after the target task is added into the abnormal task set by the abnormal processing unit;
and the task retry unit is used for transferring the target task from the abnormal task set to the normal task set after the time length of adding the target task into the abnormal task set at this time reaches the delayed retry time length of the target task.
In an alternative, the apparatus may further include:
the number updating unit is used for updating the entering number of the target task entering the abnormal task set after the target task is added into the abnormal task set by the abnormal processing unit;
the delay setting unit is specifically configured to set a delay retry duration for the target task according to the updated entry times, where the greater the entry times, the longer the delay retry duration.
In still another alternative, the delay setting unit includes:
the delay setting subunit is used for setting a delay retry duration for the target task through the exception monitoring thread;
a task retry unit, comprising:
the time length monitoring unit is used for monitoring the time length of each task in the abnormal task set, which is added into the abnormal task set at present, through the abnormal monitoring thread;
and the task transfer unit is used for transferring the target task from the abnormal task set to the normal task set after the abnormal monitoring thread confirms that the time length of adding the target task into the abnormal task set at this time reaches the delayed retry time length of the target task.
In another possible implementation manner, the exception handling unit includes:
the frequency updating unit is used for updating the distribution failure frequency of the target task if the target task fails to distribute;
a first exception handling unit, configured to, if the distribution failure number does not reach a set number, return to execute information of the task execution node configured according to the target task, and execute an operation of distribution processing of the target task until no undistributed task exists in the normal task set;
and the second exception processing unit is used for adding the target task into the exception task set if the distribution failure frequency reaches the set frequency, and returning to execute the operation of taking out the target task to be scheduled from the normal task set until no undistributed task exists in the normal task set.
In yet another possible implementation manner, the task storage unit includes:
the task determining unit is used for determining at least one task to be scheduled and processed in the workflow to be processed;
the queue inserting unit is used for adding the at least one task into a normal task queue according to the respective scheduling priority of the at least one task, wherein the task with higher scheduling priority is more front in the normal task queue;
and the task scheduling unit is specifically used for taking out the target task to be scheduled from the normal task queue according to the sequence of the tasks in the normal task queue.
The task scheduling device comprises a processor and a memory, wherein the task storage unit, the task scheduling unit, the task distribution unit, the exception handling unit and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions.
The processor comprises a kernel, and the kernel calls the corresponding program unit from the memory. The kernel can be set to be one or more than one, and the situation that task scheduling is blocked due to the exception of the task execution node is reduced in the task scheduling process by adjusting kernel parameters.
An embodiment of the present invention provides a storage medium on which a program is stored, the program implementing the task scheduling method when executed by a processor.
The embodiment of the invention provides a processor, which is used for running a program, wherein the task scheduling method is executed when the program runs.
The embodiment of the invention provides an electronic device, wherein the electronic device 600 comprises at least one processor 601, at least one memory 602 connected with the processor 601, and a bus 603; the processor and the memory complete mutual communication through a bus; the processor is used for calling the program instructions in the memory to execute the task scheduling method. The electronic device herein may be a server, a PC, a PAD, a mobile phone, etc.
The present application further provides a computer program product adapted to perform a program for initializing the following method steps when executed on an electronic device:
storing at least one task in a workflow to be processed into a normal task set, wherein the at least one task has respective scheduling priority, and each task is configured with information of a task execution node for executing the task;
according to the scheduling priority corresponding to the tasks in the normal task set, taking out target tasks to be scheduled from the normal task set;
according to the information of the task execution node configured by the target task, executing the distribution processing of the target task;
and if the target task fails to be distributed, adding the target task into an abnormal task set, and returning to execute the operation of taking out the target task to be scheduled from the normal task set until the undistributed task does not exist in the normal task set.
In one possible case, after the adding the target task to the abnormal task set, the method further includes:
setting a delay retry duration for the target task;
and after the time length of the target task added into the abnormal task set reaches the delayed retry time length of the target task, transferring the target task from the abnormal task set to the normal task set.
In another possible case, after adding the target task to the abnormal task set, the method further includes:
updating the entering times of the target task entering the abnormal task set;
the setting of the delayed retry duration for the target task includes:
and setting a delayed retry time length for the target task according to the updated entry times, wherein the more the entry times, the longer the delayed retry time length.
In yet another possible scenario, adding the target task to an exception task set if the target task fails to be distributed, includes:
if the target task fails to be distributed, updating the distribution failure times of the target task;
if the distribution failure times do not reach the set times, returning the information for executing the task execution node configured according to the target task, and executing the distribution processing operation of the target task;
and if the distribution failure times reach the set times, adding the target task into an abnormal task set.
In another possible case, the storing at least one task in the workflow to be processed into the normal task set includes:
determining at least one task to be scheduled and processed in a workflow to be processed;
adding the at least one task into a normal task queue according to the respective scheduling priority of the at least one task, wherein the task with higher scheduling priority is ranked in the normal task queue more ahead;
the taking out a target task to be scheduled from the normal task set according to the scheduling priority corresponding to the task in the normal task set includes:
and taking out the target task to be scheduled from the normal task queue according to the sequence of the tasks in the normal task queue.
In another possible case, the setting a delay retry duration for the target task includes:
setting a delayed retry duration for the target task through an exception monitoring thread;
after the time length for adding the target task into the abnormal task set at this time reaches the delayed retry time length of the target task, adding the target task into the normal task set, including:
monitoring the duration of each task in the abnormal task set which is currently added into the abnormal task set through the abnormal monitoring thread;
and after the abnormal monitoring thread confirms that the time length of the target task added to the abnormal task set at this time reaches the delayed retry time length of the target task, transferring the target task from the abnormal task set to the normal task set.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus, electronic devices (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, an electronic device includes one or more processors (CPUs), memory, and a bus. The electronic device may also include input/output interfaces, network interfaces, and the like.
The memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip. The memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (10)
1. A task scheduling method is applied to a scheduling node of a task scheduling system, the task scheduling system comprises the scheduling node and a plurality of task execution nodes, and the method comprises the following steps:
storing at least one task in a workflow to be processed into a normal task set, wherein the at least one task has respective scheduling priority, and each task is configured with information of a task execution node for executing the task;
according to the scheduling priority corresponding to the tasks in the normal task set, taking out target tasks to be scheduled from the normal task set;
according to the information of the task execution node configured by the target task, executing the distribution processing of the target task;
and if the target task fails to be distributed, adding the target task into an abnormal task set, and returning to execute the operation of taking out the target task to be scheduled from the normal task set until the undistributed task does not exist in the normal task set.
2. The method of claim 1, after said adding said target task to an exception task set, further comprising:
setting a delay retry duration for the target task;
and after the time length of the target task added into the abnormal task set reaches the delayed retry time length of the target task, transferring the target task from the abnormal task set to the normal task set.
3. The method of claim 2, after adding the target task to a set of exception tasks, further comprising:
updating the entering times of the target task entering the abnormal task set;
the setting of the delayed retry duration for the target task includes:
and setting a delayed retry time length for the target task according to the updated entry times, wherein the more the entry times are, the longer the delayed retry time length is.
4. The method of claim 1, wherein adding the target task to an exception task set if the target task fails to be distributed comprises:
if the target task fails to be distributed, updating the distribution failure times of the target task;
if the distribution failure times do not reach the set times, returning the information for executing the task execution node configured according to the target task, and executing the distribution processing operation of the target task;
and if the distribution failure times reach the set times, adding the target task into an abnormal task set.
5. The method of claim 1, wherein storing at least one task in the pending workflow to a normal task set comprises:
determining at least one task to be scheduled and processed in a workflow to be processed;
adding the at least one task into a normal task queue according to the respective scheduling priority of the at least one task, wherein the task with higher scheduling priority is ranked in the normal task queue more ahead;
the taking out the target task to be scheduled from the normal task set according to the scheduling priority corresponding to the task in the normal task set comprises:
and taking out the target task to be scheduled from the normal task queue according to the sequence of the tasks in the normal task queue.
6. The method of claim 2, wherein setting a delay retry duration for the target task comprises:
setting a delayed retry duration for the target task through an exception monitoring thread;
after the time length for adding the target task into the abnormal task set at this time reaches the delayed retry time length of the target task, adding the target task into the normal task set, including:
monitoring the duration of each task in the abnormal task set which is currently added into the abnormal task set through the abnormal monitoring thread;
and after the abnormal monitoring thread confirms that the time length of the target task added into the abnormal task set at this time reaches the delayed retry time length of the target task, transferring the target task from the abnormal task set to the normal task set.
7. A task scheduling apparatus, applied to a scheduling node of a task scheduling system, where the task scheduling system includes the scheduling node and a plurality of task execution nodes, the apparatus comprising:
the task storage unit is used for storing at least one task in a workflow to be processed into a normal task set, the at least one task has respective scheduling priority, and each task is provided with information of a task execution node used for executing the task;
the task scheduling unit is used for taking out a target task to be scheduled from the normal task set according to the scheduling priority corresponding to the tasks in the normal task set;
the task distribution unit is used for executing distribution processing of the target task according to the information of the task execution node configured by the target task;
and the exception processing unit is used for adding the target task into an exception task set if the target task fails to be distributed, and returning to execute the operation of taking out the target task to be scheduled from the normal task set until the undistributed task does not exist in the normal task set.
8. The apparatus of claim 7, further comprising:
the delay setting unit is used for setting a delay retry time length for the target task after the target task is added into an abnormal task set by the abnormal processing unit;
and the task retry unit is used for transferring the target task from the abnormal task set to the normal task set after the time length of adding the target task into the abnormal task set at this time reaches the delayed retry time length of the target task.
9. A storage medium characterized by comprising a stored program, wherein the program executes the task scheduling method according to any one of claims 1 to 7.
10. An electronic device comprising at least one processor, at least one memory connected to the processor, and a bus;
the processor and the memory complete mutual communication through a bus;
the processor is configured to call program instructions in the memory to perform a task scheduling method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011548515.7A CN114661432A (en) | 2020-12-24 | 2020-12-24 | Task scheduling method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011548515.7A CN114661432A (en) | 2020-12-24 | 2020-12-24 | Task scheduling method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114661432A true CN114661432A (en) | 2022-06-24 |
Family
ID=82024854
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011548515.7A Pending CN114661432A (en) | 2020-12-24 | 2020-12-24 | Task scheduling method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114661432A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115099693A (en) * | 2022-07-21 | 2022-09-23 | 宁波迈泰克磁材科技有限公司 | Production control method and system for sintered neodymium-iron-boron magnetic steel material |
-
2020
- 2020-12-24 CN CN202011548515.7A patent/CN114661432A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115099693A (en) * | 2022-07-21 | 2022-09-23 | 宁波迈泰克磁材科技有限公司 | Production control method and system for sintered neodymium-iron-boron magnetic steel material |
| CN115099693B (en) * | 2022-07-21 | 2023-10-13 | 宁波迈泰克磁材科技有限公司 | Production control method and system for sintered NdFeB magnetic steel material |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107450979B (en) | Block chain consensus method and device | |
| US20070240143A1 (en) | Migratable unit based application migration | |
| US9535754B1 (en) | Dynamic provisioning of computing resources | |
| CN107273195B (en) | Big data batch processing method and device and computer system | |
| CN111090502B (en) | Stream data task scheduling method and device | |
| CN109710394A (en) | Timing task processing system and method | |
| CN108491254A (en) | A kind of dispatching method and device of data warehouse | |
| CN107168777B (en) | Method and device for scheduling resources in distributed system | |
| CN111104227B (en) | Resource control method and device of K8s platform and related components | |
| CN111045811A (en) | Task allocation method and device, electronic equipment and storage medium | |
| CN112748993A (en) | Task execution method and device, storage medium and electronic equipment | |
| CN113867957A (en) | Method and device for realizing elastic expansion of number of cross-cluster containers | |
| CN114661432A (en) | Task scheduling method, device, equipment and storage medium | |
| CN110968406B (en) | Method, device, storage medium and processor for processing task | |
| CN111831408A (en) | Asynchronous task processing method and device, electronic equipment and medium | |
| CN116737345A (en) | Distributed task processing system, distributed task processing method, distributed task processing device, storage medium and storage device | |
| US10901972B2 (en) | Table partition configuration method, apparatus and system for database system | |
| CN111400032A (en) | Resource allocation method and device | |
| CN107958414B (en) | Method and system for eliminating long transactions of CICS (common integrated circuit chip) system | |
| CN115220887A (en) | Processing method of scheduling information, task processing system, processor and electronic equipment | |
| CN114791884A (en) | Test environment construction method and device, storage medium and electronic equipment | |
| CN112860779A (en) | Batch data importing method and device | |
| CN112596915A (en) | Distributed lock scheduling method, device, equipment and medium | |
| CN112579145A (en) | Application deployment method and device | |
| CN116319266A (en) | Distributed scheduling method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |