CN109542600B

CN109542600B - Distributed task scheduling system and method

Info

Publication number: CN109542600B
Application number: CN201811360588.6A
Authority: CN
Inventors: 许朋; 董荣海; 胡松
Original assignee: Koubei Shanghai Information Technology Co Ltd
Current assignee: Koubei Shanghai Information Technology Co Ltd
Priority date: 2018-11-15
Filing date: 2018-11-15
Publication date: 2020-12-25
Anticipated expiration: 2038-11-15
Also published as: CN109542600A

Abstract

The invention discloses a distributed task scheduling system and a distributed task scheduling method. The system comprises: the system comprises a task distribution module, a plurality of task fetching modules and a plurality of task execution modules; the task distribution module is suitable for distributing at least one task to be executed contained in the task set to the plurality of task fishing modules; the task fetching module is suitable for calculating the maximum available quota number of the tasks according to the available quota number of the task set and the task priority weight corresponding to the tasks to be executed aiming at each task to be executed, and further determining a subtask instance set to be fetched; distributing the retrieved subtask instances to a task execution module; dynamically adjusting task priority weights corresponding to the tasks to be executed according to the number of subtask instances in which the tasks to be executed are not fished; the task execution module is suitable for executing the subtasks corresponding to the acquired subtask instances, and realizes the release of resources at the low peak period of the task and the accelerated scheduling at the high peak period of the task by dynamically adjusting the priority weight of the task, thereby overcoming the overstock problem of the task instances.

Description

Distributed task scheduling system and method

Technical Field

The invention relates to the technical field of computers, in particular to a distributed task scheduling system and a distributed task scheduling method.

Background

In the conventional task scheduling technology, a task distribution layer generally triggers and fetches a subtask when receiving a message, the task fetching layer fetches a subtask instance to be executed according to subtask information sent by the task distribution layer, and a task execution layer executes a corresponding subtask.

Chinese patent application publication No. CN 105808335 a discloses a dynamic scheduling method, however, the scheduling weight of each job is dynamically adjusted according to the job size and the waiting time of the job, wherein the larger the job size is, the smaller the waiting time is, and the smaller the scheduling weight is; the smaller the job size, the greater the latency, and the greater the scheduling weight, and this dynamic scheduling does not take into account other tasks.

Therefore, the existing task scheduling technology still has the problems of resource isolation, limited flow control capability, incapability of sensing system pressure during scheduling and the like, and the problem of task instance overstock easily caused by incapability of scheduling task execution when high-load and large-concurrency occurs.

Disclosure of Invention

In view of the above, the present invention has been made to provide a distributed task scheduling system and method that overcomes or at least partially solves the above problems.

According to an aspect of the present invention, there is provided a distributed task scheduling system, including: the system comprises a task distribution module, a plurality of task fetching modules and a plurality of task execution modules; the task distribution module is suitable for distributing at least one task to be executed contained in the task set to the plurality of task fetching modules; the task fetching module is suitable for calculating the maximum available quota number of the tasks according to the available quota number of the task set and the task priority weight corresponding to the task to be executed aiming at each task to be executed, and further determining a subtask instance set to be fetched; distributing the retrieved subtask instances to a task execution module; dynamically adjusting the task priority weight corresponding to the task to be executed according to the number of the subtask instances of the task to be executed which are not fished yet; and the task execution module is suitable for executing the subtasks corresponding to the acquired subtask instances.

Optionally, the task fishing module is further adapted to: and dynamically adjusting the task priority weight corresponding to the task to be executed according to the number of the subtask instances which are not fished yet in the task to be executed and the maximum fishing number of the subtask instances in unit time.

Optionally, the task fishing module is further adapted to: determining the initial weight of the task; calculating a weight change value according to the initial weight of the task, the number of subtask instances of which the task to be executed is not fished yet and the maximum fishing number of the subtask instances in unit time; and calculating the task priority weight according to the task initial weight and the weight change value.

Optionally, the sum of the initial weights of the tasks corresponding to all the tasks to be executed in the task set is equal to 1; the sum of the weight change values corresponding to all the tasks to be executed in the task set is less than or equal to 1 and is greater than or equal to-1.

Optionally, the task fishing module is further adapted to: judging whether the maximum available quota number of the tasks is less than or equal to the maximum fishing number of the subtask instances in unit time; if so, determining a subtask instance set to be fished according to the maximum available quota number of the task; and if not, determining the subtask instance set to be fished according to the maximum fishing quantity of the subtask instances in unit time.

Optionally, the task fishing module is further adapted to: and counting the number of the subtask instances which are not fished yet in the task to be executed according to the maximum available quota number of the task or the maximum fishing number of the subtask instances in unit time.

Optionally, the task fishing module is further adapted to: judging whether the maximum available quota quantity of the tasks is greater than 0; if yes, further judging whether the maximum available quota quantity of the tasks is smaller than or equal to the maximum fishing quantity of the subtask instances in unit time; if not, finishing fetching the subtask instance of the task to be executed.

Optionally, the task fishing module is further adapted to: judging whether the number of available quotas of the task set is greater than 0 or not; if yes, calculating the maximum available quota quantity of the task according to the available quota quantity of the task set and the task priority weight corresponding to the task to be executed; if not, the sub-task instance is finished being fished.

According to another aspect of the present invention, a distributed task scheduling method is provided, including: the task distribution module distributes at least one task to be executed contained in the task set to the plurality of task fishing modules; the task fetching module calculates the maximum available quota number of the tasks according to the available quota number of the task set and the task priority weight corresponding to the task to be executed aiming at each task to be executed, and further determines a subtask instance set to be fetched; distributing the retrieved subtask instances to a task execution module; dynamically adjusting the task priority weight corresponding to the task to be executed according to the number of the subtask instances of the task to be executed which are not fished yet; and the task execution module executes the subtasks corresponding to the acquired subtask instances.

Optionally, dynamically adjusting the task priority weight corresponding to the task to be executed according to the number of the subtask instances in which the task to be executed has not been retrieved further includes: and dynamically adjusting the task priority weight corresponding to the task to be executed according to the number of the subtask instances which are not fished yet in the task to be executed and the maximum fishing number of the subtask instances in unit time.

Optionally, dynamically adjusting the task priority weight corresponding to the task to be executed according to the number of the subtask instances that have not been salvaged of the task to be executed and the maximum salvage number of the subtask instances in unit time further includes: determining the initial weight of the task; calculating a weight change value according to the initial weight of the task, the number of subtask instances of which the task to be executed is not fished yet and the maximum fishing number of the subtask instances in unit time; and calculating the task priority weight according to the task initial weight and the weight change value.

Optionally, the sum of the initial weights of the tasks corresponding to all the tasks to be executed in the task set is equal to 1;

the sum of the weight change values corresponding to all the tasks to be executed in the task set is less than or equal to 1 and is greater than or equal to-1.

Optionally, before determining the set of subtask instances to be fished, the method further comprises: judging whether the maximum available quota number of the tasks is less than or equal to the maximum fishing number of the subtask instances in unit time;

determining a set of subtask instances to be fished further comprises: if the number of the maximum available quotas of the tasks is less than or equal to the maximum salvaging number of the subtask instances in unit time, determining a subtask instance set to be salvaged according to the number of the maximum available quotas of the tasks; and if the maximum available quota quantity of the task is larger than the maximum salvaging quantity of the subtask instances in unit time, determining a subtask instance set to be salvaged according to the maximum salvaging quantity of the subtask instances in unit time.

Optionally, dynamically adjusting the task priority weight corresponding to the task to be executed according to the number of the subtask instances in which the task to be executed has not been retrieved further includes: counting the number of subtask instances which are not fished yet for the task to be executed according to the maximum available quota number of the task or the maximum fishing number of the subtask instances in unit time; and dynamically adjusting the task priority weight corresponding to the task to be executed according to the number of the subtask instances of the task to be executed which are not fished yet.

Optionally, after calculating the maximum available quota number of the task according to the available quota number of the task set and the task priority weight corresponding to the task to be executed, the method further includes: judging whether the maximum available quota quantity of the tasks is greater than 0; if yes, executing a step of judging whether the maximum available quota quantity of the tasks is less than or equal to the maximum fishing quantity of the subtask instances in unit time; if not, finishing fetching the subtask instance of the task to be executed.

Optionally, before calculating the maximum available quota number of the task according to the available quota number of the task set and the task priority weight corresponding to the task to be executed, the method further includes: judging whether the number of available quotas of the task set is greater than 0 or not; if yes, calculating the maximum available quota quantity of the task according to the available quota quantity of the task set and the task priority weight corresponding to the task to be executed; if not, the sub-task instance is finished being fished.

According to yet another aspect of the present invention, there is provided a computing device comprising: the processor, the memory and the communication interface complete mutual communication through the communication bus;

the memory is used for storing at least one executable instruction, and the executable instruction enables the processor to execute the operation corresponding to the distributed task scheduling method.

According to still another aspect of the present invention, there is provided a computer storage medium having at least one executable instruction stored therein, the executable instruction causing a processor to perform operations corresponding to the distributed task scheduling method.

According to the scheme provided by the invention, the task distribution module is suitable for distributing at least one task to be executed contained in the task set to the plurality of task fishing modules; the task fetching module is suitable for calculating the maximum available quota number of the tasks according to the available quota number of the task set and the task priority weight corresponding to the task to be executed aiming at each task to be executed, and further determining a subtask instance set to be fetched; distributing the retrieved subtask instances to a task execution module; dynamically adjusting the task priority weight corresponding to the task to be executed according to the number of the subtask instances of the task to be executed which are not fished yet; and the task execution module is suitable for executing the subtasks corresponding to the acquired subtask instances. Based on the scheme provided by the invention, the fishing probability of each task to be executed can be dynamically adjusted by dynamically adjusting the priority weight of the task, so that the resource is released at the low peak period of the task, the scheduling is accelerated at the high peak period of the task, the problem of resource allocation among multiple tasks is solved on the premise of ensuring the integral stability of the system, and the problem of overstock of task instances is solved. The problem of resource isolation among tasks to be executed during task scheduling is solved, and the effect that the tasks to be executed are independent from each other and can be perceived mutually is achieved

The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.

Drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:

FIG. 1 is a schematic diagram illustrating the structure of a distributed task scheduling system according to one embodiment of the present invention;

FIG. 2 is a schematic diagram illustrating a scheduling process of the distributed task scheduling system;

FIG. 3 illustrates a flow diagram of a distributed task scheduling method according to one embodiment of the present invention;

FIG. 4 is a flowchart illustrating a distributed task scheduling method according to another embodiment of the present invention;

FIG. 5 shows a schematic structural diagram of a computing device according to one embodiment of the invention.

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

Fig. 1 is a schematic structural diagram of a distributed task scheduling system according to an embodiment of the present invention, and fig. 2 is a schematic diagram illustrating a scheduling process of the distributed task scheduling system. As shown in fig. 1 and 2, the distributed task scheduling system includes: a task distribution module 100, a plurality of task fishing modules 110, and a plurality of task execution modules 120.

The task distributing module 100 is adapted to distribute at least one task to be executed included in the task set to the plurality of task fetching modules.

The task set is a set of tasks, and is to divide the tasks according to task dimensions, for example, dividing the tasks according to services, for example, corresponding store tasks may be divided into one task set, corresponding audit tasks may be divided into one task set, and corresponding staff management tasks may be divided into one task set. As shown in fig. 2, task set a contains the following tasks: a batch processing task A, a batch processing task B and a batch processing task C; task set B contains the following tasks: a general task A, a general task B and a general task C; task set C contains the following tasks: the process task a, the process task B, and the process task C are only for illustration and have no limiting effect.

In this embodiment, the tasks are divided according to the task dimension, and the task set may include a plurality of tasks, where each task corresponds to a corresponding task execution time, and for some tasks, the task execution time is not reached, and for other tasks, the task execution time is reached, where the task that needs to be executed when the task execution time is reached is referred to as a task to be executed.

Specifically, the task distribution module 100 queries the database, reads all task sets from the database, and determines all to-be-executed tasks (which may also be referred to as activation tasks and refer to tasks that reach task execution time) in each task set, for example, determines that a batch task a and a batch task B are to-be-executed tasks for a task set a; determining the general task A as a task to be executed for the task set B; determining that the flow task a is a task to be executed for the task set C, querying the database again to determine a task body of the task to be executed, and then distributing at least one task to be executed included in the task set to the plurality of task fetching modules by the task distributing module 100 according to the task dimension. Wherein the task body is a definition, template, etc. of the task. Fig. 2 is only a schematic illustration, and in general, a task set includes a plurality of tasks to be executed.

In an optional implementation manner of the present invention, the scheduling center may send an instance salvage message to the task distribution module at regular time, for example, the scheduling center sends the instance salvage message to the task distribution module every 1 second, and the task distribution module triggers and executes the task distribution operation to be executed after receiving the instance salvage message sent by the scheduling center.

The task fetching module 110 is adapted to calculate, for each task to be executed, the maximum available quota number of the task according to the available quota number of the task set and the task priority weight corresponding to the task to be executed, and further determine a subtask instance set to be fetched; distributing the retrieved subtask instances to a task execution module; and dynamically adjusting the task priority weight corresponding to the task to be executed according to the number of the subtask instances of the task to be executed which are not fished yet.

In this embodiment, the number of available quotas for each task set is set, for example, the number of available quotas for each task set is set for each task set a, B, and C, where the number of available quotas for each task set may be different.

At the beginning (before the sub-task instances are fished), the available quota quantity of the task set is the maximum quota quantity of the task set, the maximum quota quantity of the task set is set according to the sub-task instance quantity of the tasks to be executed in the task set, for example, the maximum quota number of the task set corresponding to the task set a is 200, the maximum quota number of the task set corresponding to the task set B is 150, and the maximum quota number of the task set corresponding to the task set C is 100, the available quota number of the task set continuously changes as the subtask instances of the task to be executed are continuously fetched, the available quota number of the task set is adjusted according to the fetched number of the subtask instances, for example, the number of the dragged subtask instances is recorded as the number of the quota existing in the task set, and then the number of the quota available in the task set is equal to the maximum quota number in the task set — the number of the quota existing in the task set.

In order to schedule tasks well and enable resources to be distributed reasonably, corresponding priorities are correspondingly set for each task to be executed in a task set, and the priorities limit the probability of each task obtaining quota quantity in the task set to which the task belongs and are expressed by task priority weights. Each task to be executed comprises at least one subtask, and the maximum available quota number of the task refers to the maximum number of subtask instances that can be fished by each task to be executed in the task set.

Specifically, the task fetching module 110 receives the to-be-executed tasks distributed by the task distribution module 100, and for each to-be-executed task, the task fetching module 110 may calculate the maximum available quota number of the task according to the available quota number of the task set corresponding to the task set to which the to-be-executed task belongs and the task priority weight corresponding to the to-be-executed task, and more specifically, may calculate the maximum available quota number of the task according to the following formula (1):

task priority weight formula (1) is the maximum available quota quantity of the task set

After the maximum available quota number of the task is calculated, the subtask instance set to be fished may be determined, for example, the subtask instance set to be fished may be determined in a time sequence, or the subtask instance set to be fished may also be determined randomly, which is not specifically limited herein.

After determining the set of subtask instances to be retrieved, the database may be queried to retrieve the subtask instances to be executed, and specifically, the subtask instances to be executed in the task body are retrieved by querying the database, where the task instance is an object to be executed generated by an operation in a service, and a part of information is read from the task body during generation.

In this embodiment, the task fetching module 110 is responsible for fetching the subtask instance, but does not execute the corresponding subtask instance, and therefore, after the task fetching module fetches the corresponding subtask instance, the task fetching module needs to distribute the fetched subtask instance to the task execution module, for example, the task fetching module 110 distributes the fetched subtask instance to the task execution module: and the subtask instances such as white list updating, store background modification and the like are distributed to the task execution module 120, and the task execution module 120 executes the subtasks corresponding to the acquired subtask instances.

After each time of fetching, the number of subtask instances of the to-be-executed task that have not been fetched changes, and the task fetching module 110 may dynamically adjust the task priority weight corresponding to the to-be-executed task according to the number of subtask instances of the to-be-executed task that have not been fetched, and dynamically adjust the fetching probability of each to-be-executed task by dynamically adjusting the task priority weight, thereby reasonably allocating the task fetching resources and avoiding the overstock of the task. The number of the dragged subtask instances affects the number of the subtask instances of which the task to be executed is not dragged yet.

And the task execution module 120 is adapted to execute the subtask corresponding to the acquired subtask instance.

After acquiring the subtask instance distributed by the task fetching module 110, the task execution module 120 loads execution information required for executing the subtask, and invokes a real task executor to complete processing operations on the related subtask.

If an exception occurs during the execution of the subtask corresponding to the obtained subtask instance, the task execution module 120 may record corresponding exception information, so as to perform a check according to the exception information.

In an alternative embodiment of the invention, the task salvage module is further adapted to: and dynamically adjusting the task priority weight corresponding to the task to be executed according to the number of the subtask instances which are not fished yet in the task to be executed and the maximum fishing number of the subtask instances in unit time.

Specifically, for each task to be executed, a maximum salvage number (TPS) of subtask instances in unit time may be set, where the maximum salvage number of subtask instances in unit time defines a maximum number of subtask instances that can be salvaged in unit time, for example, for a batch processing task a, the maximum salvage number of subtask instances in unit time is 5; aiming at the batch processing task B, the maximum fishing quantity of the subtask instances in the corresponding unit time is 6; aiming at the general task A, the maximum fishing quantity of the subtask instances in unit time is 4; for the process task a, the maximum fishing number of the subtask instances in the unit time corresponding to the process task a is 10, which is only an example and does not have any limitation, and a person skilled in the art can flexibly set the value corresponding to the TPS according to actual needs.

The number of the subtask instances of the task to be executed which are not fished reflects the task backlog condition of the task to be executed, if the task backlog is more or less, the TPS reflects the number of the subtask instances of each task to be executed which are fished, so the task priority weight dynamically adjusted according to the number of the subtask instances of the task to be executed which are not fished and the maximum fishing number of the subtask instances in unit time more meets the task scheduling requirement, and the subsequent resource allocation is more reasonable.

In an alternative embodiment of the invention, the task salvage module is further adapted to: determining the initial weight of the task; calculating a weight change value according to the initial weight of the task, the number of subtask instances of which the task to be executed is not fished yet and the maximum fishing number of the subtask instances in unit time; and calculating the task priority weight according to the task initial weight and the weight change value.

The task initial weights are set according to task priorities, where the priorities may be embodied as task importance degrees, different tasks, different priorities, and different corresponding task initial weights, and it should be noted that the sum of the task initial weights corresponding to all the tasks to be executed included in the task set is equal to 1.

For example, the tasks to be executed in the task set a are a batch processing task a and a batch processing task B, and then the sum of the initial weights of the tasks corresponding to the batch processing task a and the batch processing task B is equal to 1, for example, the initial weight of the task corresponding to the batch processing task a is 0.7, and the initial weight of the task corresponding to the batch processing task B is 0.3; if the task set a further includes other tasks to be executed, such as a batch task C and a batch task D, the initial weight sum of the tasks corresponding to the batch task a, the batch task B, the batch task C and the batch task D is equal to 1, which is only an example and does not have any limiting effect.

The weight change value can be increment or decrement or 0, and reflects the weight change condition, wherein the sum of the weight change values corresponding to all the tasks to be executed in the task set is less than or equal to 1 and is greater than or equal to-1.

In this alternative embodiment, the weight change value may be calculated according to the following formula (2):

wherein, Delta i is the priority of the ith task to be executed in the task setWeight change value, β, corresponding to a level_iThe initial weight of the task corresponding to the priority of the ith task to be executed in the task set, N_iFor the number of subtask instances for which the ith task to be executed has not been fished, S_iThe value of TPS corresponding to the ith task to be executed, i ═ 1,2,3 … n, where,

after the weight change value is calculated, the task priority weight of the ith task to be executed can be calculated according to the following formula (3):

task priority weight is equal to task initial weight and weight change value is equal to beta_i+ delta i formula (3)

Therefore, the number of subtask instances of which the task to be executed is not fished can obviously change the corresponding task fishing possibility beta_i+ Δ i: if the task to be executed has no sub-task instance which is not fished, namely N_iIf 0, the possibility of fishing β_i+ Δ i ═ 0; if the number of the subtask instances of which the task to be executed is not fished is equal to the preset TPS, namely N_i＝S_iThen the possibility of fishing is beta_i+Δi＝β_iAnd the initial weight of the task is consistent; n if the number of subtask instances of the task to be executed which are not fished yet is very large_i→ infinity, then the possibility of fishing out is β_i+Δi＝2β_iCompared with the case that the initial weight of the task is doubled, for the case, the adjusted priority weight of the task is more than or equal to 1, which indicates that the subtask instance of the task to be executed has a backlog phenomenon and needs to be fished, and for the case that the adjusted priority weight of the task is more than 1, the priority weight of the task corresponding to the task to be executed can be set to 1; and the task priority weight of other tasks to be executed in the task set is set to 0, and the task to be executed is preferably fished. After the fishing is finished, if enough subtask instances are fished, the subtask instances which are not fished are reduced, and after the task priority weight is calculated dynamically again, the adjusted task priority weight can beCan no longer be larger than 1, at this moment, can continue to catch other subtask instances of waiting to carry out the task. By introducing the weight change value, a feedback factor is added for task scheduling, and real dynamic scheduling is realized.

The number of subtask instances of the tasks to be executed which are not fished yet is compared with the preset TPS in real time, and the fishing probability of each task to be executed is dynamically adjusted, so that resources are released at a low-peak period of the tasks, scheduling is accelerated at a high-peak period of the tasks, and the problem of resource allocation among multiple tasks is solved on the premise of ensuring the overall stability of the system. In fig. 2, priority scheduling is a process of dynamically adjusting task priority weights.

In an optional implementation manner of the present invention, a maximum salvage number (TPS) of subtask instances in a unit time is set for each task to be executed, and one purpose of setting TPS is to limit the number of the subtask instances to be salvaged, so as to avoid a phenomenon that a task to be executed occupies too many resources, which causes too few available resources of other tasks to be executed, and causes a task backlog.

Therefore, the task fetching module can judge whether the maximum available quota number of the task is less than or equal to the maximum fetching number of the subtask instances in unit time before fetching the subtask instances, and determine the number of the fetched subtask instances according to the judgment result, so as to determine the subtask instance set to be fetched:

if the number of the maximum available quotas of the tasks is less than or equal to the maximum salvaging number of the subtask instances in unit time, determining a subtask instance set to be salvaged according to the number of the maximum available quotas of the tasks; and if the maximum available quota quantity of the task is larger than the maximum salvaging quantity of the subtask instances in unit time, determining a subtask instance set to be salvaged according to the maximum salvaging quantity of the subtask instances in unit time. In fig. 2, TPS control is a process of determining whether the maximum available quota for a task is less than or equal to the maximum salvaged number of subtask instances in a unit time, and determining the number of the salvaged subtask instances according to the determination result.

In an alternative embodiment of the invention, the task salvage module is further adapted to: and counting the number of the subtask instances which are not fished yet in the task to be executed according to the maximum available quota number of the task or the maximum fishing number of the subtask instances in unit time.

Specifically, after the subtask instance is fished according to the maximum available quota number of the task or the maximum fishing number of the subtask instances in unit time, the number of the subtask instances in which the task to be executed has not been fished may change correspondingly, after this fishing is finished, the number of the subtask instances in which the task to be executed has not been fished is equal to the number of the subtask instances in which the task to be executed has not been fished last-the maximum available quota number of the task or the maximum fishing number of the subtask instances in unit time, which is described in an example, the number of the subtask instances in which the task to be executed has not been fished last time is 90, this fishing is performed according to the maximum available quota number of the task, where the maximum available quota number of the task is 3, after this fishing is finished, the number of the subtask instances in which the task to be executed has not been fished last is 87, and the number of the subtask instances in which the task to be executed has not been fished can be used to dynamically adjust the task priority weight (ii) a

For another example, the number of the subtask instances that have not been salvaged for the last task to be executed is 90, and the salvage is performed according to the maximum salvage number of the subtask instances in unit time, where the maximum salvage number of the subtask instances in unit time is 5, after the salvage is completed, the number of the subtask instances that have not been salvaged for the task to be executed is 85, and the number of the subtask instances that have not been salvaged for the task to be executed in the statistics may be used to dynamically adjust the task priority weight.

The task priority weight can be accurately adjusted by counting the number of subtask instances of which the task to be executed is not fished in real time, so that the maximum available quota number of the task is calculated according to the adjusted task priority weight, the subtask instances with proper number are fished, and the resource allocation is more reasonable.

In an alternative embodiment of the invention, the task salvage module is further adapted to: judging whether the maximum available quota quantity of the tasks is greater than 0; if yes, further judging whether the maximum available quota quantity of the tasks is smaller than or equal to the maximum fishing quantity of the subtask instances in unit time; if not, finishing fetching the subtask instance of the task to be executed.

Specifically, after the maximum available quota number of the task is calculated according to the available quota number of the task set and the task priority weight corresponding to the task to be executed, whether the maximum available quota number of the task is greater than 0 needs to be further judged to determine whether the subtask instance of the task to be executed is fished up, and if the maximum available quota number of the task is greater than 0, it is indicated that there are subtask instances which have not been fished up, and the subtask instances need to be fished up again; if the maximum available quota number of the task is equal to 0, the subtask instances of the task to be executed are all fished, and the fishing of the subtask instances of the task to be executed can be finished.

In an alternative embodiment of the invention, the task salvage module is further adapted to: judging whether the number of available quotas of the task set is greater than 0 or not; if yes, calculating the maximum available quota quantity of the task according to the available quota quantity of the task set and the task priority weight corresponding to the task to be executed; if not, the sub-task instance is finished being fished.

Specifically, before calculating the maximum available quota number of the task according to the available quota number of the task set and the task priority weight corresponding to the task to be executed, whether the available quota number of the task set is greater than 0 or not may be first judged to determine whether all subtask instances of the task to be executed in the task set have been fished, and if the available quota number of the task set indicates that there are subtask instances that have not been fished, the subtask instances need to be fished again; if the number of the available quota of the task set is equal to 0, the subtask instances of all the tasks to be executed in the task set are all fished, and the fishing of the subtask instances can be finished. The resource check in fig. 2 is a process of determining whether the number of available quotas of the task set is greater than 0.

The following describes the distributed task scheduling system in detail with reference to specific embodiments:

the task distribution module 110 distributes the tasks to be executed included in the task set: and distributing the task A to be executed, the task B to be executed and the task C to be executed to a task fetching module, wherein the number of subtask instances corresponding to the task A to be executed is 25, the number of subtask instances corresponding to the task B to be executed is 15, the number of subtask instances corresponding to the task C to be executed is 10, and initially, the number of available quotas of the task set is 50. The initial weight of the task corresponding to the task A to be executed is 0.4, and the corresponding TPS is 8; the initial weight of the task corresponding to the task B to be executed is 0.3, and the corresponding TPS is 5; the initial weight of the task corresponding to the task C to be executed is 0.3, and the corresponding TPS is 6.

For each task to be executed, when each time the task is picked up, it is required to first determine whether the number of the available quotas of the task set is greater than 0, and when it is determined that the number of the available quotas of the task set is greater than 0, calculate the maximum number of the available quotas of the task according to the number of the available quotas of the task set and the task priority weight corresponding to the task to be executed, where the maximum number of the available quotas of the task to be processed a is 50 × 0.4 — 20, the maximum number of the available quotas of the task to be processed B is 50 × 0.3 — 15, and the maximum number of the available quotas of the task to be processed C is 50 × 0.3 — 15. Under the condition that the maximum available quota quantities of the tasks corresponding to the task A to be processed, the task B to be processed and the task C to be processed are all larger than 0, comparing the maximum available quota quantity of the task corresponding to the task A to be processed with the TPS corresponding to the task A to be processed, and if the comparison result is that the maximum available quota quantity of the task is larger than the TPS, according to the numerical value corresponding to the TPS: 8, fishing the subtask instance of the task A to be processed; and comparing the maximum available quota quantity of the task corresponding to the task B to be processed with the TPS corresponding to the task B, wherein the comparison result shows that the maximum available quota quantity of the task is greater than that of the TPS, and the maximum available quota quantity of the task is determined according to a numerical value corresponding to the TPS: 5, fishing the subtask instance of the task B to be processed; comparing the maximum available quota quantity of the task corresponding to the task C to be processed with the TPS corresponding to the task C, wherein the comparison result shows that the maximum available quota quantity of the task is greater than that of the TPS, and according to the numerical value corresponding to the TPS: 6, fishing the subtask instance of the task C to be processed; and distributing the dragged subtask instance to a task execution module.

And counting the number of subtask instances which are not fished yet by the to-be-processed task A, the to-be-processed task B and the to-be-processed task C, wherein the number of the subtask instances is 17, 10 and 4 respectively, and the number of the available quota of the task set is 50-19 and 31.

According to the number of subtask instances which are not fished yet by the task A to be processed: 17, task initial weight: 0.4, TPS: 8,

the dynamically adjusted task priority weight is 0.4+ 0.144-0.544; according to the number of subtask instances which are not fished yet by the task B to be processed: 10, task initial weight: 0.3, TPS: 5,

the dynamically adjusted task priority weight is 0.3+ 0.1-0.4; according to the number of subtask instances which are not fished yet by the task C to be processed: 4, task initial weight: 0.3, TPS: 6,

the dynamically adjusted task priority weight is 0.3-0.06-0.24

And calculating the maximum available quota number of the task again according to the changed available quota number of the task set and the adjusted task priority weight, wherein the subsequent execution process is similar to the process described above, and is not described in detail herein.

According to the system provided by the embodiment of the invention, the fishing probability of each task to be executed is dynamically adjusted by calculating the weight change value, so that the resources are released at the low peak period of the task, the scheduling is accelerated at the high peak period of the task, and the problem of resource allocation among multiple tasks is solved on the premise of ensuring the integral stability of the system. In addition, by introducing the weight change value, a feedback factor is added for task scheduling, real dynamic scheduling is realized, high-backlog tasks can be processed preferentially, and the problem that the task backlog is possibly caused because the scheduling cannot feel the load pressure of the tasks is solved. Meanwhile, the problem of resource isolation among tasks to be executed during task scheduling is solved, and the effect that the tasks to be executed are independent from each other and can be perceived mutually is achieved; the task flow control problem under different system loads is solved, and the effects that all low loads can be executed at full speed and all high loads and different priorities have probability fishing are achieved; by comparing the maximum available quota quantity of the tasks with the maximum salvaging quantity of the subtask instances in unit time in real time, the quantity of the salvaging subtask instances can be reasonably controlled, so that the resource allocation is more reasonable.

FIG. 3 is a flowchart illustrating a distributed task scheduling method according to an embodiment of the present invention. As shown in fig. 3, the method comprises the steps of:

step S300, the task distribution module distributes at least one task to be executed contained in the task set to a plurality of task fetching modules.

In this embodiment, the tasks are divided according to the task dimension, and the task set may include a plurality of tasks, where each task corresponds to a corresponding task execution time, and for some tasks, the task execution time is not reached, and for other tasks, the task execution time is reached, where the task that needs to be executed when the task execution time is reached is referred to as a task to be executed. And the task distribution module distributes at least one task to be executed contained in the task set to the plurality of task fishing modules.

Step S301, a task fetching module calculates the maximum available quota number of the task according to the available quota number of the task set and the task priority weight corresponding to the task to be executed aiming at each task to be executed, and further determines a subtask instance set to be fetched; distributing the retrieved subtask instances to a task execution module; and dynamically adjusting the task priority weight corresponding to the task to be executed according to the number of the subtask instances of the task to be executed which are not fished yet.

In order to schedule tasks well and enable resources to be distributed reasonably, corresponding priorities are correspondingly set for each task to be executed in a task set, and the priorities limit the probability of each task obtaining quota quantity in the task set to which the task belongs and are expressed by task priority weights.

Each task to be executed comprises at least one subtask, and the maximum available quota number of the task refers to the maximum number of subtask instances that can be fished by each task to be executed in the task set.

Specifically, the task fetching module receives the to-be-executed tasks distributed by the task distribution module, and for each to-be-executed task, the task fetching module may calculate the maximum available quota number of the task according to the available quota number of the task set corresponding to the task set to which the to-be-executed task belongs and the task priority weight corresponding to the to-be-executed task, and more specifically, may calculate the maximum available quota number of the task according to the following formula (4):

task maximum available quota quantity is task set available quota quantity is task priority weight formula (4)

After determining the set of subtask instances to be fished, the subtask instances to be executed can be fished, wherein the task instance is an object to be executed generated by an operation on the business, and a part of information is read from the task body during generation.

In this embodiment, the task fetching module is responsible for fetching the subtask instance, but does not execute the corresponding subtask instance, and therefore, after the task fetching module fetches the corresponding subtask instance, the task fetching module needs to distribute the fetched subtask instance to the task execution module, for example, the task fetching module distributes the fetched subtask instance to the task execution module: and the subtask instances such as white list updating, store background modification and the like are distributed to the task execution module, and the task execution module executes the subtasks corresponding to the acquired subtask instances.

After each time of fishing, the number of subtask instances of the task to be executed, which are not yet fished, changes, the task fishing module can dynamically adjust the task priority weight corresponding to the task to be executed according to the number of subtask instances of the task to be executed, which are not yet fished, and the fishing probability of each task to be executed is dynamically adjusted by dynamically adjusting the task priority weight, so that the task fishing resources are reasonably distributed, and the task overstock is avoided. The number of the dragged subtask instances affects the number of the subtask instances of which the task to be executed is not dragged yet.

Step S302, the task execution module executes the subtask corresponding to the acquired subtask instance.

After the task execution module obtains the subtask instance distributed by the task fetching module, the task execution module loads execution information required by the subtask execution and invokes a real task executor to complete processing operation of the related subtask.

And if an exception occurs in the process of executing the subtask corresponding to the acquired subtask instance, the task execution module records corresponding exception information so as to conveniently perform troubleshooting according to the exception information.

According to the method provided by the embodiment of the invention, the fishing probability of each task to be executed can be dynamically adjusted by dynamically adjusting the task priority weight, so that the resources are released at the low peak period of the task, the scheduling is accelerated at the high peak period of the task, the problem of resource allocation among multiple tasks is solved on the premise of ensuring the integral stability of the system, and the problem of overstock of task instances is solved.

Fig. 4 is a flowchart illustrating a distributed task scheduling method according to another embodiment of the present invention. As shown in fig. 4, the method comprises the steps of:

step S400, the task distribution module distributes at least one task to be executed contained in the task set to a plurality of task fetching modules.

Specifically, the task distribution module queries a database, reads all task sets from the database, and determines all to-be-executed tasks (which may be called activation tasks and refer to tasks that reach task execution time) in each task set, for example, determines a batch task a and a batch task B as to-be-executed tasks for a task set a; determining the general task A as a task to be executed for the task set B; and determining the flow task A as a task to be executed for the task set C, querying the database again to determine a task body of the task to be executed, and then distributing at least one task to be executed included in the task set to the plurality of task fetching modules by the task distribution module according to the task dimension. Wherein the task body is a definition, template, etc. of the task. Fig. 2 is only a schematic illustration, and in general, a task set includes a plurality of tasks to be executed.

Step S401, the task fetching module judges whether the number of available quota of the task set is greater than 0 or not for each task to be executed, if yes, step S402 is executed; if not, go to step S412.

Specifically, before calculating the maximum available quota number of the task according to the available quota number of the task set and the task priority weight corresponding to the task to be executed, whether the available quota number of the task set is greater than 0 or not may be first judged to determine whether all subtask instances of the task to be executed in the task set have been fished, and if the available quota number of the task set indicates that there are subtask instances that have not been fished, the subtask instances need to be fished again; if the number of the available quota of the task set is equal to 0, the subtask instances of all the tasks to be executed in the task set are all fished, and the fishing of the subtask instances can be finished.

Step S402, calculating the maximum available quota quantity of the task according to the available quota quantity of the task set and the task priority weight corresponding to the task to be executed.

This step is similar to the corresponding step in step S301 in the embodiment shown in fig. 3, and is not described here again.

Step S403, judging whether the maximum available quota quantity of the task is greater than 0, if so, executing step S404; if not, step S413 is executed.

Step S404, judging whether the maximum available quota quantity of the task is less than or equal to the maximum fishing quantity of the subtask instances in unit time, if so, executing step S405; if not, go to step S406.

The method comprises the steps that the maximum salvaging quantity (TPS) of subtask instances in unit time is set for each task to be executed, and one purpose of setting the TPS is to limit the quantity of the subtask instances to be salvaged, so that the phenomenon that the task to be executed is overstocked due to the fact that a certain task to be executed occupies too much resources and other tasks to be executed are too few in available resources is avoided.

Before the task fetching module fetches the subtask instances, whether the maximum available quota number of the task is smaller than or equal to the maximum fetching number of the subtask instances in unit time needs to be judged, the number of the fetched subtask instances is determined according to a judgment result, and then a subtask instance set to be fetched is determined, wherein the judgment result influences the number of the fetched subtask instances.

And S405, determining a subtask instance set to be fished according to the maximum available quota number of the task.

If the maximum available quota number of the task is less than or equal to the maximum salvaging number of the subtask instances in the unit time, determining a subtask instance set to be salvaged according to the maximum available quota number of the task, for example, the maximum available quota number of the task is 4, the maximum salvaging number of the subtask instances in the unit time is 5, the maximum available quota number of the task is less than the maximum salvaging number of the subtask instances in the unit time, the salvaging number of the subtask instances is 4, and determining the subtask instance set to be salvaged.

And step S406, determining a subtask instance set to be fished according to the maximum fishing quantity of the subtask instances in unit time.

If the maximum available quota number of the task is greater than the maximum salvaging number of the subtask instances in unit time, determining a subtask instance set to be salvaged according to the maximum salvaging number of the subtask instances in unit time, for example, the maximum available quota number of the task is 10, the maximum salvaging number of the subtask instances in unit time is 5, the maximum available quota number of the task is greater than the maximum salvaging number of the subtask instances in unit time, the salvaging number of the subtask instances is 5, and determining the subtask instance set to be salvaged.

And step S407, distributing the retrieved subtask instances to a task execution module.

In an optional implementation manner of the present invention, the task priority weight corresponding to the task to be executed may be dynamically adjusted according to the number of the subtask instances that have not been retrieved for the task to be executed and the maximum retrieved number of the subtask instances in unit time.

In this embodiment, the task priority weights will be dynamically adjusted according to steps S408-S411:

step S408, counting the number of the subtask instances which are not fished yet in the task to be executed according to the maximum available quota number of the task or the maximum fishing number of the subtask instances in unit time.

And step S409, determining the task initial weight.

And step S410, calculating a weight change value according to the initial weight of the task, the number of subtask instances which are not fished yet in the task to be executed and the maximum fishing number of the subtask instances in unit time.

In this alternative embodiment, the weight change value may be calculated according to the following formula (5):

wherein, Δ i is a weight change value corresponding to the priority of the ith task to be executed in the task set, β_iThe initial weight of the task corresponding to the priority of the ith task to be executed in the task set, N_iFor the number of subtask instances for which the ith task to be executed has not been fished, S_iThe value of TPS corresponding to the ith task to be executed, i ═ 1,2,3 … n, where,

step S411, calculating task priority weight according to the task initial weight and the weight change value.

After the weight change value is calculated, the task priority weight of the ith task to be executed may be calculated according to the following formula (6):

task priority weight is equal to task initial weight and weight change value is equal to beta_i+ Delta i formula (6)

Therefore, the number of subtask instances of which the task to be executed is not fished can obviously change the corresponding task fishing possibility beta_i+ Δ i: if the task to be executed has no sub-task instance which is not fished, namely N_iIf 0, the possibility of fishing β_i+ Δ i ═ 0; if the number of the subtask instances of which the task to be executed is not fished is equal to the preset TPS, namely N_i＝S_iThen the possibility of fishing is beta_i+Δi＝β_iAnd the initial weight of the task is consistent; n if the number of subtask instances of the task to be executed which are not fished yet is very large_i→ infinity, then the possibility of fishing out is β_i+Δi＝2β_iCompared with the case that the initial weight of the task is doubled, for the case, the adjusted priority weight of the task is more than or equal to 1, which means that the subtask instance of the task to be executed has a backlog phenomenon and needs to be fished, and for the case that the adjusted priority weight of the task is more than 1, the priority weight of the task corresponding to the task to be executed can be reset, and the task is concentrated with other tasks to be executedThe task priority of (1) is reset to 0, and the task to be executed is preferably fished. After the fishing is finished, if enough subtask instances are fished, the subtask instances which are not fished are reduced, after the task priority weight is calculated dynamically again, the adjusted task priority weight may not be more than 1 any more, and at this time, the subtask instances of other tasks to be executed can be continuously fished. By introducing the weight change value, a feedback factor is added for task scheduling, and real dynamic scheduling is realized.

The number of subtask instances of the tasks to be executed which are not fished yet is compared with the preset TPS in real time, and the fishing probability of each task to be executed is dynamically adjusted, so that resources are released at a low-peak period of the tasks, scheduling is accelerated at a high-peak period of the tasks, and the problem of resource allocation among multiple tasks is solved on the premise of ensuring the overall stability of the system.

In this embodiment, the execution sequence of steps S408-S411 and S407 is not limited, for example, steps S408-S411 and S407 may be executed simultaneously, step S408-S411 may be executed first, step S407 may be executed later, step S407 may be executed first, and step S408-S411 may be executed later.

And step S412, finishing the fishing of the subtask instance.

If the number of the available quota of the task set is equal to 0, the subtask instances of all the tasks to be executed in the task set are all fished, and the fishing of the subtask instances can be finished.

And step S413, ending the fetching of the subtask instance of the task to be executed.

If the maximum available quota number of the task is equal to 0, the subtask instances of the task to be executed are all fished, and the fishing of the subtask instances of the task to be executed can be finished.

In step S414, the task execution module executes the subtask corresponding to the acquired subtask instance.

In this embodiment, the execution sequence of steps S408-S411 and S414 is not limited, for example, steps S408-S411 and S414 may be executed simultaneously, steps S408-S411 and S414 may be executed first, and step S414 and step S408-S411 may be executed first.

the task distribution module is used for distributing the tasks to be executed contained in the task set: and distributing the task A to be executed, the task B to be executed and the task C to be executed to a task fetching module, wherein the number of subtask instances corresponding to the task A to be executed is 25, the number of subtask instances corresponding to the task B to be executed is 15, the number of subtask instances corresponding to the task C to be executed is 10, and initially, the number of available quotas of the task set is 50. The initial weight of the task corresponding to the task A to be executed is 0.4, and the corresponding TPS is 8; the initial weight of the task corresponding to the task B to be executed is 0.3, and the corresponding TPS is 5; the initial weight of the task corresponding to the task C to be executed is 0.3, and the corresponding TPS is 6.

According to the number of subtask instances which are not fished yet by the task A to be processed: 17, task initial weight: 0.4, TPS: 8, calculating the weight

The dynamically adjusted task priority weight is 0.4+ 0.144-0.544; according to the number of subtask instances which are not fished yet by the task B to be processed: 10, task initial weight: 0.3, TPS: 5, calculating the weight

The dynamically adjusted task priority weight is 0.3+ 0.1-0.4; according to the number of subtask instances which are not fished yet by the task C to be processed: 4, task initial weight: 0.3, TPS: 6, calculating the weight

And the dynamically adjusted task priority weight is 0.3-0.06-0.24.

According to the method provided by the embodiment of the invention, the fishing probability of each task to be executed is dynamically adjusted by calculating the weight change value, so that the resources are released at the low peak period of the task, the scheduling is accelerated at the high peak period of the task, and the problem of resource allocation among multiple tasks is solved on the premise of ensuring the integral stability of the system. In addition, by introducing the weight change value, a feedback factor is added for task scheduling, real dynamic scheduling is realized, high-backlog tasks can be processed preferentially, and the problem that the task backlog is possibly caused because the scheduling cannot feel the load pressure of the tasks is solved. Meanwhile, the problem of resource isolation among tasks to be executed during task scheduling is solved, and the effect that the tasks to be executed are independent from each other and can be perceived mutually is achieved; the task flow control problem under different system loads is solved, and the effects that all low loads can be executed at full speed and all high loads and different priorities have probability fishing are achieved; by comparing the maximum available quota quantity of the tasks with the maximum salvaging quantity of the subtask instances in unit time in real time, the quantity of the salvaging subtask instances can be reasonably controlled, so that the resource allocation is more reasonable.

The embodiment of the present application further provides a non-volatile computer storage medium, where at least one executable instruction is stored in the computer storage medium, and the computer executable instruction may execute the distributed task scheduling method in any method embodiment described above.

Fig. 5 is a schematic structural diagram of a computing device according to an embodiment of the present invention, and the specific embodiment of the present invention does not limit the specific implementation of the computing device.

As shown in fig. 5, the computing device may include: a processor (processor)502, a Communications Interface 504, a memory 506, and a communication bus 508.

Wherein:

the processor 502, communication interface 504, and memory 506 communicate with one another via a communication bus 508.

A communication interface 504 for communicating with network elements of other devices, such as clients or other servers.

The processor 502 is configured to execute the program 510, and may specifically perform relevant steps in the above-described distributed task scheduling method embodiment.

In particular, program 510 may include program code that includes computer operating instructions.

The processor 502 may be a central processing unit CPU, or an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits configured to implement an embodiment of the present invention. The computing device includes one or more processors, which may be the same type of processor, such as one or more CPUs; or may be different types of processors such as one or more CPUs and one or more ASICs.

And a memory 506 for storing a program 510. The memory 506 may comprise high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.

The program 510 may specifically be configured to cause the processor 502 to perform the distributed task scheduling method in any of the above-described method embodiments. For specific implementation of each step in the program 510, reference may be made to corresponding steps and corresponding descriptions in units in the foregoing distributed task scheduling embodiment, which are not described herein again. It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described devices and modules may refer to the corresponding process descriptions in the foregoing method embodiments, and are not described herein again.

The algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose systems may also be used with the teachings herein. The required structure for constructing such a system will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.

In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

The various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It will be appreciated by those skilled in the art that a microprocessor or Digital Signal Processor (DSP) may be used in practice to implement some or all of the functionality of some or all of the components in a distributed task scheduling apparatus according to embodiments of the present invention. The present invention may also be embodied as apparatus or device programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing the present invention may be stored on computer-readable media or may be in the form of one or more signals. Such a signal may be downloaded from an internet website or provided on a carrier signal or in any other form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names.

Claims

1. A distributed task scheduling system, comprising: the system comprises a task distribution module, a plurality of task fetching modules and a plurality of task execution modules;

the task distribution module is suitable for distributing at least one task to be executed contained in the task set to the plurality of task fetching modules;

the task salvaging module is suitable for calculating the maximum available quota number of the tasks according to the available quota number of the task set and the task priority weight corresponding to the task to be executed aiming at each task to be executed, and further determining a subtask instance set to be salvaged; distributing the retrieved subtask instances to a task execution module; dynamically adjusting the task priority weight corresponding to the task to be executed according to the number of the subtask instances of the task to be executed which are not fished yet, and adjusting the number of the available quota of the task set according to the number of the fished subtask instances;

and the task execution module is suitable for executing the subtasks corresponding to the acquired subtask instances.

2. The system of claim 1, wherein the task salvage module is further adapted to:

and dynamically adjusting the task priority weight corresponding to the task to be executed according to the number of the subtask instances which are not fished yet in the task to be executed and the maximum fishing number of the subtask instances in unit time.

3. The system of claim 2, wherein the task salvage module is further adapted to:

determining the initial weight of the task;

calculating a weight change value according to the task initial weight, the number of subtask instances which are not fished yet in the task to be executed and the maximum fishing number of the subtask instances in unit time;

and calculating task priority weight according to the task initial weight and the weight change value.

4. The system according to claim 3, wherein the task set comprises a total of the initial weights of all tasks to be executed, which are equal to 1;

5. The system of any one of claims 2-4, wherein the task salvage module is further adapted to:

judging whether the maximum available quota number of the tasks is less than or equal to the maximum fishing number of the subtask instances in unit time;

if yes, determining a subtask instance set to be fished according to the maximum available quota number of the task;

and if not, determining the subtask instance set to be fished according to the maximum fishing quantity of the subtask instances in unit time.

6. The system of claim 5, wherein the task salvage module is further adapted to:

and counting the number of the subtask instances which are not fished yet in the task to be executed according to the maximum available quota number of the task or the maximum fishing number of the subtask instances in unit time.

7. The system of claim 5, wherein the task salvage module is further adapted to: judging whether the maximum available quota quantity of the tasks is larger than 0;

if yes, further judging whether the maximum available quota quantity of the task is smaller than or equal to the maximum fishing quantity of the subtask instances in unit time;

if not, finishing fetching the subtask instance of the task to be executed.

8. The system of any one of claims 1-4, wherein the task salvage module is further adapted to: judging whether the number of available quotas of the task set is greater than 0 or not;

if yes, calculating the maximum available quota quantity of the task according to the available quota quantity of the task set and the task priority weight corresponding to the task to be executed;

if not, the sub-task instance is finished being fished.

9. A distributed task scheduling method comprises the following steps:

the task distribution module distributes at least one task to be executed contained in the task set to the plurality of task fishing modules;

the task fetching module calculates the maximum available quota number of the tasks according to the available quota number of the task set and the task priority weight corresponding to the task to be executed aiming at each task to be executed, and further determines a subtask instance set to be fetched; distributing the retrieved subtask instances to a task execution module; dynamically adjusting the task priority weight corresponding to the task to be executed according to the number of the subtask instances of the task to be executed which are not fished yet, and adjusting the number of the available quota of the task set according to the number of the fished subtask instances;

and the task execution module executes the subtasks corresponding to the acquired subtask instances.

10. The method of claim 9, wherein the dynamically adjusting the task priority weight corresponding to the task to be executed according to the number of subtask instances in which the task to be executed has not been dragged further comprises:

11. The method of claim 10, wherein the dynamically adjusting the task priority weight corresponding to the task to be executed according to the number of subtask instances that have not been fished for the task to be executed and the maximum fishing number of subtask instances per unit time further comprises:

determining the initial weight of the task;

12. The method according to claim 11, wherein the task set includes a total of the initial weights of all tasks to be executed, which are equal to 1;

13. The method of any of claims 10-12, wherein prior to determining the set of subtask instances to be fished, the method further comprises:

the determining the set of subtask instances to be fished further comprises:

if the number of the task maximum available quotas is less than or equal to the maximum salvaging number of the subtask instances in unit time, determining a subtask instance set to be salvaged according to the number of the task maximum available quotas;

and if the maximum available quota quantity of the task is larger than the maximum salvaging quantity of the subtask instances in unit time, determining a subtask instance set to be salvaged according to the maximum salvaging quantity of the subtask instances in unit time.

14. The method of claim 13, wherein the dynamically adjusting the task priority weight corresponding to the task to be executed according to the number of subtask instances for which the task to be executed has not been dragged further comprises:

counting the number of the subtask instances which are not fished yet in the task to be executed according to the maximum available quota number of the task or the maximum fishing number of the subtask instances in unit time;

and dynamically adjusting the task priority weight corresponding to the task to be executed according to the number of the subtask instances of which the task to be executed is not fished yet.

15. The method according to claim 13, wherein after calculating the maximum available quota amount of the task according to the available quota amount of the task set and the task priority weight corresponding to the task to be executed, the method further comprises: judging whether the maximum available quota quantity of the tasks is larger than 0;

if yes, executing a step of judging whether the maximum available quota quantity of the task is less than or equal to the maximum fishing quantity of the subtask instances in unit time;

if not, finishing fetching the subtask instance of the task to be executed.

16. The method according to any one of claims 9-12, wherein before calculating the maximum available quota amount for a task according to the available quota amount for a task set and the task priority weight corresponding to the task to be executed, the method further comprises: judging whether the number of available quotas of the task set is greater than 0 or not;

if not, the sub-task instance is finished being fished.

17. A computing device, comprising: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;

the memory is used for storing at least one executable instruction, and the executable instruction causes the processor to execute the operation corresponding to the distributed task scheduling method according to any one of claims 9-16.

18. A computer storage medium having stored therein at least one executable instruction for causing a processor to perform operations corresponding to the distributed task scheduling method of any one of claims 9-16.