CN115686790A

CN115686790A - Distributed task processing method and system, electronic device and storage medium

Info

Publication number: CN115686790A
Application number: CN202211361769.7A
Authority: CN
Inventors: 王龙龙; 孙能林
Original assignee: Qingdao Haier Technology Co Ltd; Haier Smart Home Co Ltd; Haier Uplus Intelligent Technology Beijing Co Ltd
Current assignee: Qingdao Haier Technology Co Ltd; Haier Smart Home Co Ltd; Haier Uplus Intelligent Technology Beijing Co Ltd
Priority date: 2022-11-02
Filing date: 2022-11-02
Publication date: 2023-02-03

Abstract

The application discloses a distributed task processing method and system, electronic equipment and a storage medium, wherein the method is applied to any distributor in the system in Airflow, the system comprises a plurality of distributors and a plurality of working nodes, and the method comprises the following steps: receiving a current task pushed by a corresponding scheduler; wherein each said distributor corresponds to one said scheduler; sequencing the working nodes according to the descending order of the load scores of the working nodes maintained locally at present to obtain a node sequencing result; the load score of each working node is obtained by calculation of each working node and fed back to the distributor; the larger the load of the working node is, the smaller the load score of the working node is; and trying to send the current task to each working node one by one according to the node sequencing result until any one working node successfully receives the current task.

Description

Distributed task processing method and system, electronic device and storage medium

Technical Field

The present application relates to the field of task scheduling technologies, and in particular, to a distributed task processing method and system, an electronic device, and a storage medium.

Background

In big data related work, task scheduling is always the infrastructure supporting the whole big data work, so the requirements on stability and concurrency of scheduling service are very high. The Airflow has better stability and functionality, so the Airflow is widely applied to task scheduling.

Because the current load balancing algorithm cannot be directly applied to the Airflow, the current Airflow mainly distributes tasks to be executed to each worker to be executed in an approximate polling manner, and the worker is responsible for starting the tasks in a sub-process manner and managing the running state of the tasks.

However, due to uncertainty of the running time of the scheduled task, in the existing manner, a situation that a plurality of tasks with long time consumption and large resource consumption are allocated to the same worker may occur, so that a certain worker overstocks too many tasks and consumes too many resources, and further a situation that a task newly connected by the worker starts identification or a running process is eliminated is caused. Therefore, the conventional method cannot effectively ensure that the task is normally executed.

Disclosure of Invention

Based on the defects of the prior art, the application provides a distributed task processing method and system, electronic equipment and a storage medium, so as to solve the problem that the task cannot be normally executed in the prior art.

In order to achieve the above object, the present application provides the following technical solutions:

a first aspect of the present application provides a distributed task processing method, which is applied to any one distributor in a distributed task processing system in an Airflow, where the distributed task processing system includes multiple distributors and multiple work nodes, and the distributed task processing method includes:

receiving a current task pushed by a corresponding scheduler; wherein each said distributor corresponds to one said scheduler;

sequencing the working nodes according to the descending order of the load scores of the working nodes maintained locally at present to obtain a node sequencing result; the load score of each working node is obtained by calculation of each working node and fed back to the distributor; the larger the load of the working node is, the smaller the load score of the working node is;

and trying to send the current task to each working node one by one according to the node sequencing result until any one working node successfully receives the current task.

Optionally, in the distributed task processing method, the method for calculating the load score of the work node includes:

every time a preset time is spaced, the current working node obtains the number of task instructions of a current local waiting queue, the number of task instructions of an operation queue and the number of task instructions of an ending queue; wherein the current working node refers to any one of the working nodes;

the current working node calculates the sum of the task instruction number of the waiting queue, the task instruction number of the running queue and the task instruction number of the ending queue to obtain the backlog of the current task instruction;

the current working node calculates the difference value between a preset limited task quantity and the current task instruction backlog quantity to obtain the current idle task quantity;

the current working node divides the larger value of the current idle task amount and zero by the preset limit task amount to obtain the current load score of the current working node; and the current load score of the current working node is used for feeding back to each distributor so as to update the load score of the current working node maintained by each distributor.

Optionally, in the distributed task processing method, the attempting to send the current task to each of the working nodes one by one according to the node sorting result until any one of the working nodes successfully receives the current task includes:

determining a current node to be received by the working node which is sequenced at the forefront in the current node sequencing result and in each working node to which the current task is not sent in the current scheduling;

sending the current task to the current node to be received;

if the response of refusing to receive fed back by the current node to be received is not received, the current node to be received is determined to successfully receive the current task; when the current self load score of the current node to be received is zero, feeding back a response of refusing to receive;

if a response of refusing to receive, which is fed back by the current node to be received, is received, whether the working node to which the current task is not sent in the current scheduling exists in the current node sequencing result is judged;

if the working nodes which have not sent the current task to the current node in the current scheduling are judged to exist in the current node sequencing result, the current node sequencing result is returned to be executed, and the working nodes which are sequenced at the forefront in the working nodes which have not sent the current task to the current node in the current scheduling determine the current nodes to be received;

and if the working nodes which do not send the current task to the current node in the current scheduling do not exist in the current node sequencing result, putting the current task back to a waiting queue, and returning to execute the current node sequencing result until next scheduling, wherein the working nodes which are not sent the current task to the current node in the current scheduling determine the current nodes to be received.

Optionally, in the distributed task processing method, the attempting to send the current task to each of the working nodes one by one according to the node sorting result until any one of the working nodes successfully receives the current task further includes:

receiving a current load score of the current target working node fed back by the current target working node; wherein the current target working node is the working node which successfully receives the current task; the current load score of the current target working node is calculated and fed back after the current target working node successfully receives the current task;

and updating the load score of the current working node maintained locally by using the current load score of the current working node.

Optionally, in the distributed task processing method, the method further includes:

receiving a target load score fed back by a node to be offline; the node to be offline refers to the working node to be offline; the target load score is a preset fixed numerical value which is less than zero and is fed back to each distributor when the working node is ready to be offline;

and updating the load score of the node to be offline maintained locally by using the target load score of the node to be offline.

A second aspect of the present application provides a distributed task processing system, where the distributed task processing system is a subsystem in an Airflow, and the distributed task processing system includes:

a plurality of distributors and a plurality of worker nodes;

the working node is used for calculating the load score of the working node, feeding the load score back to the distributor and receiving the tasks distributed by the distributor; the larger the load of the working node is, the smaller the load score of the working node is;

the dispatcher is used for sorting the working nodes according to the descending order of the load scores of the working nodes maintained locally at present when the current task pushed by the corresponding scheduler is received, obtaining a node sorting result, and trying to send the current task to each working node one by one according to the node sorting result until any one working node successfully receives the current task; wherein each of the distributors corresponds to one of the schedulers.

Optionally, in the above distributed task processing system, when the work node performs the calculation of its own load score and feeds back the calculated load score to the distributor, the work node is configured to:

acquiring the number of task instructions of a current local waiting queue, the number of task instructions of an operation queue and the number of task instructions of an ending queue at intervals of preset time;

calculating the sum of the number of the task instructions of the waiting queue, the number of the task instructions of the running queue and the number of the task instructions of the ending queue to obtain the backlog of the current task instructions;

calculating a difference value between a preset limited task quantity and the current task instruction backlog quantity to obtain a current idle task quantity;

and dividing the larger value of the current idle task amount and zero by the preset limit task amount to obtain a current load score, and feeding the current load score back to each distributor so as to update the load score maintained by each distributor.

Optionally, in the distributed task processing system, the distributor executes the step of attempting to send the current task to each of the work nodes one by one according to the node sorting result until any one of the work nodes successfully receives the current task, and is configured to:

sending the current task to the current node to be received;

Optionally, in the distributed task processing system, the distributor is further configured to:

and updating the locally maintained load score of the current working node by using the current load score of the current working node.

A third aspect of the present application provides an electronic device comprising:

a memory and a processor;

wherein the memory is used for storing programs;

the processor is configured to execute the program, and when the program is executed, the program is specifically configured to implement the distributed task processing method according to any one of the above.

A fourth aspect of the present application provides a computer storage medium storing a computer program for implementing a distributed task processing method as claimed in any one of the preceding claims when executed.

The application provides a distributed task processing method which is applied to any one distributor in a distributed task processing system in Airflow, wherein the distributed task processing system comprises a plurality of distributors and a plurality of working nodes. When receiving the current task pushed by the corresponding scheduler, the distributor sorts the working nodes according to the descending order of the load scores of the working nodes maintained locally at present to obtain a node sorting result, and tries to send the current task to the working nodes one by one according to the node sorting result until any one working node successfully receives the current task. Because the load scores of the working nodes are respectively calculated by the working nodes and fed back to the distributor, and the larger the load of the working nodes is, the smaller the load score of the working nodes is, the distributor sorts the working nodes based on the load scores, and then sequentially tries to distribute tasks to the working nodes according to the sorting result, so that the task can be preferentially distributed to the working nodes with smaller loads, the load balance of the working nodes is realized, the phenomenon that one working node overstocks too many tasks and consumes too much resources is avoided, and the task can be effectively guaranteed to be successfully executed.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, it is obvious that the drawings in the following description are only the embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.

Fig. 1 is a flowchart of a distributed task processing method according to an embodiment of the present application;

FIG. 2 is a schematic diagram of a distributed task processing system including a distributor and work nodes according to the prior art;

fig. 3 is a schematic architecture diagram of a distributed task processing system according to an embodiment of the present application;

fig. 4 is a schematic architecture diagram of a network model according to an embodiment of the present application;

fig. 5 is a schematic structural diagram of a protocol packet according to an embodiment of the present application;

fig. 6 is a flowchart of a method for calculating a load score of a work node according to an embodiment of the present application;

fig. 7 is a flowchart of a method for task distribution of case node sorting results according to the embodiment of the present application;

fig. 8 is an architecture diagram of a distributed architecture composed of a distributor and a work node according to an embodiment of the present application;

fig. 9 is a flowchart of a method for updating a load score in real time according to an embodiment of the present disclosure;

fig. 10 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

In this application, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a … …" does not exclude the presence of another identical element in a process, method, article, or apparatus that comprises the element.

The distributed task processing method provided by the embodiment of the application is applied to any distributor (execution) in a distributed task processing system in the Airflow, and the distributed task processing system comprises a plurality of distributors (execution) and a plurality of working nodes (Worker). As shown in fig. 1, a distributed task processing method provided in an embodiment of the present application includes:

and S101, receiving the current task pushed by the corresponding scheduler.

It should be noted that, for high availability, multiple schedulers (schedulers) are set in the distributed architecture Airflow, and each dispatcher (executive) corresponds to one Scheduler (Scheduler).

In the Airflow, a scheduler is mainly used for scanning a job flow DAG and an instance object of a task, and sending a scanned task instruction (TaskCmd) to a corresponding distributor for execution. So for any distributor, it receives the current task pushed by the corresponding scheduler and goes to execute step S102.

The distributor (executive) is mainly responsible for managing and running tasks, and distributes the tasks to the work nodes (Worker) in a specific process. In which, the executors are usually bound to the Scheduler in a process, and the life cycle thereof is controlled by the Scheduler.

The working node (Worker) is not a component of the ariflo architecture, but a component of the executer, because the running of the task is completely managed by the executer, and the executer-Worker constitutes a traditional Master-Worker distributed processing model.

It should be noted that, as shown in fig. 2, a distributed task processing system constructed by an existing distributor and a work node distributes a task to the work node through a message queue. In the distributed task processing system constructed by the distributors and the working nodes according to the embodiment of the present application, as shown in fig. 3, each distributor communicates with each working node. The working nodes feed back the load conditions of the working nodes to the distributors, and the distributors distribute the current tasks according to the load conditions of the working nodes.

Optionally, in this embodiment of the present application, the distributor and the working node may directly implement distribution of the task and feedback of the load condition through a Transmission Control Protocol (TCP). And TCP ensures the survival of the connection and the dynamic perception of the offline of the working node through a timing heartbeat mechanism.

Specifically, in the embodiment of the present application, an architecture of the adopted network model may be as shown in fig. 4. The protocol processor (protocol processor) mainly solves the problems of TCP half-packet and sticky packet, and transmits a complete protocol packet to the serializer (SerDes). And the serializer is primarily responsible for converting protocol packets from a byte stream to the appropriate data structure. The structure of the protocol packet may be as shown in fig. 5, where the content of the protocol header fixedly occupies one bit, and the following is int-type data with a length of 4 bits, which is used to represent the byte length of the protocol packet, and finally is a body of the protocol, which may be any length.

S102, sequencing all the working nodes according to the descending order of the load scores of all the working nodes maintained locally at present to obtain a node sequencing result.

The load score of each working node is obtained by calculating each working node and fed back to the distributor, and the larger the load of the working node is, the smaller the load score of the working node is.

It should be noted that, in the embodiment of the present application, each working node calculates a calculated load score according to its own load condition, and then feeds back the calculated load score to each distribution node. Each distributor will accordingly maintain the load scores of the various work nodes locally and will be constantly updated.

Since the load scores of the work nodes can reflect the load conditions, when the distributor needs to distribute the current tasks, the work nodes are firstly sorted according to the order of the scores from high to low, so that the tasks with higher scores can be preferentially distributed to the work nodes with lower loads.

Optionally, another embodiment of the present application provides a method for calculating a load score of a work node, as shown in fig. 6, including:

s601, every preset time, the current working node obtains the number of task instructions of the current local waiting queue, the number of task instructions of the running queue and the number of task instructions of the ending queue.

The current working node refers to any one of the working nodes, namely, each working node needs to calculate the load score at regular time.

It should be noted that, in the embodiment of the present application, the load condition of the work node is mainly reflected in consideration of the task backlog of the work node. However, the load condition of the load work node is not only reflected by the task backlog amount, and may also be fed back according to the usage amount of the resource, for example, so the embodiment of the present application is only one of the optional manners, and is not limited to this manner.

Since each work node is processing tasks continuously, that is, the load condition of the work node is changing continuously, in the embodiment of the present application, each work node needs to calculate its current load score periodically.

S602, the current working node calculates the sum of the number of the task instructions of the waiting queue, the number of the task instructions of the running queue and the number of the task instructions of the ending queue to obtain the backlog of the current task instructions.

S603, the current working node calculates the difference value between the preset limit task amount and the current task instruction backlog amount to obtain the current idle task amount.

And S604, dividing the larger value of the current idle task amount and zero by the preset limit task amount by the current working node to obtain the current load score of the current working node.

Since the current idle task amount is less than zero, it indicates that the current task instruction backlog amount is greater than the preset limit task amount, and no matter how much the current task instruction backlog amount is greater than the preset limit task amount, the working node can no longer receive tasks.

The current load score of the current working node is used for feeding back to each distributor to update the load score of the current working node maintained by each distributor, that is, after the current working node obtains the current load score, the current load score is fed back to each distributor to enable each distributor to update the load score.

S103, trying to send the current task to each working node one by one according to the node sequencing result until any working node successfully receives the current task.

In order to achieve load balancing, it is necessary to preferentially distribute the current task to the work node having the highest load score. However, because the current unknown load score of the distributor and the current actual load score of the working node may not be completely synchronized, or due to the existence of other reasons, the working node may not successfully receive the current task, in this embodiment of the present application, the current task is sequentially tried to be sent to each working node according to the sequence of each working node in the node sorting result until any one working node successfully receives the current task. Specifically, a current task is sent to a working node ranked at the first, if the current working node refuses to receive the current task, the current task is sent to a working node ranked at the second, if the current working node refuses to receive the current task, the current task is sent to a working node ranked at the third, and the sending is continuously tried until a node successfully receives the current task.

If no working node can receive the current task finally, the current task can be placed back into the waiting queue first, and the next scheduling and the resending are waited for.

Optionally, when the load score is calculated in the manner shown in fig. 6, correspondingly, in this embodiment of the present application, a specific implementation manner of step S103, as shown in fig. 7, includes:

and S701, determining the current node to be received from the working nodes which are not sent with the current task in the current scheduling in the current node sequencing result and are sequenced at the forefront.

And S702, sending the current task to the current node to be received.

And S703, judging whether a response of refusing to receive fed back by the current node to be received is received.

And if the current task cannot be received due to other reasons, the response of refusing to be received can be fed back.

Specifically, if the response of rejecting reception fed back by the current node to be received is not received, step S704 is executed. If a response of rejecting reception fed back by the current node to be received is received, step S705 is executed.

And S704, determining that the current node to be received successfully receives the current task.

S705, whether a working node to which the current task is not sent in the current scheduling exists in the current node sequencing result is judged.

If it is determined that there is a working node to which the current task has not been sent in the current scheduling in the current node ranking result, the step S701 is executed again. If it is determined that there is no working node to which the current task has not been sent in the current scheduling in the current node ranking result, step S706 is executed.

And S706, putting the current task back into the waiting queue to wait for next scheduling.

Note that, at the time of the next scheduling, the process returns to step S701.

Optionally, in order to implement the dynamic task distribution provided by the embodiment of the present application, a distributed architecture composed of a distributor and a work node is correspondingly provided in the embodiment of the present application, as shown in fig. 8. The distributor specifically comprises a distribution module, a communication model interacting with the working nodes and a Worker cache module. The distribution module is used for distributing according to the sequencing result, and the Worker cache module is used for caching the information of the working nodes and the load scores. And in the work node, a communication module is also included, a load calculation module for calculating the load score is also included, and a corresponding task queue is also included.

Optionally, since after step S103 is executed, after a working node successfully receives the current task, a task is added to the working node that receives the current task, and therefore the load score of the working node changes accordingly, in another embodiment of the present application, after step S103 is executed, an instant update of the load score may be further included, as shown in fig. 9, an instant update method of the load score provided in this embodiment of the present application includes:

s901, receiving the current load score of the current target working node fed back by the current target working node.

The current target working node is a working node which successfully receives the current task; and the current load score of the current target working node is calculated and fed back after the current target working node successfully receives the current task. That is, in the embodiment of the present application, a work node, upon receiving a task, calculates a current load score and feeds back to the dispatcher to which the task is sent.

And S902, updating the load score of the current working node maintained locally by using the current load score of the current working node.

In consideration of the problem of offline of the work node, in another embodiment of the present application, the method may further include:

and receiving the target load score fed back by the node to be offline, and updating the load score of the locally maintained node to be offline by using the target load score of the node to be offline.

The node to be offline refers to a working node to be offline. The target load score is a preset fixed numerical value which is less than zero and is fed back to each distributor when the working node is ready to be offline. Note that, in order to be distinguishable from the calculated load score, the target load score should be a value that cannot be calculated by a given algorithm. For example, when the load score is calculated in the manner shown in fig. 6, the calculated load is not less than zero, so the target load score may be set to a number less than zero, for example: -1.

Specifically, when the work node is to be stopped, it may actively send a heartbeat (heartbeat) of a target load score to each distributor, and may be formally closed after all backlogged tasks are allowed to end.

Alternatively, when the load score is the target load score, it is no longer receiving tasks because it is ready to be offline, so it may optionally not participate in the ranking, or it may be culled from the node ranking results after the ranking, so it may not be attempted to send the current task to it.

The embodiment of the application provides a distributed task processing method, which is applied to any one distributor in a distributed task processing system in Airflow, wherein the distributed task processing system comprises a plurality of distributors and a plurality of working nodes. When receiving the current task pushed by the corresponding scheduler, the distributor sorts the working nodes according to the descending order of the load scores of the working nodes maintained locally at present to obtain a node sorting result, and tries to send the current task to the working nodes one by one according to the node sorting result until any one working node successfully receives the current task. Because the load scores of the working nodes are respectively calculated by the working nodes and fed back to the distributor, and the larger the load of the working nodes is, the smaller the load score of the working nodes is, the distributor sorts the working nodes based on the load scores, and then sequentially tries to distribute tasks to the working nodes according to the sorting result, so that the task can be preferentially distributed to the working nodes with smaller loads, the load balance of the working nodes is realized, the phenomenon that one working node overstocks too many tasks and consumes too much resources is avoided, and the task can be effectively guaranteed to be successfully executed.

Another embodiment of the present application provides a distributed task processing system, which is a subsystem in an Airflow. Referring also to fig. 3, a distributed task processing system provided in an embodiment of the present application includes:

a plurality of distributors and a plurality of worker nodes.

The work nodes are used for calculating the load scores of the work nodes, feeding the load scores back to the distributor and receiving the tasks distributed by the distributor. Wherein, the larger the load of the working node is, the smaller the load score of the working node is.

And the distributor is used for sorting the working nodes according to the descending order of the load scores of the working nodes maintained locally at present when the current task pushed by the corresponding scheduler is received, obtaining a node sorting result, and trying to send the current task to each working node one by one according to the node sorting result until any one working node successfully receives the current task.

Wherein each distributor corresponds to a scheduler.

Optionally, in the distributed task processing system provided in another embodiment of the present application, when the work node performs calculation of its own load score and feeds back the load score to the distributor, the work node is configured to:

and acquiring the number of task instructions of the current local waiting queue, the number of task instructions of the running queue and the number of task instructions of the ending queue at preset time intervals.

And calculating the sum of the number of the task instructions of the waiting queue, the number of the task instructions of the running queue and the number of the task instructions of the ending queue to obtain the backlog of the current task instructions.

And calculating the difference value between the preset limited task quantity and the current task instruction backlog quantity to obtain the current idle task quantity.

And dividing the larger value of the current idle task amount and zero by the preset limit task amount to obtain a current load score, and feeding back the current load score to each distributor so as to update the load score maintained by each distributor.

Optionally, in the distributed task processing system provided in another embodiment of the present application, the distributor performs, according to the node sorting result, an attempt to send the current task to each working node one by one, until any one working node successfully receives the current task, and is configured to:

and in the current node sequencing result, determining the current node to be received by the working node which is sequenced at the forefront in all the working nodes which have not sent the current task to the current node in the current scheduling.

And sending the current task to the current node to be received.

And if the response of refusing to receive fed back by the current node to be received is not received, determining that the current node to be received successfully receives the current task. And feeding back a response of refusing to receive when the current self load score of the current node to be received is zero.

And if a response of refusing to receive fed back by the current node to be received is received, judging whether a working node which does not send the current task to the current node in the current node sequencing result exists or not.

And if the working nodes which do not send the current task to the current node in the current scheduling exist in the current node sequencing result, returning to execute the step of determining the current node to be received by the working node which is sequenced at the forefront in the current node sequencing result and in each working node which does not send the current task to the current node in the current scheduling.

And if the current node sequencing result does not have a working node which does not send the current task to the current node in the current scheduling, the current task is put back into the waiting queue, and the current node sequencing result is returned to execute until next scheduling, and the current node sequencing result is sequenced in each working node which does not send the current task to the current node in the current scheduling, so that the current node to be received is determined by the working node which is sequenced at the forefront.

Optionally, in a distributed task processing system provided in another embodiment of the present application, the distributor is further configured to:

and receiving the current load score of the current target working node fed back by the current target working node. And the current target working node is a working node which successfully receives the current task. And the current load score of the current target working node is calculated and fed back after the current target working node successfully receives the current task.

and receiving the target load score fed back by the node to be offline.

Wherein, the node to be offline refers to the working node to be offline. The target load score is a preset fixed numerical value which is less than zero and is fed back to each distributor when the work node is ready to be offline.

And updating the load score of the locally maintained node to be offline by using the target load score of the node to be offline.

It should be noted that, for the specific working processes of the distributor and the working node provided in the foregoing embodiments of the present application, corresponding steps in the foregoing method embodiments may be referred to accordingly, and details are not described here again.

Another embodiment of the present application provides an electronic device, as shown in fig. 10, including:

a memory 1001 and a processor 1002.

The memory 1001 is used for storing programs.

The processor 1002 is configured to execute a program stored in the memory 1001, and when the program is executed, the program is specifically configured to implement the distributed task processing method provided in any one of the above embodiments.

Another embodiment of the present application provides a computer storage medium for storing a computer program, which when executed, is used to implement the distributed task processing method provided in any one of the above embodiments.

Computer storage media, including permanent and non-permanent, removable and non-removable media, may implement the information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A distributed task processing method is applied to any one distributor in a distributed task processing system in Airflow, wherein the distributed task processing system comprises a plurality of distributors and a plurality of working nodes, and the distributed task processing method comprises the following steps:

2. The method of claim 1, wherein the calculating of the load score of the work node comprises:

the current working node calculates the sum of the number of the task instructions of the waiting queue, the number of the task instructions of the running queue and the number of the task instructions of the ending queue to obtain the backlog of the current task instructions;

3. The method according to claim 2, wherein said attempting to send the current task to each of the working nodes one by one according to the node ranking result until any one of the working nodes successfully receives the current task comprises:

sending the current task to the current node to be received;

if the receiving refusing response fed back by the current node to be received is not received, the current node to be received is determined to successfully receive the current task; when the current self load score of the current node to be received is zero, feeding back a response of refusing to receive;

4. The method according to claim 1, wherein said attempting to send the current task to each of the working nodes one by one according to the node ranking result until any one of the working nodes successfully receives the current task further comprises:

5. The method of claim 2, further comprising:

6. A distributed task processing system, wherein the distributed task processing system is a subsystem in an Airflow, and the distributed task processing system comprises:

a plurality of distributors and a plurality of worker nodes;

7. The distributed task processing system of claim 6, wherein the work nodes, when performing the computation of their own load scores and feeding back to the distributor, are configured to:

acquiring the number of task instructions of a current local waiting queue, the number of task instructions of an operating queue and the number of task instructions of an ending queue at intervals of preset time;

8. The distributed task processing system of claim 7, wherein the dispatcher, when executing the attempt to send the current task to each of the working nodes one by one according to the node ranking result until any one of the working nodes successfully receives the current task, is configured to:

sending the current task to the current node to be received;

9. An electronic device, comprising:

a memory and a processor;

wherein the memory is used for storing programs;

the processor is configured to execute the program, which when executed is specifically configured to implement the distributed task processing method according to any one of claims 1 to 5.

10. A computer storage medium storing a computer program which, when executed, implements the distributed task processing method of any one of claims 1 to 5.