CN111782360B

CN111782360B - Distributed task scheduling method and device

Info

Publication number: CN111782360B
Application number: CN202010594993.5A
Authority: CN
Inventors: 王轶凡; 张楠; 陈灿; 申木川
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2020-06-28
Filing date: 2020-06-28
Publication date: 2023-08-11
Anticipated expiration: 2040-06-28
Also published as: CN111782360A

Abstract

The embodiment of the application provides a distributed task scheduling method and device, wherein the method comprises the following steps: individually accessing a database for storing task configuration information to obtain configuration information of a target task; if the target task is determined to not be a designated server, adding the target task into an asynchronous queue after determining that the target task meets a preset trigger condition, so that one of a plurality of servers in distributed arrangement acquires the target task from the asynchronous queue in a competitive random access mode; and monitoring the server executing the target task in real time, and controlling the target task to resume the normal running state again when the target task is in the abnormal running state. The application can effectively improve the efficiency, reliability and effectiveness of the task distribution process, the task execution process and the exception handling process, thereby comprehensively and effectively improving the efficiency of the whole task scheduling process and improving the robustness and task handling efficiency of the task scheduling system of enterprises.

Description

Distributed task scheduling method and device

Technical Field

The application relates to the technical field of data processing, in particular to a distributed task scheduling method and device.

Background

In most enterprises, a task scheduling system is a very important infrastructure system, and most of the existing task scheduling tools are realized based on a java thread pool and a time slice rotation principle.

Currently, a task scheduling process often designates a server to sequentially execute a plurality of independent tasks, but does not effectively control the task allocation process, the processing timeliness and the abnormal situation. Therefore, in the existing task scheduling process, task acquisition failure is often caused by simultaneous acquisition of tasks by multiple servers, overload operation of a server is caused by too long execution of a single task, or abnormal task execution cannot be found in time, so that the task scheduling process is blocked, for example, a queue is blocked and resources cannot be released. Therefore, in either case, the task processing efficiency in the whole task scheduling process is low.

Disclosure of Invention

Aiming at the problems in the prior art, the application provides the distributed task scheduling method and the distributed task scheduling device, which can effectively improve the efficiency, the reliability and the effectiveness of a task distribution process, a task execution process and an exception handling process, further can comprehensively and effectively improve the efficiency of the whole task scheduling process, and can improve the robustness and the task processing efficiency of a task scheduling system of an enterprise.

In order to solve the technical problems, the application provides the following technical scheme:

in a first aspect, the present application provides a distributed task scheduling method, including:

individually accessing a database for storing task configuration information to obtain configuration information of a target task from the database;

if the target task is determined to not be a designated server according to the configuration information of the target task, adding the target task into an asynchronous queue after determining that the target task meets a preset trigger condition, so that one of a plurality of servers in distributed setting acquires the target task from the asynchronous queue in a competitive random access mode;

and monitoring the server executing the target task in real time, and controlling the target task to resume the normal running state again by applying a preset scheduling mode when the target task is monitored to be in the abnormal running state.

Further, the individually accessing the database for storing task configuration information to obtain the configuration information of the target task from the database includes:

checking whether the database is currently in a locked state or not, and if not, locking the database by using a database lock;

And reading the configuration information of the target task from the database.

Further, the method further comprises the following steps:

if the database is found to be in the locked state currently, checking whether the database is in the locked state currently or not again after a preset time interval.

Further, if it is determined that the target task does not designate a server according to the configuration information of the target task, after determining that the target task meets a preset trigger condition, adding the target task into an asynchronous queue, so that one of the servers in distributed setting obtains the target task from the asynchronous queue in a competitive random access manner, including:

judging whether the target task is a designated server or not from the configuration information of the target task, if not, determining that the resource occupancy rate of the target task is smaller than an occupancy threshold;

judging whether the current task number of the asynchronous queue is equal to or greater than a preset maximum concurrency threshold, if not, adding the target task into the asynchronous queue after determining that the target task meets a preset trigger condition according to the configuration information of the target task, so that one of a plurality of servers in distributed arrangement acquires the target task from the asynchronous queue in a competitive random access mode.

Further, the method further comprises the following steps:

if the target task is judged to be the designated server, determining that the resource occupancy rate of the target task is greater than or equal to an occupancy threshold;

and acquiring the identification of the designated server of the target task from the configuration information of the target task, and adding the target task and the identification of the corresponding designated server into an asynchronous queue so that the designated server of the target task acquires the target task from the asynchronous queue.

Further, after the target task is added to the asynchronous queue, the method further comprises:

modifying the state information of the target task in the asynchronous queue into information for representing that the task is in a state to be processed;

and releasing the database lock of the database.

Further, before the monitoring the server executing the target task in real time, the method further comprises:

when or after a server acquires a target task in the asynchronous queue, modifying the state information of the target task in the asynchronous queue into information for representing that the task is in an executing state;

recording the starting time of the server to execute the target task and the identification of the server;

Correspondingly, the monitoring the server executing the target task in real time, and when the target task is in the abnormal running state, controlling the target task to resume the normal running state by applying a preset scheduling mode comprises the following steps:

if the time for the server to execute the target task exceeds the execution time threshold according to the starting time of the server to execute the target task and the identification of the server, determining that the target task is in an abnormal running state currently;

performing interrupt processing on the current execution process of the target task;

and re-adding the target task into the asynchronous queue, and/or restarting a server executing the target task.

In a second aspect, the present application provides a distributed task scheduling device, including:

the task acquisition module is used for independently accessing a database for storing task configuration information to acquire the configuration information of a target task from the database;

the distributed task execution module is used for adding the target task into an asynchronous queue after determining that the target task meets a preset trigger condition if the target task does not designate a server according to the configuration information of the target task, so that one of a plurality of servers in distributed setting acquires the target task from the asynchronous queue in a competitive random access mode;

The exception handling module is used for monitoring the server executing the target task in real time, and controlling the target task to resume the normal running state again by applying a preset scheduling mode when the target task is monitored to be in the abnormal running state.

Further, the task acquisition module includes:

the locking access unit is used for checking whether the database is currently in a locked state or not, and if not, locking the database by using a database lock;

and the information reading unit is used for reading the configuration information of the target task from the database.

Further, the task acquisition module further includes:

and the repeated query unit is used for checking whether the database is currently in the locked state or not after a preset time interval if the database is checked to be in the locked state currently.

Further, the distributed task execution module is configured to execute the following:

Further, the distributed task execution module is further configured to execute the following:

Further, the method further comprises the following steps:

the first task state modification module is used for modifying the state information of the target task in the asynchronous queue into information for representing that the task is in a state to be processed;

and the unlocking module is used for releasing the database lock of the database.

Further, the method further comprises the following steps:

the second task state modifying module is used for modifying the state information of the target task in the asynchronous queue into information for representing that the task is in an executing state when or after the server acquires the target task in the asynchronous queue;

the data recording module is used for recording the starting time of the server to execute the target task and the identification of the server;

Correspondingly, the exception handling module is configured to execute the following:

In a third aspect, the present application provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the distributed task scheduling method when executing the program.

In a fourth aspect, the present application provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the distributed task scheduling method.

As can be seen from the above technical solutions, the method and apparatus for scheduling distributed tasks provided by the present application, the method includes: individually accessing a database for storing task configuration information to obtain configuration information of a target task from the database; if the target task is determined to not be a designated server according to the configuration information of the target task, adding the target task into an asynchronous queue after determining that the target task meets a preset trigger condition, so that one of a plurality of servers in distributed setting acquires the target task from the asynchronous queue in a competitive random access mode for execution; and monitoring the server executing the target task in real time, and controlling the target task to resume the normal running state again by applying a preset scheduling mode when the target task is monitored to be in the abnormal running state. According to the method and the device, the database for storing the task configuration information is independently accessed, so that the problem of task allocation failure caused by the fact that a plurality of servers access the database to acquire tasks simultaneously can be effectively avoided, the reliability and the effectiveness of task allocation can be effectively improved, and the efficiency of a task scheduling process can be further effectively improved; by dividing tasks into two types of tasks which can be executed by a designated server and can be acquired by the server in a competition manner, the tasks with large data volume can be effectively executed by the designated server, the tasks with small data volume can be executed by an idle server, if the tasks with large data volume cause queue blocking or downtime of the designated server, other servers which are distributed can still normally work, the operation of the whole task scheduling process is not influenced, the tasks with larger consumption resources can not be executed on the same server, the risk of downtime of the server can be effectively reduced, and the efficiency of the task scheduling process can be effectively improved; abnormal tasks can be processed in time by monitoring the abnormal tasks in execution, so that the task execution efficiency and the reliability of the whole task scheduling process are improved; that is, the application can effectively improve the efficiency, reliability and effectiveness of the task distribution process, the task execution process and the exception handling process, thereby comprehensively and effectively improving the efficiency of the whole task scheduling process and the robustness and task handling efficiency of the enterprise task scheduling system.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a flow chart of a distributed task scheduling method in an embodiment of the present application.

Fig. 2 is a schematic diagram of a first specific flow of step 100 in the distributed task scheduling method according to an embodiment of the present application.

Fig. 3 is a second specific flowchart of step 100 in the distributed task scheduling method according to the embodiment of the present application.

Fig. 4 is a schematic diagram of a first specific flow of step 200 in the distributed task scheduling method according to an embodiment of the present application.

Fig. 5 is a second specific flowchart of step 200 in the distributed task scheduling method according to the embodiment of the present application.

Fig. 6 is a flow chart of a distributed task scheduling method including step 010 and step 020 in an embodiment of the present application.

Fig. 7 is a flow chart of a distributed task scheduling method including steps 030, 040 and 310 to 330 according to an embodiment of the application.

Fig. 8 is a schematic diagram of a first configuration of a distributed task scheduling device in an embodiment of the present application.

Fig. 9 is a schematic diagram of a first configuration of the task acquisition module 10 in the distributed task scheduling device according to the embodiment of the present application.

Fig. 10 is a schematic diagram of a second configuration of the task acquisition module 10 in the distributed task scheduling device according to the embodiment of the present application.

Fig. 11 is a schematic diagram of a second structure of a distributed task scheduling device in an embodiment of the present application.

Fig. 12 is a schematic diagram of a third configuration of a distributed task scheduling device in an embodiment of the present application.

FIG. 13 is a schematic illustration of the execution of a distributed task scheduling system based on bypass asynchronous queue monitoring provided by an example application of the present application.

Fig. 14 is a schematic diagram of a data processing procedure of the task computing module provided by the application example of the present application.

FIG. 15 is a diagram of the data processing process for each atomic task in an asynchronous queue provided by an embodiment of the present application.

FIG. 16 is a flow chart of a task asynchronous execution and bypass monitoring service provided by an embodiment of the present application.

Fig. 17 is a schematic structural diagram of an electronic device in an embodiment of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

In consideration of the problem of low task processing efficiency in the whole task scheduling process existing in the existing task scheduling process, the embodiment of the application respectively provides a distributed task scheduling method, a distributed task scheduling device, electronic equipment and a computer readable storage medium, wherein a database for storing task configuration information is independently accessed to acquire the configuration information of a target task from the database; if the target task is determined to not be a designated server according to the configuration information of the target task, after the target task is determined to meet a preset trigger condition, adding the target task into an asynchronous queue, so that one of a plurality of servers arranged in a distributed mode can acquire the target task from the asynchronous queue in a competitive random access mode, dividing the task into two modes of executing the designated server and competing the servers to acquire the task, effectively enabling the task with large data quantity to be executed by the designated server, executing the task with small data quantity by an idle server, and if the task with large data quantity causes the queue of the designated server to be blocked or down, enabling the other servers arranged in a distributed mode to still work normally, so that the operation of the whole task scheduling process is not influenced, and the task with large consumption resources can not be executed on the same server, and the risk of down of the servers can be effectively reduced, so that the efficiency of the task scheduling process can be effectively improved; the method comprises the steps of monitoring a server executing the target task in real time, and when the target task is monitored to be in an abnormal running state, controlling the target task to resume the normal running state by applying a preset scheduling mode, and timely processing the abnormal task by performing abnormal monitoring on the executing task, thereby improving the task execution efficiency and the reliability of the whole task scheduling process; that is, the application can effectively improve the efficiency, reliability and effectiveness of the task distribution process, the task execution process and the exception handling process, thereby comprehensively and effectively improving the efficiency of the whole task scheduling process and the robustness and task handling efficiency of the enterprise task scheduling system.

The following description will be made with reference to various examples.

In order to avoid the problem of low task scheduling efficiency caused by task acquisition failure due to the fact that a plurality of servers acquire tasks simultaneously, overload operation of the servers due to the fact that a single task is executed for too long, incapability of being found in time, and the like, and the task scheduling process is blocked, the application provides an embodiment of a distributed task scheduling method, which specifically comprises the following steps:

step 100: a database for storing task configuration information is individually accessed to obtain configuration information of a target task from the database.

In step 100, individual access means that only one server is allowed to access the database for the current period of time.

It will be appreciated that in one or more embodiments of the present application, the task refers to a basic unit of work to be performed by a computer in a multiprogramming or multiprocessing environment, which is one or more sequences of instructions processed by a control program, and may specifically be a data file. Each task represents a single thread that is executed by a program or a group of programs. The first program executed under each task is the main program, and the others are the auxiliary programs.

Step 200: if the target task is determined to not be a designated server according to the configuration information of the target task, after the target task is determined to meet the preset trigger condition, the target task is added into an asynchronous queue, so that one of the servers in distributed setting can acquire the target task from the asynchronous queue in a competitive random access mode.

It will be appreciated that the database is primarily used to store various configuration information for each task, including the identity of the task, the task trigger conditions, the number of concurrences allowed, the start and end times of task execution, the completion of each instance and the server address that needs to be executed, and that the database provides database locks to the currently accessed servers.

And then, the server which acquires the target task from the asynchronous queue executes the target task.

In step 200, a specific determining process of whether the target task meets the preset triggering condition is: and reading task configuration in configuration information of the target task, and acquiring task number, last completion time, trigger condition and maximum concurrency number information. Inquiring the number of the tasks being executed in the current asynchronous queue through the task number, if the number is larger than the maximum concurrency number, not continuing to judge, otherwise, continuing to judge whether the trigger condition is met. And judging whether the task meets the condition of needing to run or not by comparing the current time, the trigger time and the last successful completion time according to the trigger conditions (such as interval type, time point type, multiple times per day, appointed date type and the like) of the task, and if so, grabbing the task and putting the task into an asynchronous queue to wait for processing.

The random access mode refers to a process from when a user sends a random access preamble to when attempting to access a network until a basic signaling connection is established between the user and the network. Random access is classified into competitive random access and non-competitive random access. The competitive random access refers to a random access procedure initiated by a server randomly selecting a preamble. In the contention-based random access mode, an asynchronous queue is a resource pool for a server, and different servers can use the same resource, resulting in resource contention. And based on a non-competitive random access mode, specific resources are reserved and allocated to a certain server at a certain moment.

Step 300: and monitoring the server executing the target task in real time, and controlling the target task to resume the normal running state again by applying a preset scheduling mode when the target task is monitored to be in the abnormal running state.

As can be seen from the above description, the distributed task scheduling method provided by the embodiment of the present application can effectively avoid the problem of task allocation failure caused by that multiple servers access the database to obtain tasks simultaneously by individually accessing the database for storing task configuration information, and can effectively improve the reliability and effectiveness of task allocation, thereby effectively improving the efficiency of the task scheduling process; by dividing tasks into two types of tasks which can be executed by a designated server and can be acquired by the server in a competition manner, the tasks with large data volume can be effectively executed by the designated server, the tasks with small data volume can be executed by an idle server, if the tasks with large data volume cause queue blocking or downtime of the designated server, other servers which are distributed can still normally work, the operation of the whole task scheduling process is not influenced, the tasks with larger consumption resources can not be executed on the same server, the risk of downtime of the server can be effectively reduced, and the efficiency of the task scheduling process can be effectively improved; abnormal tasks can be processed in time by monitoring the abnormal tasks in execution, so that the task execution efficiency and the reliability of the whole task scheduling process are improved; that is, the application can effectively improve the efficiency, reliability and effectiveness of the task distribution process, the task execution process and the exception handling process, thereby comprehensively and effectively improving the efficiency of the whole task scheduling process and the robustness and task handling efficiency of the enterprise task scheduling system.

In order to implement the process of locking and accessing when the database is unoccupied in the individual access, in one embodiment of the distributed task scheduling method provided by the present application, referring to fig. 2, step 100 in the distributed task scheduling method specifically includes the following:

step 110: checking whether the database is currently in a locked state, and if not, executing step 120.

Step 120: and locking the database by using a database lock.

Step 130: and reading the configuration information of the target task from the database.

As can be seen from the above description, the distributed task scheduling method provided by the embodiment of the present application can further avoid the problem of task allocation failure caused by that a plurality of servers access a database to acquire tasks at the same time, and can effectively improve the reliability and effectiveness of task allocation, thereby effectively improving the efficiency of the task scheduling process.

In order to implement the re-query process when the database is occupied in the separate access, in one embodiment of the distributed task scheduling method provided by the present application, referring to fig. 3, step 110 in the distributed task scheduling method further specifically includes the following:

If the database is currently in a locked state as seen from a view of step 110, step 140 is performed.

Step 140: and checking whether the database is currently in a locked state or not again after a preset time interval.

As can be seen from the above description, the distributed task scheduling method provided by the embodiment of the present application provides a subsequent processing manner when accessing a database fails, which can effectively ensure the reliability and effectiveness of accessing the database, further improve the reliability and effectiveness of task allocation, and further effectively improve the efficiency of the task scheduling process.

In order to randomly acquire the task joining queues of the unspecified servers, in one embodiment of the distributed task scheduling method provided by the present application, referring to fig. 4, step 200 in the distributed task scheduling method specifically includes the following:

step 210: judging whether the target task is a designated server or not from the configuration information of the target task, and if not, determining that the resource occupancy rate of the target task is smaller than an occupancy threshold.

Step 220: and judging whether the current task number of the asynchronous queue is equal to or greater than a preset maximum concurrency threshold, and if not, executing step 230.

Step 230: after the target task is determined to meet the preset trigger condition according to the configuration information of the target task, the target task is added into an asynchronous queue, so that one of the servers in distributed setting can acquire the target task from the asynchronous queue in a competitive random access mode.

As can be seen from the above description, the distributed task scheduling method provided by the embodiment of the present application can enable the task with small data volume to be executed by the idle server, and if the task with large data volume causes the queue of the designated server to be blocked or down, other servers with distributed settings can still work normally, so that the operation of the whole task scheduling process is not affected.

In order to obtain the large task joining queue of the designated server by the designated server, in one embodiment of the distributed task scheduling method provided by the present application, referring to fig. 5, step 220 in the distributed task scheduling method further specifically includes the following:

if it is determined in step 220 that the server is specified by the target task, it is determined that the resource occupancy rate of the target task is greater than or equal to the occupancy threshold, and step 240 is performed.

Step 240: and acquiring the identification of the designated server of the target task from the configuration information of the target task, and adding the target task and the identification of the corresponding designated server into an asynchronous queue so that the designated server of the target task acquires the target task from the asynchronous queue.

As can be seen from the above description, the distributed task scheduling method provided by the embodiment of the present application can enable a task with large data volume to be executed by a designated server, and if the task with large data volume causes the queue of the designated server to be blocked or down, other servers with distributed settings can still work normally, without affecting the operation of the whole task scheduling process.

In order to unlock the database, in one embodiment of the distributed task scheduling method provided by the present application, referring to fig. 6, after step 200 and before step 300 in the distributed task scheduling method, the following are specifically included:

step 010: and modifying the state information of the target task in the asynchronous queue into information for representing that the task is in a pending state.

Step 020: and releasing the database lock of the database.

From the above description, it can be seen that the distributed task scheduling method provided by the embodiment of the present application can effectively perform timely unlocking processing on the database, so that other servers can access the database timely, further, the reliability and effectiveness of accessing the database can be effectively ensured, and further, the reliability and effectiveness of task allocation can be further improved.

In order to process abnormal tasks, in one embodiment of the distributed task scheduling method provided by the present application, referring to fig. 7, after step 020 and before step 300 in the distributed task scheduling method, the following is specifically included:

step 030: and when or after the server acquires the target task in the asynchronous queue, modifying the state information of the target task in the asynchronous queue into information for representing that the task is in an executing state.

Step 040: recording the starting time of the server to execute the target task and the identification of the server.

Correspondingly, the step 300 specifically includes the following:

step 310: and if the time for the server to execute the target task exceeds the execution time threshold according to the starting time for the server to execute the target task and the identification of the server, determining that the target task is in an abnormal running state currently.

Step 320: and carrying out interrupt processing on the current execution process of the target task.

Step 330: and re-adding the target task into the asynchronous queue, and/or restarting a server executing the target task.

From the above description, it can be seen that the distributed task scheduling method provided by the embodiment of the present application can process abnormal tasks in time, and improve the efficiency of task execution and the reliability of the whole task scheduling process.

In order to solve the problem of low task scheduling efficiency caused by task acquisition failure due to simultaneous task acquisition of a plurality of servers, overload operation of a server due to too long execution of a single task, failure in timely discovery of task execution, and the like, and blocked task scheduling process, the application provides an embodiment of a distributed task scheduling device for executing all or part of the content in the distributed task scheduling method, referring to fig. 8, wherein the distributed task scheduling device specifically comprises the following contents:

the task acquisition module 10 is used for individually accessing a database for storing task configuration information to acquire the configuration information of the target task from the database.

And the distributed task execution module 20 is configured to, if it is determined that the target task does not designate a server according to the configuration information of the target task, add the target task to an asynchronous queue after determining that the target task meets a preset trigger condition, so that one of the servers in the distributed arrangement acquires the target task from the asynchronous queue in a competitive random access manner.

The exception handling module 30 is configured to monitor, in real time, a server executing the target task, and when it is monitored that the target task is in an abnormal running state, control the target task to resume a normal running state by applying a preset scheduling mode.

As can be seen from the above description, the distributed task scheduling device provided by the embodiment of the present application, by individually accessing the database for storing the task configuration information, can effectively avoid the problem of task allocation failure caused by that multiple servers access the database to acquire tasks at the same time, can effectively improve the reliability and effectiveness of task allocation, and further can effectively improve the efficiency of the task scheduling process; by dividing tasks into two types of tasks which can be executed by a designated server and can be acquired by the server in a competition manner, the tasks with large data volume can be effectively executed by the designated server, the tasks with small data volume can be executed by an idle server, if the tasks with large data volume cause queue blocking or downtime of the designated server, other servers which are distributed can still normally work, the operation of the whole task scheduling process is not influenced, the tasks with larger consumption resources can not be executed on the same server, the risk of downtime of the server can be effectively reduced, and the efficiency of the task scheduling process can be effectively improved; abnormal tasks can be processed in time by monitoring the abnormal tasks in execution, so that the task execution efficiency and the reliability of the whole task scheduling process are improved; that is, the application can effectively improve the efficiency, reliability and effectiveness of the task distribution process, the task execution process and the exception handling process, thereby comprehensively and effectively improving the efficiency of the whole task scheduling process and the robustness and task handling efficiency of the enterprise task scheduling system.

In order to implement the process of locking and accessing when the database is unoccupied in the individual access, in one embodiment of the distributed task scheduling device provided by the present application, referring to fig. 9, the task obtaining module 10 in the distributed task scheduling device specifically includes the following contents:

and the locking access unit 11 is used for checking whether the database is currently in a locked state, and if not, locking the database by using a database lock.

An information reading unit 12, configured to read configuration information of the target task from the database.

As can be seen from the above description, the distributed task scheduling device provided by the embodiment of the present application can further avoid the problem of task allocation failure caused by that a plurality of servers access a database to acquire tasks at the same time, and can effectively improve the reliability and effectiveness of task allocation, thereby effectively improving the efficiency of the task scheduling process.

In order to implement the re-query process when the database is occupied in the separate access, in one embodiment of the distributed task scheduling device provided by the present application, referring to fig. 10, the task obtaining module 10 in the distributed task scheduling device further specifically includes the following contents:

And the repeated query unit 13 is configured to check whether the database is currently in the locked state again after a preset time interval if the database is known to be currently in the locked state through checking.

As can be seen from the above description, the distributed task scheduling device provided by the embodiment of the present application provides a subsequent processing manner when accessing a database fails, which can effectively ensure the reliability and effectiveness of accessing the database, further improve the reliability and effectiveness of task allocation, and further effectively improve the efficiency of the task scheduling process.

In order to randomly acquire the small task joining queues of the unspecified servers, in one embodiment of the distributed task scheduling device provided by the present application, the distributed task execution module 20 in the distributed task scheduling device is specifically configured to execute the following:

As can be seen from the above description, the distributed task scheduling device provided by the embodiment of the present application can enable a task with small data volume to be executed by an idle server, and if a task with large data volume causes a queue of a designated server to be blocked or down, other servers in distributed arrangement can still work normally, so that the operation of the whole task scheduling process is not affected.

In order to obtain the large task joining queue of the designated server by the designated server, in one embodiment of the distributed task scheduling device provided by the present application, the distributed task execution module 20 in the distributed task scheduling device is further specifically configured to execute the following:

As can be seen from the above description, the distributed task scheduling device provided by the embodiment of the present application can enable a task with large data volume to be executed by a designated server, and if the task with large data volume causes the queue of the designated server to be blocked or down, other servers with distributed settings can still work normally, without affecting the operation of the whole task scheduling process.

In order to unlock the database, in one embodiment of the distributed task scheduling device provided by the present application, referring to fig. 11, the distributed task scheduling device further specifically includes the following contents:

a first task state modification module 01, configured to modify state information of a target task in the asynchronous queue into information for indicating that the task is in a state to be processed;

and the unlocking module 02 is used for releasing the database lock of the database.

As can be seen from the above description, the distributed task scheduling device provided by the embodiment of the present application can effectively perform timely unlocking processing on the database, so that other servers can access the database timely, further, the reliability and the effectiveness of accessing the database can be effectively ensured, and further, the reliability and the effectiveness of task allocation can be further improved.

In order to process abnormal tasks, in one embodiment of the distributed task scheduling device provided by the present application, referring to fig. 12, the distributed task scheduling device further specifically includes the following contents:

a second task state modifying module 03, configured to modify, when or after the server obtains the target task in the asynchronous queue, state information of the target task in the asynchronous queue into information for indicating that the task is in an executing state;

a data recording module 04, configured to record a start time of the server to execute the target task and an identifier of the server;

correspondingly, the exception handling module 30 is specifically configured to perform the following:

As can be seen from the above description, the distributed task scheduling device provided by the embodiment of the present application can process abnormal tasks in time, thereby improving the efficiency of task execution and the reliability of the whole task scheduling process.

In order to further explain the scheme, the application also provides a specific application example for realizing the distributed task scheduling method by applying the distributed task scheduling system. The method has the advantages that the normal scheduling of the tasks is guaranteed, the execution state of the tasks is monitored in real time, the situations that the task queue is interrupted or resources cannot be released and the like due to overlong task execution time are avoided, abnormal tasks are timely interrupted and early-warned, automatic processing or manual intervention is carried out, and system resources are recovered.

Referring to fig. 13, a distributed task scheduling system based on bypass asynchronous queue monitoring includes a database module, a task calculation module, and an asynchronous execution and monitoring module. The database module is mainly used for storing a data source, a task configuration table and a parameter table of the task scheduling system, wherein the task configuration table comprises task trigger conditions, allowable concurrency numbers, starting and ending time of task execution, each time of completion conditions and a server address required to be executed, and providing database locks. The task computing module is used for extracting tasks, establishing task queues, concurrency control and exception handling. And (3) circularly traversing the database configuration table to obtain tasks to be processed, adding the tasks into an asynchronous queue, wherein the asynchronous queue is formed with tasks 1 to n, and n is a positive integer greater than 2. Multiple servers (machines 1-N, N being a positive integer greater than 2) compete for tasks in the task queue, asynchronously execute tasks using an asynchronous execution and monitoring module, monitor execution, and update a task parameter table. Recording an execution result, performing automatic/manual intervention on the abnormal task, and re-joining the abnormal task into a task queue to wait for execution.

Referring to fig. 14, the data processing procedure of the task computing module specifically includes the following:

Step 201: and acquiring the database lock, and if the database lock is idle, locking to start a calculation task. Namely: and acquiring the database lock from the database, namely, the other servers are calculating the task to be executed, and checking the database lock again after waiting for three seconds, so that only one server is ensured to calculate the task at the same time. And if the current lock is idle, locking the database and continuing to execute.

Step 202: and reading the task being executed in the asynchronous queue, and judging whether the maximum concurrency number allowed by the task is exceeded. Namely: and reading task configuration in the database, acquiring task numbers, last completion time, trigger conditions and maximum concurrence number information. Inquiring the number of the tasks being executed in the current asynchronous queue through the task number, if the number is larger than the maximum concurrency number, not continuing to judge, otherwise, continuing to judge whether the trigger condition is met.

Step 203: and acquiring the configuration of each task from the database, comparing the trigger conditions, the last time of completion and the current time. And grabbing out tasks meeting the conditions and putting the tasks into an asynchronous queue. Namely: and judging whether the task meets the condition of needing to run or not by comparing the current time, the trigger time and the last successful completion time according to the trigger conditions (such as interval type, time point type, multiple times per day, appointed date type and the like) of the task, and if so, grabbing the task and putting the task into an asynchronous queue to wait for processing.

Step 204: releasing the database lock, recording the execution log and entering the next cycle. Namely: after completion of the calculation, the task to be processed is recorded in the task processing list log table, and the status is set to "to be processed". Releasing the database lock, ready to enter a calculation.

Referring to fig. 15, the data processing process of each atomic task in the asynchronous queue specifically includes the following:

step 301: beforecall preparation stage, the task state is set as "executing". Namely: in the preparation stage before task execution, the task state value in the task processing list log table is set as 'executing', and the starting time and the executing server IP are recorded.

Step 302: and the Call execution stage is used for executing tasks, acquiring an execution result and monitoring the execution time. Namely: executing task content, waiting for a return result, capturing abnormal conditions (such as internal error report of the task, overlong execution time and the like) in the execution process, and transmitting the execution result to the step 303.

Step 303: and (5) an Aftercall execution completion stage, and recording the completion time and task execution condition. Namely: recording the task execution completion condition, including execution completion time, execution result (success, failure) and error information.

Referring to fig. 16, the task asynchronous execution and bypass monitoring business process specifically includes the following:

step 401: and acquiring the task to be executed in the task queue. Namely: and each server competes in the task queue for acquiring the task, and the database locking processing is adopted when the task is acquired, so that the same task is prevented from being acquired and repeatedly executed by a plurality of servers at the same time.

Step 402: and submitting the task to a thread pool for execution. Namely: the acquired tasks are packaged into a Callable interface and submitted to a thread pool for ready execution, if the number of threads in the thread pool is greater than a preset threshold (5) of the system, the tasks are queued for execution, and the situation that the task is blocked due to too high resource occupation of a server is avoided

Step 403: and presetting task timeout time, and asynchronously waiting for a processing result. Namely: the timeout time is preset, the execution of the threads is asynchronously executed and monitored, the next cycle is started in step 401, the task execution is grabbed, and the problem that the grabbing efficiency of the task is affected due to the fact that an execution result is waited is avoided, so that the task queue is overlong is avoided. Asynchronous bypass processing is performed.

Step 404: and asynchronously executing, waiting for a processing result, and carrying out repair processing manually/automatically aiming at abnormal condition early warning. Namely: and acquiring an asynchronous execution result (Future) of the task, if the execution time of the task exceeds the preset time (24 hours in the system), automatically interrupting the thread, releasing the memory resource, freeing the thread pool, throwing out the overtime error report to step 303, recording a task processing log, and if the task is interrupted due to the internal abnormality of the task, reporting and recording the internal error. The system can judge whether the task can be repeatedly executed or not according to the parameter table, if so, the system automatically adds the failed task into the thread queue to wait for re-execution. If the task errors in the same category reach a certain threshold or the task cannot be repeatedly executed, an early warning is sent out, manual intervention is carried out, a manual interface is provided by the system, and the task can be manually added into a thread queue to be re-executed after the errors are checked.

In a specific example, the application example of the application is mainly used for executing asynchronous tasks such as data file loading and file asynchronous generation, and a distributed technology is adopted, so that a plurality of servers compete for reading a database lock to acquire tasks needing to be executed asynchronously and join a thread team. For each task, the executing server can be flexibly configured through parameter table configuration. For example, in production, four servers of machine 1, machine 2, machine 3 and machine 4 compete to obtain task execution in the configuration table, task a may access a large number of data file loads every day, there is a risk that the servers execute for a long time or downtime, task B needs to generate a large number of service files every day and send out, and there is such a risk. We can specify via the configuration table that task A can only be obtained by machine 1 and task B can only be obtained by machine 2. Other less resource consuming tasks are acquired by machine 1, machine 2, machine 3, and machine 4 in competition at random. If the task A causes the queue of the machine 1 to be blocked or down, other three machines can still work normally, and the operation of the whole task scheduling system is not affected. The task A and the task B which consume larger resources cannot be executed on the same server, and the risk of downtime of the server is reduced. If the task A causes the machine 1 to send out an interruption or overtime early warning, the method such as reloading the task or restarting the server can remove faults and continue to operate normally. The system adopts a distributed task scheduling mechanism, and is safer and more reliable.

The application example designs a task scheduling tool and provides a distributed task scheduling system for bypass asynchronous queue monitoring based on a task degree scheduling principle of time slice rotation and a task queue asynchronous execution mechanism, fully utilizes server resources to calculate and schedule tasks, asynchronously monitors task execution conditions, timely processes abnormal tasks, solves the problems that resources cannot be released due to certain task abnormality on a single server in the task scheduling process, and has low scheduling tool operation efficiency, and improves the robustness and task processing efficiency of the whole system.

In order to solve the problem of low task scheduling efficiency caused by task scheduling process blocking due to reasons such as task acquisition failure caused by simultaneous task acquisition of a plurality of servers, overload operation of a server caused by too long execution of a single task, incapability of being found in time, and the like, in terms of hardware level, the application provides an embodiment of an electronic device for realizing all or part of contents in the distributed task scheduling method, wherein the electronic device specifically comprises the following contents:

fig. 17 is a schematic block diagram of a system configuration of an electronic device 9600 according to an embodiment of the present application. As shown in fig. 17, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. Notably, this fig. 17 is exemplary; other types of structures may also be used in addition to or in place of the structures to implement telecommunications functions or other functions.

In one embodiment, the distributed task scheduling functionality may be integrated into the central processor. Wherein the central processor may be configured to control:

As can be seen from the above description, by individually accessing the database for storing the task configuration information, the electronic device provided by the embodiment of the present application can effectively avoid the problem of task allocation failure caused by that multiple servers access the database to acquire tasks at the same time, and can effectively improve the reliability and effectiveness of task allocation, thereby effectively improving the efficiency of the task scheduling process; by dividing tasks into two types of tasks which can be executed by a designated server and can be acquired by the server in a competition manner, the tasks with large data volume can be effectively executed by the designated server, the tasks with small data volume can be executed by an idle server, if the tasks with large data volume cause queue blocking or downtime of the designated server, other servers which are distributed can still normally work, the operation of the whole task scheduling process is not influenced, the tasks with larger consumption resources can not be executed on the same server, the risk of downtime of the server can be effectively reduced, and the efficiency of the task scheduling process can be effectively improved; abnormal tasks can be processed in time by monitoring the abnormal tasks in execution, so that the task execution efficiency and the reliability of the whole task scheduling process are improved; that is, the application can effectively improve the efficiency, reliability and effectiveness of the task distribution process, the task execution process and the exception handling process, thereby comprehensively and effectively improving the efficiency of the whole task scheduling process and the robustness and task handling efficiency of the enterprise task scheduling system.

In another embodiment, the distributed task scheduling device may be configured separately from the central processor 9100, for example, the distributed task scheduling device may be configured as a chip connected to the central processor 9100, and the distributed task scheduling function is implemented by control of the central processor.

As shown in fig. 17, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is noted that the electronic device 9600 need not include all of the components shown in fig. 17; in addition, the electronic device 9600 may further include components not shown in fig. 17, and reference may be made to the related art.

As shown in fig. 17, the central processor 9100, sometimes also referred to as a controller or operational control, may include a microprocessor or other processor device and/or logic device, which central processor 9100 receives inputs and controls the operation of the various components of the electronic device 9600.

The memory 9140 may be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information about failure may be stored, and a program for executing the information may be stored. And the central processor 9100 can execute the program stored in the memory 9140 to realize information storage or processing, and the like.

The input unit 9120 provides input to the central processor 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used for displaying display objects such as images and characters. The display may be, for example, but not limited to, an LCD display.

The memory 9140 may be a solid state memory such as Read Only Memory (ROM), random Access Memory (RAM), SIM card, etc. But also a memory which holds information even when powered down, can be selectively erased and provided with further data, an example of which is sometimes referred to as EPROM or the like. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application/function storage portion 9142, the application/function storage portion 9142 storing application programs and function programs or a flow for executing operations of the electronic device 9600 by the central processor 9100.

The memory 9140 may also include a data store 9143, the data store 9143 for storing data, such as contacts, digital data, pictures, sounds, and/or any other data used by an electronic device. The driver storage portion 9144 of the memory 9140 may include various drivers of the electronic device for communication functions and/or for performing other functions of the electronic device (e.g., messaging applications, address book applications, etc.).

The communication module 9110 is a transmitter/receiver 9110 that transmits and receives signals via an antenna 9111. A communication module (transmitter/receiver) 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, as in the case of conventional mobile communication terminals.

Based on different communication technologies, a plurality of communication modules 9110, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, etc., may be provided in the same electronic device. The communication module (transmitter/receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and to receive audio input from the microphone 9132 to implement usual telecommunications functions. The audio processor 9130 can include any suitable buffers, decoders, amplifiers and so forth. In addition, the audio processor 9130 is also coupled to the central processor 9100 so that sound can be recorded locally through the microphone 9132 and sound stored locally can be played through the speaker 9131.

An embodiment of the present application also provides a computer readable storage medium capable of implementing all steps in the distributed task scheduling method in the above embodiment, where the computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements all steps in the distributed task scheduling method in which an execution subject in the above embodiment is a server or a client, for example, the processor implements the following steps when executing the computer program:

As can be seen from the foregoing description, the computer readable storage medium provided by the embodiments of the present application can effectively avoid the problem of task allocation failure caused by that multiple servers access the database to obtain tasks at the same time by individually accessing the database for storing task configuration information, can effectively improve the reliability and effectiveness of task allocation, and further can effectively improve the efficiency of the task scheduling process; by dividing tasks into two types of tasks which can be executed by a designated server and can be acquired by the server in a competition manner, the tasks with large data volume can be effectively executed by the designated server, the tasks with small data volume can be executed by an idle server, if the tasks with large data volume cause queue blocking or downtime of the designated server, other servers which are distributed can still normally work, the operation of the whole task scheduling process is not influenced, the tasks with larger consumption resources can not be executed on the same server, the risk of downtime of the server can be effectively reduced, and the efficiency of the task scheduling process can be effectively improved; abnormal tasks can be processed in time by monitoring the abnormal tasks in execution, so that the task execution efficiency and the reliability of the whole task scheduling process are improved; that is, the application can effectively improve the efficiency, reliability and effectiveness of the task distribution process, the task execution process and the exception handling process, thereby comprehensively and effectively improving the efficiency of the whole task scheduling process and the robustness and task handling efficiency of the enterprise task scheduling system.

It will be apparent to those skilled in the art that embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The principles and embodiments of the present invention have been described in detail with reference to specific examples, which are provided to facilitate understanding of the method and core ideas of the present invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.

Claims

1. A distributed task scheduling method, comprising:

judging whether the target task is a designated server or not from the configuration information of the target task, if not, determining that the resource occupancy rate of the target task is smaller than an occupancy threshold; judging whether the current task number of the asynchronous queue is equal to or greater than a preset maximum concurrency threshold, if not, adding the target task into the asynchronous queue after determining that the target task meets a preset trigger condition according to the configuration information of the target task, so that one of a plurality of servers in distributed arrangement acquires the target task from the asynchronous queue in a competitive random access mode;

if the target task is judged to be the designated server, determining that the resource occupancy rate of the target task is greater than or equal to an occupancy threshold; acquiring the identification of the designated server of the target task from the configuration information of the target task, and adding the target task and the identification of the corresponding designated server into an asynchronous queue so that the designated server of the target task acquires the target task from the asynchronous queue;

2. A distributed task scheduling method according to claim 1, wherein the individually accessing a database for storing task configuration information to obtain configuration information of a target task from the database includes:

and reading the configuration information of the target task from the database.

3. The distributed task scheduling method of claim 2, further comprising:

4. A method of distributed task scheduling according to claim 1, further comprising, after said adding the target task to an asynchronous queue:

And releasing the database lock of the database.

5. A distributed task scheduling method according to claim 1, further comprising, prior to said monitoring in real-time of the server that is executing the target task:

6. A distributed task scheduling device, comprising:

the distributed task execution module is used for judging whether the target task is a designated server or not from the configuration information of the target task, if not, determining that the resource occupancy rate of the target task is smaller than an occupancy threshold; judging whether the current task number of the asynchronous queue is equal to or greater than a preset maximum concurrency threshold, if not, adding the target task into the asynchronous queue after determining that the target task meets a preset trigger condition according to the configuration information of the target task, so that one of a plurality of servers in distributed arrangement acquires the target task from the asynchronous queue in a competitive random access mode;

7. A distributed task scheduling apparatus according to claim 6, wherein the task acquisition module includes:

8. The distributed task scheduling device of claim 7, wherein the task acquisition module further comprises:

9. A distributed task scheduling apparatus as defined in claim 6, further comprising:

10. A distributed task scheduling apparatus as defined in claim 6, further comprising:

11. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the distributed task scheduling method of any one of claims 1 to 5 when the program is executed by the processor.

12. A computer readable storage medium having stored thereon a computer program, which when executed by a processor implements the distributed task scheduling method of any one of claims 1 to 5.