CN112231080A

CN112231080A - Task scheduling method and device based on multiple hash rings with different precisions

Info

Publication number: CN112231080A
Application number: CN202011031176.5A
Authority: CN
Inventors: 顾海; 王�义; 聂磊
Original assignee: Wuhan Wuyi Yuntong Network Technology Co ltd
Current assignee: Wuhan Wuyi Yuntong Network Technology Co ltd
Priority date: 2020-09-27
Filing date: 2020-09-27
Publication date: 2021-01-15
Anticipated expiration: 2040-09-27
Also published as: CN112231080B

Abstract

The invention provides a task scheduling method and a device based on a plurality of hash rings with different precisions, wherein the method comprises the following steps: after receiving the task, the task receiver writes the task number and the task ending time into a local cache, and writes the execution frequency and the execution logic into a buffer queue; a timing pulling thread pulls a task from a buffer queue at a fixed frequency, a matched hash ring timer is selected according to the execution frequency of the task, and the hash ring timer puts the task on a corresponding slot position on a corresponding hash ring memory; and each Hash ring timer rotates the execution pointer on the corresponding Hash ring memory at a specified frequency, and takes out the task of which the execution time has reached in the slot position pointed by the execution pointer to execute. According to the scheme, a large number of tasks can be uniformly distributed in the Hash ring memory, so that the accumulation of execution time of a large number of tasks is avoided, load balance is realized, and a large number of tasks can be efficiently scheduled; and a plurality of Hash ring timers are arranged, so that tasks with different execution frequencies can meet the precision requirement.

Description

Task scheduling method and device based on multiple hash rings with different precisions

[ technical field ] A method for producing a semiconductor device

The invention relates to the technical field of information systems, in particular to a task scheduling method and device based on a plurality of hash rings with different precisions.

[ background of the invention ]

At present, in many application scenes, a large batch of tasks need to be scheduled for many times. For example, in medium and long haul transportation, a Geographic Information System (GIS) platform needs to accurately pull a truck track from a national Global Positioning System (GPS) big data platform to monitor and conduct behavioral analysis of the track of the truck during loading and unloading. Generally, vehicles needing to be monitored by the GIS platform are in hundreds of thousands of levels, monitoring frequency is different according to the transportation distance of the vehicles, and track pulling actions to be executed per second can reach thousands of times on average. In the task scheduling of this magnitude, if the current conventional means is adopted, a large number of tasks are stored in the database, and then the database is scanned at a high frequency so as to pull the tasks that have reached the execution time to execute, a great reading and writing pressure is undoubtedly caused on the database, and efficient scheduling of the large number of tasks is difficult to realize.

In view of the above, it is an urgent problem in the art to overcome the above-mentioned drawbacks of the prior art.

[ summary of the invention ]

In the current application scenario of scheduling a large batch of tasks, all tasks are usually stored in a database, and then the database is scanned and pulled frequently to acquire the tasks with the execution time, which causes great reading and writing pressure on the database, and is difficult to implement efficient scheduling of the large batch of tasks. In view of this, the present invention provides a task scheduling method and apparatus based on multiple hash rings with different precisions, and aims to achieve stable and efficient scheduling of a large number of tasks.

The invention achieves the above purpose by the following technical scheme:

on one hand, the invention provides a task scheduling method based on a plurality of hash rings with different precisions, wherein a plurality of hash ring timers with different precisions are arranged in a task scheduler, each hash ring timer corresponds to a hash ring memory, an execution pointer and a plurality of slots which are annularly arranged at equal intervals are arranged in the hash ring memory, and the time of one circle of rotation of the execution pointer is consistent with the timing time of the corresponding hash ring timer; the task scheduling method comprises the following steps:

after receiving the task scheduling information, the task receiver writes the task number and the task ending time into a local cache, and writes the execution frequency and the execution logic of the task into a local buffer queue;

a timing pulling thread pulls the tasks from the buffer queue at a fixed frequency, a matched hash ring timer is selected according to the execution frequency of the tasks, and the tasks are placed in a task list of a corresponding slot position on a corresponding hash ring memory by the hash ring timer;

each Hash ring timer rotates an execution pointer on a corresponding Hash ring memory at a specified frequency, and takes out the task of which the execution time is up in a task list of a slot position pointed by the execution pointer;

and each Hash ring timer inquires whether the taken task is stopped or not to the task receiver based on the task number, and if the task is judged not to be stopped by the corresponding task ending time, the execution logic of the task is distributed to the asynchronous executor to be asynchronously executed.

Preferably, the timing pull thread pulls the task from the buffer queue at a fixed frequency, selects a hash ring timer matched with the timing pull thread according to the execution frequency of the task, and places the task in a task list of a corresponding slot on a corresponding hash ring memory by the hash ring timer, specifically:

a task scheduler starts a timing pull thread to pull a task from the buffer queue at a fixed frequency, and judges a hash ring timer which is used by the task according to the execution frequency of the task;

after the judgment is finished, the timing pulling thread puts the pulled task into a corresponding hash ring timer according to the judgment result;

and the Hash ring timer calculates the position of the task to be placed in the slot position corresponding to the Hash ring memory based on the execution frequency of the task, and places the task in the task list of the slot position on the corresponding Hash ring memory after calculation.

Preferably, after the timing pull thread pulls the task from the buffer queue at a fixed frequency, if a matching hash ring timer cannot be found according to the execution frequency of the task, the method further includes:

selecting the nearest hash ring timer according to the execution frequency of the task, placing the task in the corresponding slot position of the corresponding hash ring memory by the hash ring timer, and marking the task;

and when the number of the tasks with the marks exceeds a first preset threshold value, establishing a hash ring timer and a corresponding hash ring memory according to the execution frequency of each task with the marks, and putting the hash ring timer and the corresponding hash ring memory into use.

Preferably, after the newly built hash ring timer and the corresponding hash ring memory are put into use, the method further includes:

counting the total number of tasks to be executed on a hash ring memory at preset time intervals for the hash ring memory corresponding to the newly-built hash ring timer;

when the total number of the tasks on the hash ring memory is smaller than a second preset threshold value, transferring each task on the hash ring memory to the nearest hash ring memory according to the execution frequency of the task;

deleting the newly-built Hash ring timer and the corresponding Hash ring memory from the task scheduler, and quitting using; wherein the second preset threshold is smaller than the first preset threshold.

Preferably, the newly establishing a hash ring timer and a corresponding hash ring memory according to the execution frequency of each marked task specifically includes:

and counting the average execution frequency of each marked task, determining the timing time of the hash ring timer needing to be newly established according to the average execution frequency, and further establishing the hash ring timer and the hash ring memory based on the timing time.

Preferably, before the timed pull thread pulls tasks from the buffer queue at a fixed frequency, the method further comprises:

dividing time periods, and determining one or more Hash ring timers which are used most in each time period according to historical data of task scheduling;

dividing one or more Hash ring timers which use the most in each time period into a Hash ring group, and marking the divided Hash ring groups with time periods;

when the task is scheduled, starting a hash ring timer in a hash ring group with a corresponding time period mark according to the current time period, and enabling the corresponding hash ring timer and a hash ring memory to be used;

and when switching to the next time period, closing the hash ring timer in the hash ring group corresponding to the previous time period, and starting the hash ring timer in the hash ring group with the next time period mark.

Preferably, four hash ring timers with different precisions are set in the task scheduler, and the four hash ring timers are respectively a 30-second hash ring timer, a 60-second hash ring timer, a 300-second hash ring timer and a 1800-second hash ring timer and correspond to four different hash ring memories; the hash ring timer selected according to the execution frequency of the task specifically includes:

when the execution frequency of the task is greater than or equal to 30 seconds and less than 60 seconds, selecting a 30-second Hash ring timer; when the execution frequency of the task is more than or equal to 60 seconds and less than 300 seconds, selecting a 60-second Hash ring timer; when the execution frequency of the task is more than or equal to 300 seconds and less than 1800 seconds, selecting a 300-second Hash ring timer; when the execution frequency of the task is 1800 seconds or more, the 1800 second hash ring timer is selected.

Preferably, if the task is not stopped, the hash ring timer distributes the execution logic of the task to the asynchronous executor, and then puts the task back to the task list of the slot where the task was located before being fetched on the corresponding hash ring memory again.

In a second aspect, the present invention further provides a task scheduling device based on multiple hash rings with different precisions, including a task receiver and a task scheduler, where the task receiver is configured to receive and store a task sent from a client, and the task scheduler is configured to perform task scheduling according to the task stored in the task receiver;

the task receiver is internally provided with a local cache and a buffer queue, the local cache is used for storing task numbers and task ending time, and the buffer queue is used for storing the execution frequency and execution logic of tasks;

the task scheduler is internally provided with a timing pull thread, a plurality of hash ring timers with different precisions, hash ring memories corresponding to the hash ring timers and an asynchronous executor;

the timing pulling thread is used for pulling the tasks from the buffer queue at a fixed frequency and putting the tasks into a matched hash ring timer according to the execution frequency of the tasks; the hash ring timer is used for placing tasks in a task list corresponding to the slot position on the corresponding hash ring memory, rotating an execution pointer on the corresponding hash ring memory at a specified frequency, further taking out the tasks with the execution time in the task list of the slot position indicated by the execution pointer, inquiring the local cache whether the taken out tasks are stopped or not, and if not, distributing the execution logic of the tasks to an asynchronous executor for asynchronous execution.

Preferably, each hash ring memory is a data structure of a ring queue + task list;

the annular queue comprises a plurality of slot positions which are arranged in an annular equal interval mode, each time interval passes, an execution pointer of the Hash ring memory moves from a current slot position to a next slot position, and the time of one circle of rotation of the execution pointer is consistent with the timing time of a corresponding Hash ring timer; and each slot position is correspondingly provided with a task list used for storing the tasks to be executed on the slot position.

Compared with the prior art, the invention has the beneficial effects that:

according to the invention, the hash ring timer is provided with a buffer queue in front to temporarily store tasks, and the tasks are pulled from the buffer queue to the hash ring memory according to a fixed frequency, so that a large number of tasks can be uniformly distributed in the hash ring memory, the execution time accumulation of the large number of tasks is avoided, the load balancing effect is realized, and the machine CPU is fully utilized to efficiently schedule the large number of tasks; in addition, a plurality of hash ring timers with different accuracies are arranged in the task scheduler, and when the tasks are stored, the matched hash ring timers are selected according to the execution frequency of the tasks and then are stored in the corresponding hash ring memories, so that the tasks with different execution frequencies can meet the accuracy requirement.

[ description of the drawings ]

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the embodiments of the present invention will be briefly described below. It is obvious that the drawings described below are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.

Fig. 1 is an architecture diagram of a task scheduling device based on a buffer queue and a hash ring according to an embodiment of the present invention;

fig. 2 is a flowchart of a task scheduling method based on a buffer queue and a hash ring according to an embodiment of the present invention;

FIG. 3 is an exemplary diagram of a conventional solution for causing tasks to be aggregated into a few slots on a hash ring;

fig. 4 is a log screenshot of a task balanced distribution effect according to an embodiment of the present invention;

fig. 5 is a flowchart of a specific implementation of task scheduling based on a buffer queue and a hash ring according to an embodiment of the present invention;

fig. 6 is a flowchart of a task scheduling method based on multiple hash rings with different precisions according to an embodiment of the present invention;

fig. 7 is a flowchart of a hash ring set dynamic adaptation adjustment method based on time slot switching according to an embodiment of the present invention;

fig. 8 is a flowchart of a task scheduling method capable of performing hash ring switching according to an embodiment of the present invention;

fig. 9 is a flowchart of another task scheduling method for hash ring switching according to an embodiment of the present invention;

fig. 10 is a schematic diagram illustrating a switching of a hash ring memory according to an embodiment of the present invention.

[ detailed description ] embodiments

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

In addition, the technical features involved in the embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other. The invention will be described in detail below with reference to the figures and examples.

Example 1:

the invention provides a task scheduling method and device based on a buffer queue and a hash ring, and aims to solve the technical problems that when large-batch task scheduling is carried out in the traditional scheme, all tasks need to be stored in a database and then the database is frequently scanned and pulled to obtain the tasks with the execution time, so that the database reading and writing pressure is large, and efficient scheduling of large-batch tasks is difficult to realize. In order to facilitate understanding of the task scheduling method, the embodiment of the present invention first introduces a task scheduling apparatus.

As shown in fig. 1, a task scheduling apparatus provided in an embodiment of the present invention mainly includes a task receiver and a task scheduler, where the task receiver is configured to receive and store task information sent by a client, where the task information includes a task execution frequency, an execution logic, and a task end time; and the task scheduler is used for scheduling tasks according to the tasks stored in the task receiver. The following introduces specific structural functions of the task receiver and the task scheduler:

referring to fig. 1, a local buffer a and a buffer queue B are provided in the task receiver.

The local cache A is a KV data structure in the memory of the local machine and is used for storing task numbers and task ending time. Specifically, the task receiver sets a unique task number for the received task, then sets the key as the task number, and sets the value as the corresponding task ending time; if the received task has no task end time, indicating that the task will be executed all the time or a request to stop the task has not been received yet, the value is set to null. In an exemplary embodiment, the hash table may be selected to store the part of information, i.e., the key of the hash table is the task number, and the value is the corresponding task end time.

The buffer queue B is also a data structure in the memory of the local machine, and is used to temporarily store the received task scheduling information, mainly the execution frequency and the execution logic. In an exemplary embodiment, a first-in, first-out unidirectional queue may be selected to store this portion of information, as shown in FIG. 1. Wherein, T1, T2,. and Tn in the buffer queue B in fig. 1 represent the task number.

The local machine is a machine running a program, and is distinguished from the remote machine. The remote machine refers to the need of network IO, and in the scheme, if the cache of the remote machine is used, the cache information reading is obviously delayed due to the network IO, and the task scheduling thread is blocked, so that the task scheduling efficiency is influenced, and therefore the cache of the local machine is selected.

Further, in a preferred embodiment, to prevent the task from being lost in case of a restart of the computing machine, an external cache, such as the database of fig. 1, may also be provided in the task receiver. After receiving the task, the task receiver can write task scheduling information into an external cache of the computer, mainly including the execution frequency and the execution logic of the task; after the computer is restarted, the task scheduling information can be read from the external cache and written into the buffer queue B, so that the task scheduling is started again.

With continued reference to fig. 1, the task scheduler is a core component of the entire task scheduling apparatus, and is mainly internally provided with several components, namely a timing pull thread C, a hash ring timer D, a hash ring memory E, and an asynchronous executor F.

The timing pull thread C is configured to pull one task at a time from the buffer queue B of the task receiver at a fixed frequency, and place the pulled task to the hash ring timer D.

The hash ring memory E is a data structure of a ring queue + task list, the ring of the hash ring memory E is divided into a plurality of equal parts, each part is called a slot, the ring queue includes a plurality of slots arranged in a ring shape at equal intervals (in the figure, 8 slots are set as an example), and the interval between the slots represents a time interval; an execution pointer in the Hash ring memory E rotates clockwise or anticlockwise, and the execution pointer moves from the current slot position to the next slot position every time interval; and each slot position is correspondingly provided with a task list used for storing one or more tasks to be executed on the slot position. The time of the execution pointer rotating for one circle is consistent with the timing time of the hash ring timer D, and the time interval is consistent with the frequency of the pull task of the timing pull thread C, that is, the fixed frequency. As shown in fig. 1, a circle represents a circular queue, solid arrows in the circle represent execution pointers inside the hash ring memory E, rectangles corresponding to T1, T2, ·, Tn, Tb represent tasks, where the tasks T6 and Tb are in a task list of the same slot in the hash ring, and T1 and Tn are in a task list of the same slot, and as can be seen from the figure, the execution pointers currently point to the slots in which T6 and Tb are located.

The hash ring timer D is used for placing tasks in a task list of a slot corresponding to the hash ring memory F based on the execution frequency of the tasks, rotating an execution pointer on the hash ring memory F clockwise or anticlockwise at the fixed frequency, further sequentially taking out the tasks with the execution time in the task list of the slot pointed by the execution pointer, inquiring the local cache A whether the taken out tasks are stopped or not, and if not, distributing the execution logic of the tasks to the asynchronous executor F for asynchronous execution. And a timing execution thread is arranged in the hash ring timer D and is responsible for scanning tasks from the hash ring memory F according to the fixed frequency, and the scanning frequency is consistent with the time represented by the slot interval in the hash ring memory E.

Further, if the task is not stopped, after the hash ring timer D distributes the task to the asynchronous executor F, the task may be placed back into the task list of the slot where the task was located before being taken out on the hash ring memory E again, and then the next time the execution time is reached (i.e., after the execution pointer on the hash ring memory E rotates one or more circles), the task may be directly scheduled and executed from the corresponding slot of the hash ring memory E. Therefore, to meet the scheduling requirement, in a preferred embodiment, the time required for executing one pointer rotation on the hash ring memory E should be less than or equal to the execution frequency of the task.

The asynchronous executor F is configured to execute the task distributed by the hash ring timer D, and execute the task through an asynchronous thread without blocking a main logic of the hash ring timer D. In an exemplary embodiment, the asynchronous executor F may be implemented using a thread pool or the like, but is not limited to this.

Further, according to the precision requirement, a plurality of hash ring timers with different precisions can be arranged in the task scheduler. For example, in one particular embodiment, the following 4 different precision hash ring timers may be set:

1)30 second hash ring timer: at this time, 512 slot positions are set in the hash ring memory E, the time length of the hash ring execution pointer moving for one turn is 30 seconds, the execution pointer is moved to the next slot position every 30 seconds/512-58 milliseconds, that is, the time interval between every two adjacent slot positions is 58 milliseconds. In this case, the maximum time error when scheduling a task is 58 ms, and the minimum time interval between two schedules of the same task is 30s, i.e., the execution frequency of the task is 30s at minimum.

2)60 second hash ring timer: at this time, 2014 slot positions are set in the hash ring memory E, the time length for the hash ring to execute the pointer to move for one circle is 60 seconds, the execution pointer is moved to the next slot position every 60 seconds/1024-58 milliseconds, that is, the time interval between every two adjacent slot positions is 58 milliseconds. In this case, the maximum time error when scheduling a task is 58 ms, and the minimum time interval between two schedules of the same task is 60s, i.e., the execution frequency of the task is 60s at minimum.

3)300 second hash ring timer: at this time, 2048 slots are set in the hash ring memory E, the time length for the hash ring to execute pointer movement for one turn is 300 seconds, the execution pointer is moved to the next slot every 300 seconds/1024-146 milliseconds, that is, the time interval between every two adjacent slots is 146 milliseconds. In this case, the maximum time error when scheduling the task is 146 ms, and the minimum time interval between two schedules of the same task is 300s, i.e. the execution frequency of the task is 300s at minimum.

4)1800 second hash ring timer: at this time, 4096 slot positions are set in the hash ring memory E, the time length for the hash ring to execute the pointer to move for one turn is 1800 seconds, the execution pointer is moved to the next slot position every 1800 seconds/4096-439 milliseconds, that is, the time interval between every two adjacent slot positions is 439 milliseconds. In this case, the maximum time error when scheduling the task is 439 milliseconds, and the minimum time interval between two schedules of the same task is 1800 seconds, that is, the execution frequency of the task is 1800 seconds at minimum.

When the execution frequency of the task is greater than or equal to 30 seconds and less than 60 seconds, the 30-second Hash ring timer can be selected; when the execution frequency of the task is more than or equal to 60 seconds and less than 300 seconds, a 60-second hash ring timer can be selected; when the execution frequency of the task is more than or equal to 300 seconds and less than 1800 seconds, a 300-second hash ring timer can be selected; when the execution frequency of the task is 1800 seconds or more, the 1800 second hash ring timer may be selected.

If the number of hash ring timers and hash ring memory settings is too high, it may result in excessive resource usage. Therefore, on the premise of satisfying the balance between the precision requirement and the resource occupation, a maximum of 4 hash ring timers with different precisions are generally set, such as the four timers described above. In actual use, only one or two of the two types of the precision sensors may be selected to meet the actual precision requirement.

In the task scheduling device provided by the embodiment of the invention, a buffer queue is arranged in front of the hash ring timer in the task receiver for temporarily storing the tasks, a timing pull thread is arranged in the task scheduler for pulling the tasks from the buffer queue according to a fixed frequency and storing the tasks into the hash ring memory, so that a large number of tasks can be uniformly distributed in the hash ring memory, the accumulation of the execution time of the large number of tasks is avoided, the effect of load balancing is realized, and a machine CPU is fully utilized to efficiently schedule the large number of tasks;

in addition, after the tasks are scheduled, the hash ring timer is put back to the slot positions on the hash ring memory before the tasks are placed back again, so that the uniform distribution in the initial state is maintained, the data skew caused by that a large number of tasks are distributed to a few slot positions on the hash ring memory due to the CPU time slice preemption is avoided, and the scheduling efficiency of the scheduler is not reduced no matter how many times the tasks are scheduled.

Example 2

On the basis of the foregoing embodiment 1, an embodiment of the present invention provides a task scheduling method based on a buffer queue and a hash ring, which can be completed by using the task scheduling apparatus in embodiment 1, as shown in fig. 2, and mainly includes the following steps:

step 101, after receiving the task scheduling information, the task receiver writes the task number and the task ending time into a local cache, and writes the execution frequency and the execution logic of the task into a local buffer queue.

With reference to fig. 1, the specific implementation process is as follows: as an initiator of task scheduling and a provider of task information, a client may continuously send task scheduling information to a task receiver according to a flow indicated by reference number 1 in fig. 1, where the task scheduling information mainly includes an execution frequency and an execution logic, so as to initiate a task scheduling request. The execution frequency of the task can be millisecond precision or higher precision, and is not limited uniquely; the execution logic of the task is not limited, and may be a calculation logic that is executed for a short time or a calculation logic that is relatively time-consuming. After receiving the task scheduling information sent by the client, the task receiver sets a unique task number for the received task, such as T1, T2, and Tn, sets key as the task number, and sets value as the corresponding task end time, thereby writing the task number and the task end time into the local cache a. Then, according to the flow shown by the number 2 in fig. 1, the task receiver writes the received task scheduling information into the buffer queue B.

The task end time is obtained when the client sends a request for stopping the task to the task receiver, and if the received task has no task end time, which indicates that the task is executed all the time or the request for stopping the task is not received at present, the value may be set to null. And after the task receiver receives the task scheduling information, the task receiver can receive a task stopping request sent by the client in real time, namely, receive the task ending time corresponding to the task in real time, and update the task ending time in the local cache a, so that when the task is scheduled next time, whether the task is stopped can be judged according to the updated task ending time in the local cache a, and whether the task is to be executed is further determined. By receiving the task receiving time in real time and updating in time, the real-time performance and effectiveness of task scheduling can be ensured.

Further, in a preferred embodiment, to prevent the loss of tasks in case of a restart of the computing machine, after the task receiver receives the task scheduling information, the task receiver may also write the task scheduling information (mainly the execution frequency and execution logic) into an external cache of the computer, such as the database in fig. 1. After the computer is restarted, the task receiver can read the task scheduling information from the external cache and write the task scheduling information into the local buffer queue B, so that the task scheduling is started again.

And 102, pulling the task from the buffer queue by a timing pulling thread at a fixed frequency, and placing the task into a task list of a corresponding slot of the hash ring memory by the hash ring timer based on the execution frequency.

With reference to fig. 1, the specific implementation process is as follows: the task scheduler starts a timing pull thread C, then pulls one task from the buffer queue B of the task receiver at a fixed frequency each time according to the flow shown by the number 3 in the figure 1, and puts the pulled task into a Hash ring timer D; and then the hash ring timer D calculates the slot position of the hash ring memory E where the task should be placed based on the execution frequency of the task, and after the calculation is finished, the task is put into the task list of the slot position according to the flow shown by the number 4 in the figure 1. The pulling frequency of the timing pulling thread C may be in the order of milliseconds, or may be longer, which is not specifically limited herein, and the pulling frequency is equal to the frequency of executing the pointer movement in the hash ring timer in step 103.

For example, if the execution pointer of the hash ring memory makes one rotation for 60 seconds and there are 12 slots, which are respectively denoted as slot 1, slot 2,. and slot 12, the time interval between every two slots is 5 s. If the current execution pointer is pointed at the slot position 1 and the execution frequency of the task is 60s, starting from the current time (namely when the execution pointer is pointed at the slot position 1), after 60s, the task starts to be executed for the first time, and the task is determined to be placed at the slot position 1 through calculation, so that the task can be guaranteed to be just executed when the execution pointer rotates for 60s (namely 1 turn) to reach the slot position 1 again, and the task can be taken out for scheduling execution. The calculation and storage principles of other tasks are similar to those of the method, and detailed description is omitted. Because the execution frequency of each task is different, the time points for starting to calculate the execution frequency may also be different (because the execution pointer continuously rotates, and the slot position pointed by the execution pointer currently may be different when each task is stored inwards), so that after the tasks are stored by the method, a large batch of tasks can be relatively uniformly distributed on each slot position of the hash ring memory.

And 103, the hash ring timer rotates the execution pointer on the hash ring memory at the fixed frequency, and takes out the task of which the execution time has been reached in the task list of the slot position indicated by the execution pointer.

With reference to fig. 1, the hash ring timer D may move the execution pointer inside the hash ring memory E clockwise or counterclockwise according to a fixed frequency, and pull out the task that has reached the execution time, i.e., the expired task, from the task list in the slot indicated by the execution pointer according to the flow indicated by the number 5 in fig. 1. Specifically, there is a timing execution thread in the hash ring timer D, and the thread scans the task from the hash ring memory E according to a fixed frequency, where the scanning frequency (or the frequency of executing pointer movement) is consistent with the time indicated by the slot interval in the hash ring memory E, and is also consistent with the pull frequency of the timing pull thread C in step 102, which are both the fixed frequency. For example, a 30 second hash ring timer scans the hash ring memory E at a 58 millisecond rate, while a timed pull thread C pulls tasks from the buffer queue B at a 58 millisecond pull rate.

For a slot position currently pointed by an execution pointer, sequentially judging whether each task in a task list corresponding to the slot position reaches execution time; if the execution time is up, pulling out the corresponding task from the task list until all tasks of which the execution time is up in the current slot position are pulled out; then the hash ring timer D moves the execution pointer in the hash ring memory E to the next slot position, so as to continuously pull the task of which the execution time is up in the task list of the next slot position. Taking fig. 1 as an example, the tasks T6 and Tb are in the task list of the same slot; during the execution, the execution pointer in the hash ring memory E points to the slot where the tasks T6 and Tb are located, the hash ring timer D finds that T6 has reached the execution time after scanning T6 and Tb, and Tb has not reached the execution time, so that T6 is taken out of the task list.

It should be noted that, in the process of executing the pointer rotation and taking out the task execution, the timing pull thread still pulls the task from the buffer queue at a fixed frequency, the hash ring timer stores the task pulled by the timing pull thread to the corresponding slot of the hash ring memory according to the calculation result, and the two processes are executed in parallel without mutual influence.

And 104, inquiring whether the taken task is stopped or not from the task receiver by the Hash ring timer based on the task number, and if the task is not stopped according to the corresponding task ending time, distributing the execution logic of the task to an asynchronous executor to execute asynchronously.

With reference to fig. 1, the specific implementation process is as follows: the Hash ring timer D searches corresponding task ending time in a local cache B of the task receiver according to the task number corresponding to the task to be taken out; and if the task ending time is empty or the task ending time is greater than the current time and indicates that the task is not stopped, the Hash ring timer D distributes the execution logic of the task to the asynchronous executor, and the asynchronous executor F executes the task according to the execution logic. Continuing with the example of fig. 1, after the hash ring timer D retrieves task T6 from the task list, it inquires of the task receiver whether task T6 has stopped according to the flow indicated by reference number 6 in fig. 1; assuming that the task end time of the task T6 stored in the local cache a of the task receiver is greater than the current time, which proves that the task T6 has not ended yet, the hash ring timer D distributes the task T6 to the asynchronous executor E for execution according to the flow indicated by the reference number 7 in fig. 1.

Further, in a preferred embodiment, if the task is not stopped, the hash ring timer may also replace the task back into the task list of the slot on the hash ring memory where the task was in before the task was fetched after distributing the execution logic of the task to the asynchronous executor. Continuing with the example of fig. 1, after the hash ring timer D distributes the task T6 to the asynchronous executor E, according to the flow denoted by number 8 in fig. 1, the task T6 is again placed back into the slot in the hash ring memory E where the task T6 is located before, i.e. at the T6' square corresponding to the dashed line in the drawing; so far, the task list corresponding to the slot currently pointed by the execution pointer stores tasks Tb and T6'. Then, the hash ring timer D moves the execution index clockwise to the next slot, i.e., the slot pointed by the dotted execution pointer in fig. 1.

If the task is inquired to reach the end time, which means that the client notifies the task receiver to stop the task, the hash ring timer D will not distribute the task to the asynchronous executor F for execution any more, and will not place the task into the hash ring memory again.

And (5) circularly executing the step 103 and the step 104, namely, scheduling the tasks for multiple times or even infinite times.

After the task is taken out from the hash ring memory and executed for the first time, if the method for putting the task back to the slot position before the task is not adopted, because each task may have time delay in each execution process, the tasks which are not originally on the same slot position may be accumulated to the same slot position after being scheduled and executed for multiple times. Referring to fig. 3, which is a corresponding task accumulation scenario, the small dotted circles on the ring of the hash ring memory E in the figure indicate that no task is to be executed at these slots, and when the execution pointer moves to these slots, the CPU of the machine idles because no task is to be executed. On the other hand, the slot in which the tasks T4, T6, … and Tn are located has a large number of tasks, which may cause the CPU to perform too many calculations when the pointer moves to the slot due to the large number of tasks to be scanned and distributed for execution, thereby causing the CPU to be too busy at this point in time and consuming too many machine resources.

As shown in fig. 4, in a scenario where 229 tasks are scheduled by using a 30-second hash ring timer according to an embodiment of the present invention, the distribution state of the tasks at slots is after the hash ring timer runs for several days. To monitor the distribution status, the embodiment of the present invention prints the distribution status into a log, as shown in fig. 4. The timer has 512 slots in total, wherein 0 means that there is no task in the slot, 1 means that there are 1 task in the slot, and 2 means that there are 2 tasks in the slot. As can be seen from fig. 4, 229 tasks are relatively uniformly distributed in the hash ring timer, and the maldistribution condition shown in fig. 3 does not occur.

For convenience of illustration, only 229 tasks are used in the above embodiment to show the effect of the present solution, but not to limit the present invention. In fact, the effect of the present solution is still effective when the number of tasks reaches the level of 10 ten thousand, and details are not described here.

It should be noted that, the task receiver and the task scheduler in the embodiment of the present invention are merely structural names introduced for convenience of description, and are not intended to limit the present invention. According to the program algorithm corresponding to the method provided by the embodiment of the present invention, the task receiver and the task scheduler may also be completed by using other virtual modules, and the basic algorithm may still refer to the method, which is not described herein in detail.

In the task scheduling method provided by the embodiment of the invention, a buffer queue is arranged in front of the hash ring timer to temporarily store tasks, and the timed pull thread pulls the tasks from the buffer queue to the hash ring memory according to a fixed frequency, so that a large number of tasks can be uniformly distributed in the hash ring memory, the execution time accumulation of a large number of tasks is avoided, the effect of load balancing is realized, and a machine CPU is fully utilized to efficiently schedule a large number of tasks;

in addition, the tasks are placed back into the slots on the hash ring memory before the tasks are scheduled again, so that the uniform distribution in the initial state is maintained, the data skew caused by that a large number of tasks are distributed to a few slots on the hash ring memory due to the CPU time slice preemption is avoided, and the scheduling efficiency of the scheduler is not reduced no matter how many times the tasks are scheduled.

Example 3:

with reference to embodiment 1 and embodiment 2, the embodiment of the present invention further provides a specific implementation manner of a task scheduling method based on a buffer queue and a hash ring, taking a task receiver using a database as an external cache and setting a hash ring timer with a precision in a task scheduler as an example, and as shown in fig. 5, the specific implementation manner specifically includes the following steps:

in step 201, the task receiver receives the execution frequency and the execution logic of the task to be scheduled from the client, as shown in the flow numbered 1 in fig. 1.

In step 202, the task receiver writes the task number and the task end time into the local cache a. The task number is a unique number generated by the task receiver for the received task and is set as a key; the task end time is obtained when the client sends a request for stopping the task to the task receiver and is set as value; if the received task has no task end time, value is set to null.

In step 203, the task receiver writes the execution frequency and execution logic of the task to the database to prevent the task from being lost in case of a restart of the computing machine. After the computer is restarted, the task receiver can read the execution frequency and the execution logic of the tasks from the database and restart the scheduling of the tasks.

In step 204, the task receiver writes the execution frequency and execution logic of the task into the buffer queue B, as shown in the flow numbered 2 in fig. 1.

In step 205, the task scheduler starts a timing pull thread C to pull the task from the buffer queue B at a fixed frequency, and puts the pulled task into the hash ring timer D, as shown in the flow numbered 3 in fig. 1.

In step 206, the hash ring timer D calculates the slot position of the hash ring memory E where the task should be placed based on the execution frequency, and puts the task into the task list of the slot, as shown in the flow numbered 4 in fig. 1.

In step 207, the hash ring timer D scans the task list in the slot currently pointed by the execution pointer in the hash ring memory E, and pulls out the task whose execution time has been reached in the task list, as shown in the flow numbered 5 in fig. 1.

In step 208, the hash ring timer D queries the local cache a in the task receiver whether the pulled task has stopped, as shown in the flow numbered 6 in fig. 1, that is, the local cache a is queried for the corresponding task ending time based on the task number, and whether the task has stopped is determined by the task ending time. If the task has not stopped, continue to step 209; if the task has stopped, it jumps to step 211.

In step 209, the execution logic of the task is distributed to the asynchronous executor F for asynchronous execution, as shown in the flow numbered 7 in fig. 1.

In step 210, the hash ring timer D puts the task back into the task list of the slot where the task is located before being taken out in the hash ring memory E again, as shown in the flow numbered 8 in fig. 1. Therefore, the uniform distribution of the initial state on the Hash ring memory can be maintained, the data skew caused by that a large number of tasks are distributed to a few slots on the Hash ring memory due to the CPU time slice preemption is avoided, and the scheduling efficiency of the scheduler is not reduced no matter how many times the tasks are scheduled.

Step 211, determining whether the task whose execution time has been reached in the slot position currently pointed by the execution pointer in the hash ring memory E has been processed; if so, continue to step 212; if not, jumping to step 207, and continuing to pull the next task which has reached the execution time from the task list of the current slot position until the pulling is finished.

In step 212, the hash ring timer D moves the execution pointer in the hash ring memory to the next slot at the fixed frequency, and jumps to step 207, so as to continue to pull the task whose execution time has been reached in the task list of the next slot.

The steps shown in fig. 5 are repeated in such a cycle, so that multiple or even infinite scheduling of tasks can be realized.

Example 4

In the task scheduling methods provided in

embodiments

2 and 3, only one hash ring timer D is set in the task scheduler. Considering that in actual use, tasks to be scheduled are in large batch, execution frequencies corresponding to the tasks are different, the frequency related range may be wide, and if only one hash ring timer is set, it may be difficult to meet the accuracy requirement. For example, if a 30 second hash ring timer is used, it is appropriate for tasks to be executed with a frequency between 30 seconds and 60 seconds; however, if the execution frequency of the task is 1800 seconds, the execution pointer needs to rotate 60 times to fetch and execute the task once, which not only occupies the slot resources of the hash ring memory for a long time, but also may affect the execution accuracy of the task if the execution pointer is fetched and executed once after too many rotations.

In view of the above problems, the embodiments of the present invention provide a task scheduler with a plurality of hash ring timers with different precisions according to the precision requirements, and further provide a task scheduling method based on a plurality of hash rings with different precisions. Each hash ring timer corresponds to a hash ring memory with a slot structure, and the time for executing one-turn rotation of the pointer in each hash ring memory is consistent with the timing time of the corresponding hash ring timer. The main differences from example 2 and example 3 are: when a plurality of hash ring timers are set, after the tasks are pulled from the buffer queue by the timing pull thread, the most appropriate hash ring timer needs to be selected according to the execution frequency of the tasks and then the most appropriate hash ring timer is placed into the corresponding hash ring memory, so that the tasks with different execution frequencies can meet the precision requirement. On the basis of the embodiment 2 and the embodiment 3, if the requirement on the precision is improved and a single hash ring timer and a hash ring memory are difficult to meet, the task scheduling method provided by the embodiment of the invention can be adopted.

As shown in fig. 6, the task scheduling method based on multiple hash rings with different precisions provided in the embodiment of the present invention mainly includes the following steps:

step 301, after receiving the task scheduling information, the task receiver writes the task number and the task ending time into a local cache, and writes the execution frequency and the execution logic of the task into a local buffer queue.

For a specific implementation process, reference may be made to step 101 in embodiment 2, which is not described herein again. In a preferred embodiment, to prevent the loss of tasks in case of a restart of the computing machine, after the task receiver receives the task scheduling information, the task receiver may also write the task scheduling information (mainly the execution frequency and execution logic) into an external cache of the computer, such as the database in fig. 1. After the computer is restarted, the task receiver can read the task scheduling information from the external cache and write the task scheduling information into the local buffer queue B, so that the task scheduling is started again.

And 302, a timing pulling thread pulls the task from the buffer queue at a fixed frequency, a matched hash ring timer is selected according to the execution frequency of the task, and the hash ring timer puts the task in a task list of a corresponding slot position on a corresponding hash ring memory.

Because a plurality of hash ring timers with different precisions are arranged and the plurality of hash ring timers run synchronously, the specific implementation process of the step is as follows: firstly, a task scheduler starts a timing pull thread C to pull a task from a buffer queue B at a fixed frequency, and judges a hash ring timer which is used by the task according to the execution frequency of the task; after the judgment is finished, the timing pulling thread puts the pulled task into a corresponding hash ring timer according to the judgment result; and then, the hash ring timer calculates the slot position of the task to be placed in the corresponding hash ring memory based on the execution frequency of the task, and after calculation, the task is placed in a task list of the slot position in the corresponding hash ring memory.

Taking four hash ring timers with different precisions set in the task scheduler as an example, the four hash ring timers are respectively a 30-second hash ring timer, a 60-second hash ring timer, a 300-second hash ring timer, and a 1800-second hash ring timer, and the hash ring memory structure corresponding to each hash ring timer can refer to embodiment 1, which is not described herein. The hash ring timer that is determined to be used by the task according to the execution frequency of the task is specifically: when the execution frequency of the task is more than or equal to 30 seconds and less than 60 seconds, using a 30-second hash ring timer; when the execution frequency of the task is more than or equal to 60 seconds and less than 300 seconds, a 60-second Hash ring timer is used; when the execution frequency of the task is more than or equal to 300 seconds and less than 1800 seconds, a 300-second Hash ring timer is used; when the execution frequency of the task is 1800 seconds or more, the 1800 second hash ring timer is used. Therefore, the corresponding hash ring timer can be flexibly selected according to the execution frequency of the task, so that the actual precision requirement can be met to the maximum extent.

When the hash ring timer calculates which slot on the hash ring memory the task should be placed in, the specific calculation method may refer to embodiment 2, which is not described herein again. And each Hash ring timer performs slot position calculation according to the method, and the tasks are uniformly distributed on the corresponding Hash ring memories.

Step 303, each hash ring timer rotates the execution pointer on the corresponding hash ring memory at the designated frequency, and takes out the task whose execution time has been reached in the task list of the slot indicated by the execution pointer.

For hash ring timers with different precisions, the corresponding "designated frequencies" are different, that is, the rotation frequencies of the execution pointers on the corresponding hash ring memories are different and need to be consistent with the slot intervals in the corresponding hash ring memories. Specifically, each hash ring timer has a timed execution thread, and the thread scans tasks from the corresponding hash ring memory according to a specified frequency, wherein the scanning frequency is consistent with the time represented by the slot interval in the corresponding hash ring memory. For example, a 30 second hash ring timer scans tasks from the corresponding hash ring memory at a frequency of 58 milliseconds, and a 300 second hash ring timer scans tasks from the corresponding hash ring memory at a frequency of 146 milliseconds.

As mentioned above, the pulling frequency of the timing pulling thread should be the same as the scanning frequency of the hash ring timer, or the interval between the timing pulling thread and the slot in the hash ring timer, so that it can be ensured that the execution pointer moves exactly one slot every time the timing pulling thread pulls one task, and it is convenient to calculate the time from the slot pointed by the execution pointer at present. In the embodiment of the invention, as a plurality of hash ring timers with different precisions are arranged, the scanning frequency corresponding to each hash ring timer is different, and the pulling frequency cannot be consistent with the scanning frequency of each hash ring timer; at this time, the pull frequency may take the minimum scan frequency among the plurality of hash ring timers.

For example, if a 30 second hash ring timer and a 300 second hash ring timer are used, the timing pull thread C may pull tasks from the buffer queue B at a pull frequency of 58 ms. For the 30-second hash ring timer, the pulling frequency is consistent with the scanning frequency, when the task is pulled and placed into the corresponding hash ring memory, the execution pointer just points to a certain slot position, the time can be calculated from the current slot position, and the task is placed at the calculated slot position. For a 300-second hash ring timer, the pulling frequency is not consistent with the scanning frequency, to be precise, the pulling frequency is less than the scanning frequency, when the task is pulled and put into the corresponding hash ring memory, the execution pointer may be positioned between two slot positions, at this time, the time can be calculated from the next slot position, and when the execution pointer is rotated to the next slot position, the task is put into the calculated slot position.

And step 304, inquiring whether the taken task is stopped or not from the task receiver by each hash ring timer based on the task number, and if the task is not stopped according to the corresponding task ending time, distributing the execution logic of the task to an asynchronous executor to execute asynchronously.

For each hash ring timer, the specific implementation process of the pull task execution may refer to step 104 in embodiment 1, which is not described herein again. In a preferred embodiment, if the task is not stopped, the hash ring timer may also replace the task again in the task list of the slot on the corresponding hash ring memory where the task was located before the fetch after distributing the execution logic of the task to the asynchronous executor. If the task is inquired to be finished, the client notifies the task receiver to stop the task, the hash ring timer does not distribute the task to the asynchronous executor F for execution, and the task is not put into the corresponding hash ring memory again.

And (6) circularly executing the step 303 and the step 304, namely, scheduling the task for multiple times or even infinite times.

In the task scheduling method provided by the embodiment of the invention, a buffer queue is arranged in front of the hash ring timer to temporarily store the tasks, and the timed pull thread pulls the tasks from the buffer queue to the hash ring memory according to a fixed frequency, so that a large number of tasks can be uniformly distributed in the hash ring memory, the execution time accumulation of a large number of tasks is avoided, the effect of load balancing is realized, and a machine CPU is fully utilized to efficiently schedule a large number of tasks. In addition, a plurality of hash ring timers with different precisions are arranged in the task scheduler, and the hash ring timer which is most matched can be selected for use according to the execution frequency of the tasks, so that the precision requirements of the tasks with different execution frequencies are met.

In the above embodiment, four hash ring timers with different precisions are set in the task scheduler as an example, but not limited to this. Considering that too many hash ring timers occupy too many resources and too few hash ring timers are difficult to meet the precision requirement, when designing the task scheduler, the specific number and type of the hash ring timers can be selectively set according to the big data statistical result, as follows:

according to historical data of task scheduling, the hash ring timers with the use times exceeding a certain preset value in the historical task scheduling process are counted, and the big data counting result shows that the hash ring timers are used more historically, so that the probability that the hash ring timers are used more in the subsequent task scheduling process is presumed to be larger, and the hash ring timers can be arranged in the task scheduler and put into use. The execution frequency of each task can be counted according to the historical data of task scheduling, and then the hash ring timer is reasonably set according to the number of tasks in different frequency ranges, namely, the frequency range of which the number of tasks exceeds a certain preset value is counted. For example, if the execution frequency range of most tasks is between 30s to 60s and between 300s to 1800s, that is, the number of tasks in the two frequency ranges exceeds a certain preset value, two kinds of 30-second hash ring timers and 300-second hash ring timers may be set in the task scheduler, and the number of tasks in the remaining frequency ranges is small, so that the corresponding hash ring timers may not be required to be set. Therefore, resources can be saved, and the execution precision of most tasks can be ensured.

Furthermore, because the number and the type of the hash ring timers in the task scheduler are preset according to big data statistics, the scheduling requirements of most tasks can be met, but if the execution frequency of more tasks deviates from the big data in the scheduling process, the accuracy requirement cannot be met depending on the existing hash ring timers. To solve the above problem, in step 302, after the timing pull thread pulls the task from the buffer queue at a fixed frequency, if a matching hash ring timer cannot be found according to the execution frequency of the task, the method further includes:

selecting the nearest hash ring timer according to the execution frequency of the task, placing the task in the corresponding slot position of the corresponding hash ring memory by the hash ring timer, and marking the task; and when the number of the tasks with the marks exceeds a first preset threshold value, establishing a hash ring timer and a corresponding hash ring memory according to the execution frequency of each task with the marks, and putting the hash ring timer and the corresponding hash ring memory into use. When the hash ring timer is newly built, the average execution frequency of each marked task can be counted, the timing time of the hash ring timer needing to be newly built is determined according to the average execution frequency, and then the hash ring timer and the hash ring memory are newly built based on the timing time; or counting the largest number of tasks falling in which frequency range, or counting the execution frequency mode of each marked task, determining the timing time of the hash ring timer to be newly built according to the frequency range with the largest number of tasks, and further newly building the hash ring timer.

In addition, considering that the execution frequency of most tasks should be still relatively close to the big data statistical result, the execution frequency is deviated from the big data only in a few cases, and after the hash ring timer is newly established for the tasks deviated from the big data, the newly established hash ring timer may become less used after a period of time. In view of the above, after the newly built hash ring timer and the corresponding hash ring memory are put into use, the method further includes:

counting the total number of tasks to be executed on a hash ring memory at preset time intervals for the hash ring memory corresponding to the newly-built hash ring timer; when the total number of the tasks on the hash ring memory is smaller than a second preset threshold value, transferring each task on the hash ring memory to the nearest hash ring memory according to the execution frequency of the task; deleting the newly-built Hash ring timer and the corresponding Hash ring memory from the task scheduler, and quitting using; wherein the second preset threshold is smaller than the first preset threshold.

For example, after statistics of big data, a 30-second hash ring timer and a 300-second hash ring timer are set in the task scheduler, but in the current task scheduling process, the execution frequency is between 30s and 60s and between 300s and 1800s, and in addition, the execution frequency of a part of tasks is between 60s and 300s, and the hash ring timer is most suitable for the part of tasks with 60 seconds. For example, the following steps are carried out: if the execution frequency of a certain task is 90s, a matched hash ring timer cannot be found, but the task still needs to be put into a certain hash ring memory first, at this time, the closest hash ring timer, namely the 30-second hash ring timer, can be selected, and the 30-second hash ring timer puts the task onto the corresponding hash ring memory to wait for scheduling execution; and marking the task at the same time to indicate that the task cannot be matched with the most appropriate hash ring timer. Counting the number of the tasks with the marks in real time, wherein if the number of the marked tasks is small, the fact that only a few tasks cannot be matched with the most appropriate Hash ring timer is shown, and the Hash ring timer does not need to be newly built considering the resource occupation problem; if the number of marked tasks exceeds the first preset threshold (e.g., 10), it is predicted that more tasks may not match the most suitable hash ring timer, and then suitable hash ring timers and hash ring memories may be newly created according to the execution frequency of the part of marked tasks. Taking the average execution frequency as an example, if the average execution frequency of the tagged task is 100s, the timing time corresponding to the hash ring timer should be less than 100s, and at this time, a 60-second hash ring timer and a corresponding hash ring memory can be newly built, so that the accuracy requirement of the tagged task can be met. After the new construction is completed, the mark with the marking task can be deleted, so that the subsequent use is not influenced.

For a newly-built 60-second hash ring timer, counting the total number of tasks on a corresponding hash ring memory every preset time (for example, 5min), and when the total number of tasks is smaller than a second preset threshold (for example, 5), proving that the tasks with execution frequency deviating from big data are fewer, the utilization rate of the 60-second hash ring timer is lower, if the tasks are continuously used, resources are occupied, and the brought effective value is not large. In order to save resources, the rest of the tasks can be transferred to the hash ring memory corresponding to the nearest 30-second hash ring timer, and then the newly-built 60-second hash ring timer and the corresponding hash ring memory are deleted from the task scheduler and quit from use.

Further, the distribution of the execution frequency of the tasks may also have a relationship with the time period, for example, in the tasks to be scheduled, which are sent by the client in the time period a, the execution frequency of each task may be concentrated in the range of 30s to 60 s; and in the tasks needing to be scheduled, which are sent by the client in the time period B, the execution frequency of each task may be concentrated in the range of 60 s-120 s, and the like. That is, the execution frequency concentration range of tasks in different time periods may be different, and accordingly, the most suitable hash ring timer is different. In consideration of the influence of the time period on the task execution frequency distribution, an embodiment of the present invention further provides a time period switching based hash ring set dynamic adaptation adjusting method, and with reference to fig. 7, the method mainly includes the following steps:

step 401, dividing time periods, and determining one or more hash ring timers which are used most in each time period according to historical data of task scheduling.

The time periods can be divided according to actual requirements, for example, one day is taken as a unit, and each hour or every two hours is taken as a time period; or may be in units of a week, daily as a time period, etc.; and is not particularly limited herein. Taking a week as an example, dividing the week into one time period every day, if it is found through big data statistics, the execution frequency of the task is mostly concentrated between 30s and 300s in the time period of the week, and accordingly, the most suitable is to use the 30-second hash ring timer and the 60-second hash ring timer. Similarly, the hash ring timer that is most used per day on tuesdays to sundays may be determined.

Step 402, dividing one or more hash ring timers which use the most in each time period into a hash ring group, and performing time period marking on the divided hash ring groups.

Still as described in the above embodiments, it is found through big data statistics that monday uses the most 30-second hash ring timers and the most 60-second hash ring timers, so that the two hash ring timers can be divided into a hash ring group, and the hash ring group is time-period marked, i.e. marked as "monday", to characterize the group as the most used hash ring timers in the monday time period. Similarly, the hash ring timers that are most used daily from tuesday to sunday may be divided into hash ring groups, and labeled as "tuesday" to "sunday", respectively. By time period marking, subsequent calling can be facilitated.

And step 403, when the task is scheduled, starting the hash ring timer in the hash ring group with the corresponding time period mark according to the current time period, so that the corresponding hash ring timer and the hash ring memory are put into use.

Still as described in the above embodiment, when performing the task scheduling of this time, if the current time period is monday, a hash ring group with a "monday" flag is found from a plurality of hash ring groups, and hash ring timers in the hash ring group, that is, a 30-second hash ring timer and a 60-second hash ring timer, are started. As can be surmised from big data statistics, it is reasonable to use both hash ring timers during this time period of monday.

Step 404, when switching to the next time period, closing the hash ring timer in the hash ring group corresponding to the previous time period, and starting the hash ring timer in the hash ring group with the next time period mark.

Still as described in the foregoing embodiments, when monday enters tuesday, the time period is switched, and the most suitable hash ring timer may also be changed, at this time, the hash ring timers in the hash ring groups with the "monday" flag may be turned off, the hash ring groups with the "tuesday" flag may be continuously found out from the hash ring groups, and the hash ring timers in the hash ring groups may be started. By analogy, when the situation that the weekday is switched to the wednesday, the wednesday is switched to the thursday and the like, the corresponding hash ring groups are synchronously switched, and the hash ring groups with the corresponding marks can be enabled in each time period, namely the most applicable hash ring timer in the time period.

Through the combination of the time periods and the hash ring groups, the dynamic adaptation adjustment of the hash ring groups in different time periods can be completed, so that in each time period, most tasks can be guaranteed to meet the requirement of execution precision, the most matched hash ring timer is used, the effective utilization of hash ring resources can be guaranteed, and unnecessary hash ring timers are prevented from being started.

Example 5

In the task scheduling methods provided in

embodiments

2 and 3, only one hash ring timer is set in the task scheduler, and a hash ring memory is correspondingly set, where the number of slots, the slot interval, and the time for executing one rotation of the pointer in the hash ring memory are all fixed, and the number of tasks that can be accommodated in the task list of each slot at most is fixed, so that the number of tasks that can be accommodated in the whole hash ring memory at most is also fixed. Theoretically, the more slots in the hash ring memory, the more tasks can be accommodated.

After the hash ring timer is set in the task scheduler, the structure of the hash ring memory is correspondingly determined, and the hash ring timer can continuously place the task to be executed into the slot of the hash ring memory. If the number of tasks stored in the hash ring memory is large or each slot has a large number of tasks stored therein, if the tasks to be scheduled sent from the client are continuously added to the hash ring memory, the task pulling and executing efficiency may be affected. In this case, if the original hash ring storage can be switched to a hash ring storage having more slots according to the actual demand of the task amount, the above problem can be solved.

Based on the above consideration, the embodiment of the present invention further provides a task scheduling method capable of performing hash ring switching. For convenience of description, a hash ring memory currently arranged in the task scheduler is recorded as a first hash ring memory, an execution pointer and M slots which are annularly arranged at equal intervals are arranged in the first hash ring memory, and the time of the execution pointer rotating for one circle is X and is consistent with the timing time of the hash ring timer. As shown in fig. 8, the task scheduling method provided in the embodiment of the present invention mainly includes the following steps:

step 501, after receiving the task, the hash ring timer places the task in a corresponding slot of the first hash ring memory, and rotates the execution pointer on the first hash ring memory at the first frequency, thereby sequentially taking out the task whose execution time has been reached in the slot to which the execution pointer points.

The specific implementation process may refer to step 101-104 in embodiment 2, and the difference is that the hash ring memory currently set in the task scheduler is denoted as a first hash ring memory, and the fixed frequency is denoted as a first frequency, including the following steps:

1) after receiving task scheduling information sent by a client, a task receiver writes a task number and task ending time into a local cache, and writes the execution frequency and execution logic of the task into a local buffer queue; 2) a timing pulling thread pulls a task from the buffer queue at a first frequency, and a hash ring timer puts the task in a task list of a corresponding slot position of the first hash ring memory based on the execution frequency; wherein the first frequency is X/M; 3) the Hash ring timer rotates an execution pointer on the first Hash ring memory at the first frequency, and takes out the task of which the execution time is up in a task list of a slot position pointed by the execution pointer; 4) the Hash ring timer inquires whether the taken task is stopped or not to the task receiver based on the task number, and if the task is judged not to be stopped by the corresponding task ending time, the execution logic of the task is distributed to the asynchronous executor to be asynchronously executed. If the task is not stopped, the hash ring timer distributes the execution logic of the task to the asynchronous executor, and then puts the task back to the task list of the slot where the task is located before being taken out on the first hash ring memory again.

The step 3) and the step 4) are executed circularly, so that the multiple or even infinite scheduling of the tasks can be realized; the specific implementation process can refer to the description in embodiment 2, and is not described herein.

Step 502, counting the number of tasks on the first hash ring memory in real time, and when the number of tasks exceeds a preset memory amount, newly building a second hash ring memory with n × M slot positions; wherein n is more than or equal to 2, and the time of executing one circle of pointer rotation of the second hash ring memory is still X.

The preset storage capacity can be set according to the actual storage capacity or the load capacity of the first hash ring memory, if the number of tasks on the first hash ring memory exceeds the preset storage capacity, the first hash ring memory is presumed to be difficult to meet the storage requirement of the current large-batch tasks only by means of the existing slot position, and at this time, a second hash ring memory needs to be newly built for hash ring switching. The process of creating the second hash ring memory is specifically as follows:

firstly, an empty second hash ring memory is newly built, and M slots are synchronously arranged on the second hash ring memory according to the slot positions on the first hash ring memory. That is, the slot of the first hash ring memory is completely copied to the second hash ring memory, which is done to ensure that the existing task on the first hash ring memory still occupies the same slot position after being switched to the second hash ring memory, and for the M slots, the time interval between every two adjacent slots is still the same (i.e. X/M), so that the tasks can be taken out and executed on time according to the execution frequency after the hash ring memory is switched, and the original execution rule cannot be damaged.

And then, according to the task quantity requirement, adding n-1 slot positions between every two adjacent slot positions of the second hash ring memory according to the principle of equal interval arrangement to obtain the second hash ring memory with the slot position quantity of n × M. The more the number of tasks needing to be stored is, the larger the n value is selected, and therefore the number of the slots can be expanded more. Because M slot positions which are set first need to be set to be consistent with the slot positions on the first hash ring memory, and newly-added slot positions can only be additionally arranged between two adjacent slot positions, the number of slot positions of the second hash ring memory is always in a proportional relation with the number of slot positions of the first hash ring memory, namely multiple switching.

Step 503, copying each task on the first hash ring memory to a corresponding slot on the second hash ring memory according to the slot position corresponding relationship.

And copying according to the corresponding relation of the slot positions, namely correspondingly copying each task on the first hash ring memory to M slot positions which are arranged in the second hash ring memory in advance, so that the same slot position is occupied on the hash ring before and after the copying of each task. And after the copying is finished, no task is stored in the newly added slot position, and when a subsequent new task needs to be stored, the new slot position is placed in the newly added slot position to share the storage pressure of the original slot position.

Step 504, after the replication is completed, deleting the task which is taken out from the first hash ring memory and executed in the replication process from the corresponding slot position of the second hash ring memory, so that the second hash ring memory is used, and the first hash ring memory is quitted to be used.

In the process of copying the task, the execution pointer on the first hash ring memory still rotates, and when the execution pointer rotates to a certain slot position, the task from the slot position to the execution time is still taken out for execution; this part of the task will also be copied to the second hash ring memory, but since execution has already been completed on the first hash ring memory, this part of the task can be temporarily deleted in the second hash ring memory. For the position of the execution pointer, it can be determined by: if the current execution pointer on the first hash ring memory points to a first position, calculating the time required for copying each task on the first hash ring memory to the second hash ring memory, and determining the angle of the execution pointer rotated on the first hash ring memory in the time; and after the task is copied, rotating the execution pointer from the first position by the angle to serve as a second position, namely the slot position pointed by the execution pointer on the second hash ring memory.

In addition, for the part of the task which is fetched and executed from the first hash ring memory in the copying process, if the task is not stopped, after the hash ring timer distributes the execution logic of the task to the asynchronous executor, because the first hash ring memory is already out of use, the task needs to be put back into the task list of the corresponding slot on the second hash ring memory, and the task can be fetched and executed from the second hash ring memory directly during subsequent scheduling.

After the second hash ring memory is put into use, because the hash ring structure of the memory changes, the frequency of the timing pull thread pulling task and the frequency of the hash ring timer rotating execution pointer both change correspondingly, and the task scheduling method is performed according to the following steps:

1) a timing pulling thread pulls the task from the buffer queue at a second frequency, and the hash ring timer puts the task in a task list of the corresponding slot position of the second hash ring memory based on the execution frequency; wherein the second frequency X/(n M); 2) the Hash ring timer rotates the execution pointer on the second Hash ring memory at the second frequency, and takes out the task of which the execution time is up in the task list of the slot position pointed by the execution pointer; 3) the Hash ring timer inquires whether the taken task is stopped or not to the task receiver based on the task number, and if the task is judged not to be stopped by the corresponding task ending time, the execution logic of the task is distributed to the asynchronous executor to be asynchronously executed. If the task is not stopped, the hash ring timer distributes the execution logic of the task to the asynchronous executor, and then the task is put back to the task list of the slot position where the task is located before being taken out on the second hash ring memory again. Except that the task pull frequency and the pointer rotation frequency are changed, the other methods are the same as those when the first hash ring memory is used, and reference may be specifically made to the related description in embodiment 2, which is not described herein again.

Further, in the above embodiment, the task on the first hash ring memory is completely copied, and then the task that has been executed by the copying process on the first hash ring memory is selectively deleted. In an alternative embodiment, the selection may be performed in advance, and then the copy switching is selectively performed, and a specific task scheduling process may refer to fig. 9, which is as follows:

step 501, after receiving the task, the hash ring timer places the task in a corresponding slot of the first hash ring memory, and rotates the execution pointer on the first hash ring memory at the first frequency, thereby sequentially taking out the task whose execution time has been reached in the slot to which the execution pointer points. The specific implementation process is the same as step 501 in the previous embodiment, and is not described herein.

Step 502, counting the number of tasks on the first hash ring memory in real time, and when the number of tasks exceeds a preset memory amount, newly building a second hash ring memory with n × M slot positions; wherein n is more than or equal to 2, and the time of executing one circle of pointer rotation of the second hash ring memory is still X. The specific implementation process is the same as step 502 in the previous embodiment, and is not described herein.

Step 503', calculating the time required for copying each task on the first hash ring memory to the second hash ring memory, and marking the time within which the executed task can be taken out from the first hash ring memory.

The time may be calculated in units of time intervals between every two adjacent slots on the first hash ring memory, that is, several time intervals are required for the replication process, then statistics is performed on which tasks are taken out of the corresponding slots during the several time intervals, and the counted tasks are marked for subsequent screening.

And step 504', copying the unmarked task to the corresponding slot position on the second hash ring memory according to the slot position corresponding relation, so that the second hash ring memory is put into use, and the first hash ring memory is quitted from use.

The marked part of tasks can be taken out from the first hash ring memory to be executed in the copying process, so that the part of tasks can be ignored in the copying process, and the rest tasks which are not marked can be directly copied into the second hash ring memory; after the copying is completed, the location of the execution pointer on the second hash ring memory is determined, and the determining method may refer to the description of the foregoing embodiment, which is not described herein again. Correspondingly, for the part of the tasks which are not copied, if the tasks are not stopped, the hash ring timer distributes the execution logic of the tasks to the asynchronous executer, and since the first hash ring memory is already out of use, the tasks need to be put back into the task list of the corresponding slot on the second hash ring memory, and the tasks are directly taken out from the second hash ring memory for execution during subsequent scheduling. The task scheduling method after the second hash ring memory is used can be described with reference to the foregoing embodiments, and details are not described here.

By the hash ring switching method, the storage pressure of the original hash ring memory can be reduced, namely the storage pressure of the original slot position is shared, and each task is more uniformly distributed on the newly-added slot position on the newly-built hash ring memory. The description is made in particular by the following examples:

as shown in fig. 10, in a specific embodiment, the original first hash ring memory is provided with 4 slots arranged at equal intervals in a ring shape, which are respectively denoted as

slots

1, 3, 5, and 7, as shown in the left drawing; and if the time for executing one circle of rotation of the pointer is 120s, the time interval between every two adjacent slot positions is 30s, the execution pointer moves to the next slot position every 30s, and the task on the next slot position is taken out for execution. At present, due to the fact that the task amount is too large, the number of the slot positions on the first hash ring memory is difficult to meet the requirement, and the number of the slot positions needs to be doubled, a second hash ring memory is newly built, the slot positions 1, 3, 5 and 7 are arranged on the second hash ring memory and are consistent with the slot positions on the first hash ring memory; then, a slot 2 is provided between the

slots

1 and 3, a slot 4 is provided between the

slots

3 and 5, a slot 6 is provided between the

slots

5 and 7, and a slot 8 is provided between the

slots

7 and 1, so that a second hash ring memory as shown in the right drawing is obtained, and 8 slots which are arranged at equal intervals in a ring shape are shared.

In the second hash ring memory, the time of executing the pointer to rotate for one circle is still 120s, the number of the slot positions is changed to 8, the time interval between every two adjacent slot positions is 15s, the executing pointer moves to the next slot position every 15s, and the task on the next slot position is taken out to execute. Taking clockwise rotation of the execution pointer in the hash ring memory as an example, it is assumed that the execution pointer currently points to the slot 5 in the first hash ring memory, as shown in the left diagram; a time interval, i.e., 30s, is required for the copy task, and after the copy is completed, the execution pointer should point to slot 7 in the second hash ring memory, as shown in the right diagram.

When hash ring switching is not performed, assuming that the hash ring timer puts a task into the first hash ring memory, after calculation, the task is found to be put between the slot position 1 and the slot position 3, and the task is more biased to the slot position 1, so that the hash ring timer can only put the task into the slot position 1; after the second hash ring memory is switched to, because the slot position 2 is newly added between the slot position 1 and the slot position 3, if the optimal position of the task is about the slot position 2, the task can be placed at the slot position 2 of the second hash ring memory, so that the storage pressure of the original slot position can be shared. Similarly, other tasks can be preferentially placed into the newly added slot positions during storage, so that the tasks can be more uniformly distributed on the slots of the second hash ring memory.

Through the hash ring switching method, when the actual task amount is excessive and the existing hash ring memory is difficult to meet the use requirement, the slot positions can be expanded according to the requirement of the actual task amount, namely, a hash ring memory with more slot positions is newly built on the basis of the original hash ring memory, the tasks to be executed are switched to the newly built hash ring memory according to the slot position corresponding relation, slot position expansion is carried out on the basis of not damaging the original execution rule, the storage pressure of the original slot positions can be shared during subsequent task storage, all the tasks are uniformly distributed on all the slot positions of the newly built hash ring memory, and the pulling and execution efficiency of the tasks can be effectively ensured.

It will be understood by those skilled in the art that the foregoing is only a preferred embodiment of the present invention, and is not intended to limit the invention, and that any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims

1. A task scheduling method based on a plurality of hash rings with different precisions is characterized in that a task scheduler is provided with a plurality of hash ring timers with different precisions, each hash ring timer corresponds to a hash ring memory, the hash ring memory is provided with an execution pointer and a plurality of slots which are annularly arranged at equal intervals, and the time of the execution pointer rotating for one circle is consistent with the timing time of the corresponding hash ring timer; the task scheduling method comprises the following steps:

2. The method for scheduling tasks based on multiple hash rings with different precisions according to claim 1, wherein the timing pull thread pulls the task from the buffer queue at a fixed frequency, and selects a hash ring timer matching with the timing pull thread according to the execution frequency of the task, and the hash ring timer puts the task in a task list corresponding to a slot on a hash ring memory, specifically:

3. The method for scheduling tasks based on multiple hash rings with different precisions according to claim 1, wherein after the timing pull thread pulls the task from the buffer queue at a fixed frequency, if a matching hash ring timer cannot be found according to the execution frequency of the task, the method further comprises:

4. The method for task scheduling based on multiple hash rings with different precisions according to claim 3, wherein after the newly built hash ring timer and the corresponding hash ring memory are put into use, the method further comprises:

5. The method for task scheduling based on multiple hash rings with different precisions according to claim 3, wherein the step of creating a hash ring timer and a corresponding hash ring memory according to the execution frequency of each marked task specifically comprises:

6. The method of task scheduling based on multiple different precision hash rings of claim 1, wherein before the timing pull thread pulls tasks from the buffer queue at a fixed frequency, the method further comprises:

7. The method for scheduling tasks according to any of claims 1 to 6, wherein four hash ring timers with different precisions are set in the task scheduler, and the four hash ring timers with different precisions are respectively a 30-second hash ring timer, a 60-second hash ring timer, a 300-second hash ring timer and an 1800-second hash ring timer, and correspond to four different hash ring memories; the hash ring timer selected according to the execution frequency of the task specifically includes:

8. The method for scheduling tasks based on multiple hash rings with different precisions as claimed in any one of claims 1 to 6, wherein if the task is not stopped, the hash ring timer distributes the execution logic of the task to the asynchronous executor, and then puts the task back to the task list of the slot of the corresponding hash ring memory where the task was located before being fetched.

9. A task scheduling device based on a plurality of hash rings with different precisions is characterized by comprising a task receiver and a task scheduler, wherein the task receiver is used for receiving and storing tasks sent by a client, and the task scheduler is used for scheduling tasks according to the tasks stored in the task receiver;

10. The task scheduler based on multiple hash rings with different precisions of claim 9, wherein each hash ring memory is a ring queue + task list data structure;