WO2022257435A1 - Procédé et appareil de traitement de lot de tâches, dispositif informatique et support de stockage - Google Patents

Procédé et appareil de traitement de lot de tâches, dispositif informatique et support de stockage Download PDF

Info

Publication number
WO2022257435A1
WO2022257435A1 PCT/CN2021/142108 CN2021142108W WO2022257435A1 WO 2022257435 A1 WO2022257435 A1 WO 2022257435A1 CN 2021142108 W CN2021142108 W CN 2021142108W WO 2022257435 A1 WO2022257435 A1 WO 2022257435A1
Authority
WO
WIPO (PCT)
Prior art keywords
task
preset
running time
tasks
target
Prior art date
Application number
PCT/CN2021/142108
Other languages
English (en)
Chinese (zh)
Inventor
曹威
Original Assignee
深圳前海微众银行股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳前海微众银行股份有限公司 filed Critical 深圳前海微众银行股份有限公司
Publication of WO2022257435A1 publication Critical patent/WO2022257435A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues

Definitions

  • the embodiments of the present application relate to the field of financial technology (Fintech), and in particular, to a batch task processing method, device, computing device, and storage medium.
  • Fetech financial technology
  • a new execution time is set based on the experience of the operation and maintenance personnel, and the new execution time is uploaded, so as to realize the adjustment of the task execution time.
  • the distribution of tasks in each preset queue is unbalanced, resulting in some preset queues being too idle when running tasks, and some preset queues being too busy when running tasks; Tasks that need to be executed first are queued to the back of the preset queue, and some tasks that do not need to be executed first are queued to the front of the preset queue.
  • the embodiment of the present application provides a batch task processing method, device, computing device and storage medium, so as to reasonably adjust the execution time of the batch task.
  • the embodiment of the present application provides a batch task processing method, the method comprising:
  • any preset queue according to the historical average starting running time and the first historical average running time of each task in the preset queue within the first period of time, determine the target starting running of each task in the preset queue Time, and based on the target starting running time, run the corresponding task.
  • the first historical average running time of the task represents the length of the running time of the task in the first period, to a certain extent, it reflects the time required for subsequent running of the task, so according to the first historical average
  • the running time and the number of preset queues assign tasks to each preset queue, so that each preset queue will not be too idle or too busy when running tasks; due to the historical average start running time of a task, it represents that the task is
  • the order of the initial running of the first period reflects to a certain extent the order of the subsequent initial running of the task.
  • the preset The target starting running time of each task in the queue, and then run the tasks in the preset queue in a more reasonable sequence, and reasonably adjust the execution time of each task in the batch task.
  • assigning the tasks to each of the preset queues based on the first historical average running time of each task in the batch task within the first period of time and the number of preset queues includes:
  • the target running time is based on The sum of the first historical average running time of all tasks in the batch task and the number of the preset queues are determined.
  • the tasks can be evenly distributed to each preset queue according to the number; since the first historical average operation of all tasks in the preset queue When the sum of the durations is too large, tasks are no longer assigned to the preset queue, which ensures the balance of the sum of the first historical average running time of all tasks in each preset queue, so that the first historical average running time of all tasks in each preset queue The sum of historical average running time will not be too much or too little, which further ensures that each preset queue will not be too idle or too busy when running tasks.
  • select tasks with the number of preset queues from the batch tasks and assign them to each of the preset queues including:
  • the median The value is the median value of the first historical average running time of the batch task
  • the selected tasks are assigned to each In the preset queue; by selecting the task corresponding to the first historical average running time of the number of preset queues adjacent to the median after the median value, the selected task will be selected according to the first historical average running time Allocate to each of the preset queues sequentially from shortest to longest, so that each batch assigns tasks to each of the preset queues in a relatively balanced manner according to the first historical average running time.
  • determining the target initial running time of each task in the preset queue according to the historical average initial running time and the first historical average running time of each task in the preset queue within the first period of time including:
  • any target task is a task that other tasks depend on;
  • the tasks other than the target task in the preset queue are arranged after the target task, and the tasks other than the target task are sorted from front to back according to the historical average starting running time; or
  • the target start running time of the previous task in the preset queue and the first historical average running time determine the target start running time of the next task, wherein the target start time of the first task in the preset queue
  • the running time is the corresponding historical average starting running time, or the preset starting running time.
  • the number of times the target task is depended on represents how many other tasks depend on the target task.
  • the target tasks that depend on a large number of times ensure the normal operation of more other tasks; since tasks other than the target task do not affect the normal operation of other tasks, these tasks are ranked after the target task; in addition, the history of these tasks
  • the average initial running time reflects to a certain extent the order in which these tasks are subsequently started to run. Therefore, these tasks are sorted according to the historical average initial running time from front to back.
  • the historical average starting running time of each task represents the order in which the task starts to run in the first period, and to a certain extent reflects the order in which the subsequent tasks start to run. Therefore, Sort each task from front to back according to the historical average start running time. After the tasks in the preset queue are sorted, the order of running each task is determined. Since the first historical average running time of each task reflects to a certain extent the time required for subsequent running of the task, so according to The target initial running time of the previous task in the preset queue and the first historical average running time can reasonably determine the target initial running time of the next task. By using the historical average starting running time of the first task in the preset queue, or the preset starting running time as its target starting running time, it is applicable to the needs of different scenarios.
  • the method before running the corresponding task based on the target starting running time, the method further includes:
  • the second time period is a time period before the first time period
  • the second time period is the time period before the first time period, if the first historical average running time of the task executed by a single thread is longer than the second history of the task executed by the single thread in the second time period
  • the increase in the average running time is greater than the preset increase, indicating that the running time of the task executed by the single thread increases more.
  • the method also includes:
  • the task corresponding to the rerun instruction is executed.
  • the information of the corresponding preset characterization mechanism and the corresponding task can be determined; thus, the corresponding task can be executed based on the information of the corresponding preset characterization mechanism, without re-running the task every time. Write rerun code while running.
  • running corresponding tasks includes:
  • the target dependency rule corresponding to the task based on the preset corresponding relationship; wherein the corresponding relationship includes Association of preset dependency information and dependency rules;
  • the target dependency rule determine whether to execute and complete the corresponding target task based on the information of the preset characterization mechanism, wherein the corresponding target task is the task on which the execution of the task depends. task;
  • the data to be processed corresponding to the information of the preset characterization mechanism is processed according to the calculation information of the task.
  • the corresponding dependency rules can be directly determined according to the task dependency information, so that when writing the calculation information of the task, there is no need to write the task
  • the corresponding dependency rules facilitate the management of dependency rules.
  • the embodiment of the present application also provides a batch task processing device, including:
  • a task allocation unit configured to allocate the tasks to each of the preset queues based on the first historical average running time of each task in the batch task within the first period of time and the number of preset queues;
  • the operation processing unit is configured to, for any preset queue, determine each task in the preset queue according to the historical average starting running time and the first historical average running time of each task in the preset queue within the first period of time.
  • the target start running time of the task and based on the target start running time, run the corresponding task.
  • the embodiment of the present application provides a computing device, including at least one processor and at least one memory, wherein the memory stores a computer program, and when the program is executed by the processor, the processing The server executes the batch task processing method described in any one of the above first aspects.
  • an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by a computing device, and when the program runs on the computing device, the computing device executes the above-mentioned first The batch task processing method described in any one of the aspects.
  • FIG. 1 is a schematic flow diagram of the first batch task processing method provided by the embodiment of the present application.
  • FIG. 2 is a schematic flow diagram of the second batch task processing method provided by the embodiment of the present application.
  • FIG. 3 is a schematic flowchart of a third batch task processing method provided in the embodiment of the present application.
  • FIG. 4 is a schematic flowchart of a fourth batch task processing method provided by the embodiment of the present application.
  • FIG. 5 is a schematic flowchart of a fifth batch task processing method provided by the embodiment of the present application.
  • FIG. 6 is a schematic structural diagram of a batch task processing device provided in an embodiment of the present application.
  • FIG. 7 is a schematic structural diagram of a computing device provided by an embodiment of the present application.
  • a and/or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. three conditions.
  • the character "/" generally indicates that the contextual objects are an "or" relationship.
  • first and second are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, a feature defined as “first” and “second” may explicitly or implicitly include one or more of these features. In the description of the present application, unless otherwise specified, "plurality” means two or more.
  • connection should be understood in a broad sense, for example, it may be a direct connection, or an indirect connection through an intermediary, or an internal connection between two devices.
  • execution time refers to the start running time of the task, and in some embodiments, "start running time” is directly used to refer to the "execution time”.
  • an embodiment of the present application proposes a batch task processing method, device, computing device, and storage medium, the method including: based on the first historical average running time of each task in the batch task within the first period of time, and the preset queue , assign the tasks to each of the preset queues; for any preset queue, according to the historical average starting running time of each task in the preset queue within the first period of time and the first historical average
  • the running time is determined by determining the target starting running time of each task in the preset queue, and running the corresponding task based on the target starting running time.
  • the first historical average running time of a task represents the length of the task’s running time in the first period, to a certain extent, it reflects the time required to run the task in the future, so according to the first historical average running time and preset The number of queues, assign tasks to each preset queue, so that each preset queue will not be too idle or too busy when running tasks; because the historical average start running time of a task, it represents the starting time of the task in the first period
  • the order in which the task starts to run reflects to a certain extent the order in which the task will start to run in the future. Therefore, according to the historical average initial running time and the first historical average running time of each task in the preset queue, determine the order of each task in the preset queue. The target start running time, and then run the tasks in the preset queue in a more reasonable order, and reasonably adjust the execution time of each task in the batch task.
  • the embodiment of the present application provides the first batch task processing method, as shown in Figure 1, including the following steps:
  • Step S101 Based on the first historical average running time of each task in the batch task within a first period and the number of preset queues, assign the tasks to each of the preset queues.
  • the first historical average running time of a task represents the length of the task’s running time in the first period, to a certain extent, it reflects the time required to run the task in the future, so according to the first historical average running time and preset
  • the number of queues determines the tasks in each preset queue, so that each preset queue will not be too idle or too busy when running tasks.
  • the above-mentioned first time period can be set according to the actual application scenario, such as a week before the current moment; correspondingly, the first historical average running time of any task is the average time of running the task every day in this week.
  • the first historical average running time of task 1 Among them, A 11 is the running time of task 1 on the first day of this week, A 12 is the running time of task 1 on the second day of this week, A 13 is the running time of task 1 on the third day of this week, A 14 is the running time of task 1 on the fourth day of the week, A 15 is the running time of task 1 on the fifth day of the week, A 16 is the running time of task 1 on the sixth day of the week, A 17 It is the running time of task 1 on the seventh day of the week.
  • This embodiment does not limit the specific implementation manner of determining the tasks in each preset queue.
  • the tasks in the preset queues meet at least one of the following conditions:
  • the ratio of the sum of the first historical average running time of all tasks in any preset queue to the target running time is less than the preset ratio; wherein, the target running time is based on the first history of all tasks in the batch task
  • the sum of the average running time and the number of the preset queues are determined. Taking a batch task including 100 tasks, there are 4 preset queues (denoted as preset queue a, preset queue b, preset queue c, and preset queue d), and the preset ratio is 1.25 as an example:
  • the sum of the first historical average running time of these 100 tasks is recorded as Target runtime
  • the sum of the first historical average running time of all tasks in the preset queue a is denoted as A a
  • the sum of the first historical average running time of all tasks in the preset queue b is denoted as A b
  • the sum of the first historical average running time of all tasks in the preset queue c is denoted as A a .
  • the sum of the first historical average running time is recorded as A c
  • the sum of the first historical average running time of all tasks in the preset queue d is recorded as A d ; among them, A a /A 0 ⁇ 1.25, A b /A 0 ⁇ 1.25, A c /A 0 ⁇ 1.25, A d /A 0 ⁇ 1.25.
  • the ratio of the sum of the first historical average running time of all tasks in each preset queue to the target running time is less than the above preset ratio, the sum of the first historical average running time of all tasks in each preset queue will not Too much or too little (for example, the difference between A a , A b , A c and A d mentioned above will not be too large), which further ensures that each preset queue will not be too idle or too busy when running tasks.
  • the number of tasks in the preset queue a is denoted as B a
  • B b the number of tasks in the preset queue b
  • B c the number of tasks in the preset queue c
  • B d the number of tasks in the preset queue d ; among them,
  • the number of tasks in each preset queue will not be too much or too little (such as the above-mentioned B a , B b , B c and The difference between B and d will not be too large), which ensures the balance of tasks in each preset queue.
  • the above-mentioned batch tasks include 100 tasks and there are 4 preset queues as an example, the application provides the following two ways of determining the tasks in each preset queue as examples :
  • the tasks corresponding to the 4 first historical average runtimes adjacent to the median value before the median value i.e. the above task 46, task 45, task 44 and task 43
  • the 4 first historical average running durations correspond to
  • the tasks i.e. the above task 55, task 56, task 57 and task 58
  • the task in the preset queue will no longer be selected; as in the above preset queue
  • the ratio of the sum of the first historical running time of the task in b to the target running time reaches the preset ratio, and from the remaining tasks, select the 3 first historical average running times before the median value and adjacent to the median value
  • the tasks corresponding to the duration are respectively regarded as tasks in the preset queue a, preset queue c and preset queue d; and select the tasks corresponding to the first three historical average running durations after the median value and adjacent to the median value , respectively as tasks in the preset queue a, preset queue c, and preset queue d, until the preset queues corresponding to the 100 tasks are determined.
  • four tasks i.e. the above-mentioned task 1, task 2, task 3 and task 4 are sequentially selected as tasks in the preset queue a, preset queue b, preset queue c and preset queue d;
  • the task in the preset queue will no longer be selected; as in the above preset queue
  • the ratio of the sum of the first historical running time of the tasks in a to the target running time reaches the preset ratio, and from the remaining tasks, three tasks are selected in turn as the preset queue b, the preset queue c and the preset queue respectively d, until the preset queue corresponding to each of the 100 tasks is determined.
  • the above method 1 and method 2 are to sort the batch tasks according to the first historical average running time from short to long. In some embodiments, the batch tasks can also be sorted according to the first historical average running time from long to short. For the manner of determining the tasks in each preset queue, reference may be made to the above-mentioned method 1 and method 2, which will not be repeated here.
  • this embodiment does not limit the way to trigger the execution of S101, which can be triggered based on user instructions; or triggered at fixed intervals, that is, the batch task processing method is executed regularly every cycle, for example, the batch task processing method is executed once every morning to determine this The target start run time for each task during the day.
  • Step S102 For any preset queue, determine the target of each task in the preset queue according to the historical average starting running time and the first historical average running time of each task in the preset queue within the first period of time start running time, and run corresponding tasks based on the target start running time.
  • the historical average start running time of a task represents the order in which the task starts to run in the first period, and to a certain extent reflects the order in which the task starts to run in the subsequent period, so according to the historical average of each task in the preset queue
  • the initial running time and the first historical average running time determine the target starting running time of each task in the preset queue, and run the tasks in the preset queue in a relatively reasonable order.
  • the first time period can be set according to the actual application scenario, or take the week before the current moment as an example.
  • the historical average starting running time of any task is the starting running time of each day in this week Average time for this task.
  • the historical average starting running time of task 1 C 11 is the time to start running task 1 on the first day of this week
  • C 12 is the time to start running task 1 on the second day of this week
  • C 13 is the time to start running task on the third day of this week 1
  • C 14 is the time to start running task 1 on the fourth day of the week
  • C 15 is the time to start running task 1 on the fifth day of the week
  • C 16 is the time to start running task 1 on the sixth day of the week
  • the time to start running task 1 C 17 is the time to start running task 1 on the seventh day of the week.
  • the device for executing the above method will load the batch tasks one by one into the job scheduling framework (such as Quartz) at startup, and manage them through the container framework (such as Spring).
  • the job scheduling framework such as Quartz
  • the container framework such as Spring
  • the job scheduling framework such as Quartz
  • the container framework such as Spring
  • the first historical average running time of the task represents the length of the running time of the task in the first period, to a certain extent, it reflects the time required for subsequent running of the task, so according to the first historical average
  • the running time and the number of preset queues determine the tasks in each preset queue, so that each preset queue will not be too idle or too busy when running tasks; due to the historical average start running time of a task, it represents that the task is
  • the order of the initial operation of a period reflects the order of the subsequent initial operation of the task to a certain extent. Therefore, the default queue is determined according to the historical average initial running time and the first historical average running time of each task in the preset queue.
  • the target starting running time of each task in the batch task is to run the tasks in the preset queue in a reasonable order, so as to reasonably adjust the execution time of each task in the batch task (that is, the starting running time).
  • the embodiment of the present application provides a second batch task processing method, as shown in Figure 2, including the following steps:
  • Step S201 Based on the first historical average running time of each task in the batch task within a first period and the number of preset queues, assign the tasks to each of the preset queues.
  • step S201 For the specific implementation manner of this step S201, reference may be made to the above step S101, which will not be repeated here.
  • Step S202 For any preset queue, if there are target tasks in the preset queue, sort the target tasks in the preset queue according to the number of dependencies from most to least.
  • the number of times the target task is depended on indicates how many other tasks depend on the target task.
  • target tasks in the preset queue a there are 5 target tasks in the preset queue a (respectively recorded as target task 1, target task 2, target task 3, target task 4, and target task 5); wherein, the number of times target task 1 is dependent is 5 , the number of times that target task 2 is dependent is 3, the number of times that target task 3 is dependent is 1, the number of times that target task 4 is dependent is 6, and the number of times that target task 5 is dependent is 2.
  • the ranking results of these 5 target tasks are : Target task 4, Target task 1, Target task 2, Target task 5, Target task 3.
  • some target tasks in the preset queue are relied on for the same number of times.
  • the target tasks with the same number of times of dependence can be randomly sorted; Sort by start time from front to back.
  • Step S203 Arranging the tasks other than the target task in the preset queue after the target task, and sorting the tasks other than the target task according to the historical average starting running time from front to back .
  • the historical average initial running time of these tasks reflects to a certain extent the order in which these tasks are subsequently started to run. Therefore, these tasks can be sorted according to the historical average initial running time from front to back.
  • the sorting results of all tasks in the preset queue a are: target task 4, target task 1, target task 2, target task 5, target task 3, common task 2, common task 7, common task 6, common task 1, common task 4.
  • Ordinary tasks 5.
  • Step S204 Determine the target starting running time of the next task according to the target starting running time of the previous task in the preset queue and the first historical average running time, wherein the first task in the preset queue
  • the target starting running time is the corresponding historical average starting running time, or the preset starting running time.
  • the order of running each task is determined. Since the first historical average running time of each task reflects to a certain extent the time required for subsequent running of the task, so according to The target initial running time of the previous task in the preset queue and the first historical average running time can reasonably determine the target initial running time of the next task.
  • This embodiment does not limit the specific implementation of determining the target initial running time of the next task based on the target initial running time of the previous task in the preset queue and the first historical average running time. For example, The time corresponding to the first historical average running time after the target starting running time of the previous task is used as the target starting running time for determining the next task.
  • Step S205 Run the corresponding task based on the target start running time.
  • the number of times the target task is depended on represents how many other tasks depend on the target task.
  • the target tasks that depend on a large number of times ensure the normal operation of more other tasks; since tasks other than the target task do not affect the normal operation of other tasks, these tasks are ranked after the target task; in addition, the history of these tasks
  • the average initial running time reflects to a certain extent the order in which these tasks are subsequently started to run. Therefore, these tasks are sorted according to the historical average initial running time from front to back. After the tasks in the preset queue are sorted, the order of running the tasks is determined.
  • the target initial running time of the next task can be reasonably determined.
  • the historical average starting running time of the first task in the preset queue, or the preset starting running time as its target starting running time it is applicable to the needs of different scenarios.
  • the embodiment of the present application provides a third batch task processing method, as shown in FIG. 3 , including the following steps:
  • Step S301 Based on the first historical average running time of each task in the batch task within a first period and the number of preset queues, assign the tasks to each of the preset queues.
  • Step S302 If there is no target task in the preset queue, sort each task in the preset queue according to the historical average starting running time from front to back.
  • Step S303 Determine the target starting running time of the next task according to the target starting running time of the previous task in the preset queue and the first historical average running time, wherein the first task in the preset queue
  • the target starting running time is the corresponding historical average starting running time, or the preset starting running time.
  • Step S304 Run the corresponding task based on the target start running time.
  • the historical average start running time of the task represents the order of the start running of the task in the first period, and reflects to a certain extent the time for the subsequent start running of the task. Sequentially, therefore, the tasks are sorted from front to back according to the historical average start running time. After the tasks in the preset queue are sorted, the order of running each task is determined. Since the first historical average running time of each task reflects to a certain extent the time required for subsequent running of the task, so according to The target initial running time of the previous task in the preset queue and the first historical average running time can reasonably determine the target initial running time of the next task. By using the historical average starting running time of the first task in the preset queue, or the preset starting running time as its target starting running time, it is applicable to the needs of different scenarios.
  • the embodiment of the present application provides a fourth batch task processing method, as shown in Figure 4, including the following steps:
  • Step S401 Based on the first historical average running time of each task in the batch task within a first period and the number of preset queues, assign the tasks to each of the preset queues.
  • step S401 may refer to the above-mentioned embodiments, which will not be repeated here.
  • Step S402 Comparing the first historical average running time of tasks executed by a single thread with the second historical average running time within a second time period, the second time period is a time period before the first time period.
  • the above-mentioned second time period is a time period before the first time period, and the duration of the second time period and the first time period may be the same or different, and there may be one or more second time periods. If there are multiple second time periods, it is necessary to compare the first historical average running time of the single-threaded task with the second historical average running time of the single-threaded task in each second time period.
  • the second period may include at least one of the following:
  • One week before the current time and one week ahead, for example, the current time is the early morning of the third Wednesday in May; the first period is from the early morning of the second Wednesday in May to the early morning of the third Wednesday in May ; The second period is from the early morning of the first Wednesday in May to the early morning of the second Wednesday in May.
  • the current time is the early morning of the third Wednesday in May; the first period is from the early morning of the second Wednesday in May to the early morning of the third Wednesday in May ; The second period is from the early morning of the second Wednesday in April to the early morning of the third Wednesday in April.
  • the above-mentioned second time period is only an illustration, and the number of the second time period and the specific second time period can be set according to the actual application scenario.
  • Step S403 If the increase of the first historical average running time compared to the second historical average running time is greater than a preset increase, switch the single-threaded execution task to multi-threaded execution.
  • the first historical average running time of the task executed by the single thread if there are multiple second time periods, it is necessary to compare the first historical average running time of the task executed by the single thread with the second historical average running time of the task executed by the single thread in each second time period. right.
  • the increase of the second historical average running time in part or all of the second period is greater than the preset increase, and the corresponding single-threaded execution task is switched to multi-threaded execution .
  • the increase in the first historical average running time of a task executed by a single thread compared to the second historical average running time of a task executed by a single thread in the second period reflects how much the running time of a task executed by a single thread has increased . If the first historical average running time of the task executed by a single thread is greater than the preset increase compared with the second historical average running time of the task executed by the single thread in the second period, it means that the task executed by the single thread is running The duration increases more.
  • the task executed by the above single thread can be executed by multiple threads, in order to avoid the time required for the subsequent execution of the task to continue to be too long, the task executed by the single thread can be switched to multi-threaded execution to reduce the need for subsequent execution of the task duration.
  • Step S404 For any preset queue, determine the target of each task in the preset queue according to the historical average starting running time and the first historical average running time of each task in the preset queue within the first period of time start running time, and run corresponding tasks based on the target start running time.
  • the second time period is the time period before the first time period, if the first historical average running time of the task executed by a single thread is longer than the second history of the task executed by the single thread in the second time period
  • the increase in the average running time is greater than the preset increase, indicating that the running time of the task executed by the single thread increases more.
  • the embodiment of the present application provides a fifth batch task processing method, as shown in Figure 5, including the following steps:
  • Step S501 Based on the first historical average running time of each task in the batch task within the first period and the number of preset queues, assign the tasks to each of the preset queues.
  • Step S502 For any preset queue, determine the target of each task in the preset queue according to the historical average starting running time and the first historical average running time of each task in the preset queue within the first period of time Start run time.
  • Step S503 For any task, after the target start running time of the task is reached, the task is executed sequentially based on the information of each preset characterization mechanism.
  • Multi-tenant architecture In order to achieve shared development and maintenance costs, multi-tenant architecture is widely used in the financial field. Multi-tenant architecture shares the same system or program components in a multi-tenant (institutional) environment and ensures the isolation of data between institutions.
  • the organization information corresponding to the task needs to be configured, and the organization corresponding to the task is determined when the task is executed, and the task is executed based on the corresponding organization.
  • developers may forget to configure the organization information corresponding to some tasks in the batch tasks, so that the data isolation of different organizations cannot be guaranteed.
  • this embodiment presets the relevant information of the characterization mechanism in advance in the device that executes the above method, so that when the developer writes the calculation information of the task, he does not need to write the mechanism information corresponding to the task, and there will be no omissions.
  • a task base class F is implemented in the device for executing the batch task processing method, and an abstract method M is provided for task scheduling of a specific organization, and all tasks inherit from the task base class F.
  • any task is executed, it is dispatched to the unified entry of the above-mentioned task base class F; after entering this entry, it first traverses the relevant information of all preset characterization mechanisms, and then calls it in the M method based on the information of each preset characterization mechanism The N method of the interface I, so as to execute the corresponding task (calculate the data to be processed corresponding to the information of the preset representation mechanism through the calculation information of the task to obtain the calculation data).
  • the execution of some tasks needs to depend on the target task, and for the information of any preset representation mechanism, the task can be performed in the following ways:
  • the target dependency rule corresponding to the task based on the preset corresponding relationship; wherein the corresponding relationship includes Association of preset dependency information and dependency rules;
  • the target dependency rule determine whether to execute and complete the corresponding target task based on the information of the preset characterization mechanism, wherein the corresponding target task is the task on which the execution of the task depends. task;
  • the data to be processed corresponding to the information of the preset characterization mechanism is processed.
  • variable attributes of the interface I defined in the task will be dispatched to the base class S when processing a single organization (taking organization A as an example) (that is, based on each predetermined Assuming the information of the representation mechanism, when the M method is called by the N method of the interface I), through the base class S, first determine whether there is dependency information corresponding to task 1, if there is dependency information, according to the preset corresponding relationship (preset dependency information Association with dependency rules), determine the target dependency rules corresponding to task 1 (during implementation, multiple dependency rules can be preset, such as type of batch task dependencies, file transfer task dependencies between institutions, internal data synchronization task dependencies, etc.
  • the initial running time of the target task is obtained, and then it is determined whether the corresponding target task is completed based on the execution of the above-mentioned mechanism A. If the corresponding target task is completed based on the execution of the above-mentioned mechanism A, according to the calculation information of task 1, the The data to be processed corresponding to A (the data to be processed includes the calculation results obtained based on organization A's execution of the above target task) is processed to obtain the calculation results based on organization A's execution of task 1.
  • the corresponding dependency rules can be directly determined according to the task dependency information, so that developers do not need to write the corresponding dependency of the task when writing the calculation information of the task. Rules, which facilitate the management of dependent rules.
  • the multi-tenant architecture needs to isolate the data of different organizations.
  • the developer does not need to write the organization information corresponding to the task when writing the calculation information of the task.
  • the tasks are executed sequentially based on the information of each preset representation agency, realizing the data isolation of different agencies. It avoids that the task cannot be performed based on the organization dimension due to the fact that the developer misses the organization information corresponding to the task, which affects the data isolation of different organizations.
  • a task when a task is executed based on the information of the preset representation mechanism, an error may occur, and at this time, the task needs to be executed again based on the information of the preset representation mechanism (ie, the task is rerun).
  • developers are required to write rerun codes each time a task is rerun, so that the device executing the above method executes the task again based on the information of the preset characterization mechanism according to the rerun code.
  • it is cumbersome to write rerun code every time a task is rerun, and it is easy to write wrong rerun code.
  • the above method also includes:
  • the task corresponding to the rerun instruction is executed.
  • the specific implementation of the task corresponding to the execution of the re-run command is similar to the above step S503, the only difference is that for the re-run command, the corresponding task is not executed based on the target starting running time, nor is it based on the information of each preset characterization mechanism in turn, but Based on the information of the preset representation mechanism corresponding to the rerun instruction, the task corresponding to the rerun instruction is executed, which will not be repeated here.
  • the information of the corresponding preset characterization mechanism and the corresponding task can be determined; thus, the corresponding task can be executed based on the information of the corresponding preset characterization mechanism, without re-running the task every time. Write rerun code while running.
  • the embodiment of the present application provides a batch task processing device.
  • the batch task processing device 600 includes:
  • a task allocation unit 601 configured to allocate the tasks to each of the preset queues based on the first historical average running time of each task in the batch task within the first period and the number of preset queues;
  • the operation processing unit 602 is configured to, for any preset queue, determine the tasks in the preset queue according to the historical average starting running time and the first historical average running time of each task in the preset queue within the first period of time.
  • the target start running time of each task and based on the target start running time, run the corresponding task.
  • the task allocation unit 601 is specifically configured to:
  • the target running time is based on The sum of the first historical average running time of all tasks in the batch task and the number of the preset queues are determined.
  • the task allocation unit 601 is specifically configured to:
  • the median The value is the median value of the first historical average running time of the batch task
  • the running processing unit 602 is specifically configured to:
  • any target task is a task that other tasks depend on;
  • the tasks other than the target task in the preset queue are arranged after the target task, and the tasks other than the target task are sorted from front to back according to the historical average starting running time; or
  • the target start running time of the previous task in the preset queue and the first historical average running time determine the target start running time of the next task, wherein the target start time of the first task in the preset queue
  • the running time is the corresponding historical average starting running time, or the preset starting running time.
  • the running processing unit 602 is further configured to:
  • the second time period is a time period before the first time period
  • the running processing unit 602 is further configured to:
  • the task corresponding to the rerun instruction is executed.
  • the running processing unit 602 is specifically configured to:
  • the target dependency rule corresponding to the task based on the preset corresponding relationship; wherein the corresponding relationship includes Association of preset dependency information and dependency rules;
  • the target dependency rule determine whether to execute and complete the corresponding target task based on the information of the preset characterization mechanism, wherein the corresponding target task is the task on which the execution of the task depends. task;
  • the data to be processed corresponding to the information of the preset characterization mechanism is processed according to the calculation information of the task.
  • the implementation of the device can refer to the implementation of the method, and the repetition will not be repeated.
  • the embodiment of the present application also provides a computing device 700, as shown in FIG.
  • the connection between the processor 701 and the memory 702 through the bus 703 in FIG. 7 is taken as an example.
  • the bus can be divided into address bus, data bus, control bus and so on. For ease of representation, only one thick line is used in FIG. 7 , but it does not mean that there is only one bus or one type of bus.
  • the processor 701 is the control center of the computing device, which can use various interfaces and lines to connect various parts of the computing device, by running or executing instructions stored in the memory 702 and calling data stored in the memory 702, thereby realizing data deal with.
  • the processor 701 may include one or more processing units, and the processor 701 may integrate an application processor and a modem processor.
  • the call processor mainly handles issuing instructions. It can be understood that the foregoing modem processor may not be integrated into the processor 701 .
  • the processor 701 and the memory 702 can be implemented on the same chip, and in some embodiments, they can also be implemented on independent chips.
  • the processor 701 can be a general processor, such as a central processing unit (CPU), a digital signal processor, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array or other programmable logic devices, discrete gates or transistors Logic devices and discrete hardware components can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application.
  • a general purpose processor may be a microprocessor or any conventional processor or the like. The steps of the method disclosed in the embodiment of the batch task processing method can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.
  • the memory 702 can be used to store non-volatile software programs, non-volatile computer-executable programs and modules.
  • Memory 702 can include at least one type of storage medium, for example, can include flash memory, hard disk, multimedia card, card-type memory, random access memory (Random Access Memory, RAM), static random access memory (Static Random Access Memory, SRAM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Magnetic Memory, Disk , CD, etc.
  • Memory 702 is, but is not limited to, any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.
  • the memory 702 in the embodiment of the present application may also be a circuit or any other device capable of implementing a storage function, and is used for storing program instructions and/or data.
  • the memory 702 stores a computer program, and when the program is executed by the processor 701, the processor 701 executes:
  • any preset queue according to the historical average starting running time and the first historical average running time of each task in the preset queue within the first period of time, determine the target starting running of each task in the preset queue Time, and based on the target starting running time, run the corresponding task.
  • processor 701 specifically executes:
  • the target running time is based on The sum of the first historical average running time of all tasks in the batch task and the number of the preset queues are determined.
  • processor 701 specifically executes:
  • the median The value is the median value of the first historical average running time of the batch task
  • processor 701 specifically executes:
  • any target task is a task that other tasks depend on;
  • the tasks other than the target task in the preset queue are arranged after the target task, and the tasks other than the target task are sorted from front to back according to the historical average starting running time; or
  • the target start running time of the previous task in the preset queue and the first historical average running time determine the target start running time of the next task, wherein the target start time of the first task in the preset queue
  • the running time is the corresponding historical average starting running time, or the preset starting running time.
  • the processor 701 before running the corresponding task based on the target start running time, the processor 701 further executes:
  • the second time period is a time period before the first time period
  • processor 701 also executes:
  • the task corresponding to the rerun instruction is executed.
  • processor 701 specifically executes:
  • the target dependency rule corresponding to the task based on the preset corresponding relationship; wherein the corresponding relationship includes Association of preset dependency information and dependency rules;
  • the target dependency rule determine whether to execute and complete the corresponding target task based on the information of the preset characterization mechanism, wherein the corresponding target task is the task on which the execution of the task depends. task;
  • the data to be processed corresponding to the information of the preset characterization mechanism is processed according to the calculation information of the task.
  • the computing device is the computing device in the method in the embodiment of this application, and the problem-solving principle of the computing device is similar to the method, the implementation of the computing device can refer to the implementation of the method, and the repetition will not be repeated.
  • an embodiment of the present application also provides a computer-readable storage medium, which stores a computer program executable by a computing device, and when the program is run on the computing device, the computing device Execute the steps of the batch task processing method above.
  • the embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
  • computer-usable storage media including but not limited to disk storage, CD-ROM, optical storage, etc.
  • These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions
  • the device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)

Abstract

Des modes de réalisation de la présente demande concernent un procédé et un appareil permettant de traiter un lot de tâches, ainsi qu'un dispositif informatique et un support de stockage. Le procédé de traitement d'un lot de tâches consiste à : d'après les premières durées d'exécution moyennes historiques de tâches dans un lot de tâches durant une première période et le nombre de files d'attente prédéfinies, attribuer les tâches aux files d'attente prédéfinies de façon à ce que les files d'attente prédéfinies ne soient pas trop inactives ou trop occupées lors de l'exécution des tâches ; et pour n'importe quelle file d'attente prédéfinie, en fonction des moments d'exécution de début moyens historiques des tâches dans la file d'attente prédéfinie durant la première période et des premières durées d'exécution moyennes historiques, déterminer des moments d'exécution de début cibles des tâches dans la file d'attente prédéfinie, et d'après les moments d'exécution de début cibles, exécuter les tâches dans la file d'attente prédéfinie dans un ordre raisonnable, ce qui permet d'ajuster raisonnablement un temps d'exécution de chaque tâche dans le lot de tâches.
PCT/CN2021/142108 2021-06-11 2021-12-28 Procédé et appareil de traitement de lot de tâches, dispositif informatique et support de stockage WO2022257435A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110651629.2A CN113326114A (zh) 2021-06-11 2021-06-11 一种批量任务处理方法及装置
CN202110651629.2 2021-06-11

Publications (1)

Publication Number Publication Date
WO2022257435A1 true WO2022257435A1 (fr) 2022-12-15

Family

ID=77421354

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/142108 WO2022257435A1 (fr) 2021-06-11 2021-12-28 Procédé et appareil de traitement de lot de tâches, dispositif informatique et support de stockage

Country Status (2)

Country Link
CN (1) CN113326114A (fr)
WO (1) WO2022257435A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116932227A (zh) * 2023-09-14 2023-10-24 西安华创马科智能控制***有限公司 一种基于单线程的任务调度方法及装置

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113326114A (zh) * 2021-06-11 2021-08-31 深圳前海微众银行股份有限公司 一种批量任务处理方法及装置
CN114880120B (zh) * 2022-05-10 2024-07-23 马上消费金融股份有限公司 数据处理方法、装置、设备及存储介质

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060190938A1 (en) * 2005-02-22 2006-08-24 Capek Peter G System and method for predictive idle-time task initiation
CN103488691A (zh) * 2013-09-02 2014-01-01 用友软件股份有限公司 任务调度装置和任务调度方法
CN107315627A (zh) * 2017-05-31 2017-11-03 北京京东尚科信息技术有限公司 一种自动化配置数据仓库并行任务队列的方法和装置
CN108829506A (zh) * 2018-07-04 2018-11-16 中国建设银行股份有限公司 批量任务处理方法、装置及服务***
CN109669767A (zh) * 2018-11-30 2019-04-23 河海大学 一种面向多类型上下文依赖的任务封装和调度方法及***
CN110096345A (zh) * 2019-03-16 2019-08-06 平安科技（深圳）有限公司 智能任务调度方法、装置、设备及存储介质
CN111258745A (zh) * 2018-11-30 2020-06-09 华为终端有限公司 一种任务处理方法及设备
CN113326114A (zh) * 2021-06-11 2021-08-31 深圳前海微众银行股份有限公司 一种批量任务处理方法及装置

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4906587B2 (ja) * 2007-05-16 2012-03-28 株式会社リコー ジョブ稼動状況評価システム、ジョブ稼動状況評価方法及びジョブ稼動状況評価プログラム
JP6241144B2 (ja) * 2013-08-30 2017-12-06 富士通株式会社 制御プログラム、制御方法および制御装置
CN106339802A (zh) * 2016-08-24 2017-01-18 深圳市讯方技术股份有限公司 任务分配的方法和装置、电子设备
US10387217B2 (en) * 2017-02-15 2019-08-20 Ca, Inc. Distributed processing of mixed serial and concurrent workloads
US10802880B2 (en) * 2017-09-19 2020-10-13 Huawei Technologies Co., Ltd. System and method for distributed resource requirement and allocation
CN107704322B (zh) * 2017-09-30 2020-08-25 上海大象金泰科技有限公司 请求分配方法及装置
CN111125444A (zh) * 2019-12-10 2020-05-08 中国平安财产保险股份有限公司 大数据任务调度管理方法、装置、设备及存储介质
CN112286672B (zh) * 2020-12-30 2021-05-07 南京智闪萤科技有限公司 用于调度任务的方法、计算设备和计算机存储介质

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060190938A1 (en) * 2005-02-22 2006-08-24 Capek Peter G System and method for predictive idle-time task initiation
CN103488691A (zh) * 2013-09-02 2014-01-01 用友软件股份有限公司 任务调度装置和任务调度方法
CN107315627A (zh) * 2017-05-31 2017-11-03 北京京东尚科信息技术有限公司 一种自动化配置数据仓库并行任务队列的方法和装置
CN108829506A (zh) * 2018-07-04 2018-11-16 中国建设银行股份有限公司 批量任务处理方法、装置及服务***
CN109669767A (zh) * 2018-11-30 2019-04-23 河海大学 一种面向多类型上下文依赖的任务封装和调度方法及***
CN111258745A (zh) * 2018-11-30 2020-06-09 华为终端有限公司 一种任务处理方法及设备
CN110096345A (zh) * 2019-03-16 2019-08-06 平安科技（深圳）有限公司 智能任务调度方法、装置、设备及存储介质
CN113326114A (zh) * 2021-06-11 2021-08-31 深圳前海微众银行股份有限公司 一种批量任务处理方法及装置

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116932227A (zh) * 2023-09-14 2023-10-24 西安华创马科智能控制***有限公司 一种基于单线程的任务调度方法及装置
CN116932227B (zh) * 2023-09-14 2023-12-22 西安华创马科智能控制***有限公司 一种基于单线程的任务调度方法及装置

Also Published As

Publication number Publication date
CN113326114A (zh) 2021-08-31

Similar Documents

Publication Publication Date Title
WO2022257435A1 (fr) Procédé et appareil de traitement de lot de tâches, dispositif informatique et support de stockage
US10911367B2 (en) Computerized methods and systems for managing cloud computer services
US11425194B1 (en) Dynamically modifying a cluster of computing nodes used for distributed execution of a program
KR102569086B1 (ko) 태스크 병렬 처리 방법, 장치, 시스템, 기억 매체 및 컴퓨터 기기
US20200167197A1 (en) Systems, methods, and apparatuses for implementing a scheduler with preemptive termination of existing workloads to free resources for high priority items
US9442760B2 (en) Job scheduling using expected server performance information
US9280390B2 (en) Dynamic scaling of a cluster of computing nodes
US11941434B2 (en) Task processing method, processing apparatus, and computer system
US9465663B2 (en) Allocating resources in a compute farm to increase resource utilization by using a priority-based allocation layer to allocate job slots to projects
JP4935595B2 (ja) ジョブ管理方法、ジョブ管理装置およびジョブ管理プログラム
US7650601B2 (en) Operating system kernel-assisted, self-balanced, access-protected library framework in a run-to-completion multi-processor environment
US7962913B2 (en) Scheduling threads in a multiprocessor computer
US20200026571A1 (en) Systems, methods, and apparatuses for implementing a scheduler and workload manager with workload re-execution functionality for bad execution runs
US7681196B2 (en) Providing optimal number of threads to applications performing multi-tasking using threads
US20090178045A1 (en) Scheduling Memory Usage Of A Workload
US11954419B2 (en) Dynamic allocation of computing resources for electronic design automation operations
WO2021208786A1 (fr) Procédé et appareil de gestion de fil
US7925755B2 (en) Peer to peer resource negotiation and coordination to satisfy a service level objective
US11561843B2 (en) Automated performance tuning using workload profiling in a distributed computing environment
US11080092B1 (en) Correlated volume placement in a distributed block storage service
GB2514585A (en) Task scheduler
US11928464B2 (en) Systems and methods for model lifecycle management
CN117827453B (zh) 存储资源的订购方法、装置、设备及存储介质
US20240126460A1 (en) Enabling persistent memory for serverless applications
US11048554B1 (en) Correlated volume placement in a distributed block storage service

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21944931

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 03.04.2024)