CN113807710A

CN113807710A - Method for sectionally paralleling and dynamically scheduling system batch tasks and storage medium

Info

Publication number: CN113807710A
Application number: CN202111107163.6A
Authority: CN
Inventors: 刘强; 钱思先; 杨强; 倪振心
Original assignee: Sichuan XW Bank Co Ltd
Current assignee: Sichuan XW Bank Co Ltd
Priority date: 2021-09-22
Filing date: 2021-09-22
Publication date: 2021-12-17
Anticipated expiration: 2041-09-22
Also published as: CN113807710B

Abstract

The invention discloses a method and a storage medium for sectionally paralleling and dynamically scheduling system batch tasks, wherein the method is implemented based on an oracle storage process and comprises the steps of automatically analyzing the data volume of each batch task and the dependency relationship among the tasks according to all the batch tasks needing to be processed after a batch processing instruction is initiated by a batch task scheduling system, automatically dividing each batch task into a plurality of sections according to the sectionalized configuration of each batch task, sequencing the batch tasks according to the dependency relationship, creating an asynchronous processing jobof the oracle according to each section, and processing each sectioned batch task by the asynchronous processing jobs. Meanwhile, the batch performance monitoring program monitors the use condition of system resources in batch processing in real time, and dynamically adjusts the segment number of subsequent batch processing tasks according to the use rate of the current resources (such as CPU, memory, IO and the like). Therefore, the efficiency of the banking system for batching tasks is improved, and the execution time of the batching tasks is reduced.

Description

Method for sectionally paralleling and dynamically scheduling system batch tasks and storage medium

Technical Field

The invention belongs to the field of bank systems developed based on oracle storage processes, and particularly relates to a method for sectionally paralleling and dynamically scheduling tasks in batches of a system and a storage medium.

Background

With the development of internet finance, the data volume of a traditional banking system realized based on a storage process is larger and larger, most banking systems developed based on a storage process technology stack are batch tasks processed by a single processing thread at present, the processing of the existing large data volume cannot be met due to the single thread mode, and the efficiency is extremely low.

However, no technology related to the implementation of batch task segmentation parallel and dynamic scheduling based on a storage process is found at present. In order to solve the technical pain point, a method for sectionally paralleling and dynamically scheduling the system batch tasks is urgently needed, the batch tasks can be processed and dynamically scheduled in a parallel mode, multi-task parallel processing among the batch processing tasks is achieved, multi-section parallel processing in the batch processing tasks is achieved, and the batch task efficiency of a banking system is improved.

Disclosure of Invention

The invention aims to overcome the defects of the prior art and provide a method for segmenting and paralleling and dynamically scheduling system batch tasks, which realizes the segmentation and paralleling of the batch tasks of a banking system and dynamically adjusts segmentation according to the available resource quantity, thereby improving the batch task efficiency of the banking system and reducing the execution time of the batch tasks.

The purpose of the invention is realized by the following technical scheme:

a system batch task segmentation parallel and dynamic scheduling method is implemented based on an oracle storage process, and comprises the following steps: s1: initiating a batch processing instruction to a batch scheduler of the system; s2: the batch scheduler is used for screening and preparing the batch tasks to be processed at this time based on the batch processing instructions, and starting a batch monitor of the system to detect the service condition of system resources in real time during the execution and processing of the batch tasks; s3: the batch scheduler calls a segmenter, and the segmenter segments each task based on data required to be processed by the batch tasks to be processed and segmentation parameters of each task; s4: the batch dispatcher inquires the segmented unprocessed segmented tasks which are segmented, screens the current executable segmented tasks based on the logic relation among the tasks, and creates asynchronous work tasks one by one to process each segmented task; s5: polling by the batch scheduler to determine whether a batch task to be processed or abnormally processed exists at present, and if the batch task which is not processed or fails to be processed exists, newly initiating an asynchronous working task to process each segmented task; s6: and after each batch of segmentation tasks to be processed is successfully processed and completed, the batch tasks are completed.

According to a preferred embodiment, the method for batch task segmentation parallel and dynamic scheduling of the system further comprises: detecting system resources in real time by a batch monitor;

and judging whether the utilization rate of the current system resources exceeds an upper limit threshold value according to the detection result so as to complete the adjustment of the number of the segmented tasks of the subsequent batch tasks.

According to a preferred embodiment, in the step of detecting the system resource in real time by the batch monitor, when the batch monitor judges that the current utilization rate of the system resource exceeds the upper limit threshold, the batch monitor decreases the number of the segmented tasks of the subsequent batch tasks.

According to a preferred embodiment, in the step of detecting the system resource in real time by the batch monitor, if the batch monitor judges that the current utilization rate of the system resource does not reach the optimal threshold value, the number of the segmented tasks of the subsequent batch tasks is increased.

According to a preferred embodiment, step S3 specifically includes: s31, the segmenter scans the batch tasks to be processed, searches the segmentation configuration according to each batch task, and confirms the segmentation basis field and initializes the segmentation parameters; s32: the segmenter counts the total amount of data to be processed of each batch of tasks according to segmentation basis, and confirms the data amount processed by each segment based on initialized segmentation parameter configuration; s33: and the segmenter confirms the initial value and the end value of each segment of the batch task according to the data volume and the total data volume processed by each segment of the batch task, and records the initial value and the end value of each segment of the field in the segmentation task table.

According to a preferred embodiment, in step S5, the batch scheduler polls the end condition that all batch segmentation tasks that need to be processed have been successfully processed.

A computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements a method of piecewise parallelism and dynamic scheduling of a system batch of tasks as described above.

The aforementioned main aspects of the invention and their respective further alternatives can be freely combined to form a plurality of aspects, all of which are aspects that can be adopted and claimed by the present invention. The skilled person in the art can understand that there are many combinations, which are all the technical solutions to be protected by the present invention, according to the prior art and the common general knowledge after understanding the scheme of the present invention, and the technical solutions are not exhaustive herein.

The invention has the beneficial effects that:

the invention is mainly used for a banking system developed based on an Oracle storage process, is used for solving the problem of the efficiency of batch processing tasks of the banking system, and realizes multi-task parallel processing among the batch processing tasks and multi-section parallel processing in the batch processing tasks.

After a batch processing instruction is initiated by the batch task scheduling system, the data volume of each batch task and the dependency relationship among the tasks are automatically analyzed according to all the batch tasks needing to be processed, each batch task is automatically divided into multiple sections according to the sectional configuration of each batch task, the batch tasks are sequenced according to the dependency relationship, the asynchronous processing jobof oracle is created according to each section, and the asynchronous processing jobprocesses the batch tasks of each section.

Meanwhile, the batch performance monitoring program monitors the use condition of system resources in batch processing in real time, and dynamically adjusts the segment number of subsequent batch processing tasks according to the use rate of the current resources (such as CPU, memory, IO and the like).

Therefore, the method for sectionalizing and paralleling the system batch tasks and dynamically scheduling the system batch tasks realizes the sectionalized paralleling of the banking system batch tasks, and dynamically adjusts the sectionalization according to the available resource amount, thereby improving the efficiency of the banking system batch tasks and reducing the execution time of the batch tasks.

Drawings

FIG. 1 is a schematic flow diagram of the process of the present invention;

FIG. 2 is a schematic diagram of a segmentation task in the method of the present invention;

FIG. 3 is a schematic diagram of the dependency relationship of batch processing in the method of the present invention.

Detailed Description

The embodiments of the present invention are described below with reference to specific embodiments, and other advantages and effects of the present invention will be easily understood by those skilled in the art from the disclosure of the present specification. The invention is capable of other and different embodiments and of being practiced or of being carried out in various ways, and its several details are capable of modification in various respects, all without departing from the spirit and scope of the present invention. It is to be noted that the features in the following embodiments and examples may be combined with each other without conflict.

It should be noted that, in order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described below, and it is obvious that the described embodiments are some embodiments of the present invention, but not all embodiments.

The invention discloses a method for sectionally paralleling and dynamically scheduling a system batch task.

The multitask parallelism in the invention is to utilize oracle to establish an asynchronous task mechanism, namely, on the basis of the segmented completion of batch processing tasks, a segmented task table is scanned, a dbms _ job.submit method is used for establishing asynchronous processing Job for data processing of each segmented task process, the asynchronous processing Job only performs establishment action in the main establishment flow, the asynchronous processing Job is executed as an additional process (similar to a new start thread of Java) and does not occupy the time of the main establishment flow, so that all segmented tasks can be quickly called up and asynchronously executed, and the aim of multitask parallelism is fulfilled.

Referring to fig. 1, the method for sectionally paralleling and dynamically scheduling the system batch tasks according to the present invention includes:

step S1: the batch task scheduling system initiates a batch processing instruction to the banking system batch scheduler dispatcher.

Step S2: the dispatcher selects and prepares the batch tasks (task 1, task 2 and task 3 … …) which need to be processed by the batch tasks according to the instruction, and simultaneously starts the service system batch monitor to detect the use condition of system resources in the execution period of the batch tasks in real time.

Step S3: the dispatcher invokes the segmenter, the segmenter segments each task according to the batch task to be processed and the segmentation parameters of each task, and the detailed segmentation process is shown in the segmentation task schematic diagram 2.

Step S3 specifically includes:

s3.1: the segmenter scans all batch tasks needing to be processed, searches the segment configuration table batch _ split _ config one by one according to each batch task, and confirms that the segment of each batch task queries the SQL according to the field split _ key, the initialized segment parameter split _ number and the segment total amount.

S3.2: and the segmenter executes the segmentation total query SQL to count the total data sum required to be processed by each batch task according to the segmentation configuration, configures the split _ number according to the initialized segmentation parameters, and confirms the data amount split _ count processed by each segment, namely the split _ count is equal to the value of sum/split _ number, and the value is taken as one.

S3.3: the segmenter determines the start value and the end value of each segment of the segmentation basis field of the batch task according to the data volume split _ count, the data total sum and the segmentation basis field split _ key processed by each segment of the batch task (for example, the account table sum is 100, the split _ key is account _ no, and the initialized segmentation parameter split _ number is 30, so that the batch process is divided into 4 segments of subtasks, the start value and the end value of the 1 st segment of account _ no are 1-30, the start value and the end value of the 2 nd segment of account _ no are 31-60, the start value and the end value of the 3 rd segment of account _ no are 61-90, and the start value and the end value of the 4 th segment of account _ no are 91-100), and records the segment subtask table.

Step S4: the dispatcher scans the segmented subtask table (subtask 1.1, subtask 1.2, subtask 2.1, and subtask 2.2 … …) in a full scale, and synthesizes the dependency relationship between tasks (for example, as shown in fig. 3, task 1 depends on task 2 and task 3, so task 1 must be executed later than task 2 and task 3, and task 2 depends on task 4 and task 5, so task 2 must be executed later than task 4 and task 5), finds the segmented subtasks which do not depend on the parent task or depend on the batch task which has been successfully processed by the task, creates asynchronous job tasks one by one to process each segmented task, and updates the asynchronous job execution result into the segmented subtask table.

Step S5: polling whether a batch task to be processed or abnormal to be processed exists or not by the dispatcher, wherein the polling termination condition is that all batch segmentation tasks needing to be processed are successfully processed and completed, and if the batch segmentation subtasks which are not processed or failed to be processed exist, newly initiating each segmentation subtask by the asynchronous jobtask.

Step S6: and after each batch of segmentation tasks to be processed is successfully processed and completed, the batch tasks are completed.

And detecting system resources in real time by the batch monitor. And detecting the system resource in real time when the monitor exists in the whole period of the batch task, judging whether the utilization rate of the current system resource exceeds an upper limit threshold or not according to the detection result, and if the utilization rate of the current system resource exceeds the upper limit threshold (for example, the cpu utilization rate is 80%), adjusting the step number adjust _ step (namely, adjusting the increased or decreased segment number each time) according to the segments in the segment configuration to reduce the number of the segment tasks of the subsequent batch task. If the optimal threshold value (for example, the cpu utilization rate is 70%) is not reached, the segmentation subtask table is locked (the dispatcher is prevented from wrongly judging that no task is to be processed), and the number of segmentation tasks of subsequent batch tasks is increased according to the number of segmentation adjustment steps in the segmentation configuration, so that the batch processing efficiency is improved.

The invention is mainly used for a banking system developed based on an Oracle storage process, is used for solving the problem of the efficiency of batch processing tasks of the banking system every day, and realizes multi-task parallel processing among the batch processing tasks and multi-section parallel processing in the batch processing tasks.

Therefore, by the method for sectionalizing and paralleling the system batch tasks and dynamically scheduling the system batch tasks, the method realizes the sectionalizing and paralleling of the final batch tasks of the banking system, and dynamically adjusts the sections according to the available resource amount, thereby improving the batch task efficiency of the banking system and reducing the batch task execution time.

The invention also provides a computer readable storage medium, on which a computer program is stored, wherein the computer program realizes the system batch task segmentation parallel and dynamic scheduling method when being executed by a processor.

Example 1

Scene: the final daily batch processing of a certain banking system comprises 3 batch processing tasks:

task 1: the account balance is updated. The corresponding segment is configured to: the segmentation basis field is an account (account _ no), the segmentation parameters are initialized to 4, the segmentation adjustment step number (adjust _ step) is 2, and the total segmentation query SQL is a select count (account _ no) from account _ list.

Task 2: the customer balance is updated. The segmentation basis field is the client number (customer no), the segmentation parameters are initialized to be 3, the segmentation adjustment step number (adjust step) is 2, and the segmentation total query SQL is the select count (customer no) from the customer list.

Task 3: accounting is performed in a flow of accounts. The segmentation basis field is a transaction sequence number (tran _ no), a segmentation parameter 5 is initialized, the segmentation adjustment step number (adjust _ step) is 2, and the segmentation total query SQL is a select count (customer _ no) from a customer _ list.

Wherein, task 2 is completed depending on the balance update of task 1, and task 3 has no dependency relationship to other tasks.

The number of the bank accounts is 1 ten thousand, and the account number (account _ no) is increased from AC000001 to AC 100000. The number of customers is 5 thousands, the number of customers (customer _ no) increases from CN00001 to CN50000, the number of accounting lines is 35 thousands, and the transaction number (tran _ no) increases from TN000001 to TN 350000.

Batch processing steps:

s1: the batch task scheduler initiates the batch processing instruction startBatch _20210901 to the business system batch scheduler dispatcher.

And S2, the dispatcher screens and prepares the batch tasks, task 1, task 2 and task 3, which need to be processed by the batch task according to the instruction, simultaneously starts the service system batch monitor, and detects the use condition of the system resources in the execution period of the batch tasks in real time.

S3: the dispatcher invokes the segmenter to segment each batch task, and finally confirms that the segmentation is as follows according to the segmentation parameter configuration of each batch task:

task 1: the amount of data to be processed sum of the batch task, which is obtained by inquiring the account balance according to the total segmentation query SQL, is 1 ten thousand, the initialization segmentation parameter is 4, then the batch task needs to be divided into 4 segments according to the segmentation initialization parameter, and the amount of data to be processed of each segment of the batch task is 100000/4-2500, that is:

subtask 1.1: and the account balance update is responsible for the account balance update from AC00001 to AC 25000.

Subtask 1.2: is responsible for balance update of the account from AC25001 to AC 50000.

Subtask 1.3: is responsible for balance update of the account from AC50001 to AC 75000.

Subtask 1.4: and the account balance updating is carried out on account numbers from AC75001 to AC 10000.

Task 2: the amount of data to be processed sum of the batch task updated by the client balance query according to the total segmentation query SQL is 5 ten thousand, the initialization segmentation parameter is 3, then the batch task needs to be divided into 3 segments according to the segmentation initialization parameter, and the amount of data to be processed of each segment of the batch task is 50000/4-16666.67, that is, 16667, that is:

subtask 2.1: is responsible for the balance update of the customer from CN00001 to CN 16667.

Subtask 2.2: is responsible for the balance update of the customer from CN16668 to CN 33335.

Subtask 2.3: is responsible for the balance update of the client between the client number from CN33336 to CN 50000.

Task 3: the data volume to be processed of the batch task is queried according to the total quantity of the segments, SQL, and the accounting workflow is 35 ten thousand, the initialization segmentation parameter is 5, then the batch task needs to be divided into 5 segments according to the segmentation initialization parameter, and the data volume to be processed of each segment of the batch task is 350000/5-70000, that is:

subtask 3.1: is responsible for accounting flow accounting of transaction sequence numbers from TN00001 to TN 70000.

Subtask 3.2: is responsible for accounting flow accounting of transaction sequence numbers from TN70001 to TN 140000.

Subtask 3.3: responsible for accounting flow accounting of transaction sequence numbers from TN140001 to TN 210000.

Subtask 3.4: is responsible for accounting flow accounting of transaction sequence numbers from TN210001 to TN 280000.

Subtask 3.5: is responsible for accounting flow accounting of transaction sequence numbers from TN280001 to TN 350000.

And recording the subtask information completed by the segmentation in a segmentation subtask table.

And S4, the dispatcher scans the subsection subtask table in a full quantity mode, scans the subtasks which are to be processed and are already processed by the parent task which depends on the dispatcher, and creates asynchronous jobs. during the dispatcher scanning, if the 4 subtasks 1.1-1.4 of the subtask 1 are found to be unprocessed and independent, 4 asynchronous processing jobs are directly created for the subtasks 1.1-1.4 respectively. The subtasks 2.1-2.3 are not processed, but depending on the completion of the execution of task 1, if it is found that task 1 is not completed, then this scan does not process task 2. Continuing to scan down to task 3, finding that the subtasks 3.1-3.5 of task 3 are not executed and have no dependency, then directly creating 4 asynchronous processing jobs for the subtasks 3.1-3.5, respectively. At this time, after the asynchronous processing job created for the subtasks of task 1 and task 3 is completed, the asynchronous processing job is already processing the data of each segment, and after the processing is completed, the state of the subtask in the segment subtask table is updated.

S6: the dispatcher scans the segmentation subtask table in full quantity to find that there are no tasks to be processed and failed tasks, so that the whole batch is processed.

And detecting system resources in real time by the batch monitor. The pointer monitors the resource use condition of batch processing all the time, if the resource use condition of the system does not reach the optimal threshold value (the CPU use ratio reaches 70 percent, and the memory use ratio reaches 70 percent) at this time, the unprocessed task in the segment subtask table is scanned, the task is extracted, the segment number is adjusted once according to the segment adjustment step number adjust _ step in the task segment configuration, and the segment is re-segmented. During the asynchronous process of task 1 and task 3, Monitor finds that the system resource utilization rate at this time is as follows: cpu 50%, memory: 60%, the optimal threshold is not reached. At this time, the Monitor scans the segmentation subtask table to find that all the subtasks of task 2 are processed, at this time, the Monitor locks the segmentation subtask table (to prevent the dispatcher from falsely judging as no task to be processed), and then performs segmentation again according to step S3, at this time, the segmentation parameter is adjusted to 5, and the segmentation data is processed by dividing into 5 subtasks. After the segmentation is completed in step S3, the asynchronous processing procedure created in step S4 is handed to process the segmented data.

The foregoing basic embodiments of the invention and their various further alternatives can be freely combined to form multiple embodiments, all of which are contemplated and claimed herein. In the scheme of the invention, each selection example can be combined with any other basic example and selection example at will. Numerous combinations will be known to those skilled in the art.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.

Claims

1. The method for sectionally paralleling and dynamically scheduling the system batch tasks is characterized in that the method for sectionally paralleling and dynamically scheduling the system batch tasks is implemented based on an oracle storage process, and the method for sectionally paralleling and dynamically scheduling the system batch tasks comprises the following steps:

s1: initiating a batch processing instruction to a batch scheduler of the system;

s2: the batch scheduler is used for screening and preparing the batch tasks to be processed at this time based on the batch processing instructions, and starting a batch monitor of the system to detect the service condition of system resources in real time during the execution and processing of the batch tasks;

s3: the batch scheduler calls up a segmenter, and the segmenter segments each task based on the batch tasks to be processed and the segmentation parameters of each task;

s4: the batch dispatcher inquires the segmented unprocessed segmented tasks which are segmented, screens the current executable segmented tasks based on the logic relation among the tasks, and creates asynchronous work tasks one by one to process each segmented task;

s5: polling by the batch scheduler to determine whether a batch task to be processed or abnormally processed exists at present, and if the batch task which is not processed or fails to be processed exists, newly initiating an asynchronous working task to process each segmented task;

s6: and after each batch of segmentation tasks to be processed is successfully processed and completed, the batch tasks are completed.

2. The method for system batch task segmentation parallel and dynamic scheduling of claim 1, wherein the method for system batch task segmentation parallel and dynamic scheduling further comprises: detecting system resources in real time by a batch monitor;

3. The method for system batch task segmentation parallel and dynamic scheduling according to claim 2, wherein in the step of the batch monitor detecting the system resources in real time, when the batch monitor judges that the current system resource utilization rate exceeds the upper limit threshold, the batch monitor reduces the number of segmentation tasks of the subsequent batch tasks.

4. The method for system batch task segmentation parallel and dynamic scheduling according to claim 3, wherein in the step of the batch monitor detecting the system resources in real time, the batch monitor increases the number of segmentation tasks of the subsequent batch tasks if the batch monitor judges that the current system resource utilization rate does not reach the optimal threshold.

5. The method for system batch task segmentation parallel and dynamic scheduling according to claim 1, wherein step S3 specifically includes:

s31, the segmenter scans the batch tasks to be processed, searches the segmentation configuration according to each batch task, and confirms the segmentation basis field and initializes the segmentation parameters;

s32: the segmenter counts the total amount of data to be processed of each batch of tasks according to segmentation basis, and confirms the data amount processed by each segment based on initialized segmentation parameter configuration;

s33: and the segmenter confirms the initial value and the end value of each segment of the batch task according to the data volume and the total data volume processed by each segment of the batch task, and records the initial value and the end value of each segment of the field in the segmentation task table.

6. The system batch task segmentation parallel and dynamic scheduling method of claim 1, wherein in step S5, the batch scheduler polls the termination condition that all batch segmentation tasks required to be processed have been successfully processed.

7. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-6.