CN113807710B - System batch task segmentation parallel and dynamic scheduling method and storage medium - Google Patents
System batch task segmentation parallel and dynamic scheduling method and storage medium Download PDFInfo
- Publication number
- CN113807710B CN113807710B CN202111107163.6A CN202111107163A CN113807710B CN 113807710 B CN113807710 B CN 113807710B CN 202111107163 A CN202111107163 A CN 202111107163A CN 113807710 B CN113807710 B CN 113807710B
- Authority
- CN
- China
- Prior art keywords
- batch
- task
- tasks
- processed
- segmented
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
- G06Q10/06312—Adjustment or analysis of established resource schedule, e.g. resource or task levelling, or dynamic rescheduling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/02—Banking, e.g. interest calculation or account maintenance
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- Human Resources & Organizations (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Economics (AREA)
- Strategic Management (AREA)
- Databases & Information Systems (AREA)
- Finance (AREA)
- Development Economics (AREA)
- Accounting & Taxation (AREA)
- Marketing (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Business, Economics & Management (AREA)
- Entrepreneurship & Innovation (AREA)
- Technology Law (AREA)
- Computational Linguistics (AREA)
- Educational Administration (AREA)
- Mathematical Physics (AREA)
- Game Theory and Decision Science (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a system batch task segmentation parallel and dynamic scheduling method and a storage medium, wherein the system batch task segmentation parallel and dynamic scheduling method is implemented based on an oracle storage process. Meanwhile, in batch processing, a batch performance monitoring program monitors the use condition of system resources in real time, and the segmentation number of subsequent batch processing tasks is dynamically adjusted according to the use rate (such as CPU, memory, IO and the like) of the current resources. Therefore, the batch task efficiency of the banking system is improved, and the execution time of the batch task is reduced.
Description
Technical Field
The invention belongs to the field of banking systems developed based on oracle storage processes, and particularly relates to a system batch task segmentation parallel and dynamic scheduling method and a storage medium.
Background
With the development of internet finance, the data volume of the traditional banking system realized based on a storage process is larger and larger, most banking systems developed based on a storage process technology stack are currently single processing threads for processing batch tasks, and the single thread mode causes that the current large data volume processing cannot be met, so that the efficiency is extremely low.
At present, a technology related to realizing batch task segmentation parallel and dynamic scheduling based on a storage process is not found. In order to solve the technical pain, a method for sectional parallel and dynamic scheduling of batch tasks of a system is needed, which can process the batch tasks in parallel and dynamically schedule, realize multi-task parallel processing among the batch processing tasks, process the batch processing tasks in multiple sections in parallel, and improve the batch task efficiency of a banking business system.
Disclosure of Invention
The invention aims to overcome the defects of the prior art, and provides a system batch task segmentation parallel and dynamic scheduling method, which realizes the batch task segmentation parallel of a banking system and dynamically adjusts segmentation according to the available resource quantity, thereby improving the batch task efficiency of the banking system and reducing the execution time of batch tasks.
The aim of the invention is achieved by the following technical scheme:
the system batch task piecewise parallel and dynamic scheduling method is implemented based on an oracle storage process, and comprises the following steps of: s1: initiating a batch processing instruction to a batch scheduler of the system; s2: the batch dispatcher screens and prepares batch tasks to be processed at this time based on batch processing instructions, and starts a batch monitor of the system to detect the use condition of system resources in real time during the execution processing of the batch tasks; s3: the batch dispatcher calls up a sectionalizer, and the sectionalizer sections each task based on data required to be processed by batch tasks to be processed and sectionalizing parameters of each task; s4: inquiring unprocessed segmented tasks which are segmented by a batch scheduler, screening out currently executable segmented tasks based on a logic relation among the tasks, and creating asynchronous work tasks one by one to process each segmented task; s5: the batch dispatcher polls whether batch tasks to be processed or abnormal to be processed exist currently, and if the batch tasks which are not processed or failed to be processed exist, an asynchronous work task is newly initiated to process each segmented task; s6: after each batch of segmented tasks to be processed are successfully processed, the batch tasks are completed.
According to a preferred embodiment, the system batch task segment parallel and dynamic scheduling method further comprises: detecting system resources in real time by a batch monitor;
and judging whether the current system resource utilization exceeds an upper limit threshold according to the detection result so as to complete the adjustment of the number of segmented tasks of the follow-up batch task.
According to a preferred embodiment, in the step of detecting the system resources in real time by the batch monitor, when the batch monitor judges that the current utilization rate of the system resources exceeds the upper limit threshold, the number of segmented tasks of the follow-up batch tasks is reduced.
According to a preferred embodiment, in the step of detecting the system resources in real time by the batch monitor, if the batch monitor judges that the current utilization rate of the system resources does not reach the optimal threshold value, the number of segmented tasks of the subsequent batch tasks is increased.
According to a preferred embodiment, step S3 specifically comprises: s31, scanning batch tasks to be processed by a segmenter, searching segmentation configuration according to each batch task, and confirming segmentation basis fields and initializing segmentation parameters; s32: the segmenter counts the total data quantity to be processed of each batch task according to the segmentation basis, and confirms the data quantity to be processed of each segment based on the initialized segmentation parameter configuration; s33: and the segmenter confirms the start value and the end value of each segment of the batch task segment according to the field according to the data quantity and the data total quantity processed by each segment of the batch task, and records the start value and the end value in the segmented task list.
According to a preferred embodiment, in step S5, the batch scheduler poll termination condition is that all batch segmentation tasks to be processed have been successfully processed.
A computer readable storage medium having stored thereon a computer program which when executed by a processor implements a system batch task segment parallelism and dynamic scheduling method as described above.
The foregoing inventive concepts and various further alternatives thereof may be freely combined to form multiple concepts, all of which are contemplated and claimed herein. Various combinations will be apparent to those skilled in the art from a review of the present disclosure, and are not intended to be exhaustive or all of the present disclosure.
The invention has the beneficial effects that:
the invention is mainly used for a banking system developed based on an Oracle storage process, and is used for solving the problem of task efficiency of batch processing of the banking system, realizing multi-task parallel processing among batch processing tasks and multi-stage parallel processing in the batch processing tasks.
After a batch task scheduling system initiates a batch processing instruction, the data volume of each batch task and the dependency relationship among tasks are automatically analyzed according to all batch tasks to be processed, each batch task is automatically divided into a plurality of sections according to the sectional configuration of each batch task, the batch tasks are ordered according to the dependency relationship, an oracle asynchronous processing job is created according to each section, and each sectional batch task is processed by the asynchronous processing job.
Meanwhile, in batch processing, a batch performance monitoring program monitors the use condition of system resources in real time, and the segmentation number of subsequent batch processing tasks is dynamically adjusted according to the use rate (such as CPU, memory, IO and the like) of the current resources.
Therefore, the system batch task segmentation parallel and dynamic scheduling method realizes the batch task segmentation parallel of the banking system, and the segmentation is dynamically adjusted according to the available resource quantity, so that the batch task efficiency of the banking system is improved, and the execution time of the batch task is reduced.
Drawings
FIG. 1 is a schematic flow chart of the method of the present invention;
FIG. 2 is a schematic illustration of a segmented task in the method of the present invention;
FIG. 3 is a schematic diagram of batch dependency in the method of the present invention.
Detailed Description
Other advantages and effects of the present invention will become apparent to those skilled in the art from the following disclosure, which describes the embodiments of the present invention with reference to specific examples. The invention may be practiced or carried out in other embodiments that depart from the specific details, and the details of the present description may be modified or varied from the spirit and scope of the present invention. It should be noted that the following embodiments and features in the embodiments may be combined with each other without conflict.
It should be noted that, for the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described below, and it is obvious that the described embodiments are some embodiments of the present invention, but not all embodiments.
The invention discloses a system batch task segment parallel and dynamic scheduling method, which is implemented based on an oracle storage process.
The invention discloses a multi-task parallel method, which is characterized in that an oracle is utilized to create an asynchronous task mechanism, namely, a segmented task list is scanned on the basis of completing batch processing tasks in a segmented mode, an asynchronous processing Job is created for data processing of each segmented task process by using a dbms_job.sub method, the asynchronous processing Job only performs creation action in a main creation flow, the execution of the asynchronous Job is a separate process (similar to a Java new starting thread), the time of the main creation flow is not occupied, and therefore, the purposes of quickly calling up all segmented tasks, asynchronously executing and multi-task parallel are achieved.
Referring to fig. 1, the system batch task segment parallel and dynamic scheduling method of the present invention includes:
step S1: the batch task scheduling system initiates a batch processing instruction to a banking system batch scheduler.
Step S2: the dispatcher screens batch tasks (task 1, task 2 and task 3 … …) which are needed to be processed by the batch tasks according to the instruction, and simultaneously starts a business system batch monitor to detect the use condition of system resources in the batch task execution period in real time.
Step S3: the dispatcher calls up a segmenter, the segmenter segments each task according to batch tasks to be processed and segmentation parameters of each task, and the detailed segmentation process is shown in a segmented task schematic diagram 2.
The step S3 specifically comprises the following steps:
s3.1: segment scans all batch tasks to be processed, searches a segment configuration table batch_split_config one by one according to each batch task, and confirms a segment basis field split_key, an initialization segment parameter split_number and a segment total amount query SQL of each batch task.
S3.2: according to the segment configuration, the segment total amount query SQL is executed to count the total amount sum of data to be processed of each batch task, then the split_number is configured according to the initialized segment parameters, and the data amount split_count processed by each segment is confirmed, namely, the split_count is equal to the sum/split_number and takes a value of one method.
S3.3: the segment confirms the start value and the end value of each segment of the batch task segment according to the data quantity split_count, the data quantity sum and the segment according to the field split_key (for example, the account table sum is 100, the split_key is accoun_no, the initialization segment parameter split_number is 30, then the batch process is divided into 4 segments of subtasks, the 1 st segment accoun_no start and end value is 1-30, the 2 nd segment accoun_no start and end value is 31-60, the 3 rd segment accoun_no start and end value is 61-90, the 4 th segment accoun_no start and end value is 91-100), and the segment subtask table is recorded.
Step S4: the dispatcher scans the segmented subtask table (subtask 1.1, subtask 1.2, subtask 2.1, subtask 2.2 … …) in full, synthesizes the dependency relationship among the tasks (for example, as shown in fig. 3, task 1 depends on task 2 and task 3, so task 1 must be executed later than task 2 and task 3, task 2 depends on task 4 and task 5, so task 2 must be executed later than task 4 and task 5), searches the segmented subtasks of the batch tasks which do not depend on the parent task or depend on the task and have been successfully processed, creates asynchronous job tasks one by one to process each segmented task, and updates the execution result of the asynchronous job to the segmented subtask table.
Step S5: the dispatcher polls whether the batch tasks to be processed or abnormal to be processed exist currently, the polling termination condition is that all batch segmentation tasks to be processed are successfully processed and completed, and if the batch segmentation subtasks which are not processed or are failed to be processed exist, an asynchronous job task is newly initiated to process each segmentation subtask.
Step S6: after each batch of segmented tasks to be processed are successfully processed, the batch tasks are completed.
And detecting system resources in real time by the batch monitor. monitor exists in the whole period of batch tasks, detects system resources in real time, judges whether the current system resource utilization exceeds an upper threshold according to the detection result, and adjusts the number of the subsequent batch tasks to be lower according to the segment adjustment step number adjust_step (i.e. the segment number which is adjusted to be increased or decreased each time) in the segment configuration if the current system resource utilization exceeds the upper threshold (e.g. cpu utilization 80%). If the optimal threshold (for example, CPU utilization rate is 70%) is not reached, locking the segmented subtask table (preventing the dispatcher from misjudging as no task to be processed), and adjusting the number of segmented tasks of the follow-up batch tasks according to the segment adjustment step number in the segment configuration so as to improve the batch processing efficiency.
The invention is mainly used for a banking system developed based on an Oracle storage process, and is used for solving the problem of task efficiency of batch processing of the banking system every day, realizing multi-task parallel processing among batch processing tasks and multi-segment parallel processing in the batch processing tasks.
After a batch task scheduling system initiates a batch processing instruction, the data volume of each batch task and the dependency relationship among tasks are automatically analyzed according to all batch tasks to be processed, each batch task is automatically divided into a plurality of sections according to the sectional configuration of each batch task, the batch tasks are ordered according to the dependency relationship, an oracle asynchronous processing job is created according to each section, and each sectional batch task is processed by the asynchronous processing job.
Meanwhile, in batch processing, a batch performance monitoring program monitors the use condition of system resources in real time, and the segmentation number of subsequent batch processing tasks is dynamically adjusted according to the use rate (such as CPU, memory, IO and the like) of the current resources.
Therefore, the system batch task segmentation parallel and dynamic scheduling method realizes the daily final batch task segmentation parallel of the banking system, and the segmentation is dynamically adjusted according to the available resource quantity, so that the batch task efficiency of the banking system is improved, and the batch task execution time is reduced.
The invention also provides a computer readable storage medium, on which a computer program is stored, which when executed by a processor, implements the system batch task segmentation parallel and dynamic scheduling method.
Example 1
Scene: the daily final batch processing of a banking system comprises 3 batch processing tasks:
task 1: account balance update. The corresponding segment is configured to: the segmentation is based on the field account number (account_no), the segmentation parameter 4 is initialized, the segmentation adjustment step number (adjust_step) is 2, and the segmentation total amount query SQL is the select count (account_no) from account_list.
Task 2: customer balance update. The segment basis field is customer number (customer_no), the segment parameter 3 is initialized, the segment adjustment step number (adjust_step) is 2, and the segment total query SQL is select count (customer_no) from customer_list.
Task 3: accounting is carried out on the accounting running water. The segment basis field is transaction sequence number (tran_no), the segment parameter 5 is initialized, the segment adjustment step number (adjust_step) is 2, and the segment total query SQL is select count (consumer_no) from consumer_list.
Wherein, task 2 is dependent on the completion of the balance update of task 1, and task 3 has no dependency on other tasks.
The number of bank accounts is 1 myriad, and the account number (accoun_no) increases from AC000001 to AC 100000. The number of clients is 5 thousands of clients, the number of clients (customer_no) increases from CN00001 to CN50000, the number of accounting flows is 35 thousands of clients, and the number of transactions (tran_no) increases from TN000001 to TN 350000.
Batch processing:
s1: the batch task scheduling system initiates a batch processing instruction startbatch_20210901 to the business system batch scheduler dispatcher.
S2, screening batch tasks, task 1, task 2 and task 3 which are needed to be processed by the batch tasks according to the instruction by the dispatcher, and starting a business system batch monitor simultaneously to detect the use condition of system resources in the batch task execution period in real time.
S3: the dispatcher calls up a segmenter segment to carry out segmentation processing on each batch task, and finally confirms the segmentation as follows according to the segmentation parameter configuration of each batch task:
task 1: according to the segmentation total amount query SQL, the account balance is queried to update the to-be-processed data volume sum of the batch task to be 1 ten thousand, and the initialization segmentation parameter is 4, so that the batch task needs to be divided into 4 segments according to the segmentation initialization parameter, and the to-be-processed data volume of each segment of the batch task is 100000/4=2500, namely:
subtask 1.1: account balance updating from AC00001 to AC25000 is responsible for account number.
Subtask 1.2: account balance updating between account numbers from AC25001 to AC50000 is responsible.
Subtask 1.3: account balance updating between the account number from the AC50001 to the AC75000 is responsible.
Subtask 1.4: account balance updating from AC75001 to AC10000 is responsible for account numbers.
Task 2: according to the segmentation total amount query SQL, the to-be-processed data volume sum of the client balance update batch task is 5 ten thousand, and the initialization segmentation parameter is 3, so that the batch task needs to be divided into 3 segments according to the segmentation initialization parameter, and the to-be-processed data volume of each segment of the batch task is 50000/4= 16666.67, namely 16667, namely:
subtask 2.1: is responsible for the balance update of the customer between the customer number from CN00001 to CN 16667.
Subtask 2.2: is responsible for customer balance updates between the customer number from CN16668 to CN 33335.
Subtask 2.3: is responsible for the balance update of the customer between the customer number from CN33336 to CN 50000.
Task 3: according to the segment total amount query SQL query accounting batch task, the to-be-processed data volume sum is 35 ten thousand, the initialization segment parameter is 5, then the batch task is required to be divided into 5 segments according to the segment initialization parameter, and the to-be-processed data volume of each segment of the batch task is 350000/5=70000, namely:
subtask 3.1: accounting flow accounting is responsible for transaction serial numbers from TN00001 to TN 70000.
Subtask 3.2: accounting is responsible for accounting of transaction sequence numbers from TN70001 to TN 140000.
Subtask 3.3: accounting is responsible for accounting of transaction sequence numbers from TN140001 to TN 210000.
Subtask 3.4: accounting is responsible for accounting of transaction sequence numbers from TN210001 to TN 280000.
Subtask 3.5: accounting is responsible for accounting of transaction sequence numbers from TN280001 to TN 350000.
And recording the subtask information completed by the segmentation in a segmentation subtask table.
S4, scanning the segmented subtask table in a full-scale mode by the dispatcher, scanning the subtasks which are to be processed and are completed by the dependent parent task, and creating an asynchronous job. When the dispatcher scans, 4 subtasks 1.1-1.4 of the subtask 1 are found to be unprocessed and have no dependence, and then 4 asynchronous processing jobs are directly created for the subtasks 1.1-1.4 respectively. Subtasks 2.1-2.3 are unprocessed, but depending on the completion of task 1 execution, it is found that task 1 is not currently being performed, and this scan does not process task 2. Continuing to scan down to task 3, finding that subtasks 3.1-3.5 of task 3 are not executing and are not dependent, then creating 4 asynchronous processing jobs for subtasks 3.1-3.5, respectively, directly. At this time, after the creation of the asynchronous processing job for the subtask creation of the task 1 and the task 3 is completed, the asynchronous processing job has already processed the data of the respective segments, and after the processing is completed, the state of the subtask in the segment subtask table is updated.
S6: the dispatcher full scan segment subtask table finds that there are no pending tasks and failed tasks, then the entire batch is processed.
And detecting system resources in real time by a batch monitor. The monitor always monitors the resource usage of batch processing, if it is found that the usage of system resources does not reach the optimal threshold (cpu usage reaches 70% and memory usage reaches 70%), the unprocessed task in the segmented subtask table is scanned, the task is lifted out, the segmentation number is adjusted once according to the segmentation adjustment step number adjust_step in the segmentation configuration of the task, and the segmentation is re-performed. When executing the asynchronous process of task 1 and task 3, monitor finds that the system resource usage is now as follows: cpu 50%, memory: 60% and does not reach the optimal threshold. At this time, the Monitor scans the segmented subtask table to find that all subtasks of the task 2 are processed, at this time, the Monitor locks the segmented subtask table (prevents the dispatcher from misjudging as no task to be processed), then segments are performed again according to the step S3, at this time, the segmentation parameter is adjusted to 5, and the segmented data are processed by the 5 subtasks. And after the segmentation in the step S3 is completed, the step S4 is used for creating an asynchronous processing procedure to process the segmented data.
The foregoing basic embodiments of the invention, as well as other embodiments of the invention, can be freely combined to form numerous embodiments, all of which are contemplated and claimed. In the scheme of the invention, each selection example can be arbitrarily combined with any other basic example and selection example. Numerous combinations will be apparent to those skilled in the art.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, and alternatives falling within the spirit and principles of the invention.
Claims (4)
1. The system batch task piecewise parallel and dynamic scheduling method is characterized in that the system batch task piecewise parallel and dynamic scheduling method is implemented based on an oracle storage process, and comprises the following steps of:
s1: initiating a batch processing instruction to a batch scheduler of the system;
s2: the batch dispatcher screens and prepares batch tasks to be processed at this time based on batch processing instructions, and starts a batch monitor of the system to detect the use condition of system resources in real time during the execution processing of the batch tasks;
s3: the batch dispatcher calls up a sectionalizer, and the sectionalizer sections each task based on batch tasks to be processed and sectionalizing parameters of each task;
s4: inquiring unprocessed segmented tasks which are segmented by a batch scheduler, screening out currently executable segmented tasks based on a logic relation among the tasks, and creating asynchronous work tasks one by one to process each segmented task;
s5: the batch dispatcher polls whether batch tasks to be processed or abnormal to be processed exist currently, and if the batch tasks which are not processed or failed to be processed exist, an asynchronous work task is newly initiated to process each segmented task;
s6: after each batch of segmented tasks to be processed are successfully processed, the batch tasks are completed;
and the step of detecting the system resources in real time by the batch monitor:
judging whether the current system resource utilization exceeds an upper limit threshold according to the detection result so as to finish the adjustment of the number of segmented tasks of the follow-up batch task;
when the batch monitor judges that the current system resource utilization exceeds the upper limit threshold, the number of segmented tasks of the follow-up batch tasks is reduced;
and the batch monitor judges that the current system resource utilization rate does not reach the optimal threshold value, and the number of the segmented tasks of the follow-up batch tasks is increased.
2. The system batch task segment parallel and dynamic scheduling method according to claim 1, wherein step S3 specifically includes:
s31, scanning batch tasks to be processed by a segmenter, searching segmentation configuration according to each batch task, and confirming segmentation basis fields and initializing segmentation parameters;
s32: the segmenter counts the total data quantity to be processed of each batch task according to the segmentation basis, and confirms the data quantity to be processed of each segment based on the initialized segmentation parameter configuration;
s33: and the segmenter confirms the start value and the end value of each segment of the batch task segment according to the field according to the data quantity and the data total quantity processed by each segment of the batch task, and records the start value and the end value in the segmented task list.
3. The system batch task segment parallelism and dynamic scheduling method of claim 1, wherein in step S5, the batch scheduler poll termination condition is that all batch segment tasks to be processed have been successfully processed.
4. A computer readable storage medium, on which a computer program is stored, which computer program, when being executed by a processor, implements the method according to any of claims 1-3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111107163.6A CN113807710B (en) | 2021-09-22 | 2021-09-22 | System batch task segmentation parallel and dynamic scheduling method and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111107163.6A CN113807710B (en) | 2021-09-22 | 2021-09-22 | System batch task segmentation parallel and dynamic scheduling method and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113807710A CN113807710A (en) | 2021-12-17 |
| CN113807710B true CN113807710B (en) | 2023-06-20 |
Family
ID=78939893
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111107163.6A Active CN113807710B (en) | 2021-09-22 | 2021-09-22 | System batch task segmentation parallel and dynamic scheduling method and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113807710B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004280457A (en) * | 2003-03-14 | 2004-10-07 | Japan Research Institute Ltd | Data management system, batch control system, and batch control program |
| JP2008234024A (en) * | 2007-03-16 | 2008-10-02 | Fujitsu Ltd | Batch processing program, batch processing method and batch processing device |
| CN102609875A (en) * | 2012-01-20 | 2012-07-25 | 中国工商银行股份有限公司 | Method and system for concurrently and dynamically processing bank data |
| CN105677752A (en) * | 2015-12-30 | 2016-06-15 | 深圳先进技术研究院 | Streaming computing and batch computing combined processing system and method |
| CN109144731A (en) * | 2018-08-31 | 2019-01-04 | 中国平安人寿保险股份有限公司 | Data processing method, device, computer equipment and storage medium |
| CN111858653A (en) * | 2020-07-15 | 2020-10-30 | 武汉众邦银行股份有限公司 | Distributed batch processing method based on database segmentation |
| CN112433838A (en) * | 2020-11-25 | 2021-03-02 | 深圳前海微众银行股份有限公司 | Batch scheduling method, device, equipment and computer storage medium |
| CN113254437A (en) * | 2020-02-11 | 2021-08-13 | 北京京东振世信息技术有限公司 | Batch processing job processing method and device |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050198636A1 (en) * | 2004-02-26 | 2005-09-08 | International Business Machines Corporation | Dynamic optimization of batch processing |
| US8078434B2 (en) * | 2008-07-17 | 2011-12-13 | Honeywell International Inc. | Systems and methods for offline and/or online batch monitoring using decomposition and signal approximation approaches |
-
2021
- 2021-09-22 CN CN202111107163.6A patent/CN113807710B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004280457A (en) * | 2003-03-14 | 2004-10-07 | Japan Research Institute Ltd | Data management system, batch control system, and batch control program |
| JP2008234024A (en) * | 2007-03-16 | 2008-10-02 | Fujitsu Ltd | Batch processing program, batch processing method and batch processing device |
| CN102609875A (en) * | 2012-01-20 | 2012-07-25 | 中国工商银行股份有限公司 | Method and system for concurrently and dynamically processing bank data |
| CN105677752A (en) * | 2015-12-30 | 2016-06-15 | 深圳先进技术研究院 | Streaming computing and batch computing combined processing system and method |
| CN109144731A (en) * | 2018-08-31 | 2019-01-04 | 中国平安人寿保险股份有限公司 | Data processing method, device, computer equipment and storage medium |
| CN113254437A (en) * | 2020-02-11 | 2021-08-13 | 北京京东振世信息技术有限公司 | Batch processing job processing method and device |
| CN111858653A (en) * | 2020-07-15 | 2020-10-30 | 武汉众邦银行股份有限公司 | Distributed batch processing method based on database segmentation |
| CN112433838A (en) * | 2020-11-25 | 2021-03-02 | 深圳前海微众银行股份有限公司 | Batch scheduling method, device, equipment and computer storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 基于子任务区域分片下的分布式空间查询处理与并行调度方法;何永强;谷春英;;科技通报;第30卷(第1期);第110-116页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113807710A (en) | 2021-12-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113238838B (en) | Task scheduling method and device and computer readable storage medium | |
| CN108280150B (en) | Distributed asynchronous service distribution method and system | |
| US8584122B2 (en) | Batch scheduling | |
| CN109814994B (en) | Method and terminal for dynamically scheduling thread pool | |
| US8863136B2 (en) | Lock control in multiple processor systems | |
| US8041686B2 (en) | Database contention and deadlock detection and reduction within application servers | |
| CN111858086B (en) | Processing method and storage medium for queue timeout in request task processing | |
| US10768974B2 (en) | Specifying an order of a plurality of resources in a transaction according to distance | |
| CN113807710B (en) | System batch task segmentation parallel and dynamic scheduling method and storage medium | |
| CN113485812B (en) | Partition parallel processing method and system based on large-data-volume task | |
| CN113467908A (en) | Task execution method and device, computer readable storage medium and terminal equipment | |
| CN111367645B (en) | Real-time task scheduling control method, system, equipment and readable storage medium | |
| CN109146672B (en) | Matching method and device based on production mode | |
| CN116541167A (en) | System flow control method, device, electronic equipment and computer readable medium | |
| CN111259032A (en) | Service processing method and device | |
| CN115934272A (en) | Online batch task processing method and device | |
| WO2022267395A1 (en) | Payment request processing method and system | |
| CN114237884A (en) | Cloud computing data center energy consumption optimization method based on gray Markov chain | |
| CN112199401B (en) | Data request processing method, device, server, system and storage medium | |
| CN115391013A (en) | Task execution method and device and related equipment | |
| CN111583003B (en) | Massive financial processing transaction parallel execution method in capital centralized control process | |
| CN108874546B (en) | Linux system and method for optimizing resource use | |
| CN113505017A (en) | Problem troubleshooting method, device, storage medium and equipment | |
| CN112350887B (en) | APM probe sampling rate determining method, computer equipment and storage medium | |
| CN116719626B (en) | Multithreading parallel processing method and processing system for splitting mass data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |