CN111209101A - Big data calculation task multi-dependence scheduling system - Google Patents
Big data calculation task multi-dependence scheduling system Download PDFInfo
- Publication number
- CN111209101A CN111209101A CN202010008151.7A CN202010008151A CN111209101A CN 111209101 A CN111209101 A CN 111209101A CN 202010008151 A CN202010008151 A CN 202010008151A CN 111209101 A CN111209101 A CN 111209101A
- Authority
- CN
- China
- Prior art keywords
- task
- module
- dependency
- scheduling
- big data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/48—Indexing scheme relating to G06F9/48
- G06F2209/485—Resource constraint
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5021—Priority
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a big data computing task multi-dependency scheduling system which comprises a user side, a Web visualization module, a task template generation module, an actual task generation module, a task dependency solution module, a scheduling optimization calculation module, an actual task scheduling module and a big data computing platform. The invention can automatically realize the processing of complex task dependency relationship by simply configuring task parameters, reasonably distribute cluster computing resources and effectively track the computing process and results. The invention greatly simplifies the scheduling management process of big data computing tasks with complex dependency relationship, improves the utilization rate of cluster computing resources, strengthens the state management of executing tasks, and simultaneously reduces the use difficulty and the possibility of task errors.
Description
Technical Field
The invention belongs to the technical field of computers, and relates to a big data computing task multi-dependency scheduling system.
Background
With the rapid development of big data technology, the storage and calculation of a huge amount of off-line data are not difficult, and the most mainstream solution is a Hadoop distributed system, the core of which is a distributed file system HDFS, a unified resource management and scheduling system YARN and a Spark memory calculation engine. However, as the computing process and the dependency relationship between different computing processes become more complex, how to simplify the management of the increasingly complex computing dependency relationship, how to accurately grasp the state of the task scheduling process, and how to accurately and efficiently complete the task scheduling of big data computing are the major problems faced by the current big data scheduling system.
At the present stage, the big data computing task scheduling modes are quite various, and the big data computing task scheduling modes comprise a program Crontab and a class library Quartz which are biased to be executed by a single machine, an open source distributed scheduling system Oozie and Azkaban, scheduling software which is self-researched by other companies or based on open source packaging, and the like. However, there are the following problems: 1. the dependence relationship of simple sequence cannot be processed or only can be processed, the complex dependence needs to be packaged and transformed, the great development cost needs to be invested, and the function is limited by the software; 2. task intensive scheduling tasks cannot reasonably distribute cluster resources, so that cluster computing resources are inclined, and scheduling is congested; 3. the operation mode is not friendly, the learning cost is high, and the development and scheduling efficiency is low; 4. the task resource management function is not needed, and the management is completely carried out by the user. Therefore, it is desirable to provide a big data computing task multi-dependency scheduling system.
Disclosure of Invention
In order to overcome the defects in the prior art, a big data computing task multi-dependency scheduling system is provided.
The invention is realized by the following scheme:
a big data calculation task multi-dependency scheduling system comprises a user side, a Web visualization module, a task template generation module, an actual task generation module, a task dependency solution module, a scheduling optimization calculation module, an actual task scheduling module and a big data calculation platform;
the Web visualization module is used for providing a simple and understandable task management Web interface, supporting task state management, supporting cluster resource management, creating a task, filling in or modifying task parameters;
the task template generating module is used for verifying and storing the task parameters filled or modified in the Web visualization module and generating a task template;
the actual task generating module is used for checking the task template according to the set execution time, generating an actual task by using the task template, organizing task parameters into an execution command which can be directly submitted to the computing cluster, and storing the execution command;
the task dependence solving module is used for solving the actual task in a bidirectional dependence manner;
the scheduling optimization calculation module is used for adjusting the execution sequence of the tasks to be scheduled, wherein the execution sequence solves the dependency relationship;
and the actual task scheduling module is used for receiving the execution sequence submitted by the scheduling optimization module, submitting the execution sequence to the big data computing platform for operation, judging that the actual task goes to the task dependence solving module or the scheduling optimization computing module according to the operation result of the big data computing platform, and returning the execution result to the Web visualization module.
The task template generation module supports setting task priority and adjusting task execution sequence, and supports setting tolerant delay time.
The tolerated delay time is used to evaluate run-out warnings and post-run result evaluations.
The bidirectional dependency resolution comprises the generation of new task dependency resolution and the batch dependency resolution of each completed task.
The method has the beneficial effects that:
1. the big data computing task multi-dependency scheduling system provides a quick big data computing task submitting method, learning cost is low, and meanwhile, the semi-automation solution of the task dependency relationship enables a user to only consider the current task dependency, so that the process of establishing the dependency relationship is simplified, and the possibility of errors caused by the complex dependency relationship is reduced;
2. the solution of the task dependency relationship in the big data computing task multi-dependency scheduling system can process the complex multi-dependency relationship, and abstract the task into the template, thereby realizing the template management of the task resource.
3. The big data computing task multi-dependence scheduling system can process tasks with different priorities and different urgency degrees set by a user through the scheduling optimization computing module, enhances the control of the user on the task execution sequence, simultaneously realizes automatic resource balance, and can fully utilize computing resources without inclination.
4. According to the big data computing task multi-dependency scheduling system, the process of solving problems due to dependency can be reminded in time through judging whether the task exceeds the preset time before execution, the scheduling sequence is optimized through analyzing the historical data after the task is executed so as to estimate the next execution time of the task, and further optimization of the task execution sequence is achieved.
Drawings
FIG. 1 is a block flow diagram of a big data computing task multi-dependency scheduling system according to the present invention;
Detailed Description
The invention is further illustrated by the following specific examples:
a big data calculation task multi-dependency scheduling system comprises a user side, a Web visualization module, a task template generation module, an actual task generation module, a task dependency solution module, a scheduling optimization calculation module, an actual task scheduling module and a big data calculation platform;
the Web visualization module is used for providing a simple and understandable task management Web interface, supporting task state management, supporting cluster resource management, creating a task, filling in or modifying task parameters; the Web visualization module provides a simple and understandable task management Web interface, supports task state management and supports cluster resource management.
The Web visualization module is a friendly operation interface provided by the scheduling system, can carry out comprehensive management on tasks, and provides functions including but not limited to new creation, modification, deletion and start-stop of the tasks, viewing and modification of real-time task states, query and statistics of historical tasks, viewing of cluster load states and the like.
The task template generating module is used for verifying and storing the task parameters filled or modified in the Web visualization module and generating a task template; the task template generation module supports setting task priority and adjusting task execution sequence, and supports setting tolerant delay time. The tolerated delay time is used to evaluate run-out warnings and post-run result evaluations. And the task template generating module is matched with the Web visualization module, checks and stores the filled or modified task parameters, and generates a corresponding task template for a subsequent actual task generating module to use.
The task template generation module mainly comprises task basic information, a task execution file, a task execution period, task execution parameters and a task dependency template. The basic information of the task comprises basic information such as a task name, a creator, creation time and the like; the task execution file detects the integrity of the executable file and stores the integrity of the executable file to the HDFS; the task execution period comprises task execution time, execution period, task priority and tolerant delay time; the task execution period supports setting of task periods according to various time intervals, supports setting of task priorities, adjusts task execution sequences, and sets tolerant delay time to provide running overtime warning and running result evaluation.
The task execution parameters comprise additional parameters required by the task and support the transfer variables. The task extra parameters support the transmission of variable parameters filled by the scheduling system, such as task execution time, time interval, submitting user, etc., and also support the transmission of custom variables.
The task dependence template is a dependent task name + time dependence expression mode. The method supports the selection of dependent task names from configured tasks, supports the time-dependent expression to set the dependency relationship between periodic tasks, and supports the self-dependent setting of different execution times of the periodic tasks.
The actual task generating module is used for checking the task template according to the set execution time, generating an actual task by using the task template, organizing task parameters into an execution command which can be directly submitted to the computing cluster, and storing the execution command; the actual task generating module is mainly used for generating a corresponding actual task by using the task template generating module according to the set execution time.
The actual task generation module has the following specific flow:
(1) checking the task template according to the actual time to obtain the task template which accords with the execution time;
(2) if the task execution parameters have variables, performing variable replacement on the task execution parameters, and if the dependency exists, reading all the dependency of the task according to the actual time;
(3) splicing the tasks to execute the actual commands, and then storing other information, the executed commands, the dependency relationships and the like of the tasks as the actual tasks;
(4) the actual task generation module is executed according to the period, so that the template is rechecked after the completion of the task and the next checking time, and a cycle is formed.
The task dependence solving module is used for solving the actual task in a bidirectional dependence manner; the bidirectional dependency resolution comprises the generation of new task dependency resolution and the batch dependency resolution of each completed task. The task dependence solving module is mainly responsible for solving the task dependence from two aspects, namely solving the dependence of all generated new tasks from top to bottom and solving the dependence of each completed task in batches from bottom to top. The task dependency solution module adopts a bidirectional dependency solution mode, ensures that the checking of the dependency relationship is finished at the highest efficiency, and can easily solve the dependency relationship even under the conditions of numerous tasks and complex dependency relationship.
The specific flow of the task dependency solution module is as follows:
(1) firstly, the dependency resolution is performed from top to bottom, after the actual task generating module generates the actual task, the task dependency resolution module performs matching on all dependency relationships of the task in the completed task to confirm which dependencies are resolved;
(2) judging whether the dependency is completely solved, if the dependency is completely solved, handing the task to a scheduling optimization calculation module, otherwise, waiting and then solving the dependency;
(3) and (3) performing batch dependency resolution on all tasks which depend on the successful task once after the task is successfully executed, wherein the dependency is processed by all tasks which depend on the successful task, and then performing the operation in the step (2) on the tasks.
The scheduling optimization calculation module is used for adjusting the execution sequence of the tasks to be scheduled, wherein the execution sequence solves the dependency relationship; and the scheduling optimization calculation module selects a proper number of tasks and an execution sequence after comprehensive consideration is given to the tasks, the resources, the residual resources and the like according to the task priority, the time urgency degree, the historical task execution time, the number of the tasks in scheduling, the predicted occupation of the task resources, the cluster residual resources and the like, and delivers the tasks and the execution sequence to the actual task scheduling module for scheduling.
The specific flow of the scheduling optimization calculation module is as follows:
(1) aiming at all the tasks to be scheduled with the dependency relationship solved, firstly calculating the urgency degree of the tasks, and then carrying out priority sequencing and urgency degree secondary sequencing on the tasks to obtain a basic execution sequence;
(2) computing resource balance is carried out after the cluster residual computing resources are obtained, tasks with insufficient resources are skipped over, and the tasks are submitted to an actual task scheduling module according to a task sequence until the residual resources are smaller than a minimum task submitting parameter;
(3) and periodically checking the cluster resources after submission, then judging whether the tasks can be continuously submitted, if so, re-pulling the tasks to be scheduled for sequencing, and entering a circulating state.
And the actual task scheduling module is used for receiving the execution sequence submitted by the scheduling optimization module, submitting the execution sequence to the big data computing platform for operation, judging that the actual task goes to the task dependence solving module or the scheduling optimization computing module according to the operation result of the big data computing platform, and returning the execution result to the Web visualization module. The actual task scheduling module is responsible for scheduling and running tasks, monitoring task execution states, managing task execution logs, retrying failed tasks and the like.
The actual task scheduling module has the following specific flow:
(1) the actual task scheduling module receives the tasks submitted from the scheduling optimization computing module and then submits the tasks to the big data computing platform for operation;
(2) judging an operation result, if the operation result is successful, proceeding to a task dependence solving module for batch dependence solving, and if the operation result is failed, judging whether to enter a retry process according to whether to retry or not;
(3) if the task needs to retry, the state of the failed task is modified according to the retry interval timing, and the failed task is reset into the queue to be scheduled of the scheduling optimization calculation module.
The specific working process of the invention is as follows:
step one, a task is newly built through a Web visualization module, task parameters are filled in, whether the task is directly executed or not is judged according to relevant regulations of a task template generation module, and if yes, the task is delivered to a scheduling optimization calculation module; if not, the task template is delivered to the task template generation module.
And secondly, generating a task template. And after the task parameters are filled, entering a task template generation module. If abnormity is detected in the task template generating process, the failure reason is informed, and the Web visualization module is handed to modify the task; and if the abnormity is not detected in the task template generating process, entering an actual task generating module.
And thirdly, generating a module for an actual task. If abnormity is detected in the actual task generation process, informing the failure reason, and handing over to a Web visualization module for task modification; and if the exception is not detected in the actual task generation process, entering a task dependency solution module.
And fourthly, a task dependence solving module. If the delay tolerant time is found to be reached in the dependency resolution process, the Web visualization module is informed that the task is overtime, but the task will continue to resolve the dependency until the next step is carried out after the dependency resolution is completed.
And fifthly, scheduling an optimization calculation module. And receiving the tasks submitted by the direct execution and dependency solution module, optimizing the scheduling sequence, and then entering the next step.
And sixthly, an actual task scheduling module. And actually submitting the task to a big data computing platform and executing the task, and returning an execution result to the Web visualization module.
And step seven, finishing.
The invention relates to a big data calculation task multi-dependency scheduling system, which is a process for establishing a dependency relationship between templates by abstracting tasks into templates and then setting a time dependency expression; the task dependence solving mode is bidirectional processing, when a task is newly built, the task dependence is scanned and solved, and when a task is completed, all the dependence modes depending on the task are solved; semi-automatic dependency solving processing, namely automatically realizing the processing of the dependency so that a user only needs to pay attention to the technology of the dependency of the current task; a scheduling optimization process for dynamically adjusting the task execution sequence according to the priority, the task urgency degree, the cluster residual resources, the historical task execution statistics and the like; the method realizes task dependence warning and task execution effect evaluation for overdue time before and after task execution.
Although the invention has been described and illustrated in some detail, it should be understood that various modifications may be made to the described embodiments or equivalents may be substituted, as will be apparent to those skilled in the art, without departing from the spirit of the invention.
Claims (4)
1. A big data computing task multi-dependency scheduling system is characterized in that: the scheduling system comprises a user side, a Web visualization module, a task template generation module, an actual task generation module, a task dependency solution module, a scheduling optimization calculation module, an actual task scheduling module and a big data calculation platform;
the Web visualization module is used for providing a simple and understandable task management Web interface, supporting task state management, supporting cluster resource management, creating a task, filling in or modifying task parameters;
the task template generating module is used for verifying and storing the task parameters filled or modified in the Web visualization module and generating a task template;
the actual task generating module is used for checking the task template according to the set execution time, generating an actual task by using the task template, organizing task parameters into an execution command which can be directly submitted to the computing cluster, and storing the execution command;
the task dependence solving module is used for solving the actual task in a bidirectional dependence manner;
the scheduling optimization calculation module is used for adjusting the execution sequence of the tasks to be scheduled, wherein the execution sequence solves the dependency relationship;
and the actual task scheduling module is used for receiving the execution sequence submitted by the scheduling optimization module, submitting the execution sequence to the big data computing platform for operation, judging that the actual task goes to the task dependence solving module or the scheduling optimization computing module according to the operation result of the big data computing platform, and returning the execution result to the Web visualization module.
2. The big data computing task multi-dependency scheduling system of claim 1, wherein: the task template generation module supports setting task priority and adjusting task execution sequence, and supports setting tolerant delay time.
3. The big data computing task multi-dependency scheduling system of claim 2, wherein: the tolerated delay time is used to evaluate run-out warnings and post-run result evaluations.
4. The big data computing task multi-dependency scheduling system of claim 1, wherein: the bidirectional dependency resolution comprises the generation of new task dependency resolution and the batch dependency resolution of each completed task.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010008151.7A CN111209101B (en) | 2020-01-06 | 2020-01-06 | Big data calculation task multi-dependency scheduling system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010008151.7A CN111209101B (en) | 2020-01-06 | 2020-01-06 | Big data calculation task multi-dependency scheduling system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111209101A true CN111209101A (en) | 2020-05-29 |
| CN111209101B CN111209101B (en) | 2023-05-02 |
Family
ID=70789541
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010008151.7A Active CN111209101B (en) | 2020-01-06 | 2020-01-06 | Big data calculation task multi-dependency scheduling system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111209101B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112559486A (en) * | 2020-11-11 | 2021-03-26 | 国网江苏省电力有限公司信息通信分公司 | Data center unified task scheduling management system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109375996A (en) * | 2018-09-27 | 2019-02-22 | 安徽省鼎众金融信息咨询服务有限公司 | A kind of support dependence managerial role scheduling system |
| CN109669767A (en) * | 2018-11-30 | 2019-04-23 | 河海大学 | A kind of task encapsulation and dispatching method and system towards polymorphic type Context-dependent |
| CN109684053A (en) * | 2018-11-05 | 2019-04-26 | 广东岭南通股份有限公司 | The method for scheduling task and system of big data |
-
2020
- 2020-01-06 CN CN202010008151.7A patent/CN111209101B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109375996A (en) * | 2018-09-27 | 2019-02-22 | 安徽省鼎众金融信息咨询服务有限公司 | A kind of support dependence managerial role scheduling system |
| CN109684053A (en) * | 2018-11-05 | 2019-04-26 | 广东岭南通股份有限公司 | The method for scheduling task and system of big data |
| CN109669767A (en) * | 2018-11-30 | 2019-04-23 | 河海大学 | A kind of task encapsulation and dispatching method and system towards polymorphic type Context-dependent |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112559486A (en) * | 2020-11-11 | 2021-03-26 | 国网江苏省电力有限公司信息通信分公司 | Data center unified task scheduling management system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111209101B (en) | 2023-05-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109992407B (en) | YARN cluster GPU resource scheduling method, device and medium | |
| EP2666083B1 (en) | Integrated environment for execution monitoring and profiling of applications running on multi-processor system-on-chip | |
| CN110413391A (en) | Deep learning task service method for ensuring quality and system based on container cluster | |
| CN106055670A (en) | Inter-system data migration method and device | |
| CN113157710B (en) | Block chain data parallel writing method and device, computer equipment and storage medium | |
| CN112445598B (en) | Task scheduling method and device based on quartz, electronic equipment and medium | |
| US20040268335A1 (en) | Modulo scheduling of multiple instruction chains | |
| CN112416585A (en) | GPU resource management and intelligent scheduling method for deep learning | |
| CN112131315A (en) | Database synchronization method and device under multi-task parallel development | |
| WO2024032781A1 (en) | Algorithm testing method and apparatus, and storage medium | |
| CN111209101A (en) | Big data calculation task multi-dependence scheduling system | |
| CN114625507A (en) | Task scheduling method, system, equipment and storage medium based on directed acyclic graph | |
| CN114625517A (en) | DAG graph computation distributed big data workflow task scheduling platform | |
| CN115934362A (en) | Deep learning-oriented server non-perception computing cluster scheduling method and product | |
| CN114781648A (en) | Automatic arranging, constructing and executing method and system for machine learning task | |
| CN111176831A (en) | Dynamic thread mapping optimization method and device based on multithread shared memory communication | |
| CN110134506A (en) | Based on the dynamic real-time Dynamic Critical Path multi-core dispatching method of processor cores | |
| CN111125097B (en) | Report scheduling method and device | |
| CN117311990A (en) | Resource adjustment method and device, electronic equipment, storage medium and training platform | |
| CN113010276A (en) | Task scheduling method and device, terminal equipment and storage medium | |
| WO2022253165A1 (en) | Scheduling method, system, server and computer readable storage medium | |
| CN114168594A (en) | Secondary index creating method, device, equipment and storage medium of horizontal partition table | |
| CN110297692A (en) | A kind of method and its system of the management of distributed software task dynamic | |
| CN116680051B (en) | Task scheduling method, device, equipment and storage medium | |
| Walser et al. | Real-time Container Orchestration Based on Time-utility Functions |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |