CN111813527B - Data-aware task scheduling method - Google Patents
Data-aware task scheduling method Download PDFInfo
- Publication number
- CN111813527B CN111813527B CN202010677871.2A CN202010677871A CN111813527B CN 111813527 B CN111813527 B CN 111813527B CN 202010677871 A CN202010677871 A CN 202010677871A CN 111813527 B CN111813527 B CN 111813527B
- Authority
- CN
- China
- Prior art keywords
- node
- task
- map
- data
- task scheduling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5066—Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs
Abstract
According to the data-aware task scheduling method disclosed by the invention, the MapReduce is embedded into the task scheduling algorithm strategy, and the improvement of the job scheduling algorithm is realized by embedding the MapReduce into the task scheduling algorithm strategy in the form of the Map task scheduling strategy, so that the problem of low efficiency of the conventional task scheduling strategy in the Reduce stage is solved, and the execution time of the MapReduce job is further shortened.
Description
Technical Field
The invention belongs to the technical field of data centers, and particularly relates to a data-aware task scheduling method.
Background
Hadoop is a software framework capable of performing distributed processing on mass data. The core component parallel programming model MapReduce provides parallel processing capacity for large data processing. The Reduce phase of the MapReduce job execution process needs to copy the intermediate data generated in the Map phase as input and calculate to generate a final result. Aiming at the defect of the default Reduce scheduling strategy of the conventional Hadoop platform, the invention provides an optimization strategy for scheduling Reduce tasks.
The scheduling process in the Hadoop platform is a three-level scheduling model, the difference of the existing FIFO scheduling algorithm, Capacity scheduling algorithm and Fair scheduling algorithm mainly lies in two layers of queue selection and job selection, and the strategies of the three scheduling algorithms on the scheduling layer of task selection are the same, namely the factors considered and the balance used for selecting a Map task or Reduce task from a specific job to be scheduled on a task scheduler node for execution are the same. The existing related Reduce task scheduling strategies are few and simple, and the existing task scheduling strategies are that when an idle Reduce target node exists on a task scheduler node, the node requests a Reduce task, and the job scheduler selects a task of a first job meeting conditions from a job queue to be allocated to the node as soon as possible.
The three existing scheduling algorithms on the Hadoop platform do not distinguish nodes for running Reduce tasks, namely, a proper Reduce task is scheduled to a node which just applies for the Reduce task at a proper time to be executed, and the node cannot be guaranteed to be the most proper node for executing the task. The scheduled node is not the optimal node for executing the task, which may cause the intermediate data generated by the Map to be forwarded through more hops, the link length of data transmission on the network is longer, the consumption of bandwidth is more, and the execution time of a single job is prolonged. The invention can select the optimal node to execute the Reduce task, so that the length of the link through which the intermediate data is transmitted is shortest, and the purpose of improving the task execution efficiency is achieved.
Disclosure of Invention
The data-aware task scheduling method disclosed by the invention realizes the improvement of the job scheduling algorithm by embedding the form of adopting the Map task scheduling strategy, thereby solving the problem of lower efficiency of the existing task scheduling strategy in the Reduce stage and further shortening the execution time of the MapReduce job.
The term in the scheme is explained as follows:
hadoop: a distributed system framework designed for deployment on inexpensive (low-cost) hardware, high-speed computation, and reliable storage.
Task scheduling: the method refers to allocating resources to be calculated to tasks meeting the conditions according to certain constraint regulations.
MapReduce: a parallel computing model and method for large-scale data processing. The calculation model is divided into two orders of Map and Reduce. Map maps key-value pairs into new key-value pairs, specifying a concurrent Reduce function to ensure that each of all mapped key-value pairs share the same key-group.
The invention discloses a data-aware task scheduling method, which comprises the following steps:
step 1, establishing an N-1 level scheduling model based on a distributed system infrastructure, determining a corresponding scheduling strategy, embedding and executing MapReduce aiming at an Nth level scheduling model, selecting a specific task queue and selected specific jobs for a Map task scheduling strategy, starting a Map task and executing, wherein N is a positive integer greater than 1;
step 2, obtaining information of a task scheduler node for executing the Map task, wherein the information of the task scheduler node at least comprises the number of the node and the position level of the node in the topological graph;
step 3, judging whether the Map tasks of the jobs are distributed on the same level in the cluster topological graph, if so, selecting the optimal node from the nodes on the level, preferentially selecting the node for executing the Map tasks during selection, if the nodes for executing the Map tasks do not have idle resources, selecting other nodes with idle resources on the level and outputting, and skipping to the step 6 for execution; if not, the next step of judgment is carried out;
step 4, if the Map tasks of the operation are distributed unevenly in the two subtrees of the root node, cutting branches with small quantity for executing the Map tasks, taking ancestor nodes of the rest branches as new root nodes, and skipping to the step 3 to continue execution;
step 5, if the Map tasks of the operation are distributed in the two subtrees of the root node in a balanced manner, cutting off a branch with a larger subtree depth, taking the ancestor node of the rest branches as a new root node, and skipping to the step 3 to continue the execution;
step 6, preparing to dispatch the Reduce task of the operation to be executed on the output optimal node;
and 7, the Reduce task obtains a Reduce resource queue to execute until the operation is finished.
The invention discloses an improvement of a data-aware task scheduling method, wherein a distributed system infrastructure in step 1 is Hadoop.
The invention discloses an improvement of a data-aware task scheduling method, wherein N is 3 in step 1, when a distributed system infrastructure is Hadoop, a Hadoop-default fair scheduler is used as a first-stage scheduling strategy and a second-stage scheduling strategy, a Map task scheduling strategy is embedded into the first-stage scheduling strategy and the second-stage scheduling strategy to form a third-stage scheduling strategy, a specific job in a specific queue is selected by the Map task scheduling strategy, and a Map task is started and executed.
The invention discloses an improvement of a data-aware task scheduling method, wherein when step 1 is executed, the number of specific tasks selected by a Map task scheduling strategy in a specific task queue is 1 at the same stage.
The invention discloses an improvement of a data-aware task scheduling method, wherein the same stage is the same clock node or the same algorithm execution process (the same algorithm execution process can refer to a cycle from step 1 to step 7 of the complete method to the end of operation).
The invention discloses an improvement of a data-aware task scheduling method, wherein in step 3, whether Map tasks of jobs are distributed in the same level in a cluster topological graph is judged, and the absolute value of the depth difference between a left sub-tree and a right sub-tree of a non-terminal node is limited to be not more than 1.
In the step 3, judging whether Map tasks of the job are distributed on the same level in the cluster topological graph includes the following situations that the depths of two subtrees are the same or the depth of one subtree is 1 more than that of the other subtree.
The invention discloses an improvement of a data-aware task scheduling method, wherein a task scheduler node for executing a Map task in step 2 is a TaskTracker node, and a TaskTracker node set is formed.
The invention discloses an improvement of a data-aware task scheduling method, wherein each TaskTracker node in a TaskTracker node set contains an idle ReduceSlot resource.
The optimized task scheduling strategy provided by the invention belongs to the third level of a scheduling model, is a general optimized scheduling strategy and can be embedded into other job scheduling algorithms. The task scheduling strategy provided by the invention solves the problem of low efficiency of the existing task scheduling strategy in the Reduce stage, and further shortens the execution time of the MapReduce operation.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a task scheduling flowchart according to an embodiment of the present application.
Detailed Description
The present invention will be described in detail below with reference to various embodiments. The embodiments are not intended to limit the present invention, and structural, methodological, or functional changes made by those skilled in the art according to the embodiments are included in the scope of the present invention.
In the implementation process, the data-aware task scheduling method disclosed by the invention can comprise the following steps:
step 1, selecting a proper distributed system infrastructure, establishing an N-1 level scheduling model on the basis, determining scheduling strategies corresponding to each level of scheduling models, embedding and executing MapReduce aiming at an Nth level scheduling model on the basis, establishing a scheduling model with a total level number of N, wherein N is a positive integer of at least 2, selecting a specific task queue and selected specific operation for a Map task scheduling strategy at the Nth level, starting a Map task and executing the Map task;
step 2, obtaining information of a task scheduler node for executing the Map task, wherein the information of the task scheduler node at least comprises the serial number of the node and the position level of the node in the topological graph; the nodes can be screened or the priority of the nodes can be determined according to preset priority determination conditions through the information, such as the level on a topological graph, and the like, and the nodes are used for participating in optimized screening between Map tasks and the nodes in subsequent execution. Preferably, the task scheduler node executing the Map task in this step is a TaskTracker node, and forms a TaskTracker node set. Further preferably, each TaskTracker node in the set of TaskTracker nodes contains a free ReduceSlot resource. This is basically an optimal ideal situation, because only one Task has an opportunity to run after a slot is obtained, and the Hadoop scheduler is used to allocate free slots on each tasktacker to the tasks for use. In this case, the scheduling method of the present scheme has the highest policy efficiency.
Step 3, judging whether the Map tasks of the operation are distributed on the same level in the cluster topological graph, if so, selecting the optimal node from the nodes on the level, thereby realizing the high-efficiency optimization of the Map tasks and the node strategies, solving the problem of low efficiency of the scheduling strategies, preferentially selecting the node for executing the Map tasks during execution, if the node for executing the Map tasks does not have idle resources, selecting other nodes with idle resources on the level and outputting the nodes, and skipping to the step 6 for execution; if not, the next step of judgment is carried out;
step 4, if the Map tasks of the operation are distributed unevenly in the two subtrees of the root node, cutting branches with small quantity of Map tasks, taking ancestor nodes of the rest branches as new root nodes, and skipping to the step 3 to continue the execution;
step 5, if the Map tasks of the operation are distributed in the two subtrees of the root node in a balanced manner, cutting off a branch with a larger subtree depth, taking the ancestor node of the rest branches as a new root node, and skipping to the step 3 to continue the execution;
step 6, preparing to dispatch the Reduce task of the operation to be executed on the output optimal node;
and 7, the Reduce task obtains the Reduce resource queue to be executed until the operation is finished, and then the next task is continued.
In the above scheme, in order to obtain higher execution efficiency, in the step 1, the number of the specific tasks selected by the Map task scheduling policy in the specific task queue is 1 at the same stage, so that the optimal scheduling scheme can be obtained for each task in task scheduling. Preferably, the same phase is the same clock node or the same algorithm execution (the same algorithm execution may refer to a cycle from step 1 to step 7 of the complete method to the end of the run).
In the above solution, in step 3, it is determined whether Map tasks of the job are distributed on the same level in the cluster topology Map, and the absolute value of the depth difference between the left sub-tree and the right sub-tree of the non-terminal node (such as a router) is not greater than 1, so that the depths of the Map tasks on the cluster topology Map during execution are close. Preferably, the step 3 of determining whether the Map tasks of the jobs are distributed on the same level in the cluster topology includes the following situations that the depths of two subtrees are the same or the depth of one subtree is 1 more than the depth of the other subtree.
As we know, the scheduling of the Reduce task and the scheduling of the Map task of the job submitted to the Hadoop platform are two separated parts and do not affect each other, so that the Map task can be considered to be scheduled well according to a corresponding scheduling strategy before the Reduce task is scheduled. And the distribution information of the Map task can be obtained from the system to serve as a known condition when the Reduce task is scheduled, and the topological structure of the cluster can be obtained from the configuration file of the cluster.
The task scheduling policy flow of the present invention is as follows, as shown in fig. 1:
step 1, selecting specific jobs in a specific queue by using a Hadoop default fair scheduler as a first-stage scheduling strategy, a second-stage scheduling strategy and a Map task scheduling strategy, starting a Map task and executing the Map task;
step 2, acquiring information of nodes of a task scheduler for executing the Map task, wherein the information mainly comprises the number of the nodes, the position levels of the nodes in a topological graph and the like, and the information is utilized for carrying out the next analysis;
step 3, judging whether the Map tasks of the jobs are distributed on the same level in the cluster topological graph, if so, selecting the optimal node from the nodes on the level, preferentially selecting the node for executing the Map tasks during selection, if the nodes for executing the Map tasks do not have idle resources, selecting other nodes with idle resources on the level and outputting, and skipping to the step 6 for execution; if not, the next step of judgment is carried out;
step 4, if the Map tasks of the operation are distributed unevenly in the two subtrees of the root node, cutting branches with small quantity for executing the Map tasks, taking ancestor nodes of the rest branches as new root nodes, and skipping to the step 3 to continue execution;
step 5, if the Map tasks of the operation are distributed in the two subtrees of the root node in a balanced manner, cutting off a branch with a larger subtree depth, taking the ancestor node of the rest branches as a new root node, and skipping to the step 3 to continue the execution;
step 6, preparing to dispatch the Reduce task of the operation to be executed on the output optimal node;
and 7, the Reduce task obtains a Reduce resource queue to execute until the operation is finished.
The constraint conditions of the task scheduling strategy of the invention are as follows:
1. the topology in the cluster is known and has the following characteristics: the absolute value of the depth difference of the left subtree and the right subtree of the non-terminal node, namely the router node, does not exceed 1, namely the depths of the two subtrees are the same or the depth of one subtree is 1 more than that of the other subtree, and the position of a leaf node of the calculation node in the topological graph is calculated. If a router of a non-terminal node takes a router node and a plurality of computation nodes as child nodes at the same time, all the computation nodes act as a computation branch of a root node uniformly, that is, the non-terminal node still has only left and right subtrees logically.
2. Each node in the selected TaskTracker node set contains a free Reduce Slot resource.
3. Only one Reduce task of the job is scheduled per execution of the algorithm.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.
Furthermore, it should be understood that although the present description refers to embodiments, not every embodiment may contain only a single embodiment, and such description is for clarity only, and those skilled in the art should integrate the description, and the embodiments may be combined as appropriate to form other embodiments understood by those skilled in the art.
Claims (9)
1. A data-aware task scheduling method is characterized by comprising the following steps:
step 1, establishing an N-1 level scheduling model based on a distributed system infrastructure, determining a corresponding scheduling strategy, embedding and executing MapReduce aiming at an Nth level scheduling model, selecting a specific task queue and selected specific jobs for a Map task scheduling strategy, starting a Map task and executing, wherein N is a positive integer greater than 1;
step 2, obtaining information of a task scheduler node for executing the Map task, wherein the information of the task scheduler node at least comprises the number of the node and the position level of the node in the topological graph;
step 3, judging whether the Map tasks of the jobs are distributed on the same level in the cluster topological graph, if so, selecting the optimal node from the nodes on the level, preferentially selecting the node for executing the Map tasks during selection, if the nodes for executing the Map tasks do not have idle resources, selecting other nodes with idle resources on the level and outputting, and skipping to the step 6 for execution; if not, the next step of judgment is carried out;
step 4, if the Map tasks of the operation are distributed unevenly in the two subtrees of the root node, cutting branches with small quantity for executing the Map tasks, taking ancestor nodes of the rest branches as new root nodes, and skipping to the step 3 to continue execution;
step 5, if the Map tasks of the operation are distributed in the two subtrees of the root node in a balanced manner, cutting off a branch with a larger subtree depth, taking the ancestor node of the rest branches as a new root node, and skipping to the step 3 to continue the execution;
step 6, preparing to dispatch the Reduce task of the operation to be executed on the output optimal node;
and 7, the Reduce task obtains a Reduce resource queue to execute until the operation is finished.
2. The data-aware task scheduling method of claim 1, wherein the distributed system infrastructure in step 1 is Hadoop.
3. The data-aware task scheduling method according to claim 1 or 2, wherein N =3 in step 1, when the distributed system infrastructure is Hadoop, a fair scheduler defaulted by Hadoop is used as a first-level scheduling policy and a second-level scheduling policy, and a Map task scheduling policy is embedded as a third level, and the Map task scheduling policy selects a specific job in a specific queue, starts a Map task, and executes the job.
4. The data-aware task scheduling method according to claim 1, wherein when step 1 is executed, the number of the specific tasks selected by the Map task scheduling policy in the specific task queue at the same stage is 1.
5. The data-aware task scheduling method of claim 4, wherein the same phase is a same clock node or a same algorithm execution process.
6. The data-aware task scheduling method according to claim 1, wherein the step 3 determines whether Map tasks of the job are distributed on the same level in the cluster topology Map and is defined that an absolute value of a depth difference between a left sub-tree and a right sub-tree of the non-terminal node is not more than 1.
7. The data-aware task scheduling method according to claim 6, wherein the determining in step 3 whether the Map tasks of the job are distributed on the same level in the cluster topology includes that the depths of two subtrees are the same or that the depth of one subtree is 1 more than the depth of the other subtree.
8. The data-aware task scheduling method of claim 1, wherein the task scheduler node that executes the Map task in step 2 is a tasktacker node, and forms a tasktacker node set.
9. The data-aware task scheduling method of claim 8, wherein each TaskTracker node in the set of TaskTracker nodes contains a free Reduce Slot resource.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010677871.2A CN111813527B (en) | 2020-07-15 | 2020-07-15 | Data-aware task scheduling method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010677871.2A CN111813527B (en) | 2020-07-15 | 2020-07-15 | Data-aware task scheduling method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111813527A CN111813527A (en) | 2020-10-23 |
| CN111813527B true CN111813527B (en) | 2022-06-14 |
Family
ID=72865112
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010677871.2A Active CN111813527B (en) | 2020-07-15 | 2020-07-15 | Data-aware task scheduling method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111813527B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103617087A (en) * | 2013-11-25 | 2014-03-05 | 华中科技大学 | MapReduce optimizing method suitable for iterative computations |
| CN104102533A (en) * | 2014-06-17 | 2014-10-15 | 华中科技大学 | Bandwidth aware based Hadoop scheduling method and system |
| CN108304253A (en) * | 2017-12-28 | 2018-07-20 | 武汉理工大学 | Map method for scheduling task based on cache perception and data locality |
-
2020
- 2020-07-15 CN CN202010677871.2A patent/CN111813527B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103617087A (en) * | 2013-11-25 | 2014-03-05 | 华中科技大学 | MapReduce optimizing method suitable for iterative computations |
| CN104102533A (en) * | 2014-06-17 | 2014-10-15 | 华中科技大学 | Bandwidth aware based Hadoop scheduling method and system |
| CN108304253A (en) * | 2017-12-28 | 2018-07-20 | 武汉理工大学 | Map method for scheduling task based on cache perception and data locality |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111813527A (en) | 2020-10-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110297699B (en) | Scheduling method, scheduler, storage medium and system | |
| TWI786564B (en) | Task scheduling method and apparatus, storage media and computer equipment | |
| CN105373426B (en) | A kind of car networking memory aware real time job dispatching method based on Hadoop | |
| CN107360031B (en) | Virtual network mapping method based on optimized overhead-to-revenue ratio | |
| CN113391914A (en) | Task scheduling method and device | |
| CN112015765A (en) | Spark cache elimination method and system based on cache value | |
| CN116700993A (en) | Load balancing method, device, equipment and readable storage medium | |
| CN113672391B (en) | Parallel computing task scheduling method and system based on Kubernetes | |
| WO2022062648A1 (en) | Automatic driving simulation task scheduling method and apparatus, device, and readable medium | |
| CN112231081B (en) | PSO-AHP-based monotonic rate resource scheduling method and system in cloud environment | |
| CN107168795B (en) | Codon deviation factor model method based on CPU-GPU isomery combined type parallel computation frame | |
| CN111813527B (en) | Data-aware task scheduling method | |
| CN110928666B (en) | Method and system for optimizing task parallelism based on memory in Spark environment | |
| CN112446484A (en) | Multitask training cluster intelligent network system and cluster network optimization method | |
| Li et al. | On scheduling of high-throughput scientific workflows under budget constraints in multi-cloud environments | |
| Lin et al. | Joint deadline-constrained and influence-aware design for allocating MapReduce jobs in cloud computing systems | |
| CN112598112B (en) | Resource scheduling method based on graph neural network | |
| CN114980216A (en) | Dependent task unloading system and method based on mobile edge calculation | |
| KR101558807B1 (en) | Processor scheduling method for the cooperation processing between host processor and cooperation processor and host processor for performing the method | |
| CN112862312A (en) | Manufacturing service resource dynamic scheduling method and system based on random online algorithm | |
| CN117891584B (en) | Task parallelism scheduling method, medium and device based on DAG grouping | |
| CN113806042B (en) | Task scheduling method of multi-core real-time embedded system | |
| CN112308304B (en) | Workflow execution time optimization method and device | |
| CN113391891B (en) | Load balancing resource scheduling method based on Rete and character string pattern matching algorithm | |
| CN117349031B (en) | Distributed super computing resource scheduling analysis method, system, terminal and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |