CN115686792A - Task scheduling method and device, electronic equipment, storage medium and product - Google Patents
Task scheduling method and device, electronic equipment, storage medium and product Download PDFInfo
- Publication number
- CN115686792A CN115686792A CN202211384101.4A CN202211384101A CN115686792A CN 115686792 A CN115686792 A CN 115686792A CN 202211384101 A CN202211384101 A CN 202211384101A CN 115686792 A CN115686792 A CN 115686792A
- Authority
- CN
- China
- Prior art keywords
- node
- task
- resource
- scheduled
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application provides a task scheduling method, a task scheduling device, electronic equipment, a storage medium and a product, and relates to the technical field of data analysis. The method is applied to a scheduler and comprises the following steps: acquiring a target task set and a node cluster comprising a plurality of nodes; the target task set comprises a plurality of tasks to be scheduled; determining a target function corresponding to a pre-established resource cost model, and determining an optimal solution corresponding to the minimum value of the target function according to an improved ant colony algorithm; and according to the optimal solution, determining a plurality of scheduling nodes matched with the tasks to be scheduled in the node cluster, and distributing the tasks to be scheduled to the scheduling nodes. The method can ensure that the resource cost of the node cluster is reduced during task scheduling.
Description
Technical Field
The present application relates to the field of data analysis technologies, and in particular, to a task scheduling method and apparatus, an electronic device, a storage medium, and a product.
Background
With the proliferation of data volume, technologies and components of big data, cloud computing are also endless and becoming more and more abundant, such as batch processing components (Hive, spark), stream processing components (Storm, spark Streaming, flink Streaming), and so on. More and more people are beginning to focus on cost issues due to similarities and differences in task scheduling for clusters. The task scheduling refers to a process of scheduling tasks according to calculation and resource allocation under specific limitation. This process can be viewed as a mapping between task placement and resources, embodied in a particular optimization objective.
Taking a distributed cluster (e.g., dolphin cluster) as an example, when performing task scheduling in a distributed cluster environment, due to different performances of nodes under the distributed cluster and different capabilities and network bandwidths of the nodes in the distributed cluster, the load borne by the resources of the nodes in the distributed cluster at each moment is greatly changed. The scheduler adopted by the Dolphin cluster at present deploys the following three algorithms: a random scheduling algorithm, a round robin scheduling algorithm and a machine resource weight algorithm. The placement of a large number of tasks varies in different scheduling modes.
The random scheduling algorithm has poor load balancing capability. Compared with a random scheduling algorithm, the round-robin scheduling algorithm can distribute tasks more uniformly to a certain extent, but has the disadvantage of high resource cost of the whole cluster. The machine resource weight algorithm is optimized based on the target of resource scheduling and load balancing, and a node with low cost may not be utilized, so that the technical problems of unreasonable resource utilization and high resource cost overhead exist in the prior art.
Disclosure of Invention
The application provides a task degree method, a task degree device, an electronic device, a storage medium and a product, which are used for solving the problem of high resource cost.
According to a first aspect of the present application, there is provided a task scheduling method applied to a scheduler, including:
acquiring a target task set and a node cluster comprising a plurality of nodes; the target task set comprises a plurality of tasks to be scheduled;
determining an objective function corresponding to a pre-established resource cost model, and determining an optimal solution corresponding to the objective function when the objective function takes the minimum value according to an improved ant colony algorithm;
and determining the scheduling nodes matched with the tasks to be scheduled in the node cluster according to the optimal solution, and distributing the tasks to be scheduled to the scheduling nodes.
Optionally, establishing the resource cost model includes:
for each node in the node cluster, acquiring the resource unit price of the node and the resource use time required by the node to execute the corresponding task to be scheduled;
and establishing a resource cost model according to the resource unit price of the node and the resource use time.
Optionally, the establishing a resource cost model according to the resource unit price of the node and the resource usage time includes:
for each node in the node cluster, determining a product result of the resource unit price of the node and the resource use time as a resource cost for the node to execute a corresponding task to be scheduled;
and determining a formula for performing accumulation calculation on the resource cost of executing the corresponding tasks to be scheduled by all the nodes, and determining the formula as a resource cost model.
Optionally, the node includes a central processing unit and a memory, and the resource usage time includes a central processing unit usage time and a memory usage time;
determining the product result of the resource unit price of the node and the resource use time as the resource cost of the node for executing the corresponding task to be scheduled, and the method comprises the following steps:
determining the product result of the resource unit price of the central processing unit and the service time of the central processing unit as the resource cost of the central processing unit for executing the corresponding task to be scheduled by the node;
determining the product result of the resource unit price of the memory and the memory use time as the memory resource cost of the node for executing the corresponding task to be scheduled;
and determining the addition result of the resource cost of the central processing unit and the resource cost of the memory as the resource cost of the node for executing the corresponding task to be scheduled.
Optionally, the determining, according to the improved ant colony algorithm, an optimal solution corresponding to when the objective function takes the minimum value includes:
initializing a task array corresponding to the target task set, a resource cost requirement array corresponding to the target task set and a node array corresponding to the node cluster;
taking the task array and the node array as the input of the improved ant colony algorithm, and taking the resource cost demand data as the constraint condition of the improved ant colony algorithm;
and performing iterative computation by using the improved ant colony algorithm to obtain an optimal solution corresponding to the minimum value of the objective function.
Optionally, the performing iterative computation by using the improved ant colony algorithm to obtain an optimal solution corresponding to the minimum value of the objective function includes:
s1: initializing the following parameters of the improved ant colony algorithm: information elicitation factors, expected elicitation factors, pheromones, the number of ants and the maximum iteration number;
s2: updating the current iteration times;
s3: randomly distributing the starting point of each ant carrying all the tasks to be scheduled to any node in the node cluster, so that the ants distribute nodes meeting preset conditions for each scheduling task in a roulette mode according to a preset probability formula, and locally updating the pheromone; wherein the parameters of the probability formula include: the information heuristic factor, the expected heuristic factor, the pheromone and a function value output by the resource cost model, wherein the preset condition comprises the constraint condition;
s4: judging whether all ants traverse the whole course; if yes, executing S5; if not, continuing to execute S3;
s5: determining a corresponding alternative optimal solution when the objective function takes the minimum value in the current iteration, and performing global updating on the pheromone;
s6: judging whether the current iteration times reach the maximum iteration times or not; if not, executing S2; if so, finishing the iteration and executing S7;
s7: and outputting the candidate optimal solution as an optimal solution.
Optionally, the obtaining the target task set includes:
establishing a topological graph for all tasks to be executed in a preset associated task group according to a pre-execution sequence;
dividing all the tasks to be executed into different task sets according to the topological graph;
and selecting one task set from all the task sets as a target task set.
According to a second aspect of the present application, there is provided a task scheduling apparatus applied to a scheduler, including:
the acquisition module is used for acquiring a target task set and a node cluster comprising a plurality of nodes; the target task set comprises a plurality of tasks to be scheduled;
the determining module is used for determining an objective function corresponding to a pre-established resource cost model and determining an optimal solution corresponding to the objective function when the objective function takes the minimum value according to an improved ant colony algorithm;
and the determining and distributing module is used for determining the scheduling nodes matched with the tasks to be scheduled in the node cluster according to the optimal solution and distributing the tasks to be scheduled to the scheduling nodes.
According to a third aspect of the present application, there is provided an electronic device comprising: at least one processor and memory;
the memory stores computer-executable instructions;
the at least one processor executing the computer executable instructions stored by the memory causes the at least one processor to perform the method of task scheduling as described above in the first aspect.
According to a fourth aspect of the present application, there is provided a computer-readable storage medium having stored therein computer-executable instructions for implementing the task scheduling method of the first aspect when executed by a processor.
According to a fifth aspect of the present application, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method of task scheduling of the first aspect.
The task scheduling method includes the steps that a target task set and a node cluster comprising a plurality of nodes are obtained; the target task set comprises a plurality of tasks to be scheduled; then determining an objective function corresponding to a pre-established resource cost model, and determining an optimal solution corresponding to the objective function when the objective function takes the minimum value according to an improved ant colony algorithm; and finally, according to the optimal solution, determining a plurality of scheduling nodes matched with the tasks to be scheduled in the node cluster, and distributing the tasks to be scheduled to the scheduling nodes.
According to the method and the device, the target function corresponding to the pre-established resource cost model is used as the heuristic function of the improved ant colony algorithm, the optimal solution corresponding to the target function when the minimum value is obtained can be output through the improved ant colony algorithm, and then the scheduling nodes matched with a plurality of tasks to be scheduled are determined in the node cluster, so that the resource of the node cluster is utilized to the maximum extent, and the purpose of reducing the resource cost of the node cluster to the maximum extent can be achieved.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present application, nor do they limit the scope of the present application. Other features of the present application will become apparent from the following description.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.
Fig. 1 is a schematic flowchart of a task scheduling method according to an embodiment of the present application;
FIG. 2 is a schematic flow chart illustrating a resource cost model according to an embodiment of the present disclosure;
fig. 3 is a schematic flowchart of S102 in fig. 1 according to an embodiment of the present disclosure;
fig. 4 is a schematic flowchart of S1023 in fig. 3 according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of a task scheduling device according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Specific embodiments of the present application have been shown by way of example in the drawings and will be described in more detail below. These drawings and written description are not intended to limit the scope of the inventive concepts in any manner, but rather to illustrate the inventive concepts to those skilled in the art by reference to specific embodiments.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application.
At present, the task scheduling problem of the cluster gradually becomes a research hotspot in recent years. Most of the current researches are task scheduling algorithms proposed for cloud computing components (kubernets, dockers) and big data components (Storm, spark), and the algorithms mainly aim at improving the execution time and resource utilization rate of a cluster and reducing communication delay among nodes.
At present, in a Dolphin cluster, a kernel component of a Dolphin scheduler includes a MasterServer and a WorkerServer, and the MasterServer and the WorkerServer both perform cluster management and fault tolerance through a ZooKeeper. The load balancing algorithm provided by the component WorkerServer is used for balancing loads (namely tasks) and distributing the loads to a plurality of machines for execution so as to achieve the capability of reasonably distributing pressure. Three scheduling strategy algorithms, namely a random scheduling algorithm, a polling scheduling algorithm and a machine resource weight algorithm, are defaulted in the Dolphin cluster, after the component MasterServer acquires the tasks, all machine information lists of a working group to which the current tasks belong are firstly acquired from the component zookeeper, and then the corresponding tasks are distributed on all nodes according to the scheduling strategies.
All three algorithms have their corresponding drawbacks and disadvantages: 1) The principle of the random scheduling algorithm is simple, namely, after the component zookeeper acquires all machines on a task working group, the total number of the machines is judged firstly, and if the total number of the machines is 1, the machines are directly distributed; if not, a node is randomly selected as the executing machine of the task. The algorithm causes uneven task placement, and more tasks may be placed on the nodes with poor performance, so that not only the execution time of the tasks is increased, but also the resources of the node cluster are prone to be seriously unbalanced. 2) The implementation mode of the polling scheduling algorithm is realized by an atomic AtomicInteger, counting is carried out through the polling scheduling algorithm, then the position of the array where the machine to execute the task is located is obtained in a mode of taking a modulus of the number of the machine, and the task is distributed to the machine. Compared with a random scheduling algorithm, the algorithm can enable tasks to be distributed more uniformly, nodes in a cluster can be better utilized, but the algorithm does not consider the problem of cost, the condition that the tasks are placed on the nodes with higher cost is easy to occur, and the overall cost of the cluster is higher. 3) The implementation mode of the machine resource weight algorithm mainly focuses on resources, the MasterServer performs heartbeat detection on the machine through the zookeeper to acquire the use information of the machine resources, and then selects the machine with the minimum resource weight value according to the use information of the resources.
Therefore, the existing Dolphin cluster has the technical problems of unreasonable resource utilization and high resource cost overhead.
In order to solve the above technical problems, the present invention provides a task scheduling method that reduces resource cost to the maximum extent.
The following describes the technical solutions of the present application and how to solve the above technical problems with specific embodiments. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
Example 1:
fig. 1 is a schematic flowchart of a task scheduling method according to an embodiment of the present disclosure. As shown in fig. 1, the method of this embodiment, applied to a scheduler, includes:
s101: acquiring a target task set and a node cluster comprising a plurality of nodes; the target task set comprises a plurality of tasks to be scheduled.
In this embodiment of the present application, a task node cluster may refer to a Dolphin cluster, and components in the Dolphin cluster may interact with each other through a Service Level Agreement (SLA). The scheduler may be referred to as a Dolphin scheduler. The embodiment of the application can perform task scheduling on all tasks to be scheduled in the same target task set, obtain the target task set, and determine the number of the tasks to be scheduled in the target task set and the attribute information of each task to be scheduled, for example: resource demand information, association relation with other tasks to be scheduled, and the like. Acquiring a node cluster, which can determine the number of nodes in the node cluster and attribute information of the nodes, for example: total resource information, used resource information, available resource information, processing speed when executing a task, and the like.
S102: and determining an objective function corresponding to the pre-established resource cost model, and determining an optimal solution corresponding to the objective function when the objective function takes the minimum value according to the improved ant colony algorithm.
In the embodiment of the application, the ant colony algorithm is an algorithm which is proposed in recent years and describes that the optimal path is found in the graph, has the characteristics of high precision and combination with other methods, and is also an algorithm for comparing hot shortest paths. The method is a method for simulating ant colony to find a path in the foraging process. The colony, in the process of foraging, leaves secretions, called pheromones, on each path they pass. The ants prefer a path with higher concentration when selecting the path, the possibility that the following ants select the path is higher, and the purpose of finding the shortest path can be finally achieved. Therefore, the method is suitable for the problem of combinatorial optimization, and the task scheduling problem in the application belongs to the problem of combinatorial optimization, so that the application improves the ant colony algorithm by using the advantages of the ant colony algorithm and combining the objective function corresponding to the pre-established resource cost model, the improved ant colony algorithm is obtained, and the improved ant colony algorithm takes the resource cost as the pheromone and the optimized resource cost as the target.
Since the number of tasks to be scheduled in the target task set can be multiple, the resource cost model adopted is called a multitask resource cost model. Since the purpose of the present application is to optimize the resource cost, the present application provides an objective function corresponding to a pre-established resource cost model. For the establishment of the resource cost model and the establishment of the objective function, the present application is specifically described in the following embodiment 2, which is not specifically limited herein.
The target task set in S101 is set as: tasks = { tsk = 1 ，tsk 2 ，tsk 3 ，...，tsk m }, the node cluster in S101 is Nodes = { node = { node = 1 ，node 2 ，node 3 ，...，node n }. m is the number of tasks to be executed in the target task set, and n is the number of nodes in the node cluster. On the basis, the formula of the resource cost model is shown as the formula (1):
wherein, cost is the total resource Cost of all the nodes executing all the corresponding tasks to be scheduled, and Cost i Executing all corresponding waited dispatchers for ith nodeThe total resource cost of the service, n is the number of nodes in the node cluster, m is the number of tasks to be executed in the target task set,
or 1, when
When the temperature of the water is higher than the set temperature,
indicating that the jth task to be executed in the target task set is executed by the ith node in the node cluster, and at the moment
Is a non-zero value and is expressed in example 2 below; when in use
When the temperature of the water is higher than the set temperature,
indicating that the jth task to be executed in the target task set is not executed by the ith node in the node cluster, at the moment
Because task scheduling can directly affect resource usage and operating efficiency of a node cluster, the optimization objective of the embodiment of the present application is to reduce resource cost of the node cluster and minimize multitask resource cost under the condition of satisfying an SLA. In the embodiment of the present application, one task scheduling result corresponds to one total resource Cost, and a formula of an objective function corresponding to a resource Cost model is shown in formula (2):
minf obj ＝min(Cost k )，k＝1，...，Num (2)
wherein f is obj One task scheduling result corresponds to one objective function value for the objective function, and when the total resource cost output by the resource cost model takes the minimum value, the corresponding objective function value corresponds to the total resource cost output by the resource cost modelThe objective function also takes the minimum value and Num is the number of ants in the improved ant colony algorithm. In the last iteration, the target function f obj Taking the corresponding optimal solution at the minimum value as tsk best 。
S103: and according to the optimal solution, determining a plurality of scheduling nodes matched with the tasks to be scheduled in the node cluster, and distributing the tasks to be scheduled to the scheduling nodes.
According to the embodiment of the application, the target function corresponding to the resource cost model is used as the heuristic function of the improved ant colony algorithm, the optimal solution can be output through the improved ant colony algorithm, and then the scheduling nodes matched with a plurality of tasks to be scheduled are determined in the node cluster, so that the resource of the node cluster is utilized to the maximum extent, and the purpose of reducing the resource cost of the node cluster to the maximum extent can be achieved.
In one possible implementation manner, in S101, acquiring the target task set includes the following steps S1011 to S1013, where:
s1011: and establishing a topological graph for all tasks to be executed in the preset associated task group according to the sequence of the pre-execution.
S1012: and dividing all the tasks to be executed into different task sets according to the topological graph.
S1013: and selecting one task set from all the task sets as a target task set.
The embodiment of the application provides a method for acquiring a target task set by executing S1011 to S1013. In addition, the target task set may also be obtained in other ways in the embodiment of the present application, which is not specifically limited in this embodiment of the present application.
In summary, in order to solve the problem that the resource cost of the node cluster is not considered during task scheduling, a resource cost model and a scheduler based on the resource cost model are designed in the embodiments of the present application. The scheduler adopts the resource cost as a heuristic function in the pheromone improved traditional ant colony algorithm, and can place more tasks to be executed on nodes with lower cost and nodes with smaller load, thereby better utilizing the resources of the nodes in the node cluster to the maximum extent under the condition of meeting SLA, and reducing the resource cost of the node cluster to the maximum extent.
On the basis of the above embodiments, the technical solution of the present application will be described in more detail with reference to several specific embodiments.
Example 2:
fig. 2 is a schematic flowchart of establishing an objective function corresponding to a resource cost model according to an embodiment of the present disclosure. As shown in fig. 2, a resource cost model is established, including:
s21: and aiming at each node in the node cluster, acquiring the resource unit price of the node and the resource use time required by the node to execute the corresponding task to be scheduled. The resource unit price of the node is recorded as p j The resource use time required by the node i in the node cluster to execute the jth task to be scheduled in the target task set is
S22: and establishing a resource cost model according to the resource unit price and the resource use time of the node.
The embodiment of the present application describes in detail factors affecting resource cost, including resource unit price of a node in a node cluster and resource usage time required for the node to execute a corresponding task to be scheduled, and by executing S21 to S22, effective establishment of a resource cost model and an objective function corresponding to the resource cost model can be completed, where the establishment process of the resource cost model is described in S221 to S222 below.
In one possible implementation manner, in S22, establishing a resource cost model according to the resource unit price and the resource usage time of the node includes:
s221: and determining the product result of the resource unit price and the resource use time of the node as the resource cost of the node for executing the corresponding task to be scheduled aiming at each node in the node cluster.
S222: and determining a formula for performing accumulation calculation on the resource cost of executing the corresponding tasks to be scheduled by all the nodes, and determining the formula as a resource cost model.
Practice of the present applicationFor example, a resource cost model is established according to the resource unit price and the resource use time of the node, wherein the resource cost model is shown as formula (1)
Is represented by formula (3):
wherein the content of the first and second substances,
indicating that the ith node in the node cluster executes the jth task to be executed in the target task set,
the resource use time required by the node i in the node cluster to execute the jth task to be scheduled in the target task set is shown, and at the moment
Indicating that the ith node in the node cluster does not execute the jth task to be executed in the target task set, wherein the jth task to be executed in the target task set is executed by the ith node
Modifying the expression of the resource cost model in the formula (1) into a formula (4) based on the formula (3):
the resource cost model established in the embodiment of the application is used as a heuristic function for improving the ant colony algorithm, and an accurate mathematical model is provided for the ant colony algorithm to find the optimal solution, so that the application achieves the purpose of maximally reducing the resource cost of the node cluster.
In one possible implementation, the node includes a central processing unit and a memory, and the resource usage time includes a central processing unit usage time and a memory usage time. S221: determining the product result of the resource unit price and the resource using time of the node as the resource cost of the node for executing the corresponding task to be scheduled, and the method comprises the following steps:
s2211: and determining the product result of the resource unit price of the central processing unit and the using time of the central processing unit as the resource cost of the central processing unit for the node to execute the corresponding task to be scheduled.
In the embodiment of the present application, the resource unit price of the central processing unit of the ith node is denoted as p cpu i The resource cost of the CPU for the ith node to execute the jth task to be executed in the target task set is recorded as
S2212: and determining the product result of the resource unit price of the memory and the memory use time as the memory resource cost of the node for executing the corresponding task to be scheduled.
In this embodiment, the resource unit price of the memory of the ith node is denoted as p mem i The memory resource cost of the ith node for executing the jth task to be executed in the target task set is recorded as
S2213: and determining the sum of the resource cost of the central processing unit and the resource cost of the memory as the resource cost of the node for executing the corresponding task to be scheduled.
Based on the concrete description of the resource cost in steps S2211 to S2213, the resource cost in the resource cost model in formula (3) is
Modified to formula (5):
on the basis of the formula (5), modifying the expression of the resource cost model in the formula (4) into the formula (6):
in addition, the node in the embodiment of the present application may also include other resource costs in addition to the central processing unit and the memory, and in addition, the embodiment of the present application may also set corresponding coefficients for the resource cost of the central processing unit and the resource cost of the memory, so that the embodiment of the present application may adaptively modify the resource cost model based on the form of the formula (6), that is, the embodiment of the present application does not specifically limit the specific structure of the resource cost model.
The resource cost model is established mainly by considering that a plurality of tasks to be scheduled are placed on a plurality of nodes by a dolphin scheduler in a task scheduling process, and the consumption of CPU (central processing unit) and memory resources of the nodes is realized during task execution.
Due to the environment, the heterogeneity and other factors of the node cluster, task scheduling is considered to be an NP-hard problem, and when a dynamic scheduling scheme is used, due to different task placements, resource usage of nodes is also different, thus resource cost overhead of the node cluster is increased invisibly. Based on the above problems, the embodiment of the application provides a multitask resource cost model, improves the traditional ant colony algorithm, and takes a target function corresponding to the multitask resource cost model as a heuristic function for improving the ant colony algorithm, so that more tasks are preferentially placed on nodes with lower resource cost, resource scheduling of a node cluster can be more balanced, and cost can be lower.
Example 3:
fig. 3 is a schematic flowchart of S102 in fig. 1 according to an embodiment of the present disclosure. On the basis of the embodiment shown in fig. 1, the present embodiment focuses on refining S102 in fig. 1. As shown in fig. 3, S102 in fig. 1 includes:
s1021: initializing a task array corresponding to the target task set, a resource cost requirement array corresponding to the target task set and a node array corresponding to the node cluster. Wherein the task array is Tasks and the node array is Nodes. According to the method and the device, the tasks to be scheduled are sequentially arranged and attached to each task to be scheduled according to the submission sequence and the resource requirements of the tasks to be scheduled, so that a resource cost requirement array corresponding to a target task set is obtained.
S1022: and taking the task array and the node array as the input of the improved ant colony algorithm, and taking the resource cost demand data as the constraint condition of the improved ant colony algorithm.
S1023: and carrying out iterative computation by using an improved ant colony algorithm to obtain an optimal solution corresponding to the minimum value of the objective function.
The method and the device for scheduling the resources of the node cluster improve the traditional ant colony algorithm by combining the Dolphin scheduler, take the multi-task resource cost model as a heuristic function in the ant colony algorithm, and preferentially place more tasks on the nodes with lower resource cost, so that the resource scheduling of the node cluster can be more balanced, and the resource cost is lower.
In one possible implementation, as shown in fig. 4, S1023: iterative computation is carried out by utilizing an improved ant colony algorithm to obtain an optimal solution corresponding to the minimum value of the objective function, and the method comprises the following steps:
s1: the following parameters of the improved ant colony algorithm are initialized: information elicitation factors, expectation elicitation factors, pheromones, ant numbers, and maximum iteration number. The information elicitation factor is k, the expected elicitation factor is l, the pheromone is e, and the number of ants is Num. The maximum number of iterations Dmax.
The setting of the pheromone e is related to the passing of ants among nodes, so the times of the ants passing through the nodes in the node cluster, which are currently attached with task information, need to be recorded. Thus, the pheromone is updated as in equation (7):
the pheromone e is a variable, and the variable is larger as the number of times that the ants carry the jth task to be scheduled to pass through the ith node is increased, namely the numerical value of the pheromone is increased, but the numerical value cannot be increased infinitely. The number of ants cannot exceed the resource load on the current node, otherwise the performance of the algorithm is greatly reduced.
S2: and updating the current iteration number. The update of the number of iterations is done in an incremental way, i.e. D = D +1.
S3: randomly distributing the starting point of each ant carrying all tasks to be scheduled to any node in the node cluster, so that the ants distribute nodes meeting preset conditions for each scheduling task in a roulette mode according to a preset probability formula, and locally updating pheromones; wherein, the parameters of the probability formula include: the information elicitation factor, the expected elicitation factor, the pheromone and the function value output by the resource cost model, and the preset condition comprises a constraint condition.
S4: judging whether all ants traverse the whole course; if yes, executing S5; if not, S3 is continuously executed.
S5: and determining the corresponding alternative optimal solution of the objective function when the objective function takes the minimum value in the current iteration, and performing global updating of the pheromone.
In this step, all tasks to be scheduled in the current iteration have selected corresponding nodes, that is, each ant provides a solution, and a candidate optimal solution corresponding to the minimum value of the objective function in the current iteration can be determined from all solutions.
S6: judging whether the current iteration times reach the maximum iteration times or not; if not, executing S2; if yes, the iteration is ended, and S7 is executed.
S7: and outputting the alternative optimal solution as the optimal solution.
At present, three scheduling algorithms of the dolphin scheduler cannot well distribute tasks evenly under the condition of ensuring low cost, so that node resources of a node cluster are utilized to the maximum extent. In order to solve the above problems, an embodiment of the present application provides a task scheduling algorithm based on resource cost, which uses a minimized multi-task resource cost model as a heuristic function in an ant colony algorithm to improve a conventional ant colony algorithm, and aims to maximally utilize node resources of a node cluster and maximally reduce resource cost of the node cluster on the premise of satisfying an SLA.
Fig. 5 is a schematic structural diagram of a task scheduling device according to an embodiment of the present application. The apparatus of the present embodiment may be in the form of software and/or hardware. As shown in fig. 5, the task scheduling apparatus provided in this embodiment is applied to a scheduler, and includes: an acquisition module 51, a determination module 52 and a determination allocation module 53.
Wherein:
an obtaining module 51, configured to obtain a target task set and a node cluster including multiple nodes; the target task set comprises a plurality of tasks to be scheduled.
The determining module 52 is configured to determine an objective function corresponding to the pre-established resource cost model, and determine an optimal solution corresponding to the objective function when the objective function takes the minimum value according to the improved ant colony algorithm.
And the determining and allocating module 53 is configured to determine, according to the optimal solution, a plurality of scheduling nodes matched with the tasks to be scheduled in the node cluster, and allocate the plurality of tasks to be scheduled to the scheduling nodes.
In a possible implementation manner, the task scheduling apparatus is further configured to:
and aiming at each node in the node cluster, acquiring the resource unit price of the node and the resource use time required by the node to execute the corresponding task to be scheduled.
And establishing a resource cost model according to the resource unit price and the resource use time of the node.
In a possible implementation manner, the task scheduling apparatus is further configured to:
and determining a product result of the resource unit price and the resource using time of the node as the resource cost of the node for executing the corresponding task to be scheduled aiming at each node in the node cluster.
And determining a formula for performing accumulation calculation on the resource cost of executing the corresponding tasks to be scheduled by all the nodes, and determining the formula as a resource cost model.
In one possible implementation, the node includes a central processing unit and a memory, and the resource usage time includes the usage time of the central processing unit and the usage time of the memory; the task scheduling device is further configured to:
and determining the product result of the resource unit price of the central processing unit and the service time of the central processing unit as the resource cost of the central processing unit for executing the corresponding task to be scheduled by the node.
And determining the product result of the resource unit price of the memory and the memory use time as the memory resource cost of the node for executing the corresponding task to be scheduled.
And determining the addition result of the resource cost of the central processing unit and the resource cost of the memory as the resource cost of the node for executing the corresponding task to be scheduled.
In a possible implementation, the determining module 52 is further configured to:
initializing a task array corresponding to the target task set, a resource cost requirement array corresponding to the target task set and a node array corresponding to the node cluster.
And taking the task array and the node array as the input of the improved ant colony algorithm, and taking the resource cost demand data as the constraint condition of the improved ant colony algorithm.
And carrying out iterative computation by using an improved ant colony algorithm to obtain an optimal solution corresponding to the minimum value of the target function.
In a possible implementation manner, the determining module 52 is further configured to perform the following steps:
s1: the following parameters of the improved ant colony algorithm are initialized: information elicitation factor, expected elicitation factor, pheromone, ant number, and maximum number of iterations.
S2: and updating the current iteration times.
S3: randomly distributing the starting point of each ant carrying all tasks to be scheduled to any node in the node cluster, so that the ants distribute nodes meeting preset conditions for each scheduling task in a roulette mode according to a preset probability formula, and locally updating pheromones; wherein the parameters of the probability formula include: the information elicitation factor, the expected elicitation factor, the pheromone and the function value output by the resource cost model, and the preset condition comprises a constraint condition.
S4: judging whether all ants traverse the whole course; if yes, executing S5; if not, S3 is continuously executed.
S5: and determining a corresponding alternative optimal solution of the objective function when the objective function takes the minimum value in the current iteration, and performing global updating of the pheromone.
S6: judging whether the current iteration times reach the maximum iteration times or not; if not, executing S2; if yes, the iteration is ended, and S7 is executed.
S7: and outputting the alternative optimal solution as the optimal solution.
In a possible implementation manner, the obtaining module 51 is further configured to:
and establishing a topological graph for all tasks to be executed in the preset associated task group according to the sequence of the pre-execution.
And dividing all tasks to be executed into different task sets according to the topological graph.
And selecting one task set from all the task sets as a target task set.
The task scheduling apparatus provided in this embodiment may be configured to execute the task scheduling method provided in any of the above method embodiments, and the implementation principle and the technical effect are similar, which are not described herein again.
In the technical scheme of the application, the collection, storage, use, processing, transmission, provision, disclosure and other processing of the personal information of the related user are all in accordance with the regulations of related laws and regulations and do not violate the good custom of the public order.
There is also provided, in accordance with an embodiment of the present application, an electronic device, a readable storage medium, and a computer program product.
Fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application. The electronic device comprises a receiver 60, a transmitter 61, at least one processor 62 and a memory 63, and the electronic device formed by the above components can be used to implement several specific embodiments of the present application, which are not described herein again.
The embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored, and when a processor executes the computer-executable instructions, the steps in the method in the foregoing embodiment are implemented.
Embodiments of the present application further provide a computer program product, which includes computer instructions, and when executed by a processor, the computer instructions implement the steps of the method in the foregoing embodiments.
Various implementations of the systems and techniques described here above may be realized in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for implementing the methods of the present application may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or electronic device.
In the context of this application, a computer readable storage medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The computer readable storage medium may be a machine readable signal medium or a machine readable storage medium. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a computer-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data electronic device), or that includes a middleware component (e.g., an application electronic device), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the Internet.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and the present disclosure is not limited herein.
The above-described embodiments should not be construed as limiting the scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made, depending on design requirements and other factors. Any modifications, equivalents, improvements, etc. that come within the spirit of the disclosure are intended to be included within the scope of the disclosure.
Claims (11)
1. A task scheduling method applied to a scheduler includes:
acquiring a target task set and a node cluster comprising a plurality of nodes; the target task set comprises a plurality of tasks to be scheduled;
determining an objective function corresponding to a pre-established resource cost model, and determining an optimal solution corresponding to the objective function when the objective function takes the minimum value according to an improved ant colony algorithm;
and determining the scheduling nodes matched with the tasks to be scheduled in the node cluster according to the optimal solution, and distributing the tasks to be scheduled to the scheduling nodes.
2. The method of claim 1, wherein establishing the resource cost model comprises:
for each node in the node cluster, acquiring the resource unit price of the node and the resource use time required by the node to execute the corresponding task to be scheduled;
and establishing a resource cost model according to the resource unit price of the node and the resource use time.
3. The method of claim 2, wherein the establishing a resource cost model based on the resource unit price and the resource usage time of the node comprises:
for each node in the node cluster, determining a product result of the resource unit price of the node and the resource using time as a resource cost for the node to execute a corresponding task to be scheduled;
and determining a formula for performing accumulation calculation on the resource cost of executing the corresponding tasks to be scheduled by all the nodes, and determining the formula as a resource cost model.
4. The method of claim 3, wherein the node comprises a central processing unit and a memory, and the resource usage time comprises a central processing unit usage time and a memory usage time;
determining the product result of the resource unit price of the node and the resource using time as the resource cost of the node for executing the corresponding task to be scheduled, wherein the method comprises the following steps:
determining the product result of the resource unit price of the central processing unit and the service time of the central processing unit as the resource cost of the central processing unit for executing the corresponding task to be scheduled by the node;
determining the product result of the resource unit price of the memory and the memory use time as the memory resource cost of the node for executing the corresponding task to be scheduled;
and determining the addition result of the resource cost of the central processing unit and the resource cost of the memory as the resource cost of the node for executing the corresponding task to be scheduled.
5. The method of claim 1, wherein determining the optimal solution corresponding to the minimization of the objective function according to the improved ant colony algorithm comprises:
initializing a task array corresponding to the target task set, a resource cost requirement array corresponding to the target task set and a node array corresponding to the node cluster;
taking the task array and the node array as the input of the improved ant colony algorithm, and taking the resource cost demand data as the constraint condition of the improved ant colony algorithm;
and carrying out iterative computation by using the improved ant colony algorithm to obtain an optimal solution corresponding to the minimum value of the target function.
6. The method of claim 5, wherein performing iterative computations using the improved ant colony algorithm to obtain an optimal solution corresponding to the minimum value of the objective function comprises:
s1: initializing the following parameters of the improved ant colony algorithm: information elicitation factors, expected elicitation factors, pheromones, the number of ants and the maximum iteration number;
s2: updating the current iteration times;
s3: for each ant carrying all the tasks to be scheduled, randomly distributing the starting point of the ant to any node in the node cluster, so that the ant distributes nodes meeting preset conditions for each scheduling task in a roulette mode according to a preset probability formula, and locally updating the pheromone; wherein the parameters of the probability formula include: the information heuristic factor, the expected heuristic factor, the pheromone and a function value output by the resource cost model, wherein the preset condition comprises the constraint condition;
s4: judging whether all ants traverse the whole course; if yes, executing S5; if not, continuing to execute S3;
s5: determining a corresponding alternative optimal solution of the objective function when the objective function takes the minimum value in the current iteration, and performing global updating of the pheromone;
s6: judging whether the current iteration times reach the maximum iteration times or not; if not, executing S2; if yes, finishing the iteration and executing S7;
s7: and outputting the alternative optimal solution as an optimal solution.
7. The method of claim 1, wherein obtaining the set of target tasks comprises:
establishing a topological graph for all tasks to be executed in a preset associated task group according to a pre-execution sequence;
dividing all the tasks to be executed into different task sets according to the topological graph;
and selecting one task set from all the task sets as a target task set.
8. A task scheduler, applied to a scheduler, comprising:
the acquisition module is used for acquiring a target task set and a node cluster comprising a plurality of nodes; the target task set comprises a plurality of tasks to be scheduled;
the determining module is used for determining a target function corresponding to a pre-established resource cost model and determining an optimal solution corresponding to the target function when the target function takes the minimum value according to an improved ant colony algorithm;
and the determining and distributing module is used for determining the scheduling nodes matched with the tasks to be scheduled in the node cluster according to the optimal solution and distributing the tasks to be scheduled to the scheduling nodes.
9. An electronic device, comprising: at least one processor and a memory;
the memory stores computer-executable instructions;
execution of the computer-executable instructions stored by the memory by the at least one processor causes the at least one processor to perform the task scheduling method of any of claims 1 to 7.
10. A computer-readable storage medium having stored thereon computer-executable instructions for implementing the method of task scheduling of any one of claims 1 to 7 when executed by a processor.
11. A computer program product comprising a computer program which, when executed by a processor, carries out the method of task scheduling of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211384101.4A CN115686792A (en) | 2022-11-07 | 2022-11-07 | Task scheduling method and device, electronic equipment, storage medium and product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211384101.4A CN115686792A (en) | 2022-11-07 | 2022-11-07 | Task scheduling method and device, electronic equipment, storage medium and product |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115686792A true CN115686792A (en) | 2023-02-03 |
Family
ID=85050044
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211384101.4A Pending CN115686792A (en) | 2022-11-07 | 2022-11-07 | Task scheduling method and device, electronic equipment, storage medium and product |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115686792A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116185596A (en) * | 2023-04-24 | 2023-05-30 | 之江实验室 | Method and device for improving task execution efficiency of wide-area multi-cluster distributed system |
| CN117724851A (en) * | 2024-02-07 | 2024-03-19 | 腾讯科技（深圳）有限公司 | Data processing method, device, storage medium and equipment |
-
2022
- 2022-11-07 CN CN202211384101.4A patent/CN115686792A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116185596A (en) * | 2023-04-24 | 2023-05-30 | 之江实验室 | Method and device for improving task execution efficiency of wide-area multi-cluster distributed system |
| CN116185596B (en) * | 2023-04-24 | 2023-08-01 | 之江实验室 | Method and device for improving task execution efficiency of wide-area multi-cluster distributed system |
| CN117724851A (en) * | 2024-02-07 | 2024-03-19 | 腾讯科技（深圳）有限公司 | Data processing method, device, storage medium and equipment |
| CN117724851B (en) * | 2024-02-07 | 2024-05-10 | 腾讯科技（深圳）有限公司 | Data processing method, device, storage medium and equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111459628B (en) | Spark platform task scheduling method based on improved quantum ant colony algorithm | |
| WO2020206705A1 (en) | Cluster node load state prediction-based job scheduling method | |
| CN115686792A (en) | Task scheduling method and device, electronic equipment, storage medium and product | |
| US9201690B2 (en) | Resource aware scheduling in a distributed computing environment | |
| CN113516250A (en) | Method, device and equipment for federated learning and storage medium | |
| WO2016082370A1 (en) | Distributed node intra-group task scheduling method and system | |
| CN107357652B (en) | Cloud computing task scheduling method based on segmentation ordering and standard deviation adjustment factor | |
| US20200059402A1 (en) | Improving performance of multi-processor computer systems | |
| CN108667878A (en) | Server load balancing method and device, storage medium, electronic equipment | |
| CN105373432B (en) | A kind of cloud computing resource scheduling method based on virtual resource status predication | |
| Li et al. | An effective scheduling strategy based on hypergraph partition in geographically distributed datacenters | |
| CN107566535B (en) | Self-adaptive load balancing method based on concurrent access timing sequence rule of Web map service | |
| US9817698B2 (en) | Scheduling execution requests to allow partial results | |
| CN112559135B (en) | Container cloud resource scheduling method based on QoS | |
| CN116302389A (en) | Task scheduling method based on improved ant colony algorithm | |
| CN113127173B (en) | Heterogeneous sensing cluster scheduling method and device | |
| Kang et al. | A multiagent brokering protocol for supporting Grid resource discovery | |
| CN112306642B (en) | Workflow scheduling method based on stable matching game theory | |
| CN112148471A (en) | Method and device for scheduling resources in distributed computing system | |
| CN117687759A (en) | Task scheduling method, device, processing equipment and readable storage medium | |
| CN116302404B (en) | Resource decoupling data center-oriented server non-perception calculation scheduling method | |
| Guo | Ant colony optimization computing resource allocation algorithm based on cloud computing environment | |
| CN106227600B (en) | A kind of multidimensional virtual resource allocation method based on Energy-aware | |
| Wan et al. | Utility-driven share scheduling algorithm in hadoop | |
| Han et al. | Elastic allocator: An adaptive task scheduler for streaming query in the cloud |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |