CN112540841A - Task scheduling method and device, processor and electronic equipment - Google Patents
Task scheduling method and device, processor and electronic equipment Download PDFInfo
- Publication number
- CN112540841A CN112540841A CN202011589305.2A CN202011589305A CN112540841A CN 112540841 A CN112540841 A CN 112540841A CN 202011589305 A CN202011589305 A CN 202011589305A CN 112540841 A CN112540841 A CN 112540841A
- Authority
- CN
- China
- Prior art keywords
- resources
- information
- target task
- task
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application provides a method, a device, a processor and an electronic device for task scheduling, wherein the method comprises the following steps: receiving target task related information, wherein the target task related information comprises target task information and information representing the emergency degree of a target task; determining the current resource utilization rate of the SPARK cluster, wherein the current resource utilization rate is the ratio of the number of currently occupied resources in the SPARK cluster; splitting the target task information according to the target task related information and the current resource utilization rate to obtain at least one target subtask information; and sending each target subtask information to the SPARK cluster for calculation, receiving each calculation result and sending out each calculation result. The method realizes task scheduling based on target task information and emergency degree, and further carries out intelligent task scheduling and automatic load distribution mechanism on cluster resources, thereby solving the problem of low task execution efficiency in the prior art.
Description
Technical Field
The present application relates to the field of task scheduling, and in particular, to a method and an apparatus for task scheduling, a computer-readable storage medium, a processor, and an electronic device.
Background
Apache Spark (open source cluster operating framework) is a fast, general-purpose computing engine designed specifically for large-scale data processing. The Apache Spark is a general parallel framework of MapReduce in Hadoop-like (Hadoop Distributed System infrastructure) sourced by UC Berkeley AMP lab (AMP laboratory of Berkeley division, university, california), and has the advantages of Hadoop MapReduce, but different from MapReduce, an output result in the middle of work can be stored in a memory, so that HDFS (Hadoop Distributed File System ) does not need to be read and written, and therefore, the Apache Spark can be better applicable to algorithms of MapReduce which need iteration, such as data mining and machine learning.
Spark has own calculation ecology, has very high market share in big data field and direction of calculation, and the technical commonality is good, can form big data ecology with other big data core component intercombinations simultaneously. However, existing implementations are inefficient.
The above information disclosed in this background section is only for enhancement of understanding of the background of the technology described herein and, therefore, certain information may be included in the background that does not form the prior art that is already known in this country to a person of ordinary skill in the art.
Disclosure of Invention
The present application mainly aims to provide a method and an apparatus for task scheduling, a computer-readable storage medium, a processor and an electronic device, so as to solve the problem of low efficiency in executing tasks in the prior art.
According to an aspect of the embodiments of the present invention, there is provided a method for task scheduling, including: receiving target task related information, wherein the target task related information comprises target task information and information representing the urgency degree of the target task; determining the current resource utilization rate of the SPARK cluster, wherein the current resource utilization rate is the ratio of the number of currently occupied resources in the SPARK cluster; splitting the target task information according to the target task related information and the current resource utilization rate to obtain at least one target subtask information; and sending each target subtask information to the SPARK cluster for calculation, receiving each calculation result and sending out each calculation result.
Optionally, determining the current resource utilization of the SPARK cluster includes: acquiring the total amount of resources in the SPARK cluster; acquiring the quantity of the currently occupied resources, wherein the currently occupied resources are resources occupied by other tasks in the SPARK cluster; and determining the current resource utilization rate according to the total resource amount and the number of the currently occupied resources.
Optionally, splitting the target task information according to the target task related information and the current resource utilization rate, including: determining the number of available resources in the SPARK cluster according to the information representing the emergency degree of the target task and the current resource utilization rate, wherein the available resources are other resources except the currently occupied resources in the SPARK cluster; determining the splitting number of the target task information according to the number of the available resources; and splitting the target task according to the split number to obtain at least one piece of target subtask information.
Optionally, determining the number of available resources in the SPARK cluster according to the information characterizing the urgency of the target task and the current resource utilization rate, including: determining the resource demand rate of the target task information according to the information representing the target task emergency degree; and multiplying the difference between the resource demand rate and the current resource utilization rate by the total amount of the resources to obtain the number of the available resources.
Optionally, determining the split number of the target task information according to the number of the available resources includes: acquiring the number of nodes in the SPARK cluster, wherein the nodes are used for calculating the target subtask information; and dividing the number of the available resources by the number of the nodes to obtain the splitting number.
Optionally, after determining the number of available resources in the SPARK cluster according to the information characterizing the urgency of the target task and the current resource utilization rate, before determining the split number of the target task information according to the number of available resources, the method further includes: acquiring the quantity of required resources, wherein the quantity of the required resources is the quantity of resources required for calculating the target task information; determining whether the quantity of the demand resources is larger than the quantity of the available resources, and suspending the calculation of non-urgent task information when the quantity of the demand resources is larger than the quantity of the available resources, wherein the non-urgent task is the other tasks with the urgency degree lower than that of the target task, so that the quantity of the demand resources is smaller than or equal to the quantity of the available resources.
Optionally, suspending the calculation of the non-urgent task information comprises: under the condition that the quantity of the demand resources is larger than the quantity of the available resources, acquiring the emergency degree of the other tasks, and determining the non-emergency tasks; and suspending the non-emergency tasks in sequence from the small emergency degree to the large emergency degree until the quantity of the demand resources is less than or equal to the quantity of the available resources.
Optionally, in a case that the required resource amount is still greater than the available resource amount after suspending all the non-emergency task information, the method further includes: and controlling the target task to enter a waiting mode until the quantity of the demand resources is less than or equal to the quantity of the available resources.
According to another aspect of the embodiments of the present invention, there is also provided a task scheduling apparatus, including: the system comprises a receiving unit, a processing unit and a processing unit, wherein the receiving unit is used for receiving target task related information, and the target task related information comprises target task information and information representing the emergency degree of the target task; the system comprises a determining unit and a processing unit, wherein the determining unit is used for determining the current resource utilization rate of the SPARK cluster, and the current resource utilization rate is the occupation ratio of the number of currently occupied resources in the SPARK cluster; the splitting unit is used for splitting the target task information according to the target task related information and the current resource utilization rate to obtain at least one target subtask information; and the sending unit is used for sending the information of each target subtask to the SPARK cluster for calculation, receiving each calculation result and sending out each calculation result.
According to still another aspect of embodiments of the present invention, there is also provided a computer-readable storage medium including a stored program, wherein the program executes any one of the methods.
According to still another aspect of the embodiments of the present invention, there is further provided a processor, configured to execute a program, where the program executes any one of the methods.
According to still another aspect of the embodiments of the present invention, there is also provided an electronic device, including: one or more processors, memory, and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods described herein.
In the embodiment of the invention, in the method for task scheduling, firstly, target task related information is received, wherein the target task related information comprises target task information and information representing the urgency degree of the target task; then, determining the current resource utilization rate of the SPARK cluster, wherein the current resource utilization rate is the ratio of the number of currently occupied resources in the SPARK cluster; then, splitting the target task information according to the target task related information and the current resource utilization rate to obtain at least one target subtask information; and finally, sending the information of each target subtask to the SPARK cluster for calculation, receiving each calculation result and sending out each calculation result. According to the task scheduling method, the target task is split according to the target task information, the target task urgency information and the current resource utilization rate to obtain the target subtask information, the target subtask information is sent to the SPARK cluster to be calculated, the target task is uniformly received, controlled and distributed, the task scheduling method based on the target task information and the urgency is achieved, the cluster resources are further intelligently scheduled and automatically distributed, and therefore the problem that the task execution efficiency is low in the prior art is solved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the application and, together with the description, serve to explain the application and are not intended to limit the application. In the drawings:
FIG. 1 shows a flow diagram of a method of task scheduling according to an embodiment of the present application;
FIG. 2 shows a schematic diagram of an apparatus for task scheduling according to an embodiment of the present application;
FIG. 3 shows a schematic diagram of a data computing system according to an embodiment of the present application.
Reference numerals:
100. a SPARK cluster; 101. calculating a node; 102. a management node; 103. a task scheduling center; 104. and a task issuing module.
Detailed Description
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It should be understood that the data so used may be interchanged under appropriate circumstances such that embodiments of the application described herein may be used. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
It will be understood that when an element such as a layer, film, region, or substrate is referred to as being "on" another element, it can be directly on the other element or intervening elements may also be present. Also, in the specification and claims, when an element is described as being "connected" to another element, the element may be "directly connected" to the other element or "connected" to the other element through a third element.
As mentioned in the background, in order to solve the above problem, in the prior art, the task execution efficiency is low, and in an exemplary embodiment of the present application, a method, an apparatus, a computer-readable storage medium, a processor, and an electronic device for task scheduling are provided.
According to an embodiment of the present application, a method of task scheduling is provided.
Fig. 1 is a flowchart of a method of task scheduling according to an embodiment of the present application. As shown in fig. 1, the method comprises the steps of:
step S101, receiving related information of a target task, wherein the related information of the target task comprises target task information and information representing the emergency degree of the target task;
step S102, determining the current resource utilization rate of the SPARK cluster, wherein the current resource utilization rate is the ratio of the number of currently occupied resources in the SPARK cluster;
step S103, splitting the target task information according to the target task related information and the current resource utilization rate to obtain at least one target subtask information;
and step S104, sending each target subtask information to the SPARK cluster for calculation, receiving each calculation result and sending out each calculation result.
The task scheduling method comprises the steps of firstly, receiving target task related information, wherein the target task related information comprises target task information and information representing the emergency degree of the target task; determining the current resource utilization rate of the SPARK cluster, wherein the current resource utilization rate is the ratio of the number of currently occupied resources in the SPARK cluster, namely, the resource utilization rate is the ratio of the number of currently occupied resources in the SPARK cluster to the total number of cluster resources; splitting the target task information according to the target task related information and the current resource utilization rate to obtain at least one target subtask information; and sending each target subtask information to the SPARK cluster for calculation, receiving each calculation result and sending out each calculation result. According to the task scheduling method, the target task is split according to the target task information, the target task urgency information and the current resource utilization rate to obtain the target subtask information, the target subtask information is sent to the SPARK cluster to be calculated, the target task is uniformly received, controlled and distributed, the task scheduling method based on the target task information and the urgency is achieved, the cluster resources are further intelligently scheduled and automatically distributed, and therefore the problem that the task execution efficiency is low in the prior art is solved.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer-executable instructions and that, although a logical order is illustrated in the flowcharts, in some cases, the steps illustrated or described may be performed in an order different than presented herein.
In an embodiment of the present application, determining a current resource utilization rate of the SPARK cluster includes: acquiring the total amount of resources in the SPARK cluster; acquiring the quantity of the currently occupied resources, wherein the currently occupied resources are resources occupied by other tasks in the SPARK cluster; and determining the current resource utilization rate according to the total resource amount and the number of the currently occupied resources. Therefore, the current resource utilization rate can be determined quickly and accurately, the tasks can be distributed more reasonably according to the current resource utilization rate, and the execution efficiency of the tasks is further improved.
In another embodiment of the present application, splitting the target task information according to the target task related information and the current resource utilization includes: determining the number of available resources in the SPARK cluster according to the information representing the urgency degree of the target task and the current resource utilization rate, wherein the available resources are other resources except for the currently occupied resources in the SPARK cluster; determining the splitting number of the target task information according to the number of the available resources; and splitting the target task according to the splitting number to obtain at least one piece of target subtask information. In the embodiment, the number of available resources in the SPARK cluster is determined according to the information representing the emergency degree of the target task and the current resource utilization rate, the splitting number of the target task information is determined according to the number of the available resources, so that a more reasonable splitting number can be obtained, the target task is split and the resource allocation is calculated according to the splitting number, and the task execution efficiency is further ensured.
In another embodiment of the present application, determining the number of available resources in the SPARK cluster according to the information indicating the urgency of the target task and the current resource utilization rate includes: determining the resource demand rate of the target task information according to the information representing the urgency degree of the target task; and multiplying the difference between the resource demand rate and the current resource utilization rate by the total amount of the resources to obtain the number of the available resources. In the embodiment, the quantity of the available resources is obtained by multiplying the difference between the resource demand rate and the current resource utilization rate by the total quantity of the resources, so that the quantity of the available resources can be more accurately obtained, the target task can be more accurately split and the resource allocation can be more accurately calculated according to the quantity of the available resources subsequently, and the task execution efficiency is further improved.
In another embodiment of the present application, determining the split number of the target task information according to the number of the available resources includes: acquiring the number of nodes in the SPARK cluster, wherein the nodes are used for calculating the target subtask information; and dividing the number of the available resources by the number of the nodes to obtain the splitting number. In the embodiment, the number of the available resources is divided by the number of the nodes, so that the splitting number can be obtained more accurately, and subsequently, the task can be scheduled more reasonably according to the splitting number, thereby further improving the efficiency of executing the task.
In an embodiment of the application, after determining the number of available resources in the SPARK cluster according to the information representing the urgency of the target task and the current resource utilization rate, before determining the split number of the target task information according to the number of available resources, the method further includes: acquiring the quantity of required resources, wherein the quantity of the required resources is the quantity of resources required for calculating the target task information; determining whether the quantity of the demand resources is greater than the quantity of the available resources, and suspending the calculation of non-urgent task information when the quantity of the demand resources is greater than the quantity of the available resources, wherein the non-urgent task is the other tasks with the urgency degree lower than that of the target task, so that the quantity of the demand resources is less than or equal to the quantity of the available resources. In the embodiment, under the condition that the quantity of the required resources is greater than the quantity of the available resources, the calculation of the non-emergency task information is suspended, so that the emergency task information can be further calculated, and meanwhile, the emergency task can be processed preferentially, so that the task scheduling is more reasonable, and the task execution efficiency is improved.
Specifically, the urgency levels may be classified into 0 to 9, which have 10 priority levels, and the higher the number in the 1 to 9 levels is, the higher the urgency level of the task is, and the 0 level is an ultra-urgent task. When the urgency level of the target task is level 0, the calculation of the number of available resources may not be performed, and all resources in the SPARK cluster may be utilized. Of course, the division of the target task urgency level may be 0 to 9, but the division of the target task urgency level is not limited to 0 to 9, and may be other reasonable numbers capable of representing the task urgency level.
In an embodiment of the present application, in a case that the number of the required resources is greater than the number of the available resources, the urgency level of the other task is obtained, and the non-urgent task is determined; and suspending the non-emergency tasks in sequence from the small emergency degree to the large emergency degree until the quantity of the required resources is less than or equal to the quantity of the available resources. In the embodiment, the non-emergency tasks are suspended in sequence from the small emergency degree to the large emergency degree, so that the number of currently available resources occupied by the non-emergency tasks is avoided, the emergency tasks can be further ensured to be smoothly executed, and the task execution efficiency is improved.
In a specific embodiment of the present application, when the quantity of the required resources is greater than the quantity of the available resources, according to the urgency of the task, the computing tasks with the urgency of less than 9 are suspended preferentially, and the steps of task computing are re-executed after the computing tasks are suspended in sequence from small to large until the quantity of the required resources is less than the quantity of the available resources.
In another embodiment of the application, after suspending all the non-urgent task information, in a case that the quantity of the required resources is still greater than the quantity of the available resources, the method further includes: and controlling the target task to enter a waiting mode until the quantity of the required resources is less than or equal to the quantity of the available resources. By controlling the target task to enter the waiting mode, the task which is being executed can be smoothly executed, the scheduling of the target task can be orderly carried out, and the execution efficiency of the task is further improved.
The embodiment of the present application further provides a device for task scheduling, and it should be noted that the device for task scheduling according to the embodiment of the present application may be used to execute the method for task scheduling according to the embodiment of the present application. The following describes a task scheduling apparatus according to an embodiment of the present application.
Fig. 2 is a schematic diagram of an apparatus for task scheduling according to an embodiment of the present application. As shown in fig. 2, the apparatus includes:
a receiving unit 10, configured to receive target task related information, where the target task related information includes target task information and information representing an urgency of the target task;
a determining unit 20, configured to determine a current resource utilization rate of the SPARK cluster, where the current resource utilization rate is a ratio of a number of currently occupied resources in the SPARK cluster;
a splitting unit 30, configured to split the target task information according to the target task related information and the current resource utilization rate, so as to obtain at least one target subtask information;
and a sending unit 40, configured to send each piece of target subtask information to the SPARK cluster for calculation, receive each calculation result, and send out each calculation result.
In the task scheduling device, a receiving unit is configured to receive target task related information, where the target task related information includes target task information and information representing an emergency degree of the target task; the determining unit is configured to determine a current resource utilization rate of the SPARK cluster, where the current resource utilization rate is an occupied ratio of the number of currently occupied resources in the SPARK cluster, that is, of course, the resource utilization rate is a ratio of the number of currently occupied resources in the SPARK cluster to a total amount of cluster resources; the splitting unit is used for splitting the target task information according to the target task related information and the current resource utilization rate to obtain at least one target subtask information; and the sending unit is used for sending each target subtask information to the SPARK cluster for calculation, receiving each calculation result and sending out each calculation result. In the task scheduling device, the target task is split according to the target task information, the target task urgency information and the current resource utilization rate to obtain the target subtask information, and the target subtask information is sent to the SPARK cluster to be calculated, so that the target task unified receiving, controlling and distributing and the task scheduling method based on the target task information and the urgency are realized, and further, an intelligent task scheduling and load automatic allocation mechanism is performed on cluster resources, so that the problem of low task execution efficiency in the prior art is solved.
In an embodiment of the present application, the determining unit further includes a first obtaining module, a second obtaining module, and a first determining module, where the first obtaining module is configured to obtain a total amount of resources in the SPARK cluster; the second acquiring module is configured to acquire the number of the currently occupied resources, where the currently occupied resources are resources occupied by other tasks in the current SPARK cluster; the first determining module is used for determining the current resource utilization rate according to the total resource amount and the number of the currently occupied resources. Therefore, the current resource utilization rate can be determined quickly and accurately, the tasks can be distributed more reasonably according to the current resource utilization rate, and the execution efficiency of the tasks is further improved.
In another embodiment of the present application, the splitting unit further includes a second determining module, a third determining module, and a splitting module, where the second determining module is configured to determine, according to the information representing the urgency of the target task and the current resource utilization rate, the number of available resources in the SPARK cluster, where the available resources are other resources in the SPARK cluster except for the currently occupied resource; the third determining module is used for determining the splitting number of the target task information according to the number of the available resources; the splitting module is used for splitting the target task according to the splitting number to obtain at least one piece of target subtask information. In the embodiment, the number of available resources in the SPARK cluster is determined according to the information representing the emergency degree of the target task and the current resource utilization rate, the splitting number of the target task information is determined according to the number of the available resources, so that a more reasonable splitting number can be obtained, the target task is split and the resource allocation is calculated according to the splitting number, and the task execution efficiency is further ensured.
In another embodiment of the present application, the second determining module further includes a determining submodule and a multiplying submodule, where the determining submodule is configured to determine a resource demand rate of the target task information according to the information representing the urgency of the target task; and the multiplication submodule is used for multiplying the difference between the resource demand rate and the current resource utilization rate and the total amount of the resources to obtain the number of the available resources. In the embodiment, the quantity of the available resources is obtained by multiplying the difference between the resource demand rate and the current resource utilization rate by the total quantity of the resources, so that the quantity of the available resources can be more accurately obtained, the target task can be more accurately split and the resource allocation can be more accurately calculated according to the quantity of the available resources subsequently, and the task execution efficiency is further improved.
In yet another embodiment of the present application, the third determining module further includes a first obtaining sub-module and a dividing sub-module, where the first obtaining sub-module is configured to obtain the number of nodes in the SPARK cluster, and the nodes are configured to calculate the target subtask information; and the division submodule is used for dividing the number of the available resources by the number of the nodes to obtain the splitting number. In the embodiment, the number of the available resources is divided by the number of the nodes, so that the splitting number can be obtained more accurately, and subsequently, the task can be scheduled more reasonably according to the splitting number, thereby further improving the efficiency of executing the task.
In an embodiment of the application, the splitting unit further includes a third obtaining module and a fourth determining module, where the third obtaining module is configured to, after determining the number of available resources in the SPARK cluster according to the information representing the urgency of the target task and the current resource utilization rate, before determining the split number of the target task information according to the number of available resources, further include: acquiring the quantity of required resources, wherein the quantity of the required resources is the quantity of resources required for calculating the target task information; the fourth determining module is configured to determine whether the quantity of the demand resources is greater than the quantity of the available resources, and suspend the calculation of non-urgent task information when the quantity of the demand resources is greater than the quantity of the available resources, where the non-urgent task is the other task whose urgency degree is lower than the target task, so that the quantity of the demand resources is less than or equal to the quantity of the available resources. In the embodiment, under the condition that the quantity of the required resources is greater than the quantity of the available resources, the calculation of the non-emergency task information is suspended, so that the emergency task information can be further calculated, and meanwhile, the emergency task can be processed preferentially, so that the task scheduling is more reasonable, and the task execution efficiency is improved.
Specifically, the urgency levels may be classified into 0 to 9, which have 10 priority levels, and the higher the number in the 1 to 9 levels is, the higher the urgency level of the task is, and the 0 level is an ultra-urgent task. When the urgency level of the target task is level 0, the calculation of the number of available resources may not be performed, and all resources in the SPARK cluster may be utilized. Of course, the division of the target task urgency level may be 0 to 9, but the division of the target task urgency level is not limited to 0 to 9, and may be other reasonable numbers capable of representing the task urgency level.
In another embodiment of the present application, in an embodiment of the present application, the splitting unit further includes a fourth obtaining module and a suspending module, where the fourth obtaining module is configured to obtain the urgency level of the other task and determine the non-urgent task when the quantity of the required resources is greater than the quantity of the available resources; the suspension module is used for suspending the non-emergency tasks in sequence from small to large according to the emergency degree until the quantity of the required resources is less than or equal to the quantity of the available resources. In the embodiment, the non-emergency tasks are suspended in sequence from the small emergency degree to the large emergency degree, so that the number of currently available resources occupied by the non-emergency tasks is avoided, the emergency tasks can be further ensured to be smoothly executed, and the task execution efficiency is improved.
In another embodiment of the application, the splitting unit further includes a control module, configured to, after suspending all the non-emergency task information, if the quantity of the required resources is still greater than the quantity of the available resources, further include: and controlling the target task to enter a waiting mode until the quantity of the required resources is less than or equal to the quantity of the available resources. By controlling the target task to enter the waiting mode, the task which is being executed can be smoothly executed, the scheduling of the target task can be orderly carried out, and the execution efficiency of the task is further improved.
The device for task scheduling comprises a processor and a memory, wherein the receiving unit, the determining unit, the splitting unit, the sending unit and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions.
The processor comprises a kernel, and the kernel calls the corresponding program unit from the memory. The kernel can be set to be one or more than one, and the problem of low efficiency of executing tasks in the prior art is solved by adjusting kernel parameters.
The memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or nonvolatile memory such as Read Only Memory (ROM) or flash memory (flash RAM), and the memory includes at least one memory chip.
An embodiment of the present invention provides a computer-readable storage medium, on which a program is stored, which, when executed by a processor, implements the method for task scheduling.
The embodiment of the invention provides a processor, which is used for running a program, wherein the method for scheduling tasks is executed when the program runs.
An embodiment of the present invention further provides an electronic device, including: one or more processors, memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing any of the above-described methods.
The embodiment of the invention provides equipment, which comprises a processor, a memory and a program which is stored on the memory and can run on the processor, wherein when the processor executes the program, at least the following steps are realized:
step S101, receiving related information of a target task, wherein the related information of the target task comprises target task information and information representing the emergency degree of the target task;
step S102, determining the current resource utilization rate of the SPARK cluster, wherein the current resource utilization rate is the ratio of the number of currently occupied resources in the SPARK cluster;
step S103, splitting the target task information according to the target task related information and the current resource utilization rate to obtain at least one target subtask information;
and step S104, sending each target subtask information to the SPARK cluster for calculation, receiving each calculation result and sending out each calculation result.
The device herein may be a server, a PC, a PAD, a mobile phone, etc.
The present application further provides a computer program product adapted to perform a program of initializing at least the following method steps when executed on a data processing device:
step S101, receiving related information of a target task, wherein the related information of the target task comprises target task information and information representing the emergency degree of the target task;
step S102, determining the current resource utilization rate of the SPARK cluster, wherein the current resource utilization rate is the ratio of the number of currently occupied resources in the SPARK cluster;
step S103, splitting the target task information according to the target task related information and the current resource utilization rate to obtain at least one target subtask information;
and step S104, sending each target subtask information to the SPARK cluster for calculation, receiving each calculation result and sending out each calculation result.
According to a specific embodiment of the present application, as shown in fig. 3, a data computing system is provided, where the system includes a SPARK cluster 100, a task scheduling center 103, and a task issuing module 104, where the SPARK cluster 100 is built for an open-source SPARK component, and can ensure technical universality, and a computing node 101 is responsible for task computation; the management node 102 is responsible for the allocation of resources and tasks. The task scheduling center 103 is configured to execute any one of the task scheduling methods, and the task issuing module 104 is configured to serve an application with a computing requirement, and may submit a computing task through a service issued by the task scheduling center 103.
In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units may be a logical division, and in actual implementation, there may be another division, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a computer-readable storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
From the above description, it can be seen that the above-described embodiments of the present application achieve the following technical effects:
1) the task scheduling method comprises the steps of firstly, receiving target task related information, wherein the target task related information comprises target task information and information representing the emergency degree of the target task; determining the current resource utilization rate of the SPARK cluster, wherein the current resource utilization rate is the ratio of the number of currently occupied resources in the SPARK cluster, namely, the resource utilization rate is the ratio of the number of currently occupied resources in the SPARK cluster to the total number of cluster resources; splitting the target task information according to the target task related information and the current resource utilization rate to obtain at least one target subtask information; and sending each target subtask information to the SPARK cluster for calculation, receiving each calculation result and sending out each calculation result. According to the task scheduling method, the target task is split according to the target task information, the target task urgency information and the current resource utilization rate to obtain the target subtask information, the target subtask information is sent to the SPARK cluster to be calculated, the target task is uniformly received, controlled and distributed, the task scheduling method based on the target task information and the urgency is achieved, the cluster resources are further intelligently scheduled and automatically distributed, and therefore the problem that the task execution efficiency is low in the prior art is solved.
2) The receiving unit is used for receiving the related information of the target task, and the related information of the target task comprises target task information and information representing the urgency degree of the target task; the determining unit is configured to determine a current resource utilization rate of the SPARK cluster, where the current resource utilization rate is an occupied ratio of the number of currently occupied resources in the SPARK cluster, that is, of course, the resource utilization rate is a ratio of the number of currently occupied resources in the SPARK cluster to a total amount of cluster resources; the splitting unit is used for splitting the target task information according to the target task related information and the current resource utilization rate to obtain at least one target subtask information; and the sending unit is used for sending each target subtask information to the SPARK cluster for calculation, receiving each calculation result and sending out each calculation result. In the task scheduling device, the target task is split according to the target task information, the target task urgency information and the current resource utilization rate to obtain the target subtask information, and the target subtask information is sent to the SPARK cluster to be calculated, so that the target task unified receiving, controlling and distributing and the task scheduling method based on the target task information and the urgency are realized, and further, an intelligent task scheduling and load automatic allocation mechanism is performed on cluster resources, so that the problem of low task execution efficiency in the prior art is solved.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (12)
1. A method of task scheduling, comprising:
receiving target task related information, wherein the target task related information comprises target task information and information representing the urgency degree of the target task;
determining the current resource utilization rate of the SPARK cluster, wherein the current resource utilization rate is the ratio of the number of currently occupied resources in the SPARK cluster;
splitting the target task information according to the target task related information and the current resource utilization rate to obtain at least one target subtask information;
and sending each target subtask information to the SPARK cluster for calculation, receiving each calculation result and sending out each calculation result.
2. The method of claim 1, wherein determining a current resource utilization of the SPARK cluster comprises:
acquiring the total amount of resources in the SPARK cluster;
acquiring the quantity of the currently occupied resources, wherein the currently occupied resources are resources occupied by other tasks in the SPARK cluster;
and determining the current resource utilization rate according to the total resource amount and the number of the currently occupied resources.
3. The method of claim 2, wherein splitting the target task information according to the target task related information and the current resource utilization comprises:
determining the number of available resources in the SPARK cluster according to the information representing the emergency degree of the target task and the current resource utilization rate, wherein the available resources are other resources except the currently occupied resources in the SPARK cluster;
determining the splitting number of the target task information according to the number of the available resources;
and splitting the target task according to the split number to obtain at least one piece of target subtask information.
4. The method of claim 2, wherein determining the amount of available resources in the SPARK cluster based on the information characterizing the urgency of the target task and the current resource utilization comprises:
determining the resource demand rate of the target task information according to the information representing the target task emergency degree;
and multiplying the difference between the resource demand rate and the current resource utilization rate by the total amount of the resources to obtain the number of the available resources.
5. The method of claim 3, wherein determining the split number of the target task information according to the number of available resources comprises:
acquiring the number of nodes in the SPARK cluster, wherein the nodes are used for calculating the target subtask information;
and dividing the number of the available resources by the number of the nodes to obtain the splitting number.
6. The method of claim 2, wherein after determining the number of available resources in the SPARK cluster based on the information characterizing the urgency of the target task and the current resource utilization, prior to determining the split number of target task information based on the number of available resources, the method further comprises:
acquiring the quantity of required resources, wherein the quantity of the required resources is the quantity of resources required for calculating the target task information;
determining whether the quantity of the demand resources is larger than the quantity of the available resources, and suspending the calculation of non-urgent task information when the quantity of the demand resources is larger than the quantity of the available resources, wherein the non-urgent task is the other tasks with the urgency degree lower than that of the target task, so that the quantity of the demand resources is smaller than or equal to the quantity of the available resources.
7. The method of claim 6, further comprising:
under the condition that the quantity of the demand resources is larger than the quantity of the available resources, acquiring the emergency degree of the other tasks, and determining the non-emergency tasks;
and suspending the non-emergency tasks in sequence from the small emergency degree to the large emergency degree until the quantity of the demand resources is less than or equal to the quantity of the available resources.
8. The method of claim 7, wherein in the case that the amount of demand resources is still greater than the amount of available resources after suspending all of the non-emergency task information, the method further comprises:
and controlling the target task to enter a waiting mode until the quantity of the demand resources is less than or equal to the quantity of the available resources.
9. An apparatus for task scheduling, comprising:
the system comprises a receiving unit, a processing unit and a processing unit, wherein the receiving unit is used for receiving target task related information, and the target task related information comprises target task information and information representing the emergency degree of the target task;
the system comprises a determining unit and a processing unit, wherein the determining unit is used for determining the current resource utilization rate of the SPARK cluster, and the current resource utilization rate is the occupation ratio of the number of currently occupied resources in the SPARK cluster;
the splitting unit is used for splitting the target task information according to the target task related information and the current resource utilization rate to obtain at least one target subtask information;
and the sending unit is used for sending the information of each target subtask to the SPARK cluster for calculation, receiving each calculation result and sending out each calculation result.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium comprises a stored program, wherein the program performs the method of any one of claims 1 to 8.
11. A processor, characterized in that the processor is configured to run a program, wherein the program when running performs the method of any of claims 1 to 8.
12. An electronic device, comprising: one or more processors, memory, and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011589305.2A CN112540841B (en) | 2020-12-28 | 2020-12-28 | Task scheduling method and device, processor and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011589305.2A CN112540841B (en) | 2020-12-28 | 2020-12-28 | Task scheduling method and device, processor and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112540841A true CN112540841A (en) | 2021-03-23 |
| CN112540841B CN112540841B (en) | 2021-11-12 |
Family
ID=75017759
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011589305.2A Active CN112540841B (en) | 2020-12-28 | 2020-12-28 | Task scheduling method and device, processor and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112540841B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113391649A (en) * | 2021-08-17 | 2021-09-14 | 成都卡莱博尔信息技术股份有限公司 | Unmanned aerial vehicle display and control integrated method, system and device and computer readable storage medium |
| CN113612732A (en) * | 2021-07-06 | 2021-11-05 | 华控清交信息科技(北京)有限公司 | Resource calling method and device and multi-party secure computing system |
| CN113687951A (en) * | 2021-09-01 | 2021-11-23 | 睿云奇智(重庆)科技有限公司 | Task processing method and device, electronic equipment and computer readable storage medium |
| CN114710485A (en) * | 2022-02-21 | 2022-07-05 | 联想(北京)有限公司 | Processing method and processing apparatus |
| CN116109110A (en) * | 2023-04-11 | 2023-05-12 | 华能信息技术有限公司 | Task scheduling method for service center |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2005115620A (en) * | 2003-10-07 | 2005-04-28 | Sony Corp | Task management method and electronic apparatus having task management means |
| CN101129033A (en) * | 2005-02-28 | 2008-02-20 | 特克拉科技公司 | A method of and a system for controlling access to a shared resource |
| CN103067468A (en) * | 2012-12-22 | 2013-04-24 | 深圳先进技术研究院 | Cloud scheduling method and system thereof |
| CN107491351A (en) * | 2016-06-13 | 2017-12-19 | 阿里巴巴集团控股有限公司 | A kind of resource allocation methods based on priority, device and equipment |
| CN109298940A (en) * | 2018-09-28 | 2019-02-01 | 考拉征信服务有限公司 | Calculation task allocating method, device, electronic equipment and computer storage medium |
| CN110187960A (en) * | 2019-04-23 | 2019-08-30 | 广东省智能制造研究所 | A kind of distributed resource scheduling method and device |
| CN110209496A (en) * | 2019-05-20 | 2019-09-06 | 中国平安财产保险股份有限公司 | Task sharding method, device and sliced service device based on data processing |
| CN111367630A (en) * | 2019-07-12 | 2020-07-03 | 北京关键科技股份有限公司 | Multi-user multi-priority distributed cooperative processing method based on cloud computing |
-
2020
- 2020-12-28 CN CN202011589305.2A patent/CN112540841B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2005115620A (en) * | 2003-10-07 | 2005-04-28 | Sony Corp | Task management method and electronic apparatus having task management means |
| CN101129033A (en) * | 2005-02-28 | 2008-02-20 | 特克拉科技公司 | A method of and a system for controlling access to a shared resource |
| CN103067468A (en) * | 2012-12-22 | 2013-04-24 | 深圳先进技术研究院 | Cloud scheduling method and system thereof |
| CN107491351A (en) * | 2016-06-13 | 2017-12-19 | 阿里巴巴集团控股有限公司 | A kind of resource allocation methods based on priority, device and equipment |
| CN109298940A (en) * | 2018-09-28 | 2019-02-01 | 考拉征信服务有限公司 | Calculation task allocating method, device, electronic equipment and computer storage medium |
| CN110187960A (en) * | 2019-04-23 | 2019-08-30 | 广东省智能制造研究所 | A kind of distributed resource scheduling method and device |
| CN110209496A (en) * | 2019-05-20 | 2019-09-06 | 中国平安财产保险股份有限公司 | Task sharding method, device and sliced service device based on data processing |
| CN111367630A (en) * | 2019-07-12 | 2020-07-03 | 北京关键科技股份有限公司 | Multi-user multi-priority distributed cooperative processing method based on cloud computing |
Non-Patent Citations (2)
| Title |
|---|
| ABDUL HAMEED 等: "A survey and taxonomy on energy efficient resource allocation techniques for cloud computing systems", 《COMPUTING》 * |
| 惠毅 等: "一种基于业务优先级的跨层资源分配算法", 《***仿真学报》 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113612732A (en) * | 2021-07-06 | 2021-11-05 | 华控清交信息科技(北京)有限公司 | Resource calling method and device and multi-party secure computing system |
| CN113612732B (en) * | 2021-07-06 | 2023-12-26 | 华控清交信息科技(北京)有限公司 | Resource calling method and device and multiparty secure computing system |
| CN113391649A (en) * | 2021-08-17 | 2021-09-14 | 成都卡莱博尔信息技术股份有限公司 | Unmanned aerial vehicle display and control integrated method, system and device and computer readable storage medium |
| CN113391649B (en) * | 2021-08-17 | 2021-11-02 | 成都卡莱博尔信息技术股份有限公司 | Unmanned aerial vehicle display and control integrated method, system and device and computer readable storage medium |
| CN113687951A (en) * | 2021-09-01 | 2021-11-23 | 睿云奇智(重庆)科技有限公司 | Task processing method and device, electronic equipment and computer readable storage medium |
| CN114710485A (en) * | 2022-02-21 | 2022-07-05 | 联想(北京)有限公司 | Processing method and processing apparatus |
| CN114710485B (en) * | 2022-02-21 | 2023-10-27 | 联想(北京)有限公司 | Processing method and processing device |
| CN116109110A (en) * | 2023-04-11 | 2023-05-12 | 华能信息技术有限公司 | Task scheduling method for service center |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112540841B (en) | 2021-11-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112540841B (en) | Task scheduling method and device, processor and electronic equipment | |
| CN107360206B (en) | Block chain consensus method, equipment and system | |
| JP6490913B2 (en) | Task execution by idle resources of grid computing system | |
| CN109408205B (en) | Task scheduling method and device based on hadoop cluster | |
| US9262220B2 (en) | Scheduling workloads and making provision decisions of computer resources in a computing environment | |
| Kaur et al. | Analysis of job scheduling algorithms in cloud computing | |
| CN105487930A (en) | Task optimization scheduling method based on Hadoop | |
| CN113377540A (en) | Cluster resource scheduling method and device, electronic equipment and storage medium | |
| US20120054770A1 (en) | High throughput computing in a hybrid computing environment | |
| CN106598735B (en) | A kind of distributed computing method, main controlled node and computing system | |
| US9229777B2 (en) | Dynamically relocating workloads in a networked computing environment | |
| CN112463375B (en) | Data processing method and device | |
| CN109840142A (en) | Thread control method, device, electronic equipment and storage medium based on cloud monitoring | |
| US11755926B2 (en) | Prioritization and prediction of jobs using cognitive rules engine | |
| CN110166507B (en) | Multi-resource scheduling method and device | |
| CN109254836B (en) | Deadline constraint cost optimization scheduling method for priority dependent tasks of cloud computing system | |
| Bok et al. | An efficient MapReduce scheduling scheme for processing large multimedia data | |
| CN112579304A (en) | Resource scheduling method, device, equipment and medium based on distributed platform | |
| CN112905342A (en) | Resource scheduling method, device, equipment and computer readable storage medium | |
| Xu et al. | Fault tolerance and quality of service aware virtual machine scheduling algorithm in cloud data centers | |
| US20130007281A1 (en) | Dynamically tuning server placement | |
| CN105740249B (en) | Processing method and system in parallel scheduling process of big data job | |
| CN109614242B (en) | Computing capacity sharing method, device, equipment and medium | |
| CN116821187A (en) | Database-based data processing method and device, medium and electronic equipment | |
| US20210149746A1 (en) | Method, System, Computer Readable Medium, and Device for Scheduling Computational Operation Based on Graph Data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |