CN117149372A - Task scheduling method, device, equipment and storage medium - Google Patents
Task scheduling method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN117149372A CN117149372A CN202310878705.2A CN202310878705A CN117149372A CN 117149372 A CN117149372 A CN 117149372A CN 202310878705 A CN202310878705 A CN 202310878705A CN 117149372 A CN117149372 A CN 117149372A
- Authority
- CN
- China
- Prior art keywords
- cluster
- task
- force
- computing
- computing power
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 60
- 238000004364 calculation method Methods 0.000 claims abstract description 48
- 239000011159 matrix material Substances 0.000 claims description 75
- 238000012216 screening Methods 0.000 claims description 32
- 238000001914 filtration Methods 0.000 claims description 13
- 239000012634 fragment Substances 0.000 claims description 11
- 230000010354 integration Effects 0.000 claims description 11
- 238000013468 resource allocation Methods 0.000 claims description 10
- 238000012544 monitoring process Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 description 10
- 238000004458 analytical method Methods 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000005457 optimization Methods 0.000 description 4
- 238000012423 maintenance Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000001934 delay Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000013467 fragmentation Methods 0.000 description 1
- 238000006062 fragmentation reaction Methods 0.000 description 1
- 239000003292 glue Substances 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000002071 nanotube Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 238000012384 transportation and delivery Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5077—Logical partitioning of resources; Management or configuration of virtualized resources
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Supply And Distribution Of Alternating Current (AREA)
Abstract
The invention belongs to the technical field of computers, and discloses a task scheduling method, device, equipment and storage medium. The method comprises the following steps: when an application deployment task is received, cluster preselection is carried out according to service arrangement information of the application deployment task, and a plurality of alternative computing force clusters are determined; performing loss calculation on the resource idle weight and the cluster balance degree of each alternative computing force cluster, and determining idle balance loss of each alternative computing force cluster; determining a target computing force cluster in a plurality of candidate computing force clusters according to the idle balance loss of each candidate computing force cluster; and carrying out task scheduling on the application deployment task according to the target computing power cluster. By the method, resources can be used among multiple centers and multiple clusters in a task mode, the whole network resources are regarded as a unified whole, unified allocation and scheduling of the clusters of each center and the resources thereof are realized, actual service demands are focused when application deployment is carried out, the use rationality of the resources is improved, and the deployment cost is reduced.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a task scheduling method, device, apparatus, and storage medium.
Background
The target PAAS (Platform as a Service, platform as service) platform realizes nano-tube of computing power resources such as CPU, memory, storage and the like through K8S cluster, provides agile delivery, total station support, intelligent operation and maintenance and other capabilities to the upper layer, realizes rapid application deployment and implementation through the PAAS platform, realizes operation and maintenance and life cycle management of the application through the platform, when the platform user deploys the application and uses the resources, the user needs to know cluster resources meeting the user requirements in advance after the resource use condition, and the platform completes resource-oriented deployment and other operations according to the resources selected by the user. However, when the calculation power target is manually specified, the actual demand of the business cannot be focused while the use experience is poor, so that the cluster resource is unreasonable to use.
Disclosure of Invention
The invention mainly aims to provide a task scheduling method, device, equipment and storage medium, and aims to solve the technical problem of how to efficiently and reasonably schedule application deployment tasks and realize reasonable use of cluster resources when application deployment is performed.
In order to achieve the above object, the present invention provides a task scheduling method, including:
when an application deployment task is received, cluster preselection is carried out according to service arrangement information of the application deployment task, and a plurality of alternative computing force clusters are determined;
performing loss calculation on the resource idle weight and the cluster balance degree of each alternative computing force cluster, and determining idle balance loss of each alternative computing force cluster;
determining a target computing force cluster in a plurality of candidate computing force clusters according to the idle balance loss of each candidate computing force cluster;
and carrying out task scheduling on the application deployment task according to the target computing power cluster.
Optionally, the cluster preselection is performed according to the service arrangement information of the application deployment task, and determining a plurality of alternative computing force clusters includes:
extracting information from the service arrangement information of the application deployment task to obtain region selection information;
determining a dispatching cluster area according to the area selection information;
acquiring the running state and occupied resource information of each computing power cluster in the dispatching cluster area;
and filtering each computing power cluster in the dispatching cluster area according to the running state and occupied resource information of each computing power cluster, and determining a plurality of alternative computing power clusters.
Optionally, the filtering each computing power cluster in the scheduling cluster area according to the running state and occupied resource information of each computing power cluster, and determining a plurality of alternative computing power clusters includes:
when the running state of each computing power cluster is a normal running state, acquiring the cluster total resources of each computing power cluster;
performing cluster filtering according to the cluster total resources and occupied resource information of each computing power cluster to determine a plurality of primary screening clusters;
and determining a plurality of alternative calculation force clusters in the plurality of primary screening clusters according to the area weight of the dispatching cluster area and the preset screening proportion.
Optionally, the calculating the loss of the resource idle weight and the cluster balance of each candidate computing power cluster, before determining the idle balance loss of each candidate computing power cluster, includes:
acquiring resource allocation information of a plurality of indexes in each alternative computing power cluster;
parameter calculation is carried out according to the resource allocation information and the index number of each index in each alternative computing power cluster, and the resource idle weight of each alternative computing power cluster and the index fragment rate of each index in each alternative computing power cluster are determined;
and calculating the balance degree according to the index fragment rate and the index number of each index in each candidate computing force cluster, and determining the cluster balance degree of each candidate computing force cluster.
Optionally, the determining the target computing force cluster from the plurality of candidate computing force clusters according to the idle balance loss of each candidate computing force cluster includes:
generating a first matrix according to the original idle weight and the original balance degree of each alternative computing force cluster;
generating a second matrix according to the original idle weight and idle balance loss of each candidate computing force cluster;
and performing cluster screening according to the first matrix and the second matrix, and determining a target computing power cluster.
Optionally, the performing cluster screening according to the first matrix and the second matrix, and determining the target computing force cluster includes:
constructing a first identity matrix and a second identity matrix according to a plurality of alternative calculation force clusters;
performing matrix calculation on the first unit matrix, the second unit matrix, the first matrix and the second matrix to determine an integration matrix;
and determining a target extremum in the integration matrix, and determining a target matrix according to a matrix queue of the target extremum.
Optionally, before the task scheduling of the application deployment task according to the target computing power cluster, the method further includes:
computing the computing power resources according to the service arrangement information, and determining the pre-occupied computing power resources;
performing resource preemption in the target computing power cluster according to the preemption computing power resource, and generating a task scheduling request;
the task scheduling of the application deployment task according to the target computing power cluster comprises the following steps:
task scheduling is carried out on the application deployment task according to the task scheduling request and the target computing power cluster;
after the task scheduling is performed on the application deployment task according to the task scheduling request and the target computing power cluster, the method further comprises:
monitoring a task deployment state of the application deployment task;
and when the task deployment state is a preset state, clearing the pre-occupied computing force resource in the target computing force cluster.
In addition, in order to achieve the above object, the present invention also proposes a task scheduling device, including:
the determining module is used for carrying out cluster preselection according to service arrangement information of the application deployment task when receiving the application deployment task, and determining a plurality of alternative calculation force clusters;
the computing module is used for carrying out loss computation on the resource idle weight and the cluster balance degree of each alternative computing force cluster and determining idle balance loss of each alternative computing force cluster;
the determining module is further used for determining a target computing force cluster from the plurality of candidate computing force clusters according to the idle balance loss of each candidate computing force cluster;
and the scheduling module is used for performing task scheduling on the application deployment task according to the target computing power cluster.
In addition, in order to achieve the above object, the present invention also proposes a task scheduling device including: a memory, a processor, and a task scheduler stored on the memory and executable on the processor, the task scheduler configured to implement the task scheduling method as described above.
In addition, in order to achieve the above object, the present invention also proposes a storage medium having stored thereon a task scheduler, which when executed by a processor, implements a task scheduling method as described above.
When an application deployment task is received, cluster preselection is carried out according to service arrangement information of the application deployment task, and a plurality of alternative calculation force clusters are determined; performing loss calculation on the resource idle weight and the cluster balance degree of each alternative computing force cluster, and determining idle balance loss of each alternative computing force cluster; determining a target computing force cluster in a plurality of candidate computing force clusters according to the idle balance loss of each candidate computing force cluster; and carrying out task scheduling on the application deployment task according to the target computing power cluster. By adopting the method, cluster preselection is carried out based on service arrangement information of application deployment tasks, corresponding idle balance loss is determined according to the idle weight and the cluster balance degree of resources of each candidate computing power cluster, cluster optimization is carried out according to the idle balance loss of each candidate computing power cluster, a target computing power cluster is determined, resources can be used in a task mode among multiple centers and multiple clusters, the whole network resources are regarded as a unified whole, unified allocation and scheduling of the clusters of each center and the resources thereof are realized, practical service demands are focused when the application is deployed, the use rationality of the resources is improved, and the deployment cost is reduced.
Drawings
FIG. 1 is a schematic diagram of a task scheduling device of a hardware running environment according to an embodiment of the present invention;
FIG. 2 is a flowchart of a task scheduling method according to a first embodiment of the present invention;
FIG. 3 is a schematic diagram of a task scheduling process according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a resource preemption procedure according to an embodiment of the task scheduling method of the present invention;
FIG. 5 is a flowchart of a task scheduling method according to a second embodiment of the present invention;
FIG. 6 is a schematic overall flow chart of an embodiment of a task scheduling method according to the present invention;
fig. 7 is a block diagram showing the construction of a first embodiment of the task scheduler according to the present invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
Referring to fig. 1, fig. 1 is a schematic diagram of a task scheduling device in a hardware running environment according to an embodiment of the present invention.
As shown in fig. 1, the task scheduling device may include: a processor 1001, such as a central processing unit (Central Processing Unit, CPU), a communication bus 1002, a user interface 1003, a network interface 1004, a memory 1005. Wherein the communication bus 1002 is used to enable connected communication between these components. The user interface 1003 may include a Display, an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may further include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a Wireless interface (e.g., a Wireless-Fidelity (Wi-Fi) interface). The Memory 1005 may be a high-speed random access Memory (Random Access Memory, RAM) Memory or a stable nonvolatile Memory (NVM), such as a disk Memory. The memory 1005 may also optionally be a storage device separate from the processor 1001 described above.
Those skilled in the art will appreciate that the structure shown in fig. 1 does not constitute a limitation of the task scheduling device, and may include more or fewer components than shown, or may combine certain components, or may be a different arrangement of components.
As shown in fig. 1, an operating system, a network communication module, a user interface module, and a task scheduler may be included in the memory 1005, which is one type of storage medium.
In the task scheduling device shown in fig. 1, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the task scheduling device of the present invention may be provided in the task scheduling device, and the task scheduling device calls the task scheduling program stored in the memory 1005 through the processor 1001 and executes the task scheduling method provided by the embodiment of the present invention.
An embodiment of the present invention provides a task scheduling method, referring to fig. 2, and fig. 2 is a schematic flow chart of a first embodiment of a task scheduling method of the present invention.
The task scheduling method comprises the following steps:
step S10: and when receiving an application deployment task, performing cluster preselection according to service arrangement information of the application deployment task, and determining a plurality of alternative computing force clusters.
It should be noted that, the execution main body of the embodiment is a task scheduling system, and the task scheduling system is built on each computing center and is a distributed computing power distribution task scheduling system based on a PAAS platform, including but not limited to a computing power analysis engine, a computing power scheduling module, a database and a redis cluster, so as to realize unified distribution scheduling of the K8S cluster and resources thereof of each computing center.
It can be appreciated that the computational power analysis engine is configured to analyze the computational power resources to calculate an optimal computational power priority schedule. The computational force analysis engine is divided into two modules: calculation force preselection and calculation force preference. The power schedule assigns the most appropriate clusters according to the scores of the power clusters. The scheduling process may execute the pre-selected policy and then execute the preferred policy to select the best computing power resource, where the preferred policy selects clusters according to the scores of the computing power clusters. The computing power dispatching module is used for abstracting dispatching behaviors to form a public system of a dispatching center, the system does not bear business logic, the dispatching center is responsible for initiating dispatching requests, abstracting tasks to be distributed JobHandler, managing the distributed JobHandler in a unified mode, and the executor is responsible for receiving the dispatching requests and executing the business logic in the corresponding JobHandler, so that the dispatching and the tasks can be decoupled from each other, and the overall stability and expansibility of the system are improved. The database is used for storing the task scheduling record and details, wherein the details comprise attributes such as task id, task name, task execution state, task execution result, task scheduling target and the like. The redis cluster is used for hot data access efficiency which needs to be operated in the task scheduling process, and meanwhile, the access frequency of a database is reduced.
In particular implementations, the business orchestration information includes, but is not limited to, basic information of the application to be deployed, unit information, and SLA requirement information of the user. The application basic information comprises application names and region selection information, wherein the region selection information refers to a region range expected to be deployed by a user when the application is deployed. The element information includes user-selected computational power models, container information, network delays, and bandwidth sizes, and the task scheduling system supports application multi-element deployment capabilities, each element can individually select computational power models, as well as other user SLA requirements (e.g., network delays), and container operation-related startup parameters, commands, health checks, information, and the like. The container information comprises a container name and a container mirror image, and mirror image resources of the container need to be uploaded in advance at Paas centers of all regions. Ensuring that the name and version of the mirrored version are consistent. When a user performs unit information arrangement, the user needs to select architecture types and classifications required by application operation, and the proportions of host CPU, MEM and GPU resources corresponding to the capability models in each classification are different, so that a calculation model is obtained, and the calculation model is divided into three types: x86 computation, ARM computation, and heterogeneous computation.
When receiving the application deployment task, the service arrangement information of the application deployment task determines a corresponding cluster area, and performs cluster preselection in the cluster area to obtain a plurality of corresponding alternative computing force clusters.
Step S20: and carrying out loss calculation on the resource idle weight and the cluster balance degree of each candidate computing force cluster, and determining the idle balance loss of each candidate computing force cluster.
In order to obtain accurate resource idle weights and cluster balances of the candidate power clusters, the method further includes, before determining idle balance losses of the candidate power clusters, performing loss calculation on the resource idle weights and the cluster balances of the candidate power clusters: acquiring resource allocation information of a plurality of indexes in each alternative computing power cluster; parameter calculation is carried out according to the resource allocation information and the index number of each index in each alternative computing power cluster, and the resource idle weight of each alternative computing power cluster and the index fragment rate of each index in each alternative computing power cluster are determined; and calculating the balance degree according to the index fragment rate and the index number of each index in each candidate computing force cluster, and determining the cluster balance degree of each candidate computing force cluster.
It may be understood that the resource allocation information of the multiple indexes refers to schedulable resource information, allocated resource information and total resource information of each index in each candidate computing power cluster, and according to the resource allocation information of each index in each candidate computing power cluster and the number of indexes in each candidate computing power cluster, the resource idle weight of each candidate computing power cluster and the index fragment rate of each index can be determined, where a specific calculation formula of the resource idle weight is:wherein n is the number of indices in each alternative calculation power cluster, +.>For schedulable resources of index j in each alternative computing power cluster,the scheduled resources for index j in each candidate cluster are calculated. The specific calculation formula of the index fragmentation rate of index j is +.>Wherein->And calculating the total resource of the index j in each alternative force cluster.
In a specific implementation, after determining the index fragment rate and the index number of each index in each candidate computing force cluster, the cluster balance degree of each candidate computing force cluster can be determined, and a specific calculation formula is as follows:whereinFor the mean value of all index fragment rates in each alternative calculation force cluster,/for each alternative calculation force cluster>。
In a specific implementation, based on the resource idle weight and cluster balance degree of each candidate computing force cluster, idle balance loss of each candidate computing force cluster can be generated, and a specific calculation formula of the idle balance loss is as follows:whereinResource free weight for alternative computing power cluster i, +.>For the cluster balance of candidate force clusters i, m is the total number of candidate force clusters.
Step S30: and determining a target computing force cluster in the plurality of candidate computing force clusters according to the idle balance loss of each candidate computing force cluster.
It should be noted that, performing cluster selection by using the idle balance loss of each candidate computing force cluster, determining a target computing force cluster for task scheduling, and in order to accurately locate the target computing force cluster, further, determining the target computing force cluster from the plurality of candidate computing force clusters according to the idle balance loss of each candidate computing force cluster includes: generating a first matrix according to the original idle weight and the original balance degree of each alternative computing force cluster; generating a second matrix according to the original idle weight and idle balance loss of each candidate computing force cluster; and performing cluster screening according to the first matrix and the second matrix, and determining a target computing power cluster.
It can be understood that the original resource idle weight (i.e. the original idle weight) and the original balance degree of each alternative computing power cluster are obtained, the original resource idle weight and the original balance degree of each alternative computing power cluster are multiplied to generate a first matrix V1 of 1×m, and after receiving an application deployment task, that is, after receiving a user SLA requirement, the idle balance loss of each alternative computing power cluster is multiplied to the original resource idle weight to generate a second matrix V2 of 1×m. The original resource idle weight and the original balance degree are both known by the basic attributes of the alternative computing power cluster, wherein the basic attributes include, but are not limited to, the GPU, the CPU, the memory and the bandwidth of the alternative computing power cluster.
In a specific implementation, in order to accurately locate the target computing force cluster according to the first matrix and the second matrix, further, the cluster screening is performed according to the first matrix and the second matrix, and determining the target computing force cluster includes: constructing a first identity matrix and a second identity matrix according to a plurality of alternative calculation force clusters; performing matrix calculation on the first unit matrix, the second unit matrix, the first matrix and the second matrix to determine an integration matrix; and determining a target extremum in the integration matrix, and determining a target matrix according to a matrix queue of the target extremum.
It should be noted that, the first identity matrix D1 of m x m is constructed according to the total number m of candidate force clusters,and constructs a second identity matrix D2 with m x m diagonal of 0 and the remaining elements of 2,。
it can be understood that v1×d2+v2×d2 is calculated, so as to obtain an integration matrix, and a minimum value is determined in the integration matrix, where the minimum value in the integration matrix is a target extremum, and a cluster corresponding to a column where the target extremum is located is an optimal scheduling cluster for receiving the SLA requirement of the user, i.e. the optimal scheduling cluster is used as a target computing power cluster.
Step S40: and carrying out task scheduling on the application deployment task according to the target computing power cluster.
It should be noted that, in order to achieve efficient response of the application deployment task and save computing power resources, before the task scheduling of the application deployment task according to the target computing power cluster, the method further includes: computing the computing power resources according to the service arrangement information, and determining the pre-occupied computing power resources; performing resource preemption in the target computing power cluster according to the preemption computing power resource, and generating a task scheduling request; the task scheduling of the application deployment task according to the target computing power cluster comprises the following steps: task scheduling is carried out on the application deployment task according to the task scheduling request and the target computing power cluster; after the task scheduling is performed on the application deployment task according to the task scheduling request and the target computing power cluster, the method further comprises: monitoring a task deployment state of the application deployment task; and when the task deployment state is a preset state, clearing the pre-occupied computing force resource in the target computing force cluster.
It can be understood that after the target computing power cluster is determined and before the application deployment task is issued, an executor in the task scheduling system queries the computing power resource table in a redis locking mode to obtain details of the current full-network computing power resource, performs computing power resource calculation according to the service arrangement information, determines the optimal computing power resource required to be allocated to the application deployment task, namely the optimal computing power resource required to be allocated to the application deployment task, performs resource preemption of the target computing power cluster and reserves the computing power resource in the redis, and prevents the computing power preemption resource from being occupied by other applications when the computing power preemption resource can be reserved to the actual deployment of the application in the application deployment task.
In a specific implementation, a task scheduling request is generated while resource preemption is performed, a scheduling process of performing application deployment tasks according to the task scheduling request and a target computing power cluster is shown in fig. 3, a scheduling module of a task scheduling system is responsible for managing scheduling information, and sends the scheduling request according to scheduling configuration without bearing service codes. The dispatching system is decoupled from the task, so that the availability and stability of the system are improved, and meanwhile, the performance of the dispatching system is not limited by a task module; the visual, simple and dynamic management scheduling information is supported, including task creation, updating, deleting, GLUE development, task alarming and the like, all the operations can be effective in real time, monitoring scheduling results and execution logs are supported, and an executor Failover is supported; the executor is responsible for receiving the scheduling request and executing task logic, the task module is focused on operations such as task execution and the like, and development and maintenance are simpler and more efficient; and receiving an execution request, a termination request, a log request and the like of the scheduling center, and finally, performing the application deployment task on the target computing power cluster.
It should be noted that, in order to ensure that the pre-occupation force resources can be reasonably utilized, it is necessary to detect a task deployment state of an application deployment task on the target force cluster in real time, and when the task deployment state is a preset state, clear the pre-occupation force resources in the target force cluster. The preset states include a successful deployment state and a failed deployment state. When the task deployment state is the un-started state, the resource preemption is still performed.
It can be understood that, as shown in fig. 4, when an application deployment task is performed, an algorithm force analysis interface is called according to service arrangement information to perform algorithm force analysis, and after the analysis is completed, the pre-occupied algorithm force resource is pre-occupied, and the actual algorithm force resource and the pre-occupied algorithm force resource of the redis cache target algorithm force cluster are used, wherein the actual algorithm force resource is updated in real time by algorithm force perception, and the pre-occupied algorithm force resource ensures that the applied resource of the target algorithm force cluster can be reserved to the application deployment task when the application deployment is performed.
In the embodiment, when an application deployment task is received, cluster preselection is carried out according to service arrangement information of the application deployment task, and a plurality of alternative calculation force clusters are determined; performing loss calculation on the resource idle weight and the cluster balance degree of each alternative computing force cluster, and determining idle balance loss of each alternative computing force cluster; determining a target computing force cluster in a plurality of candidate computing force clusters according to the idle balance loss of each candidate computing force cluster; and carrying out task scheduling on the application deployment task according to the target computing power cluster. By adopting the method, cluster preselection is carried out based on service arrangement information of application deployment tasks, corresponding idle balance loss is determined according to the idle weight and the cluster balance degree of resources of each candidate computing power cluster, cluster optimization is carried out according to the idle balance loss of each candidate computing power cluster, a target computing power cluster is determined, resources can be used in a task mode among multiple centers and multiple clusters, the whole network resources are regarded as a unified whole, unified allocation and scheduling of the clusters of each center and the resources thereof are realized, practical service demands are focused when the application is deployed, the use rationality of the resources is improved, and the deployment cost is reduced.
Referring to fig. 5, fig. 5 is a flowchart of a task scheduling method according to a second embodiment of the present invention.
Based on the above first embodiment, the step S10 in the task scheduling method of this embodiment includes:
step S11: and when receiving an application deployment task, extracting information from service arrangement information of the application deployment task to acquire region selection information.
When receiving an application deployment task, the method obtains region selection information in the application deployment task. When the user performs application programming, a plurality of areas can be designated, and area selection information is generated according to the areas designated by the user.
Step S12: and determining a dispatching cluster area according to the area selection information.
It can be understood that the area for cluster pre-selection is determined according to the area selection information in the service arrangement information, and the area for cluster pre-selection is the scheduling cluster area, and one or more scheduling cluster areas may be provided.
Step S13: and acquiring the running state and occupied resource information of each computing power cluster in the dispatching cluster area.
It should be noted that, the running state may represent whether each computing power cluster can normally run, and the occupied resource information may represent occupied resources of each computing power cluster, where the occupied computing power resource is also used as the occupied computing power resource.
Step S14: and filtering each computing power cluster in the dispatching cluster area according to the running state and occupied resource information of each computing power cluster, and determining a plurality of alternative computing power clusters.
It should be noted that, in order to further filter each computing power cluster in the scheduling cluster area according to the running state and the occupied resource information of each computing power cluster, determining a plurality of candidate computing power clusters includes: when the running state of each computing power cluster is a normal running state, acquiring the cluster total resources of each computing power cluster; performing cluster filtering according to the cluster total resources and occupied resource information of each computing power cluster to determine a plurality of primary screening clusters; and determining a plurality of alternative calculation force clusters in the plurality of primary screening clusters according to the area weight of the dispatching cluster area and the preset screening proportion.
It can be understood that the total resources of the clusters refer to total resources of all the computing power clusters, primary screening is performed on computing power according to the running state of each computing power cluster, the faulty computing power clusters are filtered, namely, when the running state of each computing power cluster is the normal running state, the total resources of all the computing power clusters in the normal running state are obtained, secondary screening is performed according to the total resources of all the computing power clusters and occupied resource information, computing power clusters with total resources equal to occupied resources are filtered, and the remaining computing power clusters after secondary screening are primary screening clusters.
In a specific implementation, different dispatching cluster areas have corresponding area weights, the area weights are calculated by an computing power cloud platform in a task dispatching system and areas where computing power centers of all areas are located, the closer the area is to a user, the higher the weight is, and a preset screening proportion is a set proportion for screening alternative computing power clusters. And sequencing the area weight of the dispatching cluster area where each primary screening cluster is positioned from large to small, determining the primary screening clusters of the preset screening proportion arranged in front, and taking the primary screening clusters of the preset screening proportion arranged in front as alternative calculation force clusters. For example, if the preset screening ratio is 10%, the first 10% of the primary screening clusters are used as candidate calculation clusters.
If the user selects a specific cluster when performing application arrangement, the cluster designated by the user is directly used as the target computing power cluster, and the processes of cluster preselection and cluster optimization are not performed. If the user does not specify an application deployment area, the user may select any of three area scheduling policies: intent region scheduling: preferentially assigning to the area intended by the user; close to data scheduling: the user may designate a priority dispatch to an area near the data; application affinity scheduling: the user may designate a priority dispatch to the area with the affinity application. And determining a corresponding scheduling area according to the area scheduling strategy selected by the user, filtering each cluster in the scheduling area according to the running state and occupied resource information of each cluster in the scheduling area, and determining a plurality of alternative computing power clusters.
It may be appreciated that, as shown in fig. 6, the computing power clusters are preselected based on the service arrangement information, the preselection includes a forced preselection strategy and a self-selection preselection strategy, a plurality of candidate computing power clusters are obtained based on the preselection result, a target computing power cluster is determined based on the balance degree and the resource idle weight of the plurality of candidate computing power clusters, and the application deployment task is completed according to the target computing power cluster.
In the embodiment, the region selection information is obtained by extracting the information of the service arrangement information of the application deployment task; determining a dispatching cluster area according to the area selection information; acquiring the running state and occupied resource information of each computing power cluster in the dispatching cluster area; and filtering each computing power cluster in the dispatching cluster area according to the running state and occupied resource information of each computing power cluster, and determining a plurality of alternative computing power clusters. By the method, the scheduling cluster area is determined based on the service arrangement information, cluster filtering is performed based on the running state and occupied resource information of each computing power cluster, and a plurality of alternative computing power clusters are determined, so that the rationality of the computing power resource preselection process is ensured.
In addition, referring to fig. 7, an embodiment of the present invention further provides a task scheduling device, where the task scheduling device includes:
and the determining module 10 is used for carrying out cluster preselection according to the service arrangement information of the application deployment task when receiving the application deployment task, and determining a plurality of alternative calculation force clusters.
And the calculation module 20 is configured to perform loss calculation on the resource idle weight and the cluster balance degree of each candidate computing power cluster, and determine idle balance loss of each candidate computing power cluster.
The determining module 10 is further configured to determine a target computing force cluster from the plurality of candidate computing force clusters according to the idle balance loss of each candidate computing force cluster.
And the scheduling module 30 is used for performing task scheduling on the application deployment task according to the target computing power cluster.
In the embodiment, when an application deployment task is received, cluster preselection is carried out according to service arrangement information of the application deployment task, and a plurality of alternative calculation force clusters are determined; performing loss calculation on the resource idle weight and the cluster balance degree of each alternative computing force cluster, and determining idle balance loss of each alternative computing force cluster; determining a target computing force cluster in a plurality of candidate computing force clusters according to the idle balance loss of each candidate computing force cluster; and carrying out task scheduling on the application deployment task according to the target computing power cluster. By adopting the method, cluster preselection is carried out based on service arrangement information of application deployment tasks, corresponding idle balance loss is determined according to the idle weight and the cluster balance degree of resources of each candidate computing power cluster, cluster optimization is carried out according to the idle balance loss of each candidate computing power cluster, a target computing power cluster is determined, resources can be used in a task mode among multiple centers and multiple clusters, the whole network resources are regarded as a unified whole, unified allocation and scheduling of the clusters of each center and the resources thereof are realized, practical service demands are focused when the application is deployed, the use rationality of the resources is improved, and the deployment cost is reduced.
In an embodiment, the determining module 10 is further configured to extract information from service orchestration information of the application deployment task, and obtain region selection information;
determining a dispatching cluster area according to the area selection information;
acquiring the running state and occupied resource information of each computing power cluster in the dispatching cluster area;
and filtering each computing power cluster in the dispatching cluster area according to the running state and occupied resource information of each computing power cluster, and determining a plurality of alternative computing power clusters.
In an embodiment, the determining module 10 is further configured to obtain a total cluster resource of each computing power cluster when the running state of each computing power cluster is a normal running state;
performing cluster filtering according to the cluster total resources and occupied resource information of each computing power cluster to determine a plurality of primary screening clusters;
and determining a plurality of alternative calculation force clusters in the plurality of primary screening clusters according to the area weight of the dispatching cluster area and the preset screening proportion.
In an embodiment, the computing module 20 is further configured to obtain resource allocation information of a plurality of indexes in each candidate computing power cluster;
parameter calculation is carried out according to the resource allocation information and the index number of each index in each alternative computing power cluster, and the resource idle weight of each alternative computing power cluster and the index fragment rate of each index in each alternative computing power cluster are determined;
and calculating the balance degree according to the index fragment rate and the index number of each index in each candidate computing force cluster, and determining the cluster balance degree of each candidate computing force cluster.
In an embodiment, the determining module 10 is further configured to generate a first matrix according to the original idle weights and the original balances of the candidate force clusters;
generating a second matrix according to the original idle weight and idle balance loss of each candidate computing force cluster;
and performing cluster screening according to the first matrix and the second matrix, and determining a target computing power cluster.
In an embodiment, the determining module 10 is further configured to construct a first identity matrix and a second identity matrix according to the plurality of candidate force clusters;
performing matrix calculation on the first unit matrix, the second unit matrix, the first matrix and the second matrix to determine an integration matrix;
and determining a target extremum in the integration matrix, and determining a target matrix according to a matrix queue of the target extremum.
In an embodiment, the scheduling module 30 is further configured to perform computing power resource calculation according to the service arrangement information, and determine to camp on computing power resources;
performing resource preemption in the target computing power cluster according to the preemption computing power resource, and generating a task scheduling request;
the task scheduling of the application deployment task according to the target computing power cluster comprises the following steps:
and carrying out task scheduling on the application deployment task according to the task scheduling request and the target computing power cluster.
In an embodiment, the scheduling module 30 is further configured to monitor a task deployment status of the application deployment task;
and when the task deployment state is a preset state, clearing the pre-occupied computing force resource in the target computing force cluster.
Because the device adopts all the technical schemes of all the embodiments, the device at least has all the beneficial effects brought by the technical schemes of the embodiments, and the description is omitted here.
In addition, the embodiment of the invention also provides a storage medium, wherein a task scheduling program is stored on the storage medium, and the task scheduling program realizes the steps of the task scheduling method when being executed by a processor.
Because the storage medium adopts all the technical schemes of all the embodiments, the storage medium has at least all the beneficial effects brought by the technical schemes of the embodiments, and the description is omitted here.
It should be noted that the above-described working procedure is merely illustrative, and does not limit the scope of the present invention, and in practical application, a person skilled in the art may select part or all of them according to actual needs to achieve the purpose of the embodiment, which is not limited herein.
In addition, technical details that are not described in detail in this embodiment may refer to the task scheduling method provided in any embodiment of the present invention, and are not described herein again.
Furthermore, it should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
From the above description of embodiments, it will be clear to a person skilled in the art that the above embodiment method may be implemented by means of software plus a necessary general hardware platform, but may of course also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. Read Only Memory (ROM)/RAM, magnetic disk, optical disk) and comprising several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method according to the embodiments of the present invention.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.
Claims (10)
1. A task scheduling method, characterized in that the task scheduling method comprises:
when an application deployment task is received, cluster preselection is carried out according to service arrangement information of the application deployment task, and a plurality of alternative computing force clusters are determined;
performing loss calculation on the resource idle weight and the cluster balance degree of each alternative computing force cluster, and determining idle balance loss of each alternative computing force cluster;
determining a target computing force cluster in a plurality of candidate computing force clusters according to the idle balance loss of each candidate computing force cluster;
and carrying out task scheduling on the application deployment task according to the target computing power cluster.
2. The task scheduling method of claim 1, wherein the performing cluster pre-selection according to the service arrangement information of the application deployment task, determining a plurality of candidate computing force clusters, includes:
extracting information from the service arrangement information of the application deployment task to obtain region selection information;
determining a dispatching cluster area according to the area selection information;
acquiring the running state and occupied resource information of each computing power cluster in the dispatching cluster area;
and filtering each computing power cluster in the dispatching cluster area according to the running state and occupied resource information of each computing power cluster, and determining a plurality of alternative computing power clusters.
3. The task scheduling method of claim 2, wherein the filtering each computing force cluster in the scheduling cluster area according to the running state and the occupied resource information of each computing force cluster to determine a plurality of candidate computing force clusters includes:
when the running state of each computing power cluster is a normal running state, acquiring the cluster total resources of each computing power cluster;
performing cluster filtering according to the cluster total resources and occupied resource information of each computing power cluster to determine a plurality of primary screening clusters;
and determining a plurality of alternative calculation force clusters in the plurality of primary screening clusters according to the area weight of the dispatching cluster area and the preset screening proportion.
4. The task scheduling method according to claim 1, wherein the step of performing a loss calculation on the resource idle weight and the cluster balance of each candidate computing power cluster, before determining the idle balance loss of each candidate computing power cluster, includes:
acquiring resource allocation information of a plurality of indexes in each alternative computing power cluster;
parameter calculation is carried out according to the resource allocation information and the index number of each index in each alternative computing power cluster, and the resource idle weight of each alternative computing power cluster and the index fragment rate of each index in each alternative computing power cluster are determined;
and calculating the balance degree according to the index fragment rate and the index number of each index in each candidate computing force cluster, and determining the cluster balance degree of each candidate computing force cluster.
5. The task scheduling method of claim 1, wherein the determining a target computing force cluster from among the plurality of candidate computing force clusters according to the idle balance loss of each candidate computing force cluster comprises:
generating a first matrix according to the original idle weight and the original balance degree of each alternative computing force cluster;
generating a second matrix according to the original idle weight and idle balance loss of each candidate computing force cluster;
and performing cluster screening according to the first matrix and the second matrix, and determining a target computing power cluster.
6. The task scheduling method according to claim 5, wherein the performing cluster screening according to the first matrix and the second matrix to determine a target computing power cluster includes:
constructing a first identity matrix and a second identity matrix according to a plurality of alternative calculation force clusters;
performing matrix calculation on the first unit matrix, the second unit matrix, the first matrix and the second matrix to determine an integration matrix;
and determining a target extremum in the integration matrix, and determining a target matrix according to a matrix queue of the target extremum.
7. The task scheduling method according to any one of claims 1 to 6, further comprising, before the task scheduling of the application deployment task according to the target computing force cluster:
computing the computing power resources according to the service arrangement information, and determining the pre-occupied computing power resources;
performing resource preemption in the target computing power cluster according to the preemption computing power resource, and generating a task scheduling request;
the task scheduling of the application deployment task according to the target computing power cluster comprises the following steps:
task scheduling is carried out on the application deployment task according to the task scheduling request and the target computing power cluster;
after the task scheduling is performed on the application deployment task according to the task scheduling request and the target computing power cluster, the method further comprises:
monitoring a task deployment state of the application deployment task;
and when the task deployment state is a preset state, clearing the pre-occupied computing force resource in the target computing force cluster.
8. A task scheduling device, characterized in that the task scheduling device comprises:
the determining module is used for carrying out cluster preselection according to service arrangement information of the application deployment task when receiving the application deployment task, and determining a plurality of alternative calculation force clusters;
the computing module is used for carrying out loss computation on the resource idle weight and the cluster balance degree of each alternative computing force cluster and determining idle balance loss of each alternative computing force cluster;
the determining module is further used for determining a target computing force cluster from the plurality of candidate computing force clusters according to the idle balance loss of each candidate computing force cluster;
and the scheduling module is used for performing task scheduling on the application deployment task according to the target computing power cluster.
9. A task scheduling device, the device comprising: a memory, a processor, and a task scheduler stored on the memory and executable on the processor, the task scheduler configured to implement the task scheduling method of any one of claims 1 to 7.
10. A storage medium having stored thereon a task scheduler, which when executed by a processor, implements the task scheduling method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310878705.2A CN117149372A (en) | 2023-07-17 | 2023-07-17 | Task scheduling method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310878705.2A CN117149372A (en) | 2023-07-17 | 2023-07-17 | Task scheduling method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117149372A true CN117149372A (en) | 2023-12-01 |
Family
ID=88906958
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310878705.2A Pending CN117149372A (en) | 2023-07-17 | 2023-07-17 | Task scheduling method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117149372A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118069371A (en) * | 2024-04-12 | 2024-05-24 | 深圳市捷易科技有限公司 | Cluster computing power scheduling method, device, equipment and storage medium |
-
2023
- 2023-07-17 CN CN202310878705.2A patent/CN117149372A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118069371A (en) * | 2024-04-12 | 2024-05-24 | 深圳市捷易科技有限公司 | Cluster computing power scheduling method, device, equipment and storage medium |
| CN118069371B (en) * | 2024-04-12 | 2024-06-25 | 深圳市捷易科技有限公司 | Cluster computing power scheduling method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11847103B2 (en) | Data migration using customizable database consolidation rules | |
| US20200104377A1 (en) | Rules Based Scheduling and Migration of Databases Using Complexity and Weight | |
| US11188392B2 (en) | Scheduling system for computational work on heterogeneous hardware | |
| Jyoti et al. | Dynamic provisioning of resources based on load balancing and service broker policy in cloud computing | |
| US11842214B2 (en) | Full-dimensional scheduling and scaling for microservice applications | |
| US11146497B2 (en) | Resource prediction for cloud computing | |
| JP2020507135A (en) | Exclusive agent pool distribution method, electronic device, and computer-readable storage medium | |
| US20140189703A1 (en) | System and method for distributed computing using automated provisoning of heterogeneous computing resources | |
| US20050132379A1 (en) | Method, system and software for allocating information handling system resources in response to high availability cluster fail-over events | |
| EP3191948A1 (en) | Computing instance launch time | |
| CN116541134B (en) | Method and device for deploying containers in multi-architecture cluster | |
| CN109614227A (en) | Task resource concocting method, device, electronic equipment and computer-readable medium | |
| US20200279173A1 (en) | Data pipeline prioritization and prediction | |
| Ding et al. | Kubernetes-oriented microservice placement with dynamic resource allocation | |
| CN117149372A (en) | Task scheduling method, device, equipment and storage medium | |
| CN114610497A (en) | Container scheduling method, cluster system, device, electronic equipment and storage medium | |
| Geetha et al. | RETRACTED ARTICLE: An advanced artificial intelligence technique for resource allocation by investigating and scheduling parallel-distributed request/response handling | |
| CN117435306A (en) | Cluster container expansion and contraction method, device, equipment and storage medium | |
| CN111709723A (en) | RPA business process intelligent processing method, device, computer equipment and storage medium | |
| CN1783121A (en) | Method and system for executing design automation | |
| Emara et al. | Genetic-Based Multi-objective Task Scheduling Algorithm in Cloud Computing Environment. | |
| Ramezani et al. | Task scheduling in cloud environments: A survey of population‐based evolutionary algorithms | |
| CN116450290A (en) | Computer resource management method and device, cloud server and storage medium | |
| CN115904708A (en) | AI platform dynamic weighting scheduling method, device and storage medium | |
| US11449777B1 (en) | Serverless inference execution across a heterogeneous fleet of devices |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |