CN113127178A - Resource preemption method and device, computer readable storage medium and electronic equipment - Google Patents

Resource preemption method and device, computer readable storage medium and electronic equipment Download PDF

Info

Publication number
CN113127178A
CN113127178A CN201911394676.2A CN201911394676A CN113127178A CN 113127178 A CN113127178 A CN 113127178A CN 201911394676 A CN201911394676 A CN 201911394676A CN 113127178 A CN113127178 A CN 113127178A
Authority
CN
China
Prior art keywords
resource
queue
preemption
allocated
allocable
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201911394676.2A
Other languages
Chinese (zh)
Other versions
CN113127178B (en
Inventor
朱久龙
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Yidu Cloud Beijing Technology Co Ltd
Original Assignee
Yidu Cloud Beijing Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Yidu Cloud Beijing Technology Co Ltd filed Critical Yidu Cloud Beijing Technology Co Ltd
Priority to CN201911394676.2A priority Critical patent/CN113127178B/en
Publication of CN113127178A publication Critical patent/CN113127178A/en
Application granted granted Critical
Publication of CN113127178B publication Critical patent/CN113127178B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/50Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G06F9/5038Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/50Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5083Techniques for rebalancing the load in a distributed system

Abstract

The disclosure belongs to the technical field of computers, and relates to a resource preemption method and device, a computer readable storage medium and an electronic device. The method comprises the following steps: acquiring queue parameters of a resource queue to be allocated and an allocable resource queue, and determining that the resource queue to be allocated performs resource preemption on the allocable resource queue according to the queue parameters; if the resource preemption is invalid preemption, resetting the queue parameters of the resource queue to be allocated so as to perform resource preemption again; and if the resource preemption is effective preemption, determining a container set to be preempted according to the queue parameters of the allocable resource queue so as to allocate the resources in the container set to be preempted to the resource queue to be allocated. On one hand, the method and the device avoid the problem of idle resource waste caused by invalid preemption, provide a better resource preemption mode, and improve the resource preemption efficiency; on the other hand, the problem of cyclic preemption is solved, the accuracy of determining that the large container preempts the resources is improved, and the effect of improving the resource preemption stability is achieved.

Description

Resource preemption method and device, computer readable storage medium and electronic equipment
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a resource preemption method and a resource preemption device, a computer-readable storage medium, and an electronic device.
Background
A Resource coordinator (Yarn) is a Resource management system provided by a distributed computing infrastructure, and can allocate and schedule resources from a distributed storage system at a lower layer to run an application program at an upper layer. In a Resource Manager (Resource Manager) of Yarn, queues are created, and each queue can set the lowest quota and the highest quota of its own parent queue, so as to improve the Resource utilization rate of the whole cluster. When no task runs in the queue, resources can be lent to other queues with higher load through the scheduler; when the queue of the lent resource receives the newly submitted application, the lent resource can be returned to meet the resource requirement of the queue through a preemption system.
However, various drawbacks may appear during resource preemption. For example, since queues are equal, it may happen that resources of the queue where the core service is located are preempted to lower the service level agreement; for another example, if the resource preempted by the queue is not enough to meet the minimum resource requirement, the preempted resource is invalid to preempt; for another example, when resources on multiple nodes need to be preempted, in the process of not preempting resources on other nodes, a part of resources may be repeatedly scheduled, and the production cannot meet the service requirement.
In view of this, there is a need in the art to develop a new resource preemption method and apparatus.
It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present disclosure, and thus may include information that does not constitute prior art known to those of ordinary skill in the art.
Disclosure of Invention
An object of the present disclosure is to provide a resource preemption method, a resource preemption device, a computer-readable storage medium, and an electronic device, thereby overcoming, at least to some extent, the problems of invalid preemption and cyclic preemption due to the limitations of the related art.
Additional features and advantages of the disclosure will be set forth in the detailed description which follows, or in part will be obvious from the description, or may be learned by practice of the disclosure.
According to an aspect of the present disclosure, there is provided a resource preemption method, the method comprising: acquiring queue parameters of a resource queue to be allocated and an allocable resource queue, and determining that the resource queue to be allocated performs resource preemption on the allocable resource queue according to the queue parameters; if the resource preemption is invalid preemption, resetting the queue parameters of the resource queue to be allocated so as to perform resource preemption again; and if the resource preemption is effective preemption, determining a container set to be preempted according to the queue parameters of the allocable resource queue so as to allocate the resources in the container set to be preempted to the resource queue to be allocated.
In an exemplary embodiment of the present disclosure, the queue parameters include a minimum quota of resources, a current resource usage amount, a historical demand amount of the queue, an ideal capacity of the queue, and a minimum demand amount of the queue; the acquiring queue parameters of a resource queue to be allocated and an allocable resource queue, and determining that the resource queue to be allocated performs resource preemption on the allocable resource queue according to the queue parameters, includes: determining a resource preemption queue, and dividing the resource preemption queue into the allocable resource queue and the resource queue to be allocated according to the current resource usage amount, the minimum resource quota, the historical queue demand amount and the ideal queue capacity; determining that the resource queue to be allocated performs resource preemption on the allocable resource queue, and determining the resource allocation amount allocated to the resource queue to be allocated by the allocable resource queue according to a resource allocation rule; and distributing the resource distribution amount to the resource queue to be distributed, and determining whether the resource is preempted effectively or not according to the resource distribution amount and the minimum requirement amount of the queue.
In an exemplary embodiment of the present disclosure, the dividing the resource preemption queue into the allocable resource queue and the to-be-allocated resource queue according to the current resource usage amount, the minimum resource quota, the historical queue demand amount, and the ideal queue capacity includes: determining an allocable resource queue from the preemption resource queue according to the current resource usage and the minimum resource quota; and determining a resource queue to be allocated from the resource queue to be preempted according to the current resource usage, the historical queue demand and the ideal queue capacity.
In an exemplary embodiment of the present disclosure, the queue parameters include a queue priority, a queue current demand, and a resource maximum quota; the determining the resource allocation amount allocated to the resource queue to be allocated by the allocable resource queue according to the resource allocation rule includes: determining the amount of allocable resources in the allocable resource queue, and determining the amount of resource allocation according to the amount of allocable resources, the current demand of the queue, the maximum quota of resources and the ideal capacity of the queue; and sequencing the resource queues to be distributed according to the queue priority so as to distribute the resource distribution amount according to a sequencing result.
In an exemplary embodiment of the present disclosure, the determining, according to a queue parameter of the allocable resource queue, a set of containers to be preempted to allocate resources in the set of containers to be preempted to the resource queue to be allocated includes: dividing the resource demand of the resource queue to be distributed into a large-load container and a small-load container according to the minimum resource quota; counting container information of the large-load container, and calculating preemption duration corresponding to the preemptible container in the allocable resource queue according to the container information; and determining the preemptible container as a container set to be preempted based on the preemption duration so as to allocate the resources in the container set to be preempted to the resource queue to be allocated.
In an exemplary embodiment of the present disclosure, after the allocating the resources in the set of containers to be preempted to the queue of resources to be allocated, the method further includes: determining an allocable resource queue where a container in the container set to be preempted is located, and determining the resource residual quantity in the allocable resource queue according to the resource; and if the resource surplus meets the resource demand corresponding to the small-load container, distributing the resource surplus to the small-load container.
In an exemplary embodiment of the present disclosure, the container information includes a current time and a container start time; the calculating a preemption duration corresponding to the preemptible container according to the container information includes: determining a plurality of preemptible containers in the allocable resource queue, and acquiring a plurality of container starting times of the plurality of preemptible containers; and obtaining a plurality of container starting durations according to the current time and the calculation results of the plurality of container starting times, and calculating the plurality of container starting durations to obtain the preemption durations.
According to an aspect of the present disclosure, there is provided a resource preemption device, the device comprising: the resource preemption module is configured to acquire a resource queue to be allocated and queue parameters of the allocable resource queue, and determine that the resource queue to be allocated performs resource preemption on the allocable resource queue according to the queue parameters; the invalid preemption module is configured to reset the queue parameters of the resource queue to be allocated to perform resource preemption again if the resource preemption is invalid preemption; and the effective preemption module is configured to determine a container set to be preempted according to the queue parameters of the allocable resource queue if the resource preemption is effective preemption, so as to allocate the resources in the container set to be preempted to the resource queue to be allocated.
According to an aspect of the present disclosure, there is provided an electronic device including: a processor and a memory; wherein the memory has stored thereon computer readable instructions which, when executed by the processor, implement the resource preemption method of any of the exemplary embodiments described above.
According to an aspect of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the resource preemption method in any of the exemplary embodiments described above.
As can be seen from the foregoing technical solutions, the resource preemption method, the resource preemption device, the computer storage medium, and the electronic device in the exemplary embodiments of the present disclosure have at least the following advantages and positive effects:
in the method and apparatus provided in the exemplary embodiment of the present disclosure, it is determined whether resource preemption is valid through a queue parameter, and further processing is performed on valid preemption and invalid preemption, so as to implement a function of allocating resources to a resource queue to be allocated. On one hand, under the condition of invalid preemption, resource preemption of the next round is carried out by resetting queue parameters, so that the problem of idle resource waste caused by invalid preemption is avoided, a better resource preemption mode is provided, and the resource preemption efficiency is improved; on the other hand, the set of the containers to be preempted is determined to realize resource allocation, so that the problem of cyclic preemption caused by the fact that the large containers cannot be started is solved, the accuracy of determining the large containers to preempt the resources is improved, and the effect of improving the resource preemption stability is achieved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure. It is to be understood that the drawings in the following description are merely exemplary of the disclosure, and that other drawings may be derived from those drawings by one of ordinary skill in the art without the exercise of inventive faculty.
Fig. 1 schematically illustrates a flow chart of a resource preemption method in an exemplary embodiment of the present disclosure;
fig. 2 schematically illustrates a flow chart of a method of resource preemption in an exemplary embodiment of the present disclosure;
fig. 3 is a flowchart schematically illustrating a method for partitioning a resource preemption queue in an exemplary embodiment of the present disclosure;
FIG. 4 is a flow chart diagram schematically illustrating a method of determining an allocation amount of resources in an exemplary embodiment of the present disclosure;
fig. 5 schematically illustrates a flowchart of a method for allocating resources to a queue of resources to be allocated in an exemplary embodiment of the present disclosure;
fig. 6 schematically illustrates a flow chart of a method of determining preemption durations in an exemplary embodiment of the present disclosure;
FIG. 7 schematically illustrates a flow diagram of a method of allocating resources to a small capacity container in an exemplary embodiment of the disclosure;
fig. 8 schematically illustrates a flowchart of a resource preemption method in an application scenario in an exemplary embodiment of the present disclosure;
fig. 9 is a schematic diagram illustrating a structure of a resource preemption device in an exemplary embodiment of the present disclosure;
fig. 10 schematically illustrates an electronic device for implementing a resource preemption method in an exemplary embodiment of the present disclosure;
fig. 11 schematically illustrates a computer-readable storage medium for implementing a resource preemption method in exemplary embodiments of the present disclosure.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the subject matter of the present disclosure can be practiced without one or more of the specific details, or with other methods, components, devices, steps, and the like. In other instances, well-known technical solutions have not been shown or described in detail to avoid obscuring aspects of the present disclosure.
The terms "a," "an," "the," and "said" are used in this specification to denote the presence of one or more elements/components/parts/etc.; the terms "comprising" and "having" are intended to be inclusive and mean that there may be additional elements/components/etc. other than the listed elements/components/etc.; the terms "first" and "second", etc. are used merely as labels, and are not limiting on the number of their objects.
Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repetitive description will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities.
In order to solve the problems in the related art, the present disclosure provides a resource preemption method. Fig. 1 shows a flow chart of a resource preemption method, which, as shown in fig. 1, comprises at least the following steps:
step S110, queue parameters of the resource queue to be distributed and the distributable resource queue are obtained, and resource preemption is performed on the distributable resource queue by the resource queue to be distributed according to the queue parameters.
And S120, if the resource preemption is invalid preemption, resetting the queue parameters of the resource queue to be allocated so as to perform resource preemption again.
And S130, if the resource preemption is effective preemption, determining a container set to be preempted according to the queue parameters of the allocable resource queue so as to allocate the resources in the container set to be preempted to the resource queue to be allocated.
In the exemplary embodiment of the present disclosure, whether resource preemption is effective is determined through a queue parameter, and effective preemption and ineffective preemption are further processed to realize a function of allocating resources to a resource queue to be allocated. On one hand, under the condition of invalid preemption, resource preemption of the next round is carried out by resetting queue parameters, so that the problem of idle resource waste caused by invalid preemption is avoided, a better resource preemption mode is provided, and the resource preemption efficiency is improved; on the other hand, the set of the containers to be preempted is determined to realize resource allocation, so that the problem of cyclic preemption caused by the fact that the large containers cannot be started is solved, the accuracy of determining the large containers to preempt the resources is improved, and the effect of improving the resource preemption stability is achieved.
The following describes each step of the resource preemption method in detail.
In step S110, the resource queue to be allocated and the queue parameter of the allocable resource queue are obtained, and the resource queue to be allocated is determined to perform resource preemption on the allocable resource queue according to the queue parameter.
In an exemplary embodiment of the present disclosure, the queue parameters include a minimum quota of resources, a current resource usage, a historical demand for the queue, an ideal capacity for the queue, and a minimum demand for the queue. In an alternative embodiment, fig. 2 is a flowchart illustrating a method for resource preemption of an allocable resource queue by a resource queue to be allocated, where as shown in fig. 2, the method at least includes the following steps:
in step S210, a resource preemption queue is determined, and the resource preemption queue is divided into an allocable resource queue and a resource queue to be allocated according to the current resource usage amount, the minimum resource quota, the historical queue demand amount, and the ideal queue capacity. The resource preemption queue may be a queue determined by the Yarn scheduler for allocating the resource task according to a preset rule. Yarn can allocate resources from the distributed storage system to run application programs, and the running application programs are different according to different used computing frameworks. For example, a computing framework using Yarn for resource allocation and scheduling may be MapReduce, Apark, Hive, streaming, and so on; the Distributed storage System may be, for example, a Hadoop Distributed File System (HDFS for short), or other systems, and this is not particularly limited in this exemplary embodiment.
The Yarn includes a resource Manager, a Node Manager, and an Application Master. The distributed storage system may include a plurality of physical nodes, and the node manager is a process running on a physical node, and is configured to manage and control resource usage, running tasks, and the like on the physical node; the resource Manager is used for scheduling and allocating resources of the entire system, and may generally include a Scheduler (Scheduler) and an Application Manager (Application Manager), where the Scheduler is used for allocating resources of running applications according to a specific rule, and the Application Manager is used for managing and controlling each Application running in the system.
In an alternative embodiment, fig. 3 is a flowchart illustrating a method for partitioning a resource preemption queue, where as shown in fig. 3, the method at least includes the following steps: in step S310, an allocable resource queue is determined from the resource preemption queue according to the current resource usage and the minimum resource quota. For example, a resource preemption queue whose current resource usage is less than or equal to the lowest resource quota may be determined as an idle queue, and a resource preemption queue whose current resource usage is greater than the lowest resource quota may be determined as an over-allocated queue. Because there is an increment between the current resource usage amount and the minimum resource quota in both the idle queue and the over-allocated queue, and this increment can be used as the amount of resources provided by the allocable resource queue to other queues, the idle queue and the over-allocated queue can be determined as the allocable resource queue. It should be noted that the increment in the idle queue is obtained by subtracting the current resource usage from the minimum resource quota, and the increment in the over-allocated queue may be obtained by subtracting the minimum resource quota from the current resource usage.
In step S320, a resource queue to be allocated is determined from the resource preemption queue according to the current resource usage amount, the historical queue demand amount, and the ideal queue capacity. For example, the specific determination method may be to add the current resource usage and the historical queue demand and compare the result with the ideal queue capacity. And when the comparison result is that the operation result is larger than the ideal capacity of the queue, determining the corresponding resource preemption queue as the resource queue to be allocated.
In the exemplary embodiment, the allocable resource queue and the resource queue to be allocated in the resource preemption queue are determined through the queue parameters, the determination mode is simple, and a preemption basis is provided for resource preemption behaviors.
In step S220, it is determined that the resource queue to be allocated performs resource preemption on the allocable resource queue, so as to determine the resource allocation amount allocated by the allocable resource queue to the resource queue to be allocated according to the resource allocation rule. The resource preemption mainly refers to a technology for ensuring that a resource queue to be allocated can access a network by releasing resources occupied by an allocable resource queue when the network is congested.
In an alternative embodiment, the queue parameters include a queue priority, a current demand of the queue, and a maximum quota of the resource, and fig. 4 is a flowchart illustrating a method for determining an allocation amount of the resource, where as shown in fig. 4, the method at least includes the following steps: in step S410, the allocable resource amount in the allocable resource queue is determined, and the resource allocation amount is determined according to the allocable resource amount, the current demand of the queue, the maximum resource quota, and the ideal capacity of the queue. When the allocable resource queue is an idle queue, the allocable resource amount can be obtained by subtracting the current resource usage amount from the minimum resource quota; when the allocable resource queue is a super-allocation queue, the amount of the allocable resource may be obtained by subtracting the minimum quota of the resource from the current resource usage amount. After determining the amount of allocable resources, the maximum quota of resources may be subtracted from the ideal capacity of the queue, and the result of the calculation may be used as one of the parameters for determining the amount of allocable resources. Further, the allocable resource amount, the current demand amount of the queue and the calculation result are compared, and the minimum value in the comparison result is used as the resource allocation amount.
In step S420, the resource queues to be allocated are sorted according to the queue priority, so as to allocate the resource allocation amount according to the sorting result. In order to avoid the problem that the low-priority queue occupies the resources of the high-priority queue and reduces the Service-Level Agreement (SLA), the queue priority can be preset to sequence the resource queues to be allocated. For example, the queue priority may be allocated with 15 levels, the corresponding effective value may be 1 to 14, and the smaller the value, the higher the identified queue priority. The sequencing result may be that the resource queue to be allocated with the high priority is arranged to the resource queue to be allocated with the low priority, or may be that the resource queue to be allocated with the low priority is arranged to the resource queue to be allocated with the high priority, which is not particularly limited in this exemplary embodiment.
In this exemplary embodiment, the determined resource allocation amount may be allocated according to the sorting result by the queue priority, so as to ensure that the high-priority queue to be allocated is preempted preferentially, and meet the requirement of core service priority implementation.
In step S230, the resource allocation amount is allocated to the resource queue to be allocated, and the resource preemption is determined to be an effective preemption or an invalid preemption according to the resource allocation amount and the minimum requirement amount of the queue. After determining the resource allocation amount, the resource allocation amount may be allocated to the resource queue to be allocated according to a resource preemption behavior initiated by the resource queue to be allocated. However, the resource preemption behavior corresponding to the resource allocation amount may be effective preemption or may be ineffective preemption, and further determination may be made. For example, when the resource allocation amount is less than the minimum requirement amount of the queue of the resource queue to be allocated, it indicates that the resource allocation amount is not enough to start the container corresponding to the resource queue to be allocated, and therefore the resource queue to be allocated may be removed from all the resource queues to be allocated. In contrast, it may be determined that the resource allocation amount is sufficient to satisfy the container corresponding to the resource queue to be allocated, that is, the corresponding resource is preempted as effective preemption.
In the exemplary embodiment, by allocating the resource allocation amount of the allocable resource queue to the resource queue to be allocated, it can be determined whether the resource preemption corresponding to the resource queue to be allocated is effective, the division basis is clear, the determination standard is simple, a better resource preemption mode can be provided, and the application experience of the user is further improved.
In step S120, if the resource preemption is invalid preemption, the queue parameter of the resource queue to be allocated is reset to perform resource preemption again.
In an exemplary embodiment of the present disclosure, when it is determined that resource preemption is ineffective preemption, a queue parameter in a queue of resources to be allocated may be reset. For example, the queue history demand is reset to zero, and the preempted resource allocation is added to the increment of the idle queue or the increment of the over-allocated queue. And, since the current resource usage plus the queue history demand equals to the ideal capacity of the queue, after the queue history demand is reset to zero, it can be determined that the queue to be allocated is no longer a queue to be allocated. Furthermore, the next resource queue to be allocated can be determined according to the queue priority, and resource preemption is performed again.
In step S130, if the resource preemption is valid preemption, determining a set of containers to be preempted according to the queue parameters of the allocable resource queue, so as to allocate the resources in the set of containers to be preempted to the resource queue to be allocated.
In an exemplary embodiment of the present disclosure, when it is determined that resource preemption is invalid preemption, a set of containers to be preempted may be further determined, and resources in the set of containers to be preempted may be allocated. In an alternative embodiment, fig. 5 shows a flowchart of a method for allocating resources to a queue of resources to be allocated, and as shown in fig. 5, the method at least includes the following steps: in step S510, the resource demand of the resource queue to be allocated is divided into a large load container and a small load container according to the minimum resource quota. For example, the predetermined resource requirement may be 25 gigabytes (GB or G). When the minimum resource quota of the resource queue to be allocated is less than or equal to 25G, determining that the resource queue to be allocated is a small-load container; and when the minimum resource quota of the resource queue to be allocated is greater than 25G. The queue of resources to be allocated may be determined to be a large capacity container.
In step S520, container information of the large-load container is counted, and a preemption duration corresponding to the preemptible container in the allocable resource queue is calculated according to the container information. The container information may include a container identifier, a unique identifier of the node manager, a current resource availability, and the like, which is not particularly limited in this exemplary embodiment.
In an alternative embodiment, the container information includes a current time and a container start time, and fig. 6 is a flowchart illustrating a method for determining preemption duration, which, as shown in fig. 6, includes at least the following steps: in step S610, a plurality of preemptible containers are determined in the allocable resource queue, and a plurality of container start times of the plurality of preemptible containers are acquired. For example, for a resource demand of a resource queue to be allocated, seven preemptible containers may be correspondingly determined in the allocable resource queue, and therefore, the container start times of the corresponding seven preemptible containers are obtained to further determine the preemption durations for preempting the preemptible containers. It should be noted that the seven determined preemptible containers may belong to multiple allocable resource queues, and therefore, the resource amount of the to-be-allocated resource queue preempting from each allocable resource queue may not exceed the to-be-preempted amount of the corresponding allocable resource queue.
In step S620, a plurality of container start durations are obtained according to the current time and the calculation result of the plurality of container start times, and the plurality of container start durations are calculated to obtain the preemption duration. For example, after seven container start times are determined, the container start duration of each preemptible container may be obtained by subtracting the container start time from the current time counted. Further, the starting durations of the seven containers are summed, and the calculation result can be determined as the preemption durations corresponding to the seven preemptible containers.
In the exemplary embodiment, the preemption durations corresponding to each resource preemption scheme can be calculated through the container starting time corresponding to a plurality of preemptible containers, the set of containers to be preempted can be determined more accurately, the resource requirement of the resources to be allocated is further met under the condition of the minimum preemption cost, and the allocation mode is more accurate and scientific.
In step S530, based on the preemption duration, the preemptible container is determined as a set of containers to be preempted, so as to allocate resources in the set of containers to be preempted to the queue of resources to be allocated. The preemptible containers with different numbers or different containers can be preempted according to the resource demand of the same resource queue to be allocated, and the preemption duration of various feasible schemes can be calculated. And comparing the determined preemption durations, and determining the corresponding preemptible container with the least preemption duration as a container set to be preempted. And further, releasing the resources in the container set to be preempted to allocate to the resource queue to be allocated.
In the exemplary embodiment, the resource allocation of the resource queue to be allocated can be completed under the condition of minimum preemption cost, the service requirement can be quickly realized, the service of the user terminal cannot be interrupted, and the satisfaction degree of the user is improved.
After the resource allocation of the large-load container is met, the resource amount of the related allocable resource queue can be synchronized continuously, so that the resource preemption requirement of the small-load container is further met. In an alternative embodiment, fig. 7 shows a flow diagram of a method of allocating resources to a small capacity container, as shown in fig. 7, the method comprising at least the steps of: in step S710, a queue where a container in the container set is to be preempted is determined, and the remaining amount of resources in the queue is determined according to the resources. After determining the allocable resource queue where the container in the set of containers to be preempted is located, determining the resource remaining amount of the allocable resource queue according to the resources in the allocated resource queue to be allocated. Specifically, the amount of allocable resources in the allocable resource queue minus the amount of allocated resources may be the corresponding remaining amount of resources.
In step S720, if the remaining amount of resources satisfies the resource demand corresponding to the small load container, the remaining amount of resources is allocated to the small load container. After determining the remaining amount of resources of the allocable resource queue, the remaining amount of resources may be compared with a preset resource demand, for example 25G. When the resource residual quantity is less than 25G, which indicates that the resource residual quantity is not enough to meet the requirement of the large-load container, only the resource demand quantity corresponding to the small-load container can be met, and therefore the resource residual quantity is released to be distributed to the corresponding small-load container. Specifically, the remaining amount of the resource released by the last started container in the allocable resource queue may be determined.
In the exemplary embodiment, the resource requirement of the small-load container can be met through the remaining amount of the resources in the distributable resource queue, the resource requirement of the large-load container can be met, the resource requirement of the small-load container can also be met, a plurality of tasks can run by resources, and service configuration is more flexible.
The following describes the resource preemption method in the embodiment of the present disclosure in detail with reference to an application scenario.
Fig. 8 shows a flowchart of a resource preemption method in an application scenario, and as shown in fig. 8, in step S801, queue parameters representing usage states of a resource queue to be allocated and an allocable resource queue are obtained, where the queue parameters may be a minimum resource quota, a current resource usage amount, a historical queue demand amount, an ideal queue capacity, a minimum queue demand amount, a queue priority, a current queue demand amount, and a maximum resource quota, respectively.
In step S802, the queue ideal capacity of the allocable resource queue and the resource queue to be allocated is calculated according to the queue priority, and it is determined according to the queue parameter that the resource queue to be allocated performs resource preemption on the allocable resource queue.
In step S803, it is determined whether resource preemption initiated by the to-be-allocated resource queue to the allocable resource queue is effective preemption or ineffective preemption.
In step S804, if the resource preemption is invalid, the queue history demand is reset to zero, and the next resource queue to be allocated is determined according to the queue priority, so as to further calculate whether the resource allocation amount can satisfy the next resource queue to be allocated.
In step S805, if the resource preemption is an effective preemption, dividing the resource demand of the resource queue to be allocated into a large load container and a small load container according to the minimum resource quota.
In step S806, performing cyclic preemption on the large-load container, where the resource preemption amount each time is the historical queue demand amount until no node manager can satisfy the demand amount or the resource allocation amount in the allocable resource queue satisfies the small-load container.
In step S807, container information of the large-load container is counted.
In step S808, resource preemption is attempted at each node manager, and preemption durations for a plurality of preemptible containers are calculated.
In step S809, the preemptible container with the minimum preemption duration is determined as the set of containers to be preempted, and the resources in the set of containers to be preempted are released.
In step S810, the node manager is deleted in the node manager, and the resource allocation amounts in the related allocable resource queues are synchronized.
In step S811, the resource remaining amount in the allocable resource queue of the allocated resource is allocated to the small load container, specifically, the resource of the newly started container is released to meet the resource requirement of the small load container.
In the exemplary embodiment of the present disclosure, whether resource preemption is effective is determined through a queue parameter, and effective preemption and ineffective preemption are further processed to realize a function of allocating resources to a resource queue to be allocated. On one hand, under the condition of invalid preemption, resource preemption of the next round is carried out by resetting queue parameters, so that the problem of idle resource waste caused by invalid preemption is avoided, a better resource preemption mode is provided, and the resource preemption efficiency is improved; on the other hand, the set of the containers to be preempted is determined to realize resource allocation, so that the problem of cyclic preemption caused by the fact that the large containers cannot be started is solved, the accuracy of determining the large containers to preempt the resources is improved, and the effect of improving the resource preemption stability is achieved.
Further, in an exemplary embodiment of the present disclosure, a resource preemption device is also provided. Fig. 9 shows a schematic diagram of the structure of the resource preemption device, and as shown in fig. 9, the resource preemption device 900 may include: a resource preemption module 910, an invalid preemption module 920, and an valid preemption module 930. Wherein:
a resource preemption module 910, configured to obtain a resource queue to be allocated and queue parameters of the allocable resource queue, and determine, according to the queue parameters, that the resource queue to be allocated performs resource preemption on the allocable resource queue; an invalid preemption module 920, configured to reset the queue parameters of the resource queue to be allocated to perform resource preemption again if the resource preemption is invalid preemption; the effective preemption module 930 is configured to determine, if the resource preemption is effective preemption, a container set to be preempted according to the queue parameter of the allocable resource queue, so as to allocate the resources in the container set to be preempted to the resource queue to be allocated.
The details of the resource preemption device have already been described in detail in the corresponding resource preemption method, and therefore are not described herein again.
It should be noted that although several modules or units of the resource preemption device 900 are mentioned in the above detailed description, such division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.
In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.
An electronic device 1000 according to such an embodiment of the invention is described below with reference to fig. 10. The electronic device 1000 shown in fig. 10 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 10, the electronic device 1000 is embodied in the form of a general purpose computing device. The components of the electronic device 1000 may include, but are not limited to: the at least one processing unit 1010, the at least one memory unit 1020, a bus 1030 connecting different system components (including the memory unit 1020 and the processing unit 1010), and a display unit 1040.
Wherein the storage unit stores program code that is executable by the processing unit 1010 to cause the processing unit 1010 to perform steps according to various exemplary embodiments of the present invention as described in the "exemplary methods" section above in this specification.
The memory unit 1020 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM)1021 and/or a cache memory unit 1022, and may further include a read-only memory unit (ROM) 1023.
Storage unit 1020 may also include a program/utility 1024 having a set (at least one) of program modules 1025, such program modules 1025 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
Bus 1030 may be any one or more of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, and a local bus using any of a variety of bus architectures.
The electronic device 1000 may also communicate with one or more external devices 1200 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 1000, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 1000 to communicate with one or more other computing devices. Such communication may occur through input/output (I/O) interfaces 1050. Also, the electronic device 1000 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the internet) via the network adapter 1060. As shown, the network adapter 1040 communicates with other modules of the electronic device 1000 via the bus 1030. It should be appreciated that although not shown, other hardware and/or software modules may be used in conjunction with the electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.
In an exemplary embodiment of the present disclosure, there is also provided a computer-readable storage medium having stored thereon a program product capable of implementing the above-described method of the present specification. In some possible embodiments, aspects of the invention may also be implemented in the form of a program product comprising program code means for causing a terminal device to carry out the steps according to various exemplary embodiments of the invention described in the above-mentioned "exemplary methods" section of the present description, when said program product is run on the terminal device.
Referring to fig. 11, a program product 1100 for implementing the above method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited in this regard and, in the present document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
A computer readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

Claims (10)

1. A method of resource preemption, the method comprising:
acquiring queue parameters of a resource queue to be allocated and an allocable resource queue, and determining that the resource queue to be allocated performs resource preemption on the allocable resource queue according to the queue parameters;
if the resource preemption is invalid preemption, resetting the queue parameters of the resource queue to be allocated so as to perform resource preemption again;
and if the resource preemption is effective preemption, determining a container set to be preempted according to the queue parameters of the allocable resource queue so as to allocate the resources in the container set to be preempted to the resource queue to be allocated.
2. The method according to claim 1, wherein the queue parameters include a minimum quota of resources, a current resource usage amount, a historical demand amount of the queue, an ideal capacity of the queue, and a minimum demand amount of the queue;
the acquiring queue parameters of a resource queue to be allocated and an allocable resource queue, and determining that the resource queue to be allocated performs resource preemption on the allocable resource queue according to the queue parameters, includes:
determining a resource preemption queue, and dividing the resource preemption queue into the allocable resource queue and the resource queue to be allocated according to the current resource usage amount, the minimum resource quota, the historical queue demand amount and the ideal queue capacity;
determining that the resource queue to be allocated performs resource preemption on the allocable resource queue, and determining the resource allocation amount allocated to the resource queue to be allocated by the allocable resource queue according to a resource allocation rule;
and distributing the resource distribution amount to the resource queue to be distributed, and determining whether the resource is preempted effectively or not according to the resource distribution amount and the minimum requirement amount of the queue.
3. The method according to claim 2, wherein the dividing the resource preemption queue into the allocable resource queue and the to-be-allocated resource queue according to the current resource usage amount, the minimum resource quota, the historical queue demand amount, and the ideal queue capacity includes:
determining an allocable resource queue from the preemption resource queue according to the current resource usage and the minimum resource quota;
and determining a resource queue to be allocated from the resource queue to be preempted according to the current resource usage, the historical queue demand and the ideal queue capacity.
4. The method according to claim 3, wherein the queue parameters include queue priority, queue current demand, and resource maximum quota;
the determining the resource allocation amount allocated to the resource queue to be allocated by the allocable resource queue according to the resource allocation rule includes:
determining the amount of allocable resources in the allocable resource queue, and determining the amount of resource allocation according to the amount of allocable resources, the current demand of the queue, the maximum quota of resources and the ideal capacity of the queue;
and sequencing the resource queues to be distributed according to the queue priority so as to distribute the resource distribution amount according to a sequencing result.
5. The method according to claim 1, wherein said determining a set of containers to be preempted according to the queue parameters of the allocable resource queue to allocate the resources in the set of containers to be preempted to the resource queue to be allocated includes:
dividing the resource demand of the resource queue to be distributed into a large-load container and a small-load container according to the minimum resource quota;
counting container information of the large-load container, and calculating preemption duration corresponding to the preemptible container in the allocable resource queue according to the container information;
and determining the preemptible container as a container set to be preempted based on the preemption duration so as to allocate the resources in the container set to be preempted to the resource queue to be allocated.
6. The method of resource preemption of claim 5, wherein after said allocating resources in the set of containers to preempt to the queue of resources to allocate, the method further comprises:
determining an allocable resource queue where a container in the container set to be preempted is located, and determining the resource residual quantity in the allocable resource queue according to the resource;
and if the resource surplus meets the resource demand corresponding to the small-load container, distributing the resource surplus to the small-load container.
7. The method according to claim 5, wherein the container information includes a current time and a container start time;
the calculating a preemption duration corresponding to the preemptible container according to the container information includes:
determining a plurality of preemptible containers in the allocable resource queue, and acquiring a plurality of container starting times of the plurality of preemptible containers;
and obtaining a plurality of container starting durations according to the current time and the calculation results of the plurality of container starting times, and calculating the plurality of container starting durations to obtain the preemption durations.
8. A resource preemption device, comprising:
the resource preemption module is configured to acquire a resource queue to be allocated and queue parameters of the allocable resource queue, and determine that the resource queue to be allocated performs resource preemption on the allocable resource queue according to the queue parameters;
the invalid preemption module is configured to reset the queue parameters of the resource queue to be allocated to perform resource preemption again if the resource preemption is invalid preemption;
and the effective preemption module is configured to determine a container set to be preempted according to the queue parameters of the allocable resource queue if the resource preemption is effective preemption, so as to allocate the resources in the container set to be preempted to the resource queue to be allocated.
9. A computer-readable storage medium, on which a computer program is stored, which, when executed by a transmitter, implements the resource preemption method of any of claims 1-7.
10. An electronic device, comprising:
a transmitter;
a memory for storing executable instructions of the transmitter;
wherein the transmitter is configured to perform the resource preemption method of any of claims 1-7 via execution of the executable instructions.
CN201911394676.2A 2019-12-30 2019-12-30 Resource preemption method and device, computer readable storage medium and electronic equipment Active CN113127178B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911394676.2A CN113127178B (en) 2019-12-30 2019-12-30 Resource preemption method and device, computer readable storage medium and electronic equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911394676.2A CN113127178B (en) 2019-12-30 2019-12-30 Resource preemption method and device, computer readable storage medium and electronic equipment

Publications (2)

Publication Number Publication Date
CN113127178A true CN113127178A (en) 2021-07-16
CN113127178B CN113127178B (en) 2024-03-29

Family

ID=76767805

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911394676.2A Active CN113127178B (en) 2019-12-30 2019-12-30 Resource preemption method and device, computer readable storage medium and electronic equipment

Country Status (1)

Country Link
CN (1) CN113127178B (en)

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7137119B1 (en) * 2000-05-02 2006-11-14 Microsoft Corporation Resource manager architecture with resource allocation utilizing priority-based preemption
US7703102B1 (en) * 1999-08-23 2010-04-20 Oracle America, Inc. Approach for allocating resources to an apparatus based on preemptable resource requirements
CN103476074A (en) * 2013-08-30 2013-12-25 京信通信***(中国)有限公司 Method and equipment for resource preemption
US20160147566A1 (en) * 2014-11-21 2016-05-26 International Business Machines Corporation Cross-Platform Scheduling with Long-Term Fairness and Platform-Specific Optimization
CN107491351A (en) * 2016-06-13 2017-12-19 阿里巴巴集团控股有限公司 A kind of resource allocation methods based on priority, device and equipment
CN108769254A (en) * 2018-06-25 2018-11-06 星环信息科技(上海)有限公司 Resource-sharing application method, system and equipment based on preemption scheduling
CN109213594A (en) * 2017-07-06 2019-01-15 阿里巴巴集团控股有限公司 Method, apparatus, equipment and the computer storage medium that resource is seized
CN109582436A (en) * 2018-10-15 2019-04-05 上海交通大学 Fine granularity preemptive type resource scheduling system and method based on container cluster platform
CN109814985A (en) * 2017-11-20 2019-05-28 杭州华为数字技术有限公司 A kind of method for scheduling task and scheduler calculate equipment, system
CN110515704A (en) * 2019-08-30 2019-11-29 广东浪潮大数据研究有限公司 Resource regulating method and device based on Kubernetes system

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7703102B1 (en) * 1999-08-23 2010-04-20 Oracle America, Inc. Approach for allocating resources to an apparatus based on preemptable resource requirements
US7137119B1 (en) * 2000-05-02 2006-11-14 Microsoft Corporation Resource manager architecture with resource allocation utilizing priority-based preemption
CN103476074A (en) * 2013-08-30 2013-12-25 京信通信***(中国)有限公司 Method and equipment for resource preemption
US20160147566A1 (en) * 2014-11-21 2016-05-26 International Business Machines Corporation Cross-Platform Scheduling with Long-Term Fairness and Platform-Specific Optimization
CN107491351A (en) * 2016-06-13 2017-12-19 阿里巴巴集团控股有限公司 A kind of resource allocation methods based on priority, device and equipment
CN109213594A (en) * 2017-07-06 2019-01-15 阿里巴巴集团控股有限公司 Method, apparatus, equipment and the computer storage medium that resource is seized
CN109814985A (en) * 2017-11-20 2019-05-28 杭州华为数字技术有限公司 A kind of method for scheduling task and scheduler calculate equipment, system
CN108769254A (en) * 2018-06-25 2018-11-06 星环信息科技(上海)有限公司 Resource-sharing application method, system and equipment based on preemption scheduling
CN109582436A (en) * 2018-10-15 2019-04-05 上海交通大学 Fine granularity preemptive type resource scheduling system and method based on container cluster platform
CN110515704A (en) * 2019-08-30 2019-11-29 广东浪潮大数据研究有限公司 Resource regulating method and device based on Kubernetes system

Also Published As

Publication number Publication date
CN113127178B (en) 2024-03-29

Similar Documents

Publication Publication Date Title
US10003500B2 (en) Systems and methods for resource sharing between two resource allocation systems
CN109471727B (en) Task processing method, device and system
CN107291545B (en) Task scheduling method and device for multiple users in computing cluster
CN110858161B (en) Resource allocation method, device, system, equipment and medium
RU2481618C2 (en) Hierarchical infrastructure of resources backup planning
US10162684B2 (en) CPU resource management in computer cluster
US8209695B1 (en) Reserving resources in a resource-on-demand system for user desktop utility demand
CN112486648A (en) Task scheduling method, device, system, electronic equipment and storage medium
US10102042B2 (en) Prioritizing and distributing workloads between storage resource classes
US11507419B2 (en) Method,electronic device and computer program product for scheduling computer resources in a task processing environment
CN111258745B (en) Task processing method and device
US9923785B1 (en) Resource scaling in computing infrastructure
CN109992418B (en) SLA-aware resource priority scheduling method and system for multi-tenant big data platform
US11010195B2 (en) K-tier architecture scheduling
WO2017010922A1 (en) Allocation of cloud computing resources
CN110166507B (en) Multi-resource scheduling method and device
CN105022668B (en) Job scheduling method and system
CN111338785A (en) Resource scheduling method and device, electronic equipment and storage medium
CN112783659A (en) Resource allocation method and device, computer equipment and storage medium
CN107665143B (en) Resource management method, device and system
CN112749002A (en) Method and device for dynamically managing cluster resources
CN114968565A (en) Resource management method, device, electronic equipment, storage medium and server
US10956228B2 (en) Task management using a virtual node
CN109343958B (en) Computing resource allocation method and device, electronic equipment and storage medium
CN113127178B (en) Resource preemption method and device, computer readable storage medium and electronic equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant