WO2024051236A1 - 资源调度方法及其相关设备 - Google Patents

资源调度方法及其相关设备 Download PDF

Info

Publication number
WO2024051236A1
WO2024051236A1 PCT/CN2023/099201 CN2023099201W WO2024051236A1 WO 2024051236 A1 WO2024051236 A1 WO 2024051236A1 CN 2023099201 W CN2023099201 W CN 2023099201W WO 2024051236 A1 WO2024051236 A1 WO 2024051236A1
Authority
WO
WIPO (PCT)
Prior art keywords
resource
microservice
node
resources
deployed
Prior art date
Application number
PCT/CN2023/099201
Other languages
English (en)
French (fr)
Inventor
厉文超
吕卫华
李成进
Original Assignee
华为云计算技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为云计算技术有限公司 filed Critical 华为云计算技术有限公司
Publication of WO2024051236A1 publication Critical patent/WO2024051236A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
    • G06F9/5022—Mechanisms to release resources
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G06F2009/4557—Distribution of virtual machine instances; Migration and load balancing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00—Indexing scheme relating to G06F9/00
    • G06F2209/50—Indexing scheme relating to G06F9/50
    • G06F2209/5021—Priority

Definitions

  • the present application relates to the field of communication technology, and in particular, to a resource scheduling method and related equipment.
  • Container as a lightweight virtualization technology, has developed rapidly in recent years.
  • Container technology creates independent operating environments for different applications, realizes resource isolation, configuration and security guarantees, and can meet the resource requirements of applications on demand and ensure the isolation and availability of applications.
  • cluster a computing device cluster
  • k8s kubernetes
  • a container collection pod
  • a pod consists of a group of containers working on the same node.
  • K8s mainly realizes automatic expansion and contraction through pod horizontal automatic scaling (Horizontal Pod Autoscaler, HPA) and pod vertical automatic scaling (Vertical Pod Autoscaler, VPA).
  • HPA Horizontal Pod Autoscaler
  • VPA Very Pod Autoscaler
  • the first aspect of the embodiment of the present application discloses a resource scheduling method, which includes: obtaining the resource information of each node in the cluster, and the resource information of the container in each node, each node is equipped with a container engine; based on each The resource information of each node and the resource information of the container in each node are used to construct a resource portrait of the microservices deployed on each node.
  • the microservices deployed on each node run based on one or more containers; if it is determined that in the cluster
  • the first microservice deployed in requires resource adjustment.
  • Based on the resource portrait of the first microservice the resource information of the first node where the first microservice is deployed, and the resource information of the container in the first node, the first microservice is generated.
  • Associated resource adjustment instructions based on the container engine and resource adjustment instructions, adjust the resources of the pod associated with the first microservice.
  • a resource portrait of the microservice deployed on each node is constructed, and resource adjustment is performed based on the resource portrait of the microservice, the resource information of the node, and the resource information of the container. It can fully utilize the resources of the node, make the resource adjustment of microservices more targeted, and improve the comprehensive resource utilization of the node and/or cluster. Resource adjustment is based on the container engine without affecting the existing scheduling capabilities of the cluster. Compared with resource adjustment through HPA/VPA, the lag is smaller. It can realize resource adjustment at the second level, enhance the ability of microservices deployed in the cluster to cope with sudden traffic, improve the stability of the cluster and network throughput, and improve the microservices. When adjusting resources, you can adjust resources for all pods associated with the microservice, or you can adjust resources to one or several pods in the microservice. The resource adjustment granularity is smaller and more accurate.
  • building a resource portrait of the microservice deployed on each node includes: based on the resource information of the container in each node , obtain the resource information of the microservices deployed on each node; based on the resource information of each node and the resource information of the microservices deployed on each node, construct a resource portrait of the microservices deployed on each node, resources The portrait includes the resource occupation dimension and the time dimension.
  • a resource portrait of the microservices deployed on each node is constructed.
  • the time dimension is combined with resource occupation.
  • Dimensions represent the characteristics of resources occupied by microservices at different times.
  • the resource scheduling method further includes: if the difference between the resources occupied by the first microservice and the resource upper limit of the first microservice is less than the first preset threshold, determining that the first microservice needs to perform resource adjustment; or If the difference between the resources occupied by the first microservice and the resource upper limit of the first microservice is greater than the second preset threshold, it is determined that the first microservice needs to perform resource adjustment.
  • microservices with insufficient resources or microservices with excess resources can be determined as microservices that require resource adjustment. For example, if the difference between the resources occupied by the microservice and the resource upper limit of the microservice is less than the first preset The threshold indicates that the resources of the microservice are tight. The microservice is expanded by triggering to meet the resource requirements required for the operation of the microservice. If the difference between the resources occupied by the microservice and the resource upper limit of the microservice is greater than the second preset The threshold indicates that when a microservice has excess resources, the microservice is triggered to scale down so that the excess resources can be allocated to other microservices with insufficient resources, which can improve the resource utilization of the node.
  • the resources occupied by the first microservice include one or more of processor resources, memory resources, disk resources, and network bandwidth resources. It is determined that the first microservice needs to perform resource adjustment, including: if the first microservice If the difference between any one of the processor resources, memory resources, disk resources, and network bandwidth resources occupied by the microservice and the corresponding resource upper limit is less than the corresponding first preset threshold, it is determined that the first microservice requires resource adjustment; Or if the difference between any one of the processor resources, memory resources, disk resources and network bandwidth resources occupied by the first microservice and the corresponding resource upper limit is greater than the corresponding second preset threshold, it is determined that the first microservice needs Make resource adjustments.
  • the existing processor resources, memory resources, disk resources and Microservices with insufficient network bandwidth resources, or microservices with excess processor resources, memory resources, disk resources, and network bandwidth resources are determined to be microservices that require resource adjustment to meet the requirements.
  • determining that the first microservice requires resource adjustment includes: if any of the processor resources, memory resources, disk resources, and network bandwidth resources occupied by the first microservice matches the corresponding resource upper limit The difference is less than the corresponding first preset threshold, it is determined that the first microservice needs to be expanded and adjusted; or if any one of the processor resources, memory resources, disk resources and network bandwidth resources occupied by the first microservice is different from the corresponding If the difference between the resource upper limits is greater than the corresponding second preset threshold, it is determined that the first microservice needs to be scaled down.
  • the difference between any one of the processor resources, memory resources, disk resources, and network bandwidth resources occupied by the microservice and the corresponding resource upper limit is less than the corresponding first preset threshold, it indicates that the microservice is A certain resource is tight, and it is determined that the resource of the microservice needs to be expanded to meet the resource requirements required for the operation of the microservice. If the microservice occupies any one of the processor resources, memory resources, disk resources, and network bandwidth resources The difference from the corresponding resource upper limit is greater than the corresponding second preset threshold, indicating that a certain resource of the microservice is in excess, and it is determined that the resource of the microservice needs to be reduced so that the excess resources can be allocated to other resources with insufficient resources. Microservices can improve the resource utilization of nodes.
  • generating a resource adjustment instruction associated with the first microservice based on the resource portrait of the first microservice, resource information of the node where the first microservice is deployed, and resource information of the container in the node includes: based on the first microservice The resource portrait of a microservice predicts the number of resources required by the first microservice; based on the predicted number of resources, the resource information of the node where the first microservice is deployed, and the resource information of the container in the node, a resource profile associated with the first microservice is generated. Resource adjustment instructions.
  • the number of resources required by the microservice is predicted through the resource portrait of the microservice, and then resources are allocated to the microservice based on the predicted number of resources, the resource information of the node where the microservice is deployed, and the resource information of the container in the node, so that The resource adjustment of microservices is more accurate, avoiding excessive or insufficient resource allocation and improving node resource utilization.
  • generating resource adjustment instructions associated with the first microservice includes: generating resource adjustment instructions for all pods associated with the first microservice; or generating resource adjustment instructions for a specified pod associated with the first microservice.
  • the resource adjustment instruction associated with the first microservice when generating the resource adjustment instruction associated with the first microservice, you can choose to adjust the resources for all pods associated with the microservice, or you can precisely adjust the resources for one or several pods in the microservice. Resource adjustment, the adjusted pod can make the microservice have enough resources to cope with the current increase in business volume/traffic.
  • adjusting the resources of the pod associated with the first microservice based on the container engine and the resource adjustment instruction includes: determining the pod associated with the resource adjustment instruction, and calling the resource adjustment application program interface through the container engine to adjust the resource. Instruct the associated pod to adjust resources.
  • the resource adjustment instruction is to adjust the resources of all pods associated with the microservice, or to adjust the resources of one or more of the microservices.
  • Resource adjustment is performed on several pods, and the resource adjustment application interface is called by the container engine to adjust resources for the specified pod.
  • the hysteresis of resource adjustment is small and second-level resource adjustment can be achieved, which enhances the ability of microservices deployed in the cluster to cope with sudden traffic. .
  • the microservices deployed on each node are configured with priorities
  • the resource scheduling method also includes: if the remaining resources of the first node cannot meet the resource adjustment requirements of the first microservice, release the resources deployed on the first node. Part of the resources occupied by the second microservice, the priority of the second microservice is lower than the priority of the first microservice.
  • each microservice can be configured with different priorities in a preset manner. If the node resources are tight and cannot meet the resource adjustment requirements of the microservices deployed on the node, the microservices deployed on the node can be released with a lower priority than the microservice. resources of a certain microservice, so that the node has enough free resources to allocate to the microservice that needs resource adjustment.
  • releasing part of the resources occupied by the second microservice deployed on the first node includes: releasing part of the resources occupied by the second microservice according to the resource portrait of the second microservice.
  • the microservices deployed on each node are configured with priorities
  • the resource scheduling method also includes: if the remaining resources of the first node cannot meet the resource adjustment requirements of the first microservice, the microservices deployed on the first node are The second microservice is scheduled to other nodes in the cluster, and the priority of the second microservice is lower than the priority of the first microservice.
  • each microservice can be configured with different priorities in a preset manner.
  • node resources are tight and cannot meet the resource adjustment needs of the microservices deployed on the node, if resources cannot be freed from low-priority microservices, Or instead of adopting the strategy of allocating resources from low-priority microservices, you can also migrate a microservice deployed on the node with a lower priority than the microservice to other nodes so that the node has enough idle resources. Assigned to microservices that require resource adjustment.
  • scheduling the second microservice deployed on the first node to other nodes in the cluster includes: based on the resource portrait of the microservice deployed on the first node, selecting the second microservice deployed on the first node to be scheduled to other nodes in the cluster. The second microservice.
  • the second aspect of the embodiment of the present application discloses a resource scheduling method, which includes: obtaining the resource information of each node in the cluster, and the resource information of the container in each node, and each node is installed There is a container engine; in response to the startup instruction of the first microservice deployed in the cluster, a first pod is created for the first microservice, and the first pod is set with a first resource upper limit; based on the first deployment of the first microservice The resource information of the node and the resource information of the container in the first node are used to generate a resource adjustment instruction associated with the first pod; based on the container engine and the resource adjustment instruction, the resource occupancy upper limit of the first pod is set to the first resource upper limit value. Adjusted to the second resource upper limit value, the second resource upper limit value is greater than the first resource upper limit value.
  • the upper limit of resource occupancy allocated by the system for the microservice is increased to achieve rapid Start microservices, shorten the startup time of microservices, and adjust resources based on the container engine.
  • the lag is smaller than resource adjustment through HPA/VPA, and second-level resource adjustment can be achieved. Quickly respond to resource requirements required for microservice startup.
  • the resource scheduling method further includes: if it is determined that the first microservice is successfully started, adjusting the resource occupation upper limit of the first pod from the second resource upper limit value to the first resource upper limit value.
  • the resources previously allocated to the microservice in order to shorten the startup time of the microservice will be recycled to avoid excessive microservice resources and improve the resource utilization of the node.
  • embodiments of the present application provide a resource scheduling device, including: a first acquisition module, configured to acquire resource information of each node in the cluster and resource information of containers in each node, where each node Container engine is installed; the building module is used to build a resource portrait of the microservices deployed on each node based on the resource information of each node and the resource information of the container in each node, in which the resource portrait deployed on each node
  • the microservice runs based on one or more containers;
  • the first generation module is used to deploy the first microservice based on the resource portrait of the first microservice and the first step of deploying the first microservice when resource adjustment is required.
  • the resource information of the node and the resource information of the container in the first node generate a resource adjustment instruction associated with the first microservice; the first adjustment module is used to adjust the pod associated with the first microservice based on the container engine and the resource adjustment instruction.
  • a resource portrait of the microservice deployed on each node is constructed, and resource adjustment is performed based on the resource portrait of the microservice, the resource information of the node, and the resource information of the container. It can fully utilize the resources of the node, make the resource adjustment of microservices more targeted, and improve the comprehensive resource utilization of the node and/or cluster. Resource adjustment is based on the container engine without affecting the existing scheduling capabilities of the cluster. Compared with resource adjustment through HPA/VPA, the lag is smaller. It can realize resource adjustment at the second level, enhance the ability of microservices deployed in the cluster to cope with sudden traffic, improve the stability of the cluster and network throughput, and improve the microservices. When adjusting resources, you can adjust resources for all pods associated with the microservice, or you can adjust resources to one or several pods in the microservice. The resource adjustment granularity is smaller and more accurate.
  • embodiments of the present application provide a resource scheduling device, including: a second acquisition module for acquiring resource information of each node in the cluster and resource information of containers in each node.
  • a container engine is installed; a module is created to create a first pod for the first microservice in response to a startup instruction of the first microservice deployed in the cluster, and the first pod is set with a first resource upper limit.
  • the second generation module is used to generate resource adjustment instructions associated with the first pod based on the resource information of the first node where the first microservice is deployed and the resource information of the container in the first node; the second adjustment module is used Based on the container engine and the resource adjustment command, the resource usage upper limit of the first pod is adjusted from the first resource upper limit value to the second resource upper limit value, and the second resource upper limit value is greater than the first resource upper limit value.
  • the upper limit of resource occupancy allocated by the system for the microservice is increased to achieve rapid Start microservices, shorten the startup time of microservices, and adjust resources based on the container engine.
  • the lag is smaller than resource adjustment through HPA/VPA, and second-level resource adjustment can be achieved. Quickly respond to resource requirements required for microservice startup.
  • embodiments of the present application provide a computer-readable storage medium, including computer program instructions.
  • the computer program instructions When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the resources described in the first or second aspect. Scheduling method.
  • embodiments of the present application provide a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device , so that the computing device cluster executes the resource scheduling method as described in the first aspect or the second aspect.
  • embodiments of the present application provide a computer program product, which when the computer program product is run by a computing device cluster, causes the computing device cluster to execute the resource scheduling method described in the first or second aspect.
  • An eighth aspect provides a device that has the function of implementing the computing device cluster behavior in the method provided in the first aspect.
  • Functions can be implemented by hardware, or by hardware executing corresponding software.
  • Hardware or software includes one or more modules corresponding to the above functions.
  • the computer-readable storage medium described in the fifth aspect, the computing device cluster described in the sixth aspect, the computer program product described in the seventh aspect, and the device described in the eighth aspect provided above can be combined with the above-mentioned third aspect.
  • the methods of the first aspect and/or the second aspect correspond to each other. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be described again here.
  • Figure 1 is a schematic architectural diagram of a cloud computing system provided by an embodiment of the present application.
  • Figure 2 is a schematic structural diagram of a computing device provided by an embodiment of the present application.
  • Figure 3 is a schematic structural diagram of a computing device cluster provided by an embodiment of the present application.
  • Figure 4 is a schematic architectural diagram of a resource scheduling system provided by an embodiment of the present application.
  • Figure 5 is a schematic diagram of the interaction flow of the resource scheduling system provided by an embodiment of the present application for resource scheduling in a microservice traffic change scenario;
  • Figure 6 is a schematic diagram of the interactive process of resource scheduling in a microservice startup scenario by the resource scheduling system provided by an embodiment of the present application;
  • Figure 7 is a schematic diagram of resource changes in the startup phase of the microservice provided by an embodiment of the present application.
  • Figure 8 is a schematic flowchart of a resource scheduling method provided by an embodiment of the present application.
  • Figure 9 is a schematic flowchart of a resource scheduling method provided by another embodiment of the present application.
  • Figure 10 is a schematic diagram of the functional modules of the first resource scheduling device provided by an embodiment of the present application.
  • Figure 11 is a schematic diagram of the functional modules of the second resource scheduling device provided by another embodiment of the present application.
  • k8s is a container cluster management system open sourced by Google. k8s can build a container scheduling service. The purpose is to allow users to manage cloud container clusters through k8s clusters without requiring users to perform complicated settings. The system will automatically select appropriate working nodes to perform specific container cluster scheduling and processing work. Its core concept is container pod.
  • a container collection (pod) consists of a group of containers working on the same worker node. Pod is the most basic deployment unit of k8s.
  • a pod can encapsulate one or more containers (containers), storage resources (volume), an independent network IP, and policy options to manage and control the running mode of the container. Pod can logically be used to identify an instance of a certain application.
  • a pod When a pod encapsulates multiple containers, usually the multiple containers in this scenario include a main container and several auxiliary containers (SideCar containers).
  • a web application consists of three components: front-end, back-end and database. These three components run in their own containers.
  • the main container can be the web application front-end. For this example, it can contain pods of three containers.
  • LTE long term evolution
  • WiMAX global interoperability for microwave access
  • 5G fifth Generation
  • NR new radio access technology
  • FIG. 1 it is a schematic architectural diagram of a cloud computing system provided by an embodiment of the present application.
  • This embodiment includes a master 101 and a node 102.
  • the node 102 can be a virtual computing device or a physical computing device, and several pods can be deployed on one node 102 .
  • the manager 101 is the central management module of the container cluster management system.
  • the manager 101 can be regarded as a set of processes that manage the container life cycle.
  • the manager 101 is a k8s master, and the manager 101 may include a control module (controller), a scheduling module (scheduler), an application programming interface server (application programming interface server, API server) module, etc.
  • These processes implement management functions such as resource management, pod deployment, and system establishment of the entire computing device cluster.
  • the API server module provides the only operation entrance for resource objects, that is, the interface module that provides functions for users.
  • the control module is responsible for the unified management and control of various container models, such as CRUD (create, read, update and delete) operations on the container model.
  • a container model may, for example, indicate one or more of the following information: the number of containers included in a pod, the type of application running in the container, the maximum value of each type of resource used by the container while working, which container requires Monopolize CPU and other information.
  • the scheduling module is responsible for selecting appropriate nodes 102 for deployed units (containers or pods), etc.
  • the manager 101 may run on a certain node 102 in the computing device cluster, or on several nodes 102 in the computing device cluster (for high availability purposes).
  • Node 102 is mainly responsible for running containers. Each node 102 can also run components such as kubelet and container engine, and is responsible for managing the life cycle of pods on this node.
  • the kubelet is used to process tasks issued by the manager 101 to this node and manage pods and containers in the pod.
  • Kubelet can register the node's own information on the application programming interface service module, regularly report the usage of the node resources to the manager 101, and can monitor container and node resources through cAdvisor.
  • a container engine may be responsible for deploying containers, and the container engine may be a Docker component, for example.
  • a microservice can run on one or more containers.
  • Microservices is a cloud-native architectural approach in which a single application is composed of many smaller components or services that are loosely coupled and independently deployable.
  • Each microservice can run in its own independent process, and microservices can communicate with each other using lightweight communication mechanisms (such as RESTful API based on HTTP).
  • Each microservice can be built around a specific business and can be independently deployed to production environments, production-like environments, etc.
  • users, tenants, or managers can issue instructions to deploy pods to the manager 101 according to business needs.
  • the instructions can include: the number of pods, the number of containers contained in each pod, the minimum resource requirements (Request) and the maximum resource value (Limit) used by each container when working, and other information.
  • the resources referred to in the embodiment of this application may include CPU resources, memory resources, network bandwidth resources, disk resources, etc.
  • the resource request or resource limit of a pod refers to the sum of the resource requests or resource limits of all containers in the pod.
  • the pods in the node 102 can be distinguished by the pod name or Internet Protocol (IP), and multiple nodes 102 can also be distinguished by the node name or IP.
  • IP Internet Protocol
  • a computing device 100 provided by an embodiment of the present invention includes: a processor 1001, a memory 1002, a bus 1003, an input and output interface 1004, and a communication interface 1005.
  • the bus 1003 is used to connect the processor 1001, the memory 1002, the input and output interface 1004 and the communication interface. 1005, and implement data transmission between the processor 1001, the memory 1002, the input and output interface 1004 and the communication interface 1005.
  • the processor 1001 receives a command from the input/output interface 1004 through the bus 1003, decrypts the received command, and performs calculation or data processing according to the decrypted command.
  • the memory 1002 may include program modules, such as kernel, middleware, application program interface (AP), and applications.
  • Program modules may be composed of software, firmware, hardware, or at least two of them.
  • the input and output interface 1004 forwards commands or data input by the user through input devices (such as sensors, keyboards, and touch screens).
  • the communication interface 1005 connects the computing device 100 to other computing devices and networks.
  • the communication interface 1005 may be connected to a network through a wired or wireless connection to connect to other external computing devices.
  • the computing device 100 may also include a display device for displaying configuration information input by the user and displaying an operation interface to the user.
  • the processor 1001 may include a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (digital signal). processor, DSP) and other processors.
  • CPU central processing unit
  • GPU graphics processing unit
  • MP microprocessor
  • DSP digital signal processor
  • Memory 1002 includes volatile memory, such as random access memory (RAM).
  • RAM random access memory
  • the memory 1002 may also include non-volatile memory (non-volatile memory), such as read-only memory (ROM), flash memory, mechanical hard disk (hard disk drive, HDD) or solid state drive (solid state drive). ,SSD).
  • ROM read-only memory
  • HDD hard disk drive
  • SSD solid state drive
  • the bus 1003 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc.
  • the bus can be divided into address bus, data bus, control bus, etc. For ease of presentation, only one line is used in Figure 2, but it does not mean that there is only one bus or one type of bus.
  • Bus 1003 may include a path that carries information between various components of computing device 100 (eg, memory 1002, processor 1001, communications interface 1005).
  • the communication interface 1005 uses transceiver modules such as, but not limited to, network interface cards and transceivers to implement communication between the computing device 100 and other devices or communication networks.
  • executable program code is stored in the memory 1002, and the processor 1001 executes the executable program code to respectively implement the first acquisition module 201, the construction module 202, and the first generation shown in Figure 10 below.
  • the functions of the module 203 and the first adjustment module 204 are implemented to implement the resource scheduling method shown in FIG. 8 . That is, the memory 1002 stores instructions for executing the resource scheduling method shown in FIG. 8 .
  • the processor 1001 executes the executable program code, it can also realize the functions of the second acquisition module 301, the creation module 302, the second generation module 303 and the second adjustment module 304 shown in Figure 11 below, thereby realizing the functions shown in Figure 9
  • the computing device cluster 1000 includes at least one computing device, such as the first computing device 100A and the second computing device 100B in FIG. 3 .
  • the computing device 100 may be a server, such as a central server, an edge server, or a local server in a local data center.
  • the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.
  • computing device cluster 1000 includes at least one computing device.
  • the memory 1002 of one or more computing devices in the computing device cluster may store the same instructions for executing the resource scheduling method shown in FIG. 8 , or the instructions for the resource scheduling method shown in FIG. 8 .
  • the memory 1002 of one or more computing devices in the computing device cluster may also store memory for executing the resource scheduling method shown in Figure 8 or the resource scheduling method shown in Figure 9. Partial instructions.
  • a combination of one or more computing devices may jointly execute instructions for performing the resource scheduling method shown in FIG. 8 , or the instructions for the resource scheduling method shown in FIG. 9 .
  • the memory 1002 in different computing devices in the computing device cluster can store different instructions, respectively used to execute part of the functions of the resource scheduling device. That is, the instructions stored in the memory 1002 in different computing devices 100 can implement the functions of one or more modules among the first acquisition module 201, the construction module 202, the first generation module 203 and the first adjustment module 204, or implement Functions of one or more of the second acquisition module 301, the creation module 302, the second generation module 303 and the second adjustment module 304.
  • one or more computing devices in a cluster of computing devices may be connected through a network.
  • the network may be a wide area network or a local area network, etc.
  • Figure 3 shows a possible implementation.
  • two computing devices hereinafter referred to as a first computing device 100A and a second computing device 100B for ease of distinction
  • the connection to the network is made through a communication interface in each computing device.
  • instructions for executing the functions of the first acquisition module 201 and the construction module 202 may be stored in the memory 1002 of the first computing device 100A.
  • instructions for performing the functions of the first generation module 203 and the first adjustment module 204 may be stored in the memory 1002 in the second computing device 100B.
  • instructions for executing the functions of the second acquisition module 301 and the creation module 302 may also be stored in the memory 1002 of the first computing device 100A.
  • instructions for executing the functions of the second generation module 303 and the second adjustment module 304 may also be stored in the memory 1002 in the second computing device 100B.
  • connection method between the computing device clusters shown in Figure 3 can be implemented by considering that the resource scheduling method provided by this application requires a large amount of data storage and reading, so the first generation module 203 and the first adjustment module 204 are considered to be implemented.
  • the functions are performed by the second computing device 100B.
  • first computing device 100A shown in FIG. 3 can also be performed by multiple computing devices 100 .
  • second computing device 100B can also be completed by multiple computing devices 100 .
  • the resource scheduling system 10 includes a manager 101, nodes, a resource portrait service component 103, a decision service component 104, a monitoring service component 105, and a paging configuration component 106.
  • the embodiment of this application does not limit the number of nodes, and the number of nodes can be set according to actual business deployment requirements.
  • Figure 3 illustrates that the resource scheduling system 10 includes n nodes 102_1 ⁇ 102_n, where n is a positive integer greater than 1. This application The value of n is not limited.
  • n nodes 102_1 ⁇ 102_n can be connected together through a communication network, and the communication network can be Wired network, or wireless network.
  • the communication network can be implemented using any known network communication protocol.
  • the above-mentioned network communication protocol can be various wired or wireless communication protocols, such as Ethernet, universal serial bus (USB), global mobile communication system (global mobile communication system). system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), Time-division code division multiple access (TD-SCDMA), long term evolution (LTE), wireless fidelity (Wi-Fi), voice over Internet protocol (voice over Internet protocol (VoIP), a communication protocol that supports network slicing architecture, or any other suitable communication protocol.
  • GSM universal serial bus
  • GPRS general packet radio service
  • CDMA code division multiple access
  • WCDMA wideband code division multiple access
  • TD-SCDMA Time-division code division multiple access
  • LTE long term
  • the manager 101, resource profiling service component 103, decision service component 104, monitoring service component 105 and paging configuration component 106 can run on the same node, or run on two or more nodes. superior.
  • the manager 101 runs on one node, and the resource portrait service component 103, decision service component 104, monitoring service component 105 and paging configuration component 106 run on another node.
  • the resource portrait service component 103, the decision service component 104, the monitoring service component 105 and the paging configuration component 106 can be deployed on the node in the form of plug-ins.
  • the monitoring service component 105 can be used to collect resource information of each node 102_1 to 102_n, resource occupation information, Request information and Limit information of containers deployed on each node 102_1 to 102_n.
  • the resource information of a node may include the total amount of each type of resource on the node, the number of used resources, the number of remaining resources, the usage rate of each type of resource, and other information.
  • the resource occupancy information, Request information, and Limit information of the pods deployed by each node 102_1 to 102_n can be obtained based on the sum of the resource occupancy information, Request information, and Limit information of the containers included in the pod.
  • the resource occupancy information, Request information, and Limit information of the microservices deployed on each node 102_1 to 102_n can also be obtained based on the sum of the resource occupancy information, Request information, and Limit information of the containers associated with the microservices.
  • the resource occupation information of containers, pods, and microservices can refer to the amount of each type of resource (CPU, memory, disk, network bandwidth) occupied by pods, containers, and microservices.
  • the monitoring service component 105 may include a Prometheus component.
  • Prometheus components are tools for collecting and aggregating specified metrics as time series data.
  • the monitoring service component 105 can use the Prometheus component to collect the resource information of each node 102_1 to 102_n in real time, as well as the resource occupancy information, Request information and Limit information of the containers deployed on each node 102_1 to 102_n.
  • Each node 102_1 ⁇ 102_n can also be deployed with an Over Load Control (OLC) component.
  • OLC Over Load Control
  • the decision service component 104 determines that a certain microservice has sudden traffic resulting in insufficient resources based on the resource occupancy information of the microservice collected by the OLC component, it can trigger the expansion of the microservice to meet the resource requirements required for the operation of the microservice.
  • Insufficient resources may refer to the shortage of one or more of the CPU resources, memory resources, disk resources, network bandwidth resources, etc. allocated to the microservice.
  • the decision service component 104 is based on OLC
  • the resource occupancy information of microservices collected by the component determines that a certain microservice has excess resources, it can also trigger the reduction of the microservice to improve the utilization of node resources.
  • the monitoring service component 105 can also collect operating indicator data of each node 102_1 to 102_n and the microservices deployed on each node 102_1 to 102_n, so as to obtain the operating status of the nodes and microservices.
  • the monitoring service component 105 can be used to determine whether the node is running abnormally (disk abnormality, network abnormality, etc.), and to check the running status of the microservice to realize real-time monitoring of the running status of the microservice before or after resource adjustment.
  • the monitoring service component 105 can also monitor events triggered by the manager 101, such as the creation, deletion, and upgrade of microservices triggered by the manager 101, and expand or shrink the resources of microservices/pods. Waiting for events, the monitoring service component 105 can report the monitored events to the decision-making service component 104, and the decision-making service component 104 makes decisions and issues instructions. The monitoring service component 105 can also notify the operation and maintenance personnel of the monitored events through text messages or emails, so that the operation and maintenance personnel can be informed of changes in the computing device cluster in a timely manner.
  • the resource scheduling system 10 also includes a message center 107, and the monitoring service component 105 also The monitored events can be notified to the message center 107, and the message center 107 notifies the operation and maintenance personnel through text messages or emails.
  • the resource portrait service component 103 is used to receive the resource information of each node 102_1 ⁇ 102_n collected by the monitoring service component 105, the resource occupancy information, Request information and Limit information of the containers deployed on each node 102_1 ⁇ 102_n, etc. .
  • the resource profiling service component 103 can summarize and analyze various information collected by the monitoring service component 105 to implement resource profiling for each microservice. Through the monitoring service component 105, the resource occupation historical data of node resources, containers, pods and microservices can be collected, and the resource portrait service component 103 can summarize and analyze it to generate resource portraits of the microservices, so that the microservices can be analyzed later. /pod can be targeted when adjusting resources.
  • the resource portrait may include resource occupation dimensions and time dimensions.
  • the resource occupation dimension can characterize the resources occupied by microservices, such as CPU, memory, network bandwidth, disk IO and other resources occupied by microservices.
  • the time dimension combined with the resource occupation dimension can characterize the resources occupied by microservices at different times. For example, the traffic of Internet-based microservices will be affected by people's production and life, showing obvious peaks and peaks, periodicity and predictability, or scheduled microservices are also cyclical and predictable, and local life microservices will There are noon and evening peaks, and the traffic peak of e-commerce microservices during the promotion phase is several times the trough. If the resource scheduling system 10 can capture this information, it can flexibly allocate resources according to the status of nodes and microservices, realize dynamic allocation of microservice resources, and meet the service level agreement (Service Level Agreement, SLA) requirements of microservices to the greatest extent.
  • SLA Service Level Agreement
  • the resource profiling service component 103 can be integrated with a data processing model to implement aggregation and analysis of information collected by the monitoring service component 105 .
  • the data processing model can use the quantile value algorithm (quantile value for microservice resource requirements) to perform resource portraits on microservices.
  • the data processing model can also use other artificial intelligence algorithms to implement resource drawing for microservices. Like, this application does not limit this.
  • the resource portrait service component 103 can report the constructed resource portrait and the information collected by the monitoring service component 105 to the decision-making service component 104.
  • the decision service component 104 can make resource scheduling decisions based on the resource portrait constructed by the resource portrait service component 103, the information collected by the monitoring service component 105, and the running status of the nodes. For example, if a node is in an abnormal operating state (disk abnormality, network abnormality, etc.), the node is scheduled and isolated, that is, the decision service component 104 will not adjust the resources of the pods under the node that meet the resource adjustment conditions. After the node resumes normal operation, the decision service component 104 then adjusts the resources of the pods under the node that meet the resource adjustment conditions.
  • Pods that meet resource adjustment conditions can refer to pods associated with microservices that need to be expanded/shrunk.
  • the decision service component 104 can predict that a certain microservice needs to be expanded in a certain period of time based on the resource portrait of the microservice, and can then issue resource adjustment instructions to the microservice at a preset time in advance to meet the business requirements of the microservice.
  • the microservice is a life-type microservice and has a traffic peak from 7 to 9 p.m.
  • the decision-making service component 104 can adjust the resources of the life-type microservice 5 minutes in advance, and the specific amount of resource adjustment can be based on Resource portraits are estimated and set.
  • the decision service component 104 can issue resource adjustment instructions to the microservice before the large promotion starts to satisfy the requirements of the microservice.
  • the specific amount of resource adjustment can be estimated and set based on the traffic data of historical large promotions.
  • the decision service component 104 can also issue resource adjustment instructions for a certain microservice when the OLC detects that the traffic of a certain microservice has increased and the resources are approaching overload, so as to meet the business processing needs of the microservice.
  • the decision service component 104 issues resource adjustment instructions to the nodes through the paging configuration component 106, and uses the docker component on the node to adjust the resources of the pods that meet the resource adjustment conditions. Second-level resource adjustment can be achieved. Compared with using HPA/VPA, Resource adjustment has less hysteresis.
  • the resources when adjusting resources for a microservice, can be adjusted for all pods associated with the microservice, or it can be precise for one or several pods in the microservice, that is, the adjustment The pod can make the microservice have enough resources to cope with the current increase in business volume/traffic.
  • the decision service component 104 can automatically issue resource adjustment instructions to the paging configuration component 106 to trigger resource adjustment for pods that meet the resource adjustment conditions.
  • the decision service component 104 can also give corresponding resource adjustment suggestions, and notify the operation and maintenance personnel of the resource adjustment suggestions through the message center 107 for decision-making, that is, the operation and maintenance personnel can manually issue resource adjustments.
  • Instructions are sent to the paging configuration component 106 to trigger resource adjustment for pods that meet resource adjustment conditions.
  • Resource adjustment instructions can include the pods that require resource adjustment and the pod's resource adjustment method.
  • the pod's resource adjustment method can mean increasing the pod's resource Limit by a1, or decreasing the pod's resource Limit by a2, or setting the pod's resource Limit. For a3 and other methods.
  • the values of a1, a2, and a3 can be set according to actual needs. It can be understood that a1, a2, and a3 can each be a set of setting values of resources such as CPU, memory, disk IO, and network bandwidth.
  • the paging configuration component 106 can be used to issue resource adjustment instructions to nodes.
  • the kubelet on the node can listen to the resource adjustment instructions issued by the paging configuration component 106 and determine whether the resource adjustment is for the pod on the node. If the kubelet determines that it is not adjusting resources for the pod on this node, it can ignore the resource adjustment instruction. If kubelet determines that it is adjusting resources for pods on this node, kubelet can continue to determine which pods on this node are adjusting resources based on the resource adjustment instructions. For pods that require resource adjustment, kubelet can control the pod to call unix-socket to communicate with the docker process (the process of the docker component), and adjust the resources by calling the docker API.
  • the paging configuration component 106 can be ZooKeeper.
  • ZooKeeper can accept the registration of listeners. Once certain data changes, ZooKeeper can notify the listeners registered on ZooKeeper to respond accordingly. That is, the kubelet on the node can be registered with ZooKeeper to monitor the resource adjustment instructions issued by ZooKeeper.
  • the docker component can set limits for resources such as CPU, memory, disk, and network bandwidth for each process, and then can set limits for process access to resources.
  • the bottom layer of the docker component uses control groups (Control grpups, Cgroups) implementation.
  • Cgroups is a feature of the Linux kernel that can be used to limit, control and separate the resources of a process group.
  • Resource control in Cgroups is implemented in units of control groups.
  • a process can join a certain control group or migrate from one process group to another control group. Processes in a process group can use Cgroups to allocate resources in units of control groups, and are subject to restrictions set by Cgroups in units of control groups.
  • Cgroups can implement functions such as limiting the number of resources that a process group can use, controlling the priority of the process group, recording the number of resources used by the process group, process group isolation, and process group control.
  • Cgroups information When deploying docker components on a node, by default, Cgroups information will be mounted in the node/sys/fs/cgroup/cpu/kubebpods/burstable/podxxx directory, so that each process will have separate configuration information, which can be called by calling docker API to update the configuration information of the process and adjust the resources of the pod.
  • each microservice is configured with different priorities in a preset manner. For example, its own priority can be configured through the Annotation option of the microservice.
  • the resource portrait service component 103 can obtain the priority of each microservice and transmit the priority information to the decision-making service component 104.
  • the decision service component 104 can release the resources of low-priority microservices to high-priority microservices based on the priority of the microservices and the resource usage data of the microservices. If resources cannot be freed from low-priority microservices, the decision-making service component 104 can also schedule the low-priority microservices that are currently inactive (sleeping state) to other nodes based on the resource portrait of the microservices.
  • high-priority microservices can be allocated more resources. For example, if the current time is night time, low-priority microservices that are only active during the day and dormant at night can be scheduled to other nodes, thereby freeing up more resources for high-priority microservices without Affects the operation of this low-priority microservice.
  • multiple microservices are deployed on node 102_1.
  • the multiple microservices include computing microservices, and the computing microservices are configured with the highest priority. If the pod associated with the computing microservice is running, the CPU Limit is too small, which limits the running speed of the process, or In situations such as when the memory usage is close to Limit, the resource portrait service component 103 can obtain the priority of each microservice on the node 102_1, and the decision service component 104 can combine the resource information of the node 102_1 and the resource occupancy information of each microservice to make decisions through The analysis reveals more resources (such as CPU or memory) that can be allocated to pods associated with computing microservices, which can then speed up the calculation of computing microservices and fully utilize node resources.
  • resources such as CPU or memory
  • the decision service component 104 can also send a pod scaling request to the manager 101 when the OLC detects that the traffic of a certain microservice has increased and the resources are approaching overload.
  • the manager 101 can use HPA and/or VPA. Implement resource adjustment for pods that meet resource adjustment conditions.
  • HAP/VPA When resources are allocated to microservices through HAP/VPA, HAP/VPA will collect the resource indicators of the microservice in the most recent preset time period and calculate the average, compare the average with the target value, and then adjust the resources based on the comparison results. Therefore, resource adjustment through HAP/VPA will cause the problem of response lag; in addition, since the default expansion cooling period of HPA/VPA is 3 minutes, the problem of response lag will be further amplified.
  • microservice S1 is an e-commerce microservice.
  • the e-commerce microservice starts a product flash sale or an e-commerce live broadcast of a certain type of product at a certain moment, resulting in a large increase in traffic.
  • the resource scheduling system 10 includes a manager 101, a node 102_1, a node 102_2, a resource portrait service component 103, a decision service component 104, a monitoring service component 105, a paging configuration component 106 and a message center 107 as an example.
  • the manager 101, resource portrait service component 103, decision service component 104, monitoring service component 105, paging configuration component 106 and message center 107 can all be deployed on node 102_1.
  • the monitoring service component 105 may include a Prometheus component, and the OLC component may be deployed on the node 102_1. Assume that microservice S1 is deployed on node 102_1, and the pod associated with microservice S1 is pod_1.
  • the internal interaction process of resource scheduling system 10 to implement resource scheduling for microservice S1 includes:
  • the Prometheus component collects resource information of node 102_1, resource occupation information, Request information and Limit information of containers deployed on node 102_1, and reports it to the resource portrait service component 103.
  • the OLC component collects the resource occupation information of the microservice S1 deployed on the node 102_1, and reports it to the decision service component 104.
  • the resource portrait service component 103 builds a resource portrait of the microservice S1 based on the information collected by the Prometheus component, and reports the resource portrait of the microservice S1 and the information collected by the Prometheus component to the decision service component 104.
  • the decision-making service component 104 determines that microservice S1 needs to adjust resources based on the resource occupancy information of microservice S1 collected by the OLC component, the decision-making service component 104 issues resources based on the resource portrait of microservice S1 and the information collected by the Prometheus component. Adjust the instructions to the paging configuration component 106, or send the pod scaling request to the manager 101.
  • microservice S1 For example, if the OLC component collects that the resource occupation information of microservice S1 is close to the limit or has reached the limit, the decision service component 104 can determine that microservice S1 has a large increase in traffic. In order to ensure that microservice S1 can operate normally, microservice S1 needs to be Make resource adjustments to ensure that microservice S1 has sufficient resources to maintain normal operation.
  • the decision service component 104 can determine that there are too many idle resources in microservice S1. In order to ensure that the resources on node 102_1 can be fully utilized, it is necessary to Microservice S1 adjusts resources so that node 102_1 has enough remaining resources to allocate to other resource-constrained microservices.
  • calling the docker API to adjust the resources of pod_1 may mean calling the docker API to adjust the Cgroup parameters of pod_1, thereby adjusting the resources of pod_1.
  • the manager 101 adjusts resources for pod_1 through HPA and/or VPA.
  • the Prometheus component can also monitor the resource adjustment results of pod_1 and notify the message center 107, so that the message center 107 can notify the operation and maintenance personnel of the resource adjustment results of pod_1 via SMS or email.
  • the Prometheus component can also monitor the running status of microservice S1 after adjusting the resources of pod_1, and feedback it to the decision service component 104, and check the running status of microservice S1 through the Prometheus component to determine the adjusted Whether the resources are sufficient for the operation of microservice S1. If the microservice S1 still runs abnormally due to insufficient resources, the decision service component 104 can be triggered again to adjust the resources of the microservice S1.
  • the resource scheduling system 10 includes a manager 101, a node 102_1, a node 102_2, a resource portrait service component 103, a decision service component 104, a monitoring service component 105 and a paging configuration component 106 as an example.
  • Microservice S1 may be an e-commerce microservice, a computing microservice, a life microservice, etc., which is not limited in this embodiment.
  • microservice S1 When microservice S1 is started, it often performs a lot of initialization work, such as pulling image files, starting Tomcat containers, initializing Spring MVC/SpringBoot, instantiating Beans, instantiating Services, instantiating container engine base components, etc., and starts
  • the time is directly proportional to the size of the microservice (such as the amount of code in the microservice), the number of Restful interfaces, the number of externally dependent components, and inversely proportional to the resource specifications of the container.
  • the shorter the startup time of microservice S1 the sooner the Pod generated for microservice S1 through HPA or VPA will be ready, which can improve the stability and throughput of the cluster.
  • microservice S1 is deployed on node 102_1, and the pod associated with microservice S1 is pod_1.
  • the internal interaction process of resource scheduling system 10 to implement resource scheduling for microservice S1 includes:
  • the monitoring service component 105 collects the resource information of the node 102, the resource occupation information of the containers deployed on the node 102_1, the Request information and the Limit information, and transmits them to the resource portrait service component 103.
  • the monitoring service component 105 includes a Prometheus component.
  • the Prometheus component can be used to collect resource information of the node 102, resource occupation information, Request information, and Limit information of the containers deployed on the node 102_1.
  • the resource portrait service component 103 aggregates various information collected by the monitoring service component 105, and transmits the information aggregation result to the decision-making service component 104.
  • the decision service component 104 listens to the creation event of pod_1, it analyzes and calculates the number of resources that node 102_1 can currently allocate to pod_1, and issues resource adjustment instructions to the paging configuration component 106.
  • the manager 101 when creating pod_1, the manager 101 sets the resource Request and resource Limit of pod_1. In order to shorten the startup time of microservice S1, the resource Limit can be expanded. That is, the resource Limit of pod_1 can be expanded by calculating the additional resources that node 102_1 can currently allocate to pod_1.
  • the monitoring service component 105 can listen to the event that the manager 101 creates pod_1, and notify the decision service component 104 of the creation event of pod_1.
  • the unused resources of 3 can be allocated to pod_1, so that the resources of pod_1 change from 1 set by the manager 101 to 2, so as to achieve the purpose of quickly starting microservice S1.
  • the decision-making service component 104 then recycles the resources allocated to pod_1, that is, the resources of pod_1 change from 2 to 1 again.
  • additional resources allocated to pod_1 may include changing the CPU limit in pod_1 from 2 cores to 4 cores, changing the memory limit from b1MB to b2MB, etc., where b2 is greater than b1.
  • the paging configuration component 106 sends the resource adjustment instructions to the pod service.
  • the pod service controls pod_1 to call unix-socket to communicate with the docker process to call the docker API to adjust the resources of pod_1.
  • a pod service may refer to a component on a node used to manage pods, such as Kubelet.
  • the resource Request and resource Limit of each microservice, the information aggregated by the resource portrait service component 103, the information decided by the decision-making service component 104, etc. can be stored in the preset database.
  • the preset database can record each pod.
  • the unique identification for example, name or IP
  • allocated resources for example, name or IP
  • an embodiment of the present application provides a resource scheduling method, which can be applied to a computing device cluster.
  • the resource scheduling method may include:
  • Step S81 Obtain the resource information of each node in the cluster and the resource information of the container in each node.
  • Each node is equipped with a container engine.
  • the Prometheus component and the OLC component can be deployed in the cluster to collect the resource information of each node in real time, as well as the resource occupancy information, Request information, and Limit information of the containers deployed on each node.
  • the container engine may refer to the docker component.
  • the resource information may include information such as the total amount of each type of resource, the number of used resources, the number of remaining resources, the usage rate of each type of resource, and other information.
  • the resources referred to in the embodiment of this application may include CPU resources, memory resources, network bandwidth resources, disk resources, etc.
  • Step S82 Based on the resource information of each node and the resource information of the containers in each node, construct a resource portrait of the microservices deployed on each node.
  • the microservices deployed on each node run based on one or more containers. .
  • a resource portrait of the microservices deployed on each node For example, you can first obtain the resource information of the microservice deployed on each node based on the resource information of the container in each node, and then based on the resource information of each node and the resource information of the microservice deployed on each node, Build a resource portrait of the microservices deployed on each node.
  • the time dimension combined with the resource occupation dimension can be used to characterize the resources occupied by the microservice at different times.
  • resource allocation can be based on the resource portrait of the microservice, so that The resource adjustment of microservices is more targeted and can improve the resource utilization of nodes.
  • Step S83 If it is determined that the first microservice deployed in the cluster requires resource adjustment, based on the resource portrait of the first microservice, the resource information of the first node where the first microservice is deployed, and the resource information of the container in the first node , generate resource adjustment instructions associated with the first microservice.
  • a microservice with insufficient resources or a microservice with excess resources can be determined as the first microservice that requires resource adjustment. For example, if the resources occupied by the first microservice are less than the resource upper limit of the first microservice The difference is less than the first preset threshold, indicating that the resources of the first microservice are tight. The first microservice is expanded by triggering to meet the resource requirements required for the operation of the first microservice. If the resources occupied by the first microservice are When the difference between the resource upper limits of the first microservice is greater than the second preset threshold, indicating that the first microservice has excess resources, the first microservice is triggered to scale down so that the excess resources can be allocated to other resources with insufficient resources. Microservices.
  • the resources occupied by the first microservice include one or more of processor resources, memory resources, disk resources, and network bandwidth resources. If the resources occupied by the first microservice include processor resources, memory resources, disk resources, and network bandwidth, If the difference between any one of the resources and the corresponding resource upper limit is less than the corresponding first preset threshold, it indicates that one or several resources of the microservice are tight, and it is determined that the microservice needs to be expanded. If the difference between any one of the processor resources, memory resources, disk resources, and network bandwidth resources occupied by the first microservice and the corresponding resource upper limit is greater than the corresponding second preset threshold, it indicates that a certain item of the microservice or Certain resources are excessive, and it is determined that microservices need to be scaled down.
  • the number of resources required by the microservice can be predicted through the resource portrait of the first microservice, and then the first microservice is deployed based on the predicted number of resources.
  • the resource information of the node and the resource information of the container in the node allocate resources to the first microservice, making the resource adjustment of the microservice more accurate, avoiding excessive or insufficient resource allocation, and improving the resource utilization of the node.
  • Step S84 Adjust the resources of the pod associated with the first microservice based on the container engine and resource adjustment instructions.
  • the resource adjustment instruction is to adjust resources for all pods associated with the microservice, or to adjust resources for one or several pods in the microservice, and call it through the container engine
  • the resource adjustment application program interface adjusts resources for specified pods.
  • the hysteresis of resource adjustment is small, and it can realize resource adjustment at the second level and enhance the ability of microservices deployed in the cluster to cope with sudden traffic.
  • the container engine is a docker component.
  • the docker component can set limits for resources such as CPU, memory, disk, and network bandwidth for each process, and then set limits for process access to resources.
  • the bottom layer of the docker component is implemented through Cgroups. Cgroups can implement functions such as limiting the number of resources that a process group can use, controlling the priority of a process group, recording the number of resources used by a process group, process group isolation, and process group control.
  • Cgroups information can be mounted in the node/sys/fs/cgroup/cpu/kubebpods/burstable/podxxx directory, so that each process will have separate configuration information, and the container engine can pass
  • the docker API is called to update the configuration information of the process to implement resource adjustment for the pod associated with the first microservice.
  • the microservices deployed on each node can be configured with different priorities in a preset manner.
  • the resources of a microservice deployed on the node with a lower priority than the first microservice can be released, so that the node has enough free resources to allocate to the first microservice. For example, if the microservice deployed on the node has a lower priority than the first microservice as the second microservice, some of the resources occupied by the second microservice can be released based on the resource portrait of the second microservice. Resource profiling can determine and estimate how many resources the second microservice can currently release without affecting the operation of the second microservice.
  • the nodes deployed on the node may be configured with a smaller number of resources than the first microservice.
  • a microservice with low priority is migrated to other nodes, so that the node has enough free resources to allocate to the microservices that need resource adjustment. For example, migrate the third microservice deployed on the node with a lower priority than the first microservice to other nodes in the cluster, so that the node has enough free resources to allocate to the microservice that needs resource adjustment.
  • a third microservice that can be scheduled to other nodes in the cluster can be selected based on the resource portrait of the microservice deployed on the node. For example, if the current time is night time, you can choose to be active only during the day and not active at night. The dormant low-priority third microservice is scheduled to other On the node, more resources can be allocated for the high-priority first microservice without affecting the operation of the low-priority third microservice.
  • the resource scheduling method may include:
  • Step S91 Obtain the resource information of each node in the cluster and the resource information of the container in each node.
  • Each node is equipped with a container engine.
  • the Prometheus component and the OLC component can be deployed in the cluster to collect the resource information of each node in real time, as well as the resource occupancy information, Request information, and Limit information of the containers deployed on each node.
  • the container engine may refer to the docker component.
  • the resource information may include information such as the total amount of each type of resource, the number of used resources, the number of remaining resources, the usage rate of each type of resource, and other information.
  • the resources referred to in the embodiment of this application may include CPU resources, memory resources, network bandwidth resources, disk resources, etc.
  • Step S92 In response to the startup instruction of the first microservice deployed in the cluster, create a first pod for the first microservice, and the first pod is set with a first resource upper limit.
  • the first microservice can refer to any microservice deployed in the cluster.
  • the cluster starts the first microservice, it can create the first pod for the first microservice, maintain the operation of the first microservice through the first pod, and the cluster sets the first resource upper limit (Limit) for the first pod.
  • Limit first resource upper limit
  • Step S93 Generate a resource adjustment instruction associated with the first pod based on the resource information of the first node where the first microservice is deployed and the resource information of the container in the first node.
  • the upper limit of resource occupancy allocated by the system for the microservice is increased to achieve Quickly start microservices and shorten the startup time of microservices.
  • Step S94 Adjust the resource usage upper limit of the first pod from the first resource upper limit value to the second resource upper limit value based on the container engine and the resource adjustment instruction.
  • the second resource upper limit value is greater than the first resource upper limit value.
  • the container engine calls the resource adjustment application program interface to adjust the upper limit of the first pod's resource usage from the first resource upper limit to the second resource upper limit.
  • the resource adjustment has little hysteresis and can realize second-level resource adjustment.
  • the container engine is a docker component, and the bottom layer of the docker component is implemented through Cgroups.
  • Cgroups information can be mounted in the node/sys/fs/cgroup/cpu/kubebpods/burstable/podxxx directory, so that each process will have separate configuration information, and the container engine can pass
  • the docker API is called to adjust the upper resource occupation limit of the first pod from the first resource upper limit value to the second resource upper limit value.
  • the resource occupation upper limit of the first pod can be adjusted from the second resource upper limit value to the first resource upper limit value. That is, after the first microservice is started, Finally, the resources previously allocated to the first microservice in order to shorten the startup time of the first microservice are recycled to avoid excess resources of the first microservice and improve the resource utilization of the node.
  • the first resource scheduling device 20 may include a first acquisition module 201, a construction module 202, a first generation module 203 and a first adjustment module 204.
  • the first acquisition module 201 is used to acquire the resource information of each node in the cluster and the resource information of the containers in each node.
  • Each node can have a container engine installed.
  • the building module 202 is configured to build a resource portrait of the microservice deployed on each node based on the resource information of each node and the resource information of the container in each node. Microservices deployed on each node can run based on one or more containers.
  • the first generation module 203 is used to, when the first microservice deployed in the cluster requires resource adjustment, based on the resource portrait of the first microservice, the resource information of the first node where the first microservice is deployed, and the resource information of the first node.
  • the resource information of the container is obtained, and a resource adjustment instruction associated with the first microservice is generated.
  • the first adjustment module 204 is used to adjust the resources of the pod associated with the first microservice based on the container engine and resource adjustment instructions.
  • the first acquisition module 201, the construction module 202, the first generation module 203 and the first adjustment module 204 can all be implemented by software, or can be implemented by hardware.
  • the implementation of the first acquisition module 201 is introduced next, taking the first acquisition module 201 as an example.
  • the implementation of the building module 202, the first generation module 203 and the first adjustment module 204 can refer to the implementation of the first acquisition module 201.
  • the first acquisition module 201 may include code running on a computing instance.
  • the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the above computing instance may be one or more.
  • the first acquisition module 201 may include code running on multiple hosts/virtual machines/containers. It should be noted that multiple hosts/virtual machines/containers used to run the code can be distributed in the same region (region) or in different regions. Furthermore, multiple hosts/virtual machines/containers used to run the code can be distributed in the same availability zone (AZ) or in different AZs. Each AZ includes one data center or multiple AZs. geographically close data centers. Among them, usually a region can include multiple AZs.
  • the multiple hosts/VMs/containers used to run the code can be distributed in the same virtual private cloud (VPC), or across multiple VPCs.
  • VPC virtual private cloud
  • Cross-region communication between two VPCs in the same region and between VPCs in different regions requires a communication gateway in each VPC, and the interconnection between VPCs is realized through the communication gateway. .
  • the first acquisition module 201 may include at least one computing device, such as a server.
  • the first acquisition module 201 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD).
  • ASIC application-specific integrated circuit
  • PLD programmable logic device
  • the above-mentioned PLD can be a complex programmable logical device (CPLD), a field-operable Implemented by field-programmable gate array (FPGA), general array logic (GAL) or any combination thereof.
  • CPLD complex programmable logical device
  • FPGA field-operable Implemented by field-programmable gate array
  • GAL general array logic
  • the multiple computing devices included in the first acquisition module 201 may be distributed in the same region or in different regions.
  • the multiple computing devices included in the first acquisition module 201 may be distributed in the same AZ or in different AZs.
  • multiple computing devices included in the first acquisition module 201 may be distributed in the same VPC or in multiple VPCs.
  • the plurality of computing devices may be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.
  • the first acquisition module 201 can be used to perform any steps in the resource scheduling method shown in Figure 8
  • the building module 202 can be used to perform any steps in the resource scheduling method shown in Figure 8
  • the first generation module 203 can be used to perform any step in the resource scheduling method shown in Figure 8
  • the first adjustment module 204 can be used to perform any step in the resource scheduling method shown in Figure 8
  • the first acquisition module 201, the construction module 202, the first generation module 203 and the first adjustment module 204 can be specified as needed, through the first acquisition module 201, the construction module 202, the first generation module 203 and the first adjustment module 204 respectively. Implement different steps in the resource scheduling method shown in Figure 8 to realize all functions of the first resource scheduling device 20.
  • the second resource scheduling device 30 includes a second acquisition module 301, a creation module 302, a second generation module 303 and a second adjustment module 304.
  • the second acquisition module 301 is used to acquire the resource information of each node in the cluster and the resource information of the containers in each node.
  • Each node can have a container engine installed.
  • the creation module 302 is configured to create a first pod for the first microservice in response to a startup instruction of the first microservice deployed in the cluster.
  • the first pod can be set with a first resource upper limit.
  • the second generation module 303 is configured to generate a resource adjustment instruction associated with the first pod based on the resource information of the first node where the first microservice is deployed and the resource information of the container in the first node.
  • the second adjustment module 304 is configured to adjust the upper resource occupation limit of the first pod from the first upper resource limit value to the second upper resource limit value based on the container engine and resource adjustment instructions.
  • the second resource upper limit value may be greater than the first resource upper limit value.
  • the second acquisition module 301, the creation module 302, the second generation module 303 and the second adjustment module 304 can all be implemented by software, or can be implemented by hardware.
  • the implementation of the second acquisition module 301 is introduced next, taking the second acquisition module 301 as an example.
  • the implementation of the creation module 302, the second generation module 303 and the second adjustment module 304 can refer to the implementation of the second acquisition module 301.
  • the second acquisition module 301 may include code running on the computing instance.
  • the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container.
  • the above computing instance may be one or more.
  • the second acquisition module 301 may include code running on multiple hosts/virtual machines/containers. It should be noted that multiple hosts/virtual machines/containers used to run the code can be distributed in the same region (region) or in different regions. Further, for running this code Multiple hosts/virtual machines/containers can be distributed in the same availability zone (AZ) or in different AZs. Each AZ includes one data center or multiple geographically close data centers. Among them, usually a region can include multiple AZs.
  • the multiple hosts/VMs/containers used to run the code can be distributed in the same VPC or across multiple VPCs.
  • a VPC is set up in a region.
  • Cross-region communication between two VPCs in the same region and between VPCs in different regions requires a communication gateway in each VPC, and the interconnection between VPCs is realized through the communication gateway. .
  • the second acquisition module 301 may include at least one computing device, such as a server.
  • the second acquisition module 301 may also be a device implemented using ASIC, or a PLD, or the like.
  • the above-mentioned PLD can be implemented by CPLD, FPGA, GAL or any combination thereof.
  • the multiple computing devices included in the second acquisition module 301 may be distributed in the same region or in different regions.
  • the multiple computing devices included in the second acquisition module 301 may be distributed in the same AZ or in different AZs.
  • multiple computing devices included in the second acquisition module 301 may be distributed in the same VPC or in multiple VPCs.
  • the plurality of computing devices may be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.
  • the second acquisition module 301 can be used to perform any steps in the resource scheduling method shown in Figure 9, and the creation module 302 can be used to perform any steps in the resource scheduling method shown in Figure 9.
  • the second generation module 303 can be used to perform any step in the resource scheduling method shown in Figure 9
  • the second adjustment module 304 can be used to perform any step in the resource scheduling method shown in Figure 9, the second acquisition
  • the steps responsible for implementation by the module 301, the creation module 302, the second generation module 303 and the second adjustment module 304 can be specified as needed, through the second acquisition module 301, the creation module 302, the second generation module 303 and the second adjustment module 304 respectively. Implement different steps in the resource scheduling method shown in Figure 9 to realize all functions of the second resource scheduling device 30.
  • Embodiments of the present application also provide a computer storage medium.
  • Computer instructions are stored in the computer storage medium.
  • the computing device cluster executes the above related method steps to implement the above embodiments. Resource scheduling methods.
  • An embodiment of the present application also provides a computer program product.
  • the computer program product When the computer program product is run on a computing device cluster, it causes the computing device cluster to perform the above related steps to implement the resource scheduling method in the above embodiment.
  • inventions of the present application also provide a device.
  • This device may be a chip, a component or a module.
  • the device may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and the device has an implementation
  • the resource scheduling method provided in the above embodiment has the function of calculating device cluster behavior.
  • Functions can be implemented by hardware, or by hardware executing corresponding software.
  • Hardware or software includes one or more modules corresponding to the above functions.
  • the computer storage media, computer program products or chips provided by the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the corresponding methods provided above. The beneficial effects will not be repeated here.
  • the disclosed devices and methods can be implemented in other ways.
  • the device embodiments described above are schematic.
  • the division of modules or units is a logical function division.
  • multiple units or components may be combined or can be integrated into another device, or some features can be ignored, or not implemented.
  • the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated.
  • the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place, or they may be distributed to multiple different places. . Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present application can be integrated into one processing unit, each unit can exist physically alone, or two or more units can be integrated into one unit.
  • the above integrated units can be implemented in the form of hardware or software functional units.
  • the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a readable storage medium.
  • the storage medium includes a number of instructions to cause a device (which may be a microcontroller, a chip, etc.) or a processor to execute all or part of the steps of the methods described in various embodiments of this application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program code. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

本申请实施例提供了一种资源调度方法及其相关设备，涉及通信技术领域。所述方法包括：通过获取集群中的每个节点的资源信息，以及每个节点中的容器的资源信息，构建在每个节点上部署的微服务的资源画像，每个节点均安装有容器引擎；当确定在集群中部署的第一微服务为需进行资源调整的微服务时，基于第一微服务的资源画像、部署第一微服务的第一节点的资源信息以及第一节点中的容器的资源信息，生成与第一微服务关联的资源调整指令，及基于容器引擎及资源调整指令调整与第一微服务关联的pod的资源。本申请基于微服务画像与容器引擎进行pod资源调整，资源调整灵敏准确，响应速度快。

Description

资源调度方法及其相关设备
本申请要求于2022年09月05日提交中国专利局，申请号为202211080778.9、申请名称为“资源调度方法及其相关设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及通信技术领域，尤其涉及一种资源调度方法及其相关设备。
背景技术
随着虚拟化技术的发展，越来越多的公司将自己的线上应用迀移到云平台上。容器(Container)作为一种轻量级的虚拟化技术，近年来发展迅速。容器技术为不同的应用程序创造了独立的运行环境，实现了资源隔离、配置与安全保障，能够满足应用按需分配的资源需求以及保证应用的隔离性和可用性。
为了满足应用的需求，实践中往往需要将很多容器部署在计算设备集群(Cluster，以下简称集群)中进行统一管理并对外提供服务。目前，较流行的容器集群管理工具是kubernetes(k8s)容器集群管理***。k8s可以构建容器的部署服务，容器集合(pod)是k8s的基本部署单元，一个pod由一组工作在同一节点的容器构成。
k8s主要是通过pod水平自动伸缩(Horizontal Pod Autoscaler，HPA)和pod垂直自动伸缩(Vertical Pod Autoscaler，VPA)实现自动扩缩容。然而，采用HPA或VPA方式对pod进行扩缩容，存在响应速度慢的缺陷，无法应对流量突发、持续时间短的场景的扩容需求。
发明内容
有鉴于此，有必要提供一种资源调度方法，解决现有技术中对pod进行扩缩容存在响应速度慢的问题。
本申请实施例第一方面公开了一种资源调度方法，包括：获取集群中的每个节点的资源信息，以及每个节点中的容器的资源信息，每个节点均安装有容器引擎；基于每个节点的资源信息和每个节点中的容器的资源信息，构建在每个节点上部署的微服务的资源画像，每个节点上部署的微服务基于一个或多个容器运行；若确定在集群中部署的第一微服务需进行资源调整，基于第一微服务的资源画像、部署第一微服务的第一节点的资源信息以及第一节点中的容器的资源信息，生成与第一微服务关联的资源调整指令；基于容器引擎及资源调整指令，调整与第一微服务关联的pod的资源。
采用上述技术方案，通过获取集群中的每个节点的资源信息，构建在每个节点上部署的微服务的资源画像，基于微服务的资源画像、节点的资源信息及容器的资源信息进行资源调整可以充分利用节点的资源，使得微服务的资源调整更有针对性，且可提高节点和/或集群的综合资源利用率，基于容器引擎进行资源调整，在不影响集群既有的调度能力下，相比通过HPA/VPA进行资源调整迟滞性更小，可以实现秒级资源调整，增强集群中部署的微服务应对突发流量的能力，提升集群的稳定性与网络吞吐量，在对微服务进行资源调整时，可以对与微服务关联的所有pod进行资源调整，也可以精确到对微服务中的某个或某几个pod进行资源调整，资源调整粒度更小、更准确。
在一些实施例中，基于每个节点的资源信息和每个节点中的容器的资源信息，构建在每个节点上部署的微服务的资源画像，包括：基于每个节点中的容器的资源信息，得到在每个节点上部署的微服务的资源信息；基于每个节点的资源信息及每个节点上部署的微服务的资源信息，构建在每个节点上部署的微服务的资源画像，资源画像包括资源占用维度与时间维度。
采用上述技术方案，基于每个节点的资源信息及每个节点上部署的微服务的资源信息，构建在每个节点上部署的微服务的资源画像，在资源画像中，采用时间维度结合资源占用维度表征微服务在不同的时间所占用的资源的特征，进而在对微服务进行资源调整时，可以基于微服务的资源画像进行资源分配，使得微服务的资源调整更有针对性，可提高节点的资源利用率，且可实现最大程度满足微服务的服务等级协议要求。
在一些实施例中，资源调度方法还包括：若第一微服务占用的资源与第一微服务的资源上限值之差小于第一预设阈值，确定第一微服务需进行资源调整；或者若第一微服务占用的资源与第一微服务的资源上限值之差大于第二预设阈值，确定第一微服务需进行资源调整。
采用上述技术方案，可以将资源不足的微服务或者资源过剩的微服务确定为需要进行资源调整的微服务，例如若微服务占用的资源与微服务的资源上限值之差小于第一预设阈值，表明微服务的资源紧张，通过触发对该微服务进行扩容，以满足微服务运行所需的资源需求，若微服务占用的资源与微服务的资源上限值之差大于第二预设阈值，表明微服务存在资源过剩时，通过触发对该微服务进行缩容，使得过剩的资源可以分配给其他资源不足的微服务，可提高节点的资源利用率。
在一些实施例中，第一微服务占用的资源包括处理器资源、内存资源、磁盘资源以及网络带宽资源中的一种或多种，确定第一微服务需进行资源调整，包括：若第一微服务占用的处理器资源、内存资源、磁盘资源以及网络带宽资源中的任意一者与对应的资源上限值之差小于对应的第一预设阈值，确定第一微服务需进行资源调整；或者若第一微服务占用的处理器资源、内存资源、磁盘资源以及网络带宽资源中的任意一者与对应的资源上限值之差大于对应的第二预设阈值，确定第一微服务需进行资源调整。
采用上述技术方案，可以将存在处理器资源、内存资源、磁盘资源以及 网络带宽资源中的任意一者不足的微服务，或者将存在处理器资源、内存资源、磁盘资源以及网络带宽资源中的任意一者过剩的微服务确定为需要进行资源调整的微服务，以满足微服务运行所需的资源需求，提高节点的资源利用率。
在一些实施例中，确定第一微服务需进行资源调整，包括：若第一微服务占用的处理器资源、内存资源、磁盘资源以及网络带宽资源中的任意一者与对应的资源上限值之差小于对应的第一预设阈值，确定第一微服务需进行扩容调整；或者若第一微服务占用的处理器资源、内存资源、磁盘资源以及网络带宽资源中的任意一者与对应的资源上限值之差大于对应的第二预设阈值，确定第一微服务需进行缩容调整。
采用上述技术方案，若微服务占用的处理器资源、内存资源、磁盘资源以及网络带宽资源中的任意一者与对应的资源上限值之差小于对应的第一预设阈值，表明微服务的某项资源紧张，确定微服务的该项资源需进行扩容，以满足微服务运行所需的资源需求，若微服务占用的处理器资源、内存资源、磁盘资源以及网络带宽资源中的任意一者与对应的资源上限值之差大于对应的第二预设阈值，表明微服务的某项资源过剩，确定微服务的该项资源需进行缩容，使得过剩的资源可以分配给其他资源不足的微服务，可提高节点的资源利用率。
在一些实施例中，基于第一微服务的资源画像、部署第一微服务的节点的资源信息以及节点中的容器的资源信息，生成与第一微服务关联的资源调整指令，包括：基于第一微服务的资源画像预测第一微服务所需的资源数量；基于预测的资源数量、部署第一微服务的节点的资源信息以及节点中的容器的资源信息，生成与第一微服务关联的资源调整指令。
采用上述技术方案，通过微服务的资源画像预测微服务所需的资源数量，再基于预测的资源数量、部署微服务的节点的资源信息以及节点中的容器的资源信息为微服务分配资源，使得微服务的资源调整更加准确，避免出现资源分配过多或者资源分配不足，提高节点的资源利用率。
在一些实施例中，生成与第一微服务关联的资源调整指令，包括：生成与第一微服务关联的所有pod的资源调整指令；或者生成与第一微服务关联的指定pod的资源调整指令。
采用上述技术方案，在生成与第一微服务关联的资源调整指令时，可以选择对与微服务关联的所有pod进行资源调整，也可以精确到对微服务中的某个或某几个pod进行资源调整，调整的pod可以使得该微服务具有足够的资源应对当前的业务量/流量增大问题即可。
在一些实施例中，基于容器引擎及资源调整指令调整与第一微服务关联的pod的资源，包括：确定与资源调整指令关联的pod，及通过容器引擎调用资源调整应用程序接口对与资源调整指令关联的pod进行资源调整。
采用上述技术方案，在对微服务进行资源调整时，先确定资源调整指令是对与微服务关联的所有pod进行资源调整，或者是对微服务中的某个或某 几个pod进行资源调整，通过容器引擎调用资源调整应用程序接口对指定的pod进行资源调整，资源调整迟滞性小，可以实现秒级资源调整，增强集群中部署的微服务应对突发流量的能力。
在一些实施例中，每个节点上部署的微服务配置有优先级，资源调度方法还包括：若第一节点的剩余资源无法满足第一微服务的资源调整需求，释放第一节点上部署的第二微服务占用的部分资源，第二微服务的优先级低于第一微服务的优先级。
采用上述技术方案，各个微服务可以通过预设方式配置不同的优先级，若节点资源紧张无法满足节点上部署的微服务的资源调整需求，可以释放节点上部署的比该微服务的优先级低的某个微服务的资源，使得节点有足够空闲的资源分配给需要进行资源调整的微服务。
在一些实施例中，释放第一节点上部署的第二微服务占用的部分资源，包括：根据第二微服务的资源画像，释放第二微服务占用的部分资源。
采用上述技术方案，在节点资源紧张无法满足节点上部署的微服务的资源调整需求时，可以根据低优先级的微服务的资源画像，来释放低优先级的微服务占用的部分资源，例如，可以基于低优先级的微服务的资源画像确定优先级的微服务可以释放多少资源，又不影响低优先级的微服务的运行。
在一些实施例中，每个节点上部署的微服务配置有优先级，资源调度方法还包括：若第一节点的剩余资源无法满足第一微服务的资源调整需求，将第一节点上部署的第二微服务调度至集群中的其他节点，第二微服务的优先级低于第一微服务的优先级。
采用上述技术方案，各个微服务可以通过预设方式配置不同的优先级，节点资源紧张无法满足节点上部署的微服务的资源调整需求时，若无法从低优先级的微服务中腾挪出资源，或者不采用从低优先级的微服务中腾挪出资源的策略，还可以将节点上部署的比该微服务的优先级低的某个微服务迁移到其他节点上，使得节点有足够空闲的资源分配给需要进行资源调整的微服务。
在一些实施例中，将第一节点上部署的第二微服务调度至集群中的其他节点，包括：基于在第一节点上部署的微服务的资源画像，选择可调度至集群中的其他节点的第二微服务。
采用上述技术方案，若无法从低优先级的微服务中腾挪出资源，或者不采用从低优先级的微服务中腾挪出资源的策略，还可以采用将低优先级的某个微服务迁移到其他节点上的策略，可以基于微服务的资源画像将当前处于未活跃的低优先级的微服务调度到其他节点上，以使得高优先级的微服务可以分配到更多的资源，例如，当前时刻为夜间时刻，可以选择将仅日间活跃而夜间处于休眠状态的低优先级的微服务调度到其他节点上，进而既可以为高优先级的微服务腾挪更多的资源，又不影响该低优先级的微服务的运行。
本申请实施例第二方面公开了一种资源调度方法，包括：获取集群中的每个节点的资源信息，以及每个节点中的容器的资源信息，每个节点均安装 有容器引擎；响应于在集群中部署的第一微服务的启动指令，为第一微服务创建第一pod，第一pod设置有第一资源上限值；基于部署第一微服务的第一节点的资源信息以及第一节点中的容器的资源信息，生成与第一pod关联的资源调整指令；基于容器引擎及资源调整指令将第一pod的资源占用上限由所述第一资源上限值调整为第二资源上限值，第二资源上限值大于第一资源上限值。
采用上述技术方案，在微服务启动过程中，基于集群中的部署的微服务的节点的资源信息及节点中的容器的资源信息，来调高***为微服务分配的资源占用上限，以达到快速启动微服务，缩短微服务的启动时间，基于容器引擎进行资源调整，在不影响集群既有的调度能力下，相比通过HPA/VPA进行资源调整迟滞性更小，可以实现秒级资源调整，快速响应微服务启动所需的资源需求。
在一些实施例中，资源调度方法还包括：若确定第一微服务启动成功，将第一pod的资源占用上限由第二资源上限值调整为第一资源上限值。
采用上述技术方案，在微服务启动完成之后，将先前为了缩短微服务的启动时间而多分配给微服务的资源进行回收，避免微服务资源过剩，可提高节点的资源利用率。
第三方面，本申请实施例提供一种资源调度装置，包括：第一获取模块，用于获取集群中的每个节点的资源信息，以及每个节点中的容器的资源信息，其中每个节点均安装有容器引擎；构建模块，用于基于每个节点的资源信息和每个节点中的容器的资源信息，构建在每个节点上部署的微服务的资源画像，其中每个节点上部署的微服务基于一个或多个容器运行；第一生成模块，用于在集群中部署的第一微服务为需进行资源调整时，基于第一微服务的资源画像、部署第一微服务的第一节点的资源信息以及第一节点中的容器的资源信息，生成与第一微服务关联的资源调整指令；第一调整模块，用于基于容器引擎及资源调整指令调整与第一微服务关联的pod的资源。
采用上述技术方案，通过获取集群中的每个节点的资源信息，构建在每个节点上部署的微服务的资源画像，基于微服务的资源画像、节点的资源信息及容器的资源信息进行资源调整可以充分利用节点的资源，使得微服务的资源调整更有针对性，且可提高节点和/或集群的综合资源利用率，基于容器引擎进行资源调整，在不影响集群既有的调度能力下，相比通过HPA/VPA进行资源调整迟滞性更小，可以实现秒级资源调整，增强集群中部署的微服务应对突发流量的能力，提升集群的稳定性与网络吞吐量，在对微服务进行资源调整时，可以对与微服务关联的所有pod进行资源调整，也可以精确到对微服务中的某个或某几个pod进行资源调整，资源调整粒度更小、更准确。
第四方面，本申请实施例提供一种资源调度装置，包括：第二获取模块，用于获取集群中的每个节点的资源信息，以及每个节点中的容器的资源信息，每个节点均安装有容器引擎；创建模块，用于响应于在集群中部署的第一微服务的启动指令，为第一微服务创建第一pod，第一pod设置有第一资源上限 值；第二生成模块，用于基于部署第一微服务的第一节点的资源信息以及第一节点中的容器的资源信息，生成与第一pod关联的资源调整指令；第二调整模块，用于基于容器引擎及资源调整指令将第一pod的资源占用上限由第一资源上限值调整为第二资源上限值，第二资源上限值大于第一资源上限值。
采用上述技术方案，在微服务启动过程中，基于集群中的部署的微服务的节点的资源信息及节点中的容器的资源信息，来调高***为微服务分配的资源占用上限，以达到快速启动微服务，缩短微服务的启动时间，基于容器引擎进行资源调整，在不影响集群既有的调度能力下，相比通过HPA/VPA进行资源调整迟滞性更小，可以实现秒级资源调整，快速响应微服务启动所需的资源需求。
第五方面，本申请实施例提供一种计算机可读存储介质，包括计算机程序指令，当计算机程序指令由计算设备集群执行时，使得计算设备集群执行如第一方面或第二方面所述的资源调度方法。
第六方面，本申请实施例提供一种计算设备集群，包括至少一个计算设备，每个计算设备包括处理器和存储器；至少一个计算设备的处理器用于执行至少一个计算设备的存储器中存储的指令，以使得计算设备集群执行如第一方面或第二方面所述的资源调度方法。
第七方面，本申请实施例提供一种计算机程序产品，当计算机程序产品被计算设备集群运行时，使得计算设备集群执行如第一方面或第二方面所述的资源调度方法。
第八方面，提供一种装置，所述装置具有实现上述第一方面所提供的方法中计算设备集群行为的功能。功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。硬件或软件包括一个或多个与上述功能相对应的模块。
可以理解地，上述提供的第五方面所述的计算机可读存储介质，第六方面所述的计算设备集群，第七方面所述的计算机程序产品，第八方面所述的装置可以与上述第一方面和/或第二方面的方法对应，因此，其所能达到的有益效果可参考上文所提供的对应的方法中的有益效果，此处不再赘述。
附图说明
图1为本申请一实施例提供的云计算***的架构示意图；
图2为本申请一实施例提供的一种计算设备的结构示意图；
图3为本申请一实施例提供的一种计算设备集群的结构示意图；
图4为本申请一实施例提供的资源调度***的架构示意图；
图5为本申请一实施例提供的资源调度***在微服务流量变化场景进行资源调度的交互流程示意图；
图6为本申请一实施例提供的资源调度***在微服务启动场景进行资源调度的交互流程示意图；
图7为本申请一实施例提供的微服务在启动阶段的资源变化示意图；
图8为本申请一实施例提供的资源调度方法的流程示意图；
图9为本申请另一实施例提供的资源调度方法的流程示意图；
图10为本申请一实施例提供的第一资源调度装置的功能模块示意图；
图11为本申请另一实施例提供的第二资源调度装置的功能模块示意图。
具体实施方式
需要说明的是，本申请中“至少一个”是指一个或者多个，“多个”是指两个或多于两个。“和/或”，描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B可以表示：单独存在A，同时存在A和B，单独存在B的情况，其中A，B可以是单数或者复数。本申请的说明书和权利要求书及附图中的术语“第一”、“第二”、“第三”、“第四”等(如果存在)是用于区别类似的对象，而不是用于描述特定的顺序或先后次序。
在本申请实施例中，“示例性的”或者“例如”等词用于表示作例子、例证或说明。本申请实施例中被描述为“示例性的”或者“例如”的任何实施例或设计方案不应被解释为比其它实施例或设计方案更优选或更具优势。确切而言，使用“示例性的”或者“例如”等词旨在以具体方式呈现相关概念。
k8s是Google公司开源的一个容器集群管理***，k8s可以构建一个容器的调度服务，目的是让用户通过k8s集群来进行云端容器集群的管理，而无需用户进行复杂的设置工作。***会自动选取合适的工作节点来执行具体的容器集群调度处理工作。其核心概念是容器仓(container pod)。一个容器集合(pod)由工作在同一工作节点上的一组容器构成。pod是k8s的最基本的部署单元，一个pod可以封装一个或多个容器(container)、存储资源(volume)、一个独立的网络IP以及管理控制容器运行方式的策略选项。pod在逻辑上可用于标识某种应用的一个实例，当一个pod封装多个容器，通常这种场景下的多个容器包含一个主容器和几个辅助容器(SideCar container)。如一个web应用由前端、后端和数据库三个组件构成，这三个组件运行在各自的容器中，主容器可以为web应用前端，针对这个实例可以包含三个容器的pod。
本申请实施例的技术方案可以应用于各种云化的通信***，例如：长期演进(longterm evolution，LTE)***，全球互联微波接入(worldwide interoperability formicrowave access，WiMAX)通信***，未来的第五代(5th Generation，5G)***，如新一代无线接入技术(new radio access technology，NR)，及未来的通信***等。
为便于理解本申请实施例，接下来对本请的应用场景进行介绍，本申请实施例描述的业务场景是为了更加清楚的说明本申请实施例的技术方案，并不构成对于本申请实施例提供的技术方案的限定，本领域普通技术人员可知，随着新业务场景的出现，本申请实施例提供的技术方案对于类似的技术问题，同样适用。
如图1所示，为本申请一实施例提供的云计算***的架构示意图。
该实施例包括管理器(master)101和节点(node)102。节点102可以是虚拟计算设备，也可以是物理的计算设备，一个节点102上可以部署若干个pod。 管理器101是容器集群管理***的中心管理模块，管理器101可以看作是对容器生命周期进行管理的一组进程。例如管理器101为k8s master，管理器101可以包括控制模块(controller)、调度模块(scheduler)、应用程序编程接口服务(applicationprogramming interface server，API server)模块等。这些进程实现了整个计算设备集群的资源管理、pod部署、***建立等管理功能。API server模块提供了资源对象的唯一操作入口，即为用户提供功能的接口模块，其他所有组件通过其提供的API接口来操作资源数据，通过对相关的资源数据监听，完成相关的网元功能。控制模块负责对各种容器模型的统一管控，例如对容器模型进行CRUD(增加Create、读取查询Read、更新Update和删除Delete)操作。容器模型例如可以指示以下信息中的一种或多种：pod中包括的容器的数量，容器中运行的应用程序的类型，容器在工作时使用的每种类型的资源的最大值，哪个容器需要独占cpu等信息。调度模块负责为部署的单元(容器或pod)选择合适的节点102等。
管理器101可以运行在计算设备集群中的某一个节点102上，或者运行在计算设备集群中的若干个节点102上(处于高可用性目的)。
节点102主要用于负责运行容器，每个节点102还可以运行有kubelet、容器引擎(container engine)等组件，负责对本节点上的pod的生命周期进行管理。kubelet用于处理管理器101下发到本节点的任务，管理pod及pod中的容器。Kubelet可以在应用程序编程接口服务模块上注册本节点自身的信息，定期向管理器101汇报本节点资源的使用情况，并可通过cAdvisor监控容器和节点资源。容器引擎可以负责部署容器，容器引擎例如可以是Docker组件。
一个微服务可以基于一个或多个容器维持运行。微服务是一种云原生架构方法，其中单个应用程序由许多松散耦合且可独立部署的较小组件或服务组成。每个微服务可运行在其独立的进程中，微服务与微服务间可以采用轻量级的通信机制互相沟通(例如基于HTTP的RESTful API)。每个微服务可以围绕着具体业务进行构建，并且能够独立地部署到生产环境、类生产环境等。
在一些实施例，用户或者租户或者管理人员可以根据业务需求，向管理器101下发部署pod的指令。指令中可以包括：pod的数量，每个pod包含的容器的数量，每个容器在工作时使用的最小资源需求(Request)与资源的最大值(Limit)等信息。本申请实施例所称的资源可以包括CPU资源、内存资源、网络带宽资源、磁盘资源等。pod的资源Request或资源Limit是指该pod中所有容器的资源Request或资源Limit的总和。节点102中的pod可以通过pod的名称或者互联网协议(Internet Protocol，IP)进行区分，多个节点102之间亦可通过节点名称或者IP进行区分。
如图2所示，为本发明实施例提供的一种计算设备100，包括：处理器1001、存储器1002、总线1003、输入输出接口1004和通信接口1005。总线1003用于连接处理器1001、存储器1002、输入输出接口1004和通信接口 1005，并且在处理器1001、存储器1002、输入输出接口1004和通信接口1005之间实现数据传输。例如，处理器1001通过总线1003从输入输出接口1004接收到命令，解密接收到的命令，根据解密的命令执行计算或数据处理。存储器1002可以包括程序模块，例如内核(kernel)，中间件(middleware)，应用程序接口(AP)和应用等。程序模块可以由软件、固件、硬件或其中的至少两种组成。输入输出接口1004转发用户通过输入设备(例如感应器、键盘、触摸屏)输入的命令或数据。通信接口1005将计算设备100与其他的计算设备、网络进行连接。例如，通信接口1005可以通过有线或无线连接到网络以连接到外部其他的计算设备。可选的，计算设备100上还可以包括显示设备，用于向用户显示由用户输入的配置信息以及向用户显示操作界面等。
在一些实施例中，处理器1001可以包括中央处理器(central processing unit，CPU)、图形处理器(graphics processing unit，GPU)、微处理器(micro processor，MP)或者数字信号处理器(digital signal processor，DSP)等处理器中的任意一种或多种。
存储器1002包括易失性存储器(volatile memory)，例如随机存取存储器(random access memory，RAM)。存储器1002还可以包括非易失性存储器(non-volatile memory)，例如只读存储器(read-only memory，ROM)，快闪存储器，机械硬盘(hard disk drive，HDD)或固态硬盘(solid state drive，SSD)。
总线1003可以是外设部件互连标准(peripheral component interconnect，PCI)总线或扩展工业标准结构(extended industry standard architecture，EISA)总线等。总线可以分为地址总线、数据总线、控制总线等。为便于表示，图2中仅用一条线表示，但并不表示仅有一根总线或一种类型的总线。总线1003可包括在计算设备100各个部件(例如，存储器1002、处理器1001、通信接口1005)之间传送信息的通路。
通信接口1005使用例如但不限于网络接口卡、收发器一类的收发模块，来实现计算设备100与其他设备或通信网络之间的通信。
在一些实施例中，存储器1002中存储有可执行的程序代码，处理器1001执行该可执行的程序代码以分别实现下述图10所示的第一获取模块201、构建模块202、第一生成模块203和第一调整模块204的功能，从而实现图8所示的资源调度方法。也即，存储器1002上存有用于执行图8所示的资源调度方法的指令。处理器1001执行该可执行的程序代码还可以分别实现下述图11所示的第二获取模块301、创建模块302、第二生成模块303和第二调整模块304的功能，从而实现图9所示的资源调度方法。也即，存储器1002上存有用于执行图9所示的资源调度方法的指令。
在一些实施例中，计算设备集群1000包括至少一台计算设备，例如图3中的第一计算设备100A、第二计算设备100B。该计算设备100可以是服务器，例如是中心服务器、边缘服务器，或者是本地数据中心中的本地服务器。在一些实施例中，计算设备也可以是台式机、笔记本电脑或者智能手机等终端设备。
如图3所示，计算设备集群1000包括至少一个计算设备。计算设备集群中的一个或多个计算设中的存储器1002中可以存有相同的用于执行图8所示的资源调度方法的指令，或者图8所示的资源调度方法的指令。
在一些可能的实现方式中，该计算设备集群中的一个或多个计算设备的存储器1002中也可以分别存有用于执行图8所示的资源调度方法，或者图9所示的资源调度方法的部分指令。换言之，一个或多个计算设备的组合可以共同执行用于执行图8所示的资源调度方法的指令，或者图9所示的资源调度方法的指令。
需要说明的是，计算设备集群中的不同的计算设备中的存储器1002可以存储不同的指令，分别用于执行资源调度装置的部分功能。也即，不同的计算设备100中的存储器1002存储的指令可以实现第一获取模块201、构建模块202、第一生成模块203和第一调整模块204中的一个或多个模块的功能，或者实现第二获取模块301、创建模块302、第二生成模块303和第二调整模块304中的一个或多个模块的功能。
在一些可能的实现方式中，计算设备集群中的一个或多个计算设备可以通过网络连接。其中，所述网络可以是广域网或局域网等等。图3示出了一种可能的实现方式。如图3所示，两个计算设备(为了便于区分，以下分别称为第一计算设备100A、第二计算设备100B)之间通过网络进行连接。具体地，通过各个计算设备中的通信接口与所述网络进行连接。在这一类可能的实现方式中，第一计算设备100A中的存储器1002中可存有执行第一获取模块201与构建模块202的功能的指令。同时，第二计算设备100B中的存储器1002中可存储有执行第一生成模块203和第一调整模块204的功能的指令。同样地，第一计算设备100A中的存储器1002中还可存有执行第二获取模块301与创建模块302的功能的指令。同时，第二计算设备100B中的存储器1002中还可存储有执行第二生成模块303和第二调整模块304的功能的指令。
图3所示的计算设备集群之间的连接方式可以是考虑到本申请提供的资源调度方法需要大量地存储数据和读取数据，因此考虑将第一生成模块203和第一调整模块204实现的功能交由第二计算设备100B执行。
应理解，图3中示出的第一计算设备100A的功能也可以由多个计算设备100完成。同样，第二计算设备100B的功能也可以由多个计算设备100完成。
下面结合图4，示例性的介绍本申请的资源调度***的架构示意图。
在该实施例中，资源调度***10包括管理器101、节点、资源画像服务组件103、决策服务组件104、监控服务组件105及分页式配置组件106。本申请实施例不对节点的数量进行限定，可以根据实际业务部署需求设置节点的数量，图3以资源调度***10包括n个节点102_1～102_n进行举例说明，n为大于1的正整数，本申请并不对n的值进行限定。
n个节点102_1～102_n可以通过通信网络连接在一起，通信网络可以是 有线网络，也可以是无线网络。通信网络可使用任何已知的网络通信协议来实现，上述网络通信协议可以是各种有线或无线通信协议，诸如以太网、通用串行总线(universal serial bus，USB)、全球移动通讯***(global system for mobile communications，GSM)、通用分组无线服务(general packet radio service，GPRS)、码分多址接入(code divisionmultiple access，CDMA)、宽带码分多址(wideband code division multiple access，WCDMA)，时分码分多址(time-division code division multiple access，TD-SCDMA)，长期演进(long term evolution，LTE)、无线保真(wireless fidelity，Wi-Fi)、基于互联网协议的语音通话(voice over Internet protocol，VoIP)、支持网络切片架构的通信协议或任何其他合适的通信协议。
在一些实施例中，管理器101、资源画像服务组件103、决策服务组件104、监控服务组件105及分页式配置组件106可以运行在同一个节点上，或者运行在两个或两个以上的节点上。例如，管理器101运行在一个节点上，资源画像服务组件103、决策服务组件104、监控服务组件105及分页式配置组件106运行在另一个节点上。资源画像服务组件103、决策服务组件104、监控服务组件105及分页式配置组件106可以以插件的形式部署在节点上。
监控服务组件105可用于收集各个节点102_1～102_n的资源信息，各个节点102_1～102_n上部署的容器的资源占用信息、Request信息和Limit信息。节点的资源信息可以包括节点上的每种类型的资源的总量、已使用的资源数量、剩余的资源数量、每种类型的资源的使用率等信息。
各个节点102_1～102_n部署的pod的资源占用信息、Request信息和Limit信息可以基于pod中所包含的容器的资源占用信息、Request信息和Limit信息的总和得到。同样的，各个节点102_1～102_n部署的微服务的资源占用信息、Request信息和Limit信息也可以基于微服务所关联的容器的资源占用信息、Request信息和Limit信息的总和得到。
容器、pod、微服务的资源占用信息可以是指pod、容器、微服务所占用的每种类型的资源(CPU、内存、磁盘、网络带宽)数量。
在一些实施例中，监控服务组件105可以包括Prometheus组件。Prometheus组件为收集和聚合指定指标作为时间序列数据的工具。例如，监控服务组件105可以通过Prometheus组件实现实时收集各个节点102_1～102_n的资源信息、以及各个节点102_1～102_n上部署的容器的资源占用信息、Request信息和Limit信息。
各个节点102_1～102_n还可以部署有过载控制(Over Load Control，OLC)组件，通过OLC组件实时收集各个节点102_1～102_n上部署的微服务的资源占用信息，可以确定是否存在资源不足的微服务。当决策服务组件104基于OLC组件收集到的微服务的资源占用信息确定某个微服务存在突发流量导致资源不足时，可以触发对该微服务进行扩容，以满足微服务运行所需的资源需求，资源不足可以是指分配给微服务的CPU资源、内存资源、磁盘资源、网络带宽资源等中的一个或多个存在不足。当决策服务组件104基于OLC 组件收集到的微服务的资源占用信息确定某个微服务存在资源过剩时，还可以触发对该微服务进行缩容，以提升节点资源的利用率。
在一些实施例中，监控服务组件105还可以收集各个节点102_1～102_n与各个节点102_1～102_n上部署的微服务的运行指标数据，以实现获取节点与微服务的运行状态。例如，通过监控服务组件105可以实现确定节点是否运行异常(磁盘异常、网络异常等)，对微服务进行运行状态检查，以实现实时监控微服务在资源调整前或者资源调整后的运行状态。对于节点以及部署在节点上的微服务，可以根据实际需求设定监控服务组件105需收集节点及微服务的哪些运行指标数据，以实现获取节点与微服务的运行状态。
在一些实施例中，监控服务组件105还可以监听由管理器101触发的事件，例如由管理器101触发的微服务的创建、删除、升级，以及对微服务/pod的资源进行扩容或者缩容等事件，监控服务组件105可将监听到的事件上报给决策服务组件104，由决策服务组件104进行决策并下发指令。监控服务组件105还可以将监听到的事件通过短信或者邮件等方式通知运维人员，以便运维人员及时获知计算设备集群的变化，例如资源调度***10还包括消息中心107，监控服务组件105还可以将监听到的事件通知至消息中心107，由消息中心107通过短信或者邮件通知运维人员。
在一些实施例中，资源画像服务组件103用于接收监控服务组件105收集到的各个节点102_1～102_n的资源信息，各个节点102_1～102_n上部署的容器的资源占用信息、Request信息和Limit信息等。资源画像服务组件103可对监控服务组件105收集到的各个信息进行汇总与分析，实现对各个微服务进行资源画像。通过监控服务组件105可以实现将节点资源、容器、pod以及微服务的资源占用历史数据收集起来，并由资源画像服务组件103进行汇总与分析，生成微服务的资源画像，使得后续在对微服务/pod进行资源调整时可以做到有的放矢。
在一些实施例中，资源画像可以包括资源占用维度与时间维度。资源占用维度可以表征微服务占用的资源的特征，例如微服务占用的CPU、内存、网络带宽、磁盘IO等资源的特征。时间维度结合资源占用维度可以表征微服务在不同的时间所占用的资源的特征。例如，互联网属性的微服务的流量会受到人们生产生活影响，呈现出明显的高低峰、周期性以及可预测性，或者定时的微服务亦具有周期性以及可预测性，本地生活类微服务会存在午间和晚间高峰，电商类微服务在促销阶段的流量波峰会是波谷的数倍之多等。若资源调度***10可以捕获到这些信息，可以根据节点和微服务状态灵活调配资源，实现微服务资源的动态分配，最大程度满足微服务的服务等级协议(Service Level Agreement，SLA)要求。
在一些实施例中，资源画像服务组件103可以集成有数据处理模型来实现对监控服务组件105收集到的信息进行汇总与分析。例如，数据处理模型可以采用分位值算法(针对微服务资源需求的分位值)对微服务进行资源画像。数据处理模型也可以采用其他的人工智能算法实现对微服务进行资源画 像，本申请对此不作限定。
资源画像服务组件103可以将构建的资源画像以及监控服务组件105收集到的信息一并上报至决策服务组件104。决策服务组件104可以基于资源画像服务组件103构建的资源画像、监控服务组件105收集的信息、以及节点的运行状态进行资源调度决策。例如，若某个节点处于异常运行状态(磁盘异常、网络异常等)，对该节点进行调度隔离，即决策服务组件104不会对该节点下的满足资源调整条件的pod进行资源调整，待该节点恢复正常运行后，决策服务组件104再对该节点下满足资源调整条件的pod进行资源调整。满足资源调整条件的pod可以是指与需要进行扩/缩容的微服务关联的pod。
例如，决策服务组件104可以根据微服务的资源画像预测某个微服务在某个时段需要进行扩容，进而可以提前预设时间下发对该微服务的资源调整指令，以满足该微服务的业务处理需求，例如该微服务为生活类微服务，并在晚间七点至九点具有流量高峰，决策服务组件104可以提前5分钟对该生活类微服务对资源调整，资源调整的具体数量可以基于资源画像进行预估并设定。例如，某个微服务为电商类微服务，并计划在某个时间段进行大促销，决策服务组件104可以在大促销开始之前下发对该微服务的资源调整指令，以满足该微服务的业务处理需求。资源调整的具体数量可以基于历史大促销的流量数据进行预估并设定。
决策服务组件104也可以在OLC检测到某个微服务的流量增大导致资源临近过载时，下发针对该微服务的资源调整指令，以满足该微服务的业务处理需求。决策服务组件104通过分页式配置组件106下发资源调整指令至节点，并通过节点上的docker组件对满足资源调整条件的pod进行资源调整，可以实现秒级资源调整，相比通过HPA/VPA进行资源调整迟滞性更小。
在一些实施例中，在对微服务进行资源调整时，可以对与微服务关联的所有pod进行资源调整，也可以精确到对微服务中的某个或某几个pod进行资源调整，即调整的pod可以使得该微服务具有足够的资源应对当前的业务量/流量增大问题即可。
在一些实施例中，对于满足资源调整条件的pod，决策服务组件104可以自动下发资源调整指令至分页式配置组件106，以触发对满足资源调整条件的pod进行资源调整。对于满足资源调整条件的pod，决策服务组件104也可以给出相应的资源调整建议，并通过消息中心107将资源调整建议通知至运维人员进行决策，即可以由运维人员手动下发资源调整指令至分页式配置组件106，以触发对满足资源调整条件的pod进行资源调整。资源调整指令可以包括需要进行资源调整的pod以及pod的资源调整方式，pod的资源调整方式可以是指将pod的资源Limit增加a1，或者将pod的资源Limit减少a2，或者将pod的资源Limit设置为a3等方式。a1、a2、a3的值均可以根据实际需求进行设定。可以理解的，a1、a2、a3均可以是由CPU、内存、磁盘IO、网络带宽等资源的设定值组成的集合。
分页式配置组件106可用于实现将资源调整指令下发至节点，节点上的kubelet可以监听分页式配置组件106下发的资源调整指令，并判断是否是对本节点上的pod进行资源调整。若kubelet确定不是对本节点上的pod进行资源调整时，可以忽略该资源调整指令。若kubelet确定是对本节点上的pod进行资源调整时，kubelet可以基于资源调整指令继续判断是对本节点上的哪些pod进行资源调整。对于需要进行资源调整的pod，kubelet可以控制pod调用unix-socket与docker进程(docker组件的进程)进行通信，通过调用docker API进行资源调整。
在一些实施例中，分页式配置组件106可以为ZooKeeper，ZooKeeper可以接受监听者的注册，一旦某些数据发生变化，ZooKeeper可以通知已经在ZooKeeper上注册的监听者做出相应的反应。即节点上的kubelet可以在ZooKeeper进行注册，实现监听ZooKeeper下发的资源调整指令。
在一些实施例中，docker组件可以实现为每个进程设定限制的CPU、内存、磁盘、网络带宽等资源，进而可以实现设置进程访问资源的Limit，docker组件底层通过控制组群(Control grpups，Cgroups)实现。Cgroups是Linux内核的一个功能，可用来限制、控制与分离一个进程组的资源。Cgroups中的资源控制是以控制族群为单位实现，一个进程可以加入到某个控制族群，也可以从一个进程组迁移到另一个控制族群。一个进程组的进程可以使用Cgroups以控制族群为单位分配资源，同时受到Cgroups以控制族群为单位设定的限制。Cgroups可以实现限制进程组可以使用的资源数量、进程组的优先级控制、记录进程组使用的资源数量、进程组隔离及进程组控制等功能。当在节点上部署docker组件，默认情况下会在节点/sys/fs/cgroup/cpu/kubebpods/burstable/podxxx目录下挂载Cgroups信息，使得每个进程都会有单独的配置信息，可以通过调用docker API来更新进程的配置信息，实现对pod进行资源调整。
在一些实施例中，各个微服务通过预设方式配置不同的优先级，例如可以通过微服务的Annotation选项配置自身的优先级。当节点资源紧张时，可以由资源画像服务组件103获取各个微服务的优先级，并将优先级信息传送给决策服务组件104。决策服务组件104可以根据微服务的优先级结合微服务的资源使用率数据，释放低优先级的微服务的资源给高优先级的微服务。若无法从低优先级的微服务中腾挪出资源，决策服务组件104还可以基于微服务的资源画像，将当前处于未活跃(休眠状态)的低优先级的微服务调度到其他节点上，以使得高优先级的微服务可以分配到更多的资源。例如，当前时刻为夜间时刻，可以将仅日间活跃而夜间处于休眠状态的低优先级的微服务调度到其他节点上，进而既可以为高优先级的微服务腾挪更多的资源，又不影响该低优先级的微服务的运行。
例如，在节点102_1上部署了多个微服务，多个微服务中包括计算类微服务，且计算类微服务配置了最高的优先级。若与计算类微服务关联的pod在运行过程中，由于CPU的Limit偏小，导致限制了进程的运行速度，或者 内存的使用率快要接近Limit等情形，资源画像服务组件103可以获取节点102_1上的各个微服务的优先级，决策服务组件104可以结合节点102_1的资源信息以及各个微服务的资源占用信息，通过决策分析得到可以多分配给与计算类微服务关联的pod的资源(例如CPU或者内存)，进而可以加快计算类微服务的计算速度，实现节点资源的充分利用。
在一些实施例中，决策服务组件104也可以在OLC检测到某个微服务的流量增大导致资源临近过载时，下发pod伸缩请求至管理器101，管理器101可以通过HPA和/或VPA实现对满足资源调整条件的pod进行资源调整。通过HAP/VPA对微服务进行资源时，HAP/VPA会收集微服务最近一预设时间段的资源指标并计算平均值，及将平均值与目标值进行比较，再根据比较结果进行资源调整，因此通过HAP/VPA进行资源调整会存在响应滞后的问题；另外由于HPA/VPA的默认扩容冷却周期为3分钟，会进一步放大响应滞后的问题。
下面结合图5，示例性的介绍本申请在微服务流量变化场景进行资源调度的交互流程示意图。
该实施例的资源调度的应用场景以微服务S1存在流量增大的情形为例进行说明。例如，微服务S1为电商类微服务，电商类微服务在某个时刻开启商品秒杀活动或者某种类型商品的电商直播而导致流量增加较大。在该实施例中，以资源调度***10包括管理器101、节点102_1、节点102_2、资源画像服务组件103、决策服务组件104、监控服务组件105、分页式配置组件106及消息中心107为例。
在一些实施例中，管理器101、资源画像服务组件103、决策服务组件104、监控服务组件105、分页式配置组件106及消息中心107均可以部署在节点102_1上。监控服务组件105可以包括Prometheus组件，节点102_1上可以部署有OLC组件。假设微服务S1部署在节点102_1上，与微服务S1关联的pod为pod_1，资源调度***10对微服务S1实现资源调度的内部交互流程包括：
I1、Prometheus组件收集节点102_1的资源信息、节点102_1上部署的容器的资源占用信息、Request信息和Limit信息，并上报给资源画像服务组件103。
I2、OLC组件收集节点102_1上部署的微服务S1的资源占用信息，并上报给决策服务组件104。
I3、资源画像服务组件103基于Prometheus组件收集的信息构建微服务S1的资源画像，将微服务S1的资源画像以及Prometheus组件收集的信息一并上报给决策服务组件104。
I4、若决策服务组件104基于OLC组件收集的微服务S1的资源占用信息，确定微服务S1需进行资源调整，决策服务组件104基于微服务S1的资源画像、Prometheus组件收集的信息，下发资源调整指令至分页式配置组件106，或者下发pod伸缩请求至管理器101。
例如，若OLC组件收集到微服务S1的资源占用信息临近Limit或者已达到Limit，决策服务组件104可以确定微服务S1存在流量增加较大，为了确保微服务S1可以正常运行，需要对微服务S1进行资源调整，以确保微服务S1具有足够的资源维持正常运行。
例如，若OLC组件收集到微服务S1的资源占用信息与Limit相差较大，决策服务组件104可以确定微服务S1存在过多的空闲资源，为了确保节点102_1上的资源可以被充分利用，需要对微服务S1进行资源调整，以使得节点102_1具有足够的剩余资源分配给其他资源紧张的微服务。
I5、若决策服务组件104下发资源调整指令至分页式配置组件106，Kubelet监听分页式配置组件106传送的资源调整指令，控制pod_1调用unix-socket与docker进程进行通信，以调用docker API对pod_1进行资源调整。
例如，调用docker API对pod_1进行资源调整可以是指调用docker API调整pod_1的Cgroup参数，进而实现调整pod_1的资源。
I6、若决策服务组件104下发pod伸缩请求至管理器101，管理器101通过HPA和/或VPA对pod_1进行资源调整。
在一些实施例中，Prometheus组件还可以监听pod_1的资源调整结果，并通知至消息中心107，使得消息中心107可以将pod_1的资源调整结果通过短信或者邮件通知至运维人员。
在一些实施例中，Prometheus组件还可以在对pod_1进行资源调整后，监听微服务S1的运行状态，并反馈给决策服务组件104，通过Prometheus组件对微服务S1进行运行状态检查，以确定调整后的资源是否满足微服务S1的运行。若微服务S1仍然因为资源不足导致运行异常时，可以再次触发决策服务组件104对微服务S1进行资源调整。
下面结合图6，示例性的介绍本申请在微服务启动阶段进行资源调度的交互流程示意图。
该实施例的资源调度的应用场景以启动微服务S1为例进行说明。在该实施例中，以资源调度***10包括管理器101、节点102_1、节点102_2、资源画像服务组件103、决策服务组件104、监控服务组件105及分页式配置组件106为例。微服务S1可以为电商类微服务、计算类微服务、生活类微服务等，本实施例对此不作限定。
微服务S1在启动时，往往会进行大量的初始化工作，比如进行镜像文件拉取、Tomcat容器启动、Spring MVC/SpringBoot初始化、实例化Bean、实例化Service、实例化容器引擎底座组件等操作，启动时间与微服务的规模(如微服务的代码量)、Restful接口的数量、外部依赖的组件数量成正比，与容器的资源规格成反比。微服务S1的启动时间越短，通过HPA或者VPA为微服务S1生成的Pod就会越快就绪，从而可以提升集群的稳定性与吞吐量。
假设微服务S1部署在节点102_1上，与微服务S1关联的pod为pod_1，资源调度***10对微服务S1实现资源调度的内部交互流程包括：
I11、监控服务组件105收集节点102的资源信息、节点102_1上部署的容器的资源占用信息、Request信息和Limit信息，并传送给资源画像服务组件103。
例如，监控服务组件105包括Prometheus组件，可以通过Prometheus组件收集节点102的资源信息、节点102_1上部署的容器的资源占用信息、Request信息和Limit信息。
I12、资源画像服务组件103对监控服务组件105收集的各个信息进行聚合，并将信息聚合结果传送至决策服务组件104。
I13、决策服务组件104在监听到pod_1的创建事件时，分析并计算出节点102_1当前可以多分配给pod_1的资源，并下发资源调整指令至分页式配置组件106。
在一些实施例中，管理器101在创建pod_1时，设定了pod_1的资源Request与资源Limit。为了实现缩短微服务S1的启动时间，可以对资源Limit进行扩增，即通过计算出节点102_1当前可以多分配给pod_1的资源实现对pod_1的资源Limit进行扩增。
在一些实施例中，监控服务组件105可以监听管理器101创建pod_1的事件，及将pod_1的创建事件通知至决策服务组件104。
例如，如图7所示，可以将③的未使用的资源分配给pod_1，使得pod_1的资源由管理器101设定的①变为②，以达到快速启动微服务S1的目的。监控服务组件105通过运行状态检测确定微服务S1启动成功后，决策服务组件104再将多分配给pod_1的资源进行回收，即pod_1的资源再次由②变为①。
例如，多分配给pod_1的资源可以包括将pod_1中的CPU的Limit由2核变更为4核，将内存的Limit由b1MB变更为b2MB等，其中b2大于b1。
I14、分页式配置组件106将资源调整指令下发至pod服务，pod服务控制pod_1调用unix-socket与docker进程进行通信，以调用docker API对pod_1进行资源调整。
在一些实施例中，pod服务可以是指节点上用于对pod进行管理的组件，例如Kubelet。
在一些实施例中，各个微服务的资源Request和资源Limit、资源画像服务组件103聚合的信息、决策服务组件104决策的信息等都可以存储到预设数据库中，同时预设数据库可以记录各个pod的唯一标识(例如，名称或者IP)与分配的资源，若资源画像服务组件103/决策服务组件104出现异常，可以从预设数据库中获取正确的资源信息，可以避免对pod进行超分或者超回收资源。
参照图8所示，本申请一实施例提供一种资源调度方法，可应用于计算设备集群。本实施例中，资源调度方法可以包括：
步骤S81、获取集群中的每个节点的资源信息，以及每个节点中的容器的资源信息，每个节点均安装有容器引擎。
在一些实施例中，集群中可以通过部署Prometheus组件与OLC组件实现实时收集各个节点的资源信息、以及各个节点上部署的容器的资源占用信息、Request信息和Limit信息。容器引擎可以是指docker组件，通过在每个节点部署docker组件，可以使得每个节点均可通过docker组件实现快速进行微服务的资源调整。
在一些实施例中，资源信息可以包括每种类型的资源的总量、已使用的资源数量、剩余的资源数量、每种类型的资源的使用率等信息。本申请实施例所称的资源可以包括CPU资源、内存资源、网络带宽资源、磁盘资源等。
步骤S82、基于每个节点的资源信息和每个节点中的容器的资源信息，构建在每个节点上部署的微服务的资源画像，每个节点上部署的微服务基于一个或多个容器运行。
在一些实施例中，当获取到集群中的每个节点的资源信息，以及每个节点中的容器的资源信息之后，可以基于每个节点的资源信息和每个节点中的容器的资源信息，构建在每个节点上部署的微服务的资源画像。例如，可以先基于每个节点中的容器的资源信息，得到在每个节点上部署的微服务的资源信息，再基于每个节点的资源信息及每个节点上部署的微服务的资源信息，构建在每个节点上部署的微服务的资源画像。在资源画像中，可以采用时间维度结合资源占用维度表征微服务在不同的时间所占用的资源的特征，进而后续在对微服务进行资源调整时，可以基于微服务的资源画像进行资源分配，使得微服务的资源调整更有针对性，可提高节点的资源利用率。
步骤S83、若确定在集群中部署的第一微服务需进行资源调整，基于第一微服务的资源画像、部署第一微服务的第一节点的资源信息以及第一节点中的容器的资源信息，生成与第一微服务关联的资源调整指令。
在一些实施例中，可以将资源不足的微服务或者资源过剩的微服务确定为需要进行资源调整的第一微服务，例如若第一微服务占用的资源与第一微服务的资源上限值之差小于第一预设阈值，表明第一微服务的资源紧张，通过触发对第一微服务进行扩容，以满足第一微服务运行所需的资源需求，若第一微服务占用的资源与第一微服务的资源上限值之差大于第二预设阈值，表明第一微服务存在资源过剩时，通过触发对第一微服务进行缩容，使得过剩的资源可以分配给其他资源不足的微服务。
例如，第一微服务占用的资源包括处理器资源、内存资源、磁盘资源以及网络带宽资源中的一种或多种，若第一微服务占用的处理器资源、内存资源、磁盘资源以及网络带宽资源中的任意一者与对应的资源上限值之差小于对应的第一预设阈值，表明微服务的某项或者某几项资源紧张，确定微服务进行扩容。若第一微服务占用的处理器资源、内存资源、磁盘资源以及网络带宽资源中的任意一者与对应的资源上限值之差大于对应的第二预设阈值，表明微服务的某项或者某几项资源过剩，确定微服务的需进行缩容。
在生成与第一微服务关联的资源调整指令，可以通过第一微服务的资源画像预测微服务所需的资源数量，再基于预测的资源数量、部署第一微服务 的节点的资源信息以及节点中的容器的资源信息为第一微服务分配资源，使得微服务的资源调整更加准确，避免出现资源分配过多或者资源分配不足，提高节点的资源利用率。在生成与第一微服务关联的资源调整指令时，可以选择对与微服务关联的所有pod进行资源调整，也可以精确到对微服务中的某个或某几个pod进行资源调整，调整的pod可以使得该微服务具有足够的资源应对当前的业务量/流量增大问题即可。
步骤S84、基于容器引擎及资源调整指令，调整与第一微服务关联的pod的资源。
在对微服务进行资源调整时，可以先确定资源调整指令是对与微服务关联的所有pod进行资源调整，或者是对微服务中的某个或某几个pod进行资源调整，通过容器引擎调用资源调整应用程序接口对指定的pod进行资源调整，资源调整迟滞性小，可以实现秒级资源调整，增强集群中部署的微服务应对突发流量的能力。
例如，容器引擎为docker组件，docker组件可以实现为每个进程设定限制的CPU、内存、磁盘、网络带宽等资源，进而可以实现设置进程访问资源的Limit，docker组件底层通过Cgroups实现。Cgroups可以实现限制进程组可以使用的资源数量、进程组的优先级控制、记录进程组使用的资源数量、进程组隔离及进程组控制等功能。当在节点上部署docker组件，默认情况下可以在节点/sys/fs/cgroup/cpu/kubebpods/burstable/podxxx目录下挂载Cgroups信息，使得每个进程都会有单独的配置信息，容器引擎可以通过响应资源调整指令调用docker API来更新进程的配置信息，实现对第一微服务关联的pod进行资源调整。
在一些实施例中，每个节点上部署的微服务可以通过预设方式配置不同的优先级，当对节点上部署的第一微服务进行资源调整时，若节点资源紧张无法满足第一微服务的资源调整需求，可以释放节点上部署的比第一微服务的优先级低的某个微服务的资源，使得节点有足够空闲的资源分配给第一微服务。例如，节点上部署的比第一微服务的优先级低的微服务为第二微服务，可以根据第二微服务的资源画像，释放第二微服务占用的部分资源，基于第二微服务的资源画像可以确定预估第二微服务当前可以释放多少资源，又不影响第二微服务的运行。
在一些实施例中，若无法从低优先级的微服务中腾挪出资源，或者不采用从低优先级的微服务中腾挪出资源的策略，还可以将节点上部署的比第一微服务的优先级低的某个微服务迁移到其他节点上，使得节点有足够空闲的资源分配给需要进行资源调整的微服务。例如，将节点上部署的比第一微服务的优先级低的第三微服务迁移到集群的其他节点上，使得节点有足够空闲的资源分配给需要进行资源调整的微服务。
在一些实施例中，可以基于节点上部署的微服务的资源画像，选择可调度至集群中的其他节点的第三微服务，例如，当前时刻为夜间时刻，可以选择将仅日间活跃而夜间处于休眠状态的低优先级的第三微服务调度到其他 节点上，进而既可以为高优先级的第一微服务腾挪更多的资源，又不影响该低优先级的第三微服务的运行。
参照图9所示，本申请另一实施例提供一种资源调度方法，可应用于计算设备集群。本实施例中，资源调度方法可以包括：
步骤S91、获取集群中的每个节点的资源信息，以及每个节点中的容器的资源信息，每个节点均安装有容器引擎。
在一些实施例中，集群中可以通过部署Prometheus组件与OLC组件实现实时收集各个节点的资源信息、以及各个节点上部署的容器的资源占用信息、Request信息和Limit信息。容器引擎可以是指docker组件，通过在每个节点部署docker组件，可以使得每个节点均可通过docker组件实现快速进行微服务的资源调整。
在一些实施例中，资源信息可以包括每种类型的资源的总量、已使用的资源数量、剩余的资源数量、每种类型的资源的使用率等信息。本申请实施例所称的资源可以包括CPU资源、内存资源、网络带宽资源、磁盘资源等。
步骤S92、响应于在集群中部署的第一微服务的启动指令，为第一微服务创建第一pod，第一pod设置有第一资源上限值。
第一微服务可以是指集群中部署的任意一个微服务。集群启动第一微服务时，可以为第一微服务创建第一pod，通过第一pod来维持第一微服务的运行，集群为第一pod设置第一资源上限值(Limit)。
步骤S93、基于部署第一微服务的第一节点的资源信息以及第一节点中的容器的资源信息，生成与第一pod关联的资源调整指令。
在微服务启动过程中，基于集群中的部署的第一微服务的第一节点的资源信息及第一节点中的容器的资源信息，来调高***为微服务分配的资源占用上限，以达到快速启动微服务，缩短微服务的启动时间。
例如，可以第一节点中未使用的空闲资源分配给第一pod，实现调高***为微服务分配的资源占用上限，以达到快速启动第一微服务的目的。
步骤S94、基于容器引擎及资源调整指令将第一pod的资源占用上限由第一资源上限值调整为第二资源上限值，第二资源上限值大于第一资源上限值。
通过容器引擎调用资源调整应用程序接口将第一pod的资源占用上限由第一资源上限值调整为第二资源上限值，资源调整迟滞性小，可以实现秒级资源调整。例如，容器引擎为docker组件，docker组件底层通过Cgroups实现。当在节点上部署docker组件，默认情况下可以在节点/sys/fs/cgroup/cpu/kubebpods/burstable/podxxx目录下挂载Cgroups信息，使得每个进程都会有单独的配置信息，容器引擎可以通过响应资源调整指令调用docker API来将第一pod的资源占用上限由第一资源上限值调整为第二资源上限值。
在一些实施例中，第一微服务启动成功之后，可以将第一pod的资源占用上限由第二资源上限值调整为第一资源上限值。即在第一微服务启动完成之 后，将先前为了缩短第一微服务的启动时间而多分配给第一微服务的资源进行回收，避免第一微服务资源过剩，可提高节点的资源利用率。
参照图10所示，本申请一实施例提供一种第一资源调度装置20。第一资源调度装置20可以包括第一获取模块201、构建模块202、第一生成模块203和第一调整模块204。
第一获取模块201，用于获取集群中的每个节点的资源信息，以及每个节点中的容器的资源信息。每个节点可以均安装有容器引擎。
构建模块202，用于基于每个节点的资源信息和每个节点中的容器的资源信息，构建在每个节点上部署的微服务的资源画像。每个节点上部署的微服务可以基于一个或多个容器运行。
第一生成模块203，用于在集群中部署的第一微服务为需进行资源调整时，基于第一微服务的资源画像、部署第一微服务的第一节点的资源信息以及第一节点中的容器的资源信息，生成与第一微服务关联的资源调整指令。
第一调整模块204，用于基于容器引擎及资源调整指令调整与第一微服务关联的pod的资源。
其中，第一获取模块201、构建模块202、第一生成模块203和第一调整模块204均可以通过软件实现，或者可以通过硬件实现。示例性的，接下来以第一获取模块201为例，介绍第一获取模块201的实现方式。类似的，构建模块202、第一生成模块203和第一调整模块204的实现方式可以参考第一获取模块201的实现方式。
模块作为软件功能单元的一种举例，第一获取模块201可以包括运行在计算实例上的代码。其中，计算实例可以包括物理主机(计算设备)、虚拟机、容器中的至少一种。进一步地，上述计算实例可以是一台或者多台。例如，第一获取模块201可以包括运行在多个主机/虚拟机/容器上的代码。需要说明的是，用于运行该代码的多个主机/虚拟机/容器可以分布在相同的区域(region)中，也可以分布在不同的region中。进一步地，用于运行该代码的多个主机/虚拟机/容器可以分布在相同的可用区(availability zone，AZ)中，也可以分布在不同的AZ中，每个AZ包括一个数据中心或多个地理位置相近的数据中心。其中，通常一个region可以包括多个AZ。
同样，用于运行该代码的多个主机/虚拟机/容器可以分布在同一个虚拟私有云(virtual private cloud，VPC)中，也可以分布在多个VPC中。其中，通常一个VPC设置在一个region内，同一region内两个VPC之间，以及不同region的VPC之间跨区通信需在每个VPC内设置通信网关，经通信网关实现VPC之间的互连。
模块作为硬件功能单元的一种举例，第一获取模块201可以包括至少一个计算设备，如服务器等。或者，第一获取模块201也可以是利用专用集成电路(application-specific integrated circuit，ASIC)实现、或可编程逻辑器件(programmable logic device，PLD)实现的设备等。其中，上述PLD可以是复杂程序逻辑器件(complex programmable logical device，CPLD)、现场可 编程门阵列(field-programmable gate array，FPGA)、通用阵列逻辑(generic array logic，GAL)或其任意组合实现。
第一获取模块201包括的多个计算设备可以分布在相同的region中，也可以分布在不同的region中。第一获取模块201包括的多个计算设备可以分布在相同的AZ中，也可以分布在不同的AZ中。同样，第一获取模块201包括的多个计算设备可以分布在同一个VPC中，也可以分布在多个VPC中。其中，所述多个计算设备可以是服务器、ASIC、PLD、CPLD、FPGA和GAL等计算设备的任意组合。
需要说明的是，在其他实施例中，第一获取模块201可以用于执行图8所示的资源调度方法中的任意步骤，构建模块202可以用于执行图8所示的资源调度方法中的任意步骤，第一生成模块203可以用于执行图8所示的资源调度方法中的任意步骤，第一调整模块204可以用于执行图8所示的资源调度方法中的任意步骤，第一获取模块201、构建模块202、第一生成模块203和第一调整模块204负责实现的步骤可根据需要指定，通过第一获取模块201、构建模块202、第一生成模块203和第一调整模块204分别实现图8所示的资源调度方法中不同的步骤来实现第一资源调度装置20的全部功能。
参照图11所示，本申请另一实施例提供一种第二资源调度装置30。第二资源调度装置30包括第二获取模块301、创建模块302、第二生成模块303和第二调整模块304。
第二获取模块301，用于获取集群中的每个节点的资源信息，以及每个节点中的容器的资源信息。每个节点可以均安装有容器引擎。
创建模块302，用于响应于在集群中部署的第一微服务的启动指令，为第一微服务创建第一pod。第一pod可以设置有第一资源上限值。
第二生成模块303，用于基于部署第一微服务的第一节点的资源信息以及第一节点中的容器的资源信息，生成与第一pod关联的资源调整指令。
第二调整模块304，用于基于容器引擎及资源调整指令将第一pod的资源占用上限由第一资源上限值调整为第二资源上限值。第二资源上限值可以大于第一资源上限值。
其中，第二获取模块301、创建模块302、第二生成模块303和第二调整模块304均可以通过软件实现，或者可以通过硬件实现。示例性的，接下来以第二获取模块301为例，介绍第二获取模块301的实现方式。类似的，创建模块302、第二生成模块303和第二调整模块304的实现方式可以参考第二获取模块301的实现方式。
模块作为软件功能单元的一种举例，第二获取模块301可以包括运行在计算实例上的代码。其中，计算实例可以包括物理主机(计算设备)、虚拟机、容器中的至少一种。进一步地，上述计算实例可以是一台或者多台。例如，第二获取模块301可以包括运行在多个主机/虚拟机/容器上的代码。需要说明的是，用于运行该代码的多个主机/虚拟机/容器可以分布在相同的区域(region)中，也可以分布在不同的region中。进一步地，用于运行该代码的 多个主机/虚拟机/容器可以分布在相同的可用区(availability zone，AZ)中，也可以分布在不同的AZ中，每个AZ包括一个数据中心或多个地理位置相近的数据中心。其中，通常一个region可以包括多个AZ。
同样，用于运行该代码的多个主机/虚拟机/容器可以分布在同一个VPC中，也可以分布在多个VPC中。其中，通常一个VPC设置在一个region内，同一region内两个VPC之间，以及不同region的VPC之间跨区通信需在每个VPC内设置通信网关，经通信网关实现VPC之间的互连。
模块作为硬件功能单元的一种举例，第二获取模块301可以包括至少一个计算设备，如服务器等。或者，第二获取模块301也可以是利用ASIC实现、或PLD实现的设备等。其中，上述PLD可以是CPLD、FPGA、GAL或其任意组合实现。
第二获取模块301包括的多个计算设备可以分布在相同的region中，也可以分布在不同的region中。第二获取模块301包括的多个计算设备可以分布在相同的AZ中，也可以分布在不同的AZ中。同样，第二获取模块301包括的多个计算设备可以分布在同一个VPC中，也可以分布在多个VPC中。其中，所述多个计算设备可以是服务器、ASIC、PLD、CPLD、FPGA和GAL等计算设备的任意组合。
需要说明的是，在其他实施例中，第二获取模块301可以用于执行图9所示的资源调度方法中的任意步骤，创建模块302可以用于执行图9所示的资源调度方法中的任意步骤，第二生成模块303可以用于执行图9所示的资源调度方法中的任意步骤，第二调整模块304可以用于执行图9所示的资源调度方法中的任意步骤，第二获取模块301、创建模块302、第二生成模块303和第二调整模块304负责实现的步骤可根据需要指定，通过第二获取模块301、创建模块302、第二生成模块303和第二调整模块304分别实现图9所示的资源调度方法中不同的步骤来实现第二资源调度装置30的全部功能。
本申请实施例还提供一种计算机存储介质，所述计算机存储介质中存储有计算机指令，当所述计算机指令被计算设备集群运行时，使得计算设备集群执行上述相关方法步骤实现上述实施例中的资源调度方法。
本申请实施例还提供了一种计算机程序产品，当所述计算机程序产品在计算设备集群上运行时，使得计算设备集群执行上述相关步骤，以实现上述实施例中的资源调度方法。
另外，本申请实施例还提供一种装置，这个装置具体可以是芯片，组件或模块，所述装置可包括相连的处理器和存储器；其中，存储器用于存储计算机执行指令，所述装置具有实现上述实施例中提供的资源调度方法中计算设备集群行为的功能。功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。硬件或软件包括一个或多个与上述功能相对应的模块。
其中，本申请实施例提供的计算机存储介质、计算机程序产品或芯片均用于执行上文所提供的对应的方法，因此，其所能达到的有益效果可参考上文所提供的对应的方法中的有益效果，此处不再赘述。
通过以上的实施方式的描述，所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将装置的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。
在本申请所提供的几个实施例中，应所述理解到，所揭露的装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例是示意性的，例如，所述模块或单元的划分，为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个装置，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是一个物理单元或多个物理单元，即可以位于一个地方，或者也可以分布到多个不同地方。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个可读取存储介质中。基于这样的理解，本申请实施例的技术方案本质上或者说对现有技术做出贡献的部分或者所述技术方案的全部或部分可以以软件产品的形式体现出来，所述软件产品存储在一个存储介质中，包括若干指令用以使得一个设备(可以是单片机，芯片等)或处理器(processor)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何在本申请揭露的技术范围内的变化或替换，都应涵盖在本申请的保护范围之内。

Claims (19)

  1. 一种资源调度方法，其特征在于，所述方法包括：
    获取集群中的每个节点的资源信息，以及所述每个节点中的容器的资源信息，所述每个节点均安装有容器引擎；
    基于所述每个节点的资源信息和所述每个节点中的容器的资源信息，构建在所述每个节点上部署的微服务的资源画像，所述每个节点上部署的微服务基于一个或多个容器运行；
    若确定在所述集群中部署的第一微服务需进行资源调整，基于所述第一微服务的资源画像、部署所述第一微服务的第一节点的资源信息以及所述第一节点中的容器的资源信息，生成与所述第一微服务关联的资源调整指令；
    基于所述容器引擎及所述资源调整指令，调整与所述第一微服务关联的容器集合pod的资源。
  2. 如权利要求1所述的资源调度方法，其特征在于，所述基于所述每个节点的资源信息和所述每个节点中的容器的资源信息，构建在所述每个节点上部署的微服务的资源画像，包括：
    基于所述每个节点中的容器的资源信息，得到在所述每个节点上部署的微服务的资源信息；
    基于所述每个节点的资源信息及所述每个节点上部署的微服务的资源信息，构建在所述每个节点上部署的微服务的资源画像，所述资源画像包括资源占用维度与时间维度。
  3. 如权利要求1或2所述的资源调度方法，其特征在于，所述方法还包括：
    若所述第一微服务占用的资源与所述第一微服务的资源上限值之差小于第一预设阈值，确定所述第一微服务需进行资源调整；或者
    若所述第一微服务占用的资源与所述第一微服务的资源上限值之差大于第二预设阈值，确定所述第一微服务需进行资源调整。
  4. 如权利要求3所述的资源调度方法，其特征在于，所述第一微服务占用的资源包括处理器资源、内存资源、磁盘资源以及网络带宽资源中的一种或多种，所述确定所述第一微服务需进行资源调整，包括：
    若所述第一微服务占用的处理器资源、内存资源、磁盘资源以及网络带宽资源中的任意一者与对应的资源上限值之差小于对应的第一预设阈值，确定所述第一微服务需进行资源调整；或者
    若所述第一微服务占用的处理器资源、内存资源、磁盘资源以及网络带宽资源中的任意一者与对应的资源上限值之差大于对应的第二预设阈值，确定所述第一微服务需进行资源调整。
  5. 如权利要求4所述的资源调度方法，其特征在于，所述确定所述第一微服务需进行资源调整，包括：
    若所述第一微服务占用的处理器资源、内存资源、磁盘资源以及网络带 宽资源中的任意一者与对应的资源上限值之差小于对应的第一预设阈值，确定所述第一微服务需进行扩容调整；或者
    若所述第一微服务占用的处理器资源、内存资源、磁盘资源以及网络带宽资源中的任意一者与对应的资源上限值之差大于对应的第二预设阈值，确定所述第一微服务需进行缩容调整。
  6. 如权利要求1至5中任意一项所述的资源调度方法，其特征在于，基于所述第一微服务的资源画像、部署所述第一微服务的节点的资源信息以及所述节点中的容器的资源信息，生成与所述第一微服务关联的资源调整指令，包括：
    基于所述第一微服务的资源画像预测所述第一微服务所需的资源数量；
    基于所述预测的资源数量、部署所述第一微服务的节点的资源信息以及所述节点中的容器的资源信息，生成与所述第一微服务关联的资源调整指令。
  7. 如权利要求6所述的资源调度方法，其特征在于，所述生成与所述第一微服务关联的资源调整指令，包括：
    生成与所述第一微服务关联的所有pod的资源调整指令；或者
    生成与所述第一微服务关联的指定pod的资源调整指令。
  8. 如权利要求1至7中任意一项所述的资源调度方法，其特征在于，所述基于所述容器引擎及所述资源调整指令调整与所述第一微服务关联的pod的资源，包括：
    确定与所述资源调整指令关联的pod，及通过所述容器引擎调用资源调整应用程序接口API对与所述资源调整指令关联的pod进行资源调整。
  9. 如权利要求1至8中任意一项所述的资源调度方法，其特征在于，所述每个节点上部署的微服务配置有优先级，所述方法还包括：
    若所述第一节点的剩余资源无法满足所述第一微服务的资源调整需求，释放所述第一节点上部署的第二微服务占用的部分资源，所述第二微服务的优先级低于所述第一微服务的优先级。
  10. 如权利要求9所述的资源调度方法，其特征在于，所述释放所述第一节点上部署的第二微服务占用的部分资源，包括：
    根据所述第二微服务的资源画像，释放所述第二微服务占用的部分资源。
  11. 如权利要求1至8中任意一项所述的资源调度方法，其特征在于，所述每个节点上部署的微服务配置有优先级，所述方法还包括：
    若所述第一节点的剩余资源无法满足所述第一微服务的资源调整需求，将所述第一节点上部署的第二微服务调度至所述集群中的其他节点，所述第二微服务的优先级低于所述第一微服务的优先级。
  12. 如权利要求11所述的资源调度方法，其特征在于，所述将所述第一节点上部署的第二微服务调度至所述集群中的其他节点，包括：
    基于在所述第一节点上部署的微服务的资源画像，选择可调度至所述集群中的其他节点的第二微服务。
  13. 一种资源调度方法，其特征在于，所述方法包括：
    获取集群中的每个节点的资源信息，以及所述每个节点中的容器的资源信息，所述每个节点均安装有容器引擎；
    响应于在所述集群中部署的第一微服务的启动指令，为所述第一微服务创建第一容器集合pod，所述第一pod设置有第一资源上限值；
    基于部署所述第一微服务的第一节点的资源信息以及所述第一节点中的容器的资源信息，生成与所述第一pod关联的资源调整指令；
    基于所述容器引擎及所述资源调整指令将所述第一pod的资源占用上限由所述第一资源上限值调整为第二资源上限值，所述第二资源上限值大于所述第一资源上限值。
  14. 如权利要求13所述的资源调度方法，其特征在于，所述方法还包括：
    若确定所述第一微服务启动成功，将所述第一pod的资源占用上限由所述第二资源上限值调整为所述第一资源上限值。
  15. 一种资源调度装置，其特征在于，所述装置包括：
    第一获取模块，用于获取集群中的每个节点的资源信息，以及所述每个节点中的容器的资源信息，其中所述每个节点均安装有容器引擎；
    构建模块，用于基于所述每个节点的资源信息和所述每个节点中的容器的资源信息，构建在所述每个节点上部署的微服务的资源画像，其中所述每个节点上部署的微服务基于一个或多个容器运行；
    第一生成模块，用于在所述集群中部署的第一微服务为需进行资源调整时，基于所述第一微服务的资源画像、部署所述第一微服务的第一节点的资源信息以及所述第一节点中的容器的资源信息，生成与所述第一微服务关联的资源调整指令；
    第一调整模块，用于基于所述容器引擎及所述资源调整指令调整与所述第一微服务关联的容器集合pod的资源。
  16. 一种资源调度装置，其特征在于，所述装置包括：
    第二获取模块，用于获取集群中的每个节点的资源信息，以及所述每个节点中的容器的资源信息，所述每个节点均安装有容器引擎；
    创建模块，用于响应于在所述集群中部署的第一微服务的启动指令，为所述第一微服务创建第一容器集合pod，所述第一pod设置有第一资源上限值；
    第二生成模块，用于基于部署所述第一微服务的第一节点的资源信息以及所述第一节点中的容器的资源信息，生成与所述第一pod关联的资源调整指令；
    第二调整模块，用于基于所述容器引擎及所述资源调整指令将所述第一pod的资源占用上限由所述第一资源上限值调整为第二资源上限值，所述第二资源上限值大于所述第一资源上限值。
  17. 一种计算设备集群，其特征在于，包括至少一个计算设备，每个计算设备包括处理器和存储器；
    所述至少一个计算设备的处理器用于执行所述至少一个计算设备的存储器中存储的指令，以使得所述计算设备集群执行如权利要求1至权利要求 12中任一项所述的资源调度方法，或者执行如权利要求13或权利要求14所述的资源调度方法。
  18. 一种包含指令的计算机程序产品，其特征在于，当所述指令被计算设备集群运行时，使得所述计算设备集群执行如权利要求1至权利要求12中任一项所述的资源调度方法，或者执行如权利要求13或权利要求14所述的资源调度方法。
  19. 一种计算机可读存储介质，其特征在于，包括计算机程序指令，当所述计算机程序指令由计算设备集群执行时，所述计算设备集群执行如权利要求1至权利要求12中任一项所述的资源调度方法，或者执行如权利要求13或权利要求14所述的资源调度方法。
PCT/CN2023/099201 2022-09-05 2023-06-08 资源调度方法及其相关设备 WO2024051236A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211080778.9A CN117687739A (zh) 2022-09-05 2022-09-05 资源调度方法及其相关设备
CN202211080778.9 2022-09-05

Publications (1)

Publication Number Publication Date
WO2024051236A1 true WO2024051236A1 (zh) 2024-03-14

Family

ID=90125131

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/099201 WO2024051236A1 (zh) 2022-09-05 2023-06-08 资源调度方法及其相关设备

Country Status (2)

Country Link
CN (1) CN117687739A (zh)
WO (1) WO2024051236A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118245197A (zh) * 2024-05-27 2024-06-25 中航信移动科技有限公司 一种多类型集群协同调度方法、存储介质及电子设备

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200387372A1 (en) * 2019-06-06 2020-12-10 Dick's Sporting Goods, Inc. Microservice file generation system
CN112948050A (zh) * 2019-11-26 2021-06-11 西安华为技术有限公司 一种部署pod的方法及装置
CN112988398A (zh) * 2021-04-26 2021-06-18 北京邮电大学 一种微服务动态伸缩及迁移方法和装置
CN113382077A (zh) * 2021-06-18 2021-09-10 广西电网有限责任公司 微服务调度方法、装置、计算机设备和存储介质
CN113395178A (zh) * 2021-06-11 2021-09-14 聚好看科技股份有限公司 一种容器云弹性伸缩的方法及装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200387372A1 (en) * 2019-06-06 2020-12-10 Dick's Sporting Goods, Inc. Microservice file generation system
CN112948050A (zh) * 2019-11-26 2021-06-11 西安华为技术有限公司 一种部署pod的方法及装置
CN112988398A (zh) * 2021-04-26 2021-06-18 北京邮电大学 一种微服务动态伸缩及迁移方法和装置
CN113395178A (zh) * 2021-06-11 2021-09-14 聚好看科技股份有限公司 一种容器云弹性伸缩的方法及装置
CN113382077A (zh) * 2021-06-18 2021-09-10 广西电网有限责任公司 微服务调度方法、装置、计算机设备和存储介质

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118245197A (zh) * 2024-05-27 2024-06-25 中航信移动科技有限公司 一种多类型集群协同调度方法、存储介质及电子设备

Also Published As

Publication number Publication date
CN117687739A (zh) 2024-03-12

Similar Documents

Publication Publication Date Title
WO2017041556A1 (zh) 虚拟资源调度方法、装置及***
CN107637111B (zh) 用于提供和分配频谱资源的***和方法
US9405563B2 (en) Resource management method and apparatus for virtual machine system, and virtual machine system
US20190324819A1 (en) Distributed-system task assignment method and apparatus
JP6538869B2 (ja) ネットワーク管理
CN112583861B (zh) 服务部署方法、资源配置方法、***、装置及服务器
CN102567072B (zh) 一种资源分配方法、装置及***
CN102427475B (zh) 一种云计算环境中负载均衡调度的***
CN106452818B (zh) 一种资源调度的方法和***
US10587462B2 (en) Method and apparatus for deploying virtual operation, administration and maintenance, and virtualized network system
CN105868004B (zh) 一种基于云计算的业务***的调度方法及调度装置
CN105893113A (zh) 虚拟机的管理***及管理方法
US10142997B2 (en) Method and apparatus for adjusting physical resource, and controller
EP4172768A1 (en) Rightsizing virtual machine deployments in a cloud computing environment
WO2022252986A1 (zh) 中断调度方法、电子设备及存储介质
WO2024051236A1 (zh) 资源调度方法及其相关设备
CN110825212B (zh) 节能调度方法及装置、计算机可存储介质
CN103973811A (zh) 一种可动态迁移的高可用集群管理方法
KR102359681B1 (ko) 멀티 클라이언트용 사이버 원격관리 장치
WO2021109125A1 (zh) 一种弹性伸缩组的管理方法、装置
WO2016176937A1 (zh) 一种云服务***的资源设备管理方法和装置
US20230388844A1 (en) Translating commands for legacy radio access network
KR102359687B1 (ko) 멀티 클라이언트용 사이버 원격관리 장치
WO2024108810A1 (zh) 智能切片***的数据参考集生成方法、装置、设备及介质
WO2023071777A1 (zh) 一种网络集成控制方法及装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23861939

Country of ref document: EP

Kind code of ref document: A1