CN104253865A - Two-level management method for hybrid desktop cloud service platform - Google Patents
Two-level management method for hybrid desktop cloud service platform Download PDFInfo
- Publication number
- CN104253865A CN104253865A CN201410479717.9A CN201410479717A CN104253865A CN 104253865 A CN104253865 A CN 104253865A CN 201410479717 A CN201410479717 A CN 201410479717A CN 104253865 A CN104253865 A CN 104253865A
- Authority
- CN
- China
- Prior art keywords
- node
- virtual machine
- server
- domain
- server node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a two-level management method for a hybrid desktop cloud service platform. The two-level management method for the hybrid desktop cloud service platform comprises the following steps: building a hybrid desktop cloud data center; sending a login request to a central management node (Vmm-Server) by a server node; receiving and processing the login request of the server node by the central management node; uniformly receiving user requests by the central management node through a service interface layer; responding to the user requests by the central management mode according to a two-tier scheduling method; maintaining system load balance by the central management node according to the current system state and the running state of a virtual machine. According to the two-level management method disclosed by the invention is designed and realized upon the open system architecture of a hybrid desktop cloud server, and the scheduling and migration efficiency of the virtual machine can be improved.
Description
Technical field
The present invention relates to desktop cloud service platform, particularly a kind of two-level management way of mixed type desktop cloud service platform.
Background technology
Cloud computing is one of research field that current computer is burning the hotest, desktop cloud as the one in cloud computing, have be easy to dispose, the feature of fail safe high and data centralization management.Traditional desktop environment can be transferred to data center by desktop cloud, makes itself and terminal equipment decoupling zero, and user can use the terminal equipments such as thin-client to be belonged to the desktop environment of individual whenever and wherever possible by network insertion, improves the flexibility of office and the utilization rate of resource.In addition, desktop cloud solves conventional P C resource and the problem such as server resource utilization rate is low, O&M cost is high, coefficient of safety is low, can make up the calculating of the terminal equipments such as thin-client and the deficiency of storage capacity simultaneously, give full play to the advantage of mobile office.
It is investigated, Shanghai Dian Ji University application " a kind of college experiment teaching plateform system based on cloud desktop and building method " patent (application number: CN201410100306), this system uses the constructing virtual group of planes subsystems such as the fully virtualized technology KVM of expansion, use is increased income RDP structure cloud tabletop experiments teaching subsystem, although efficiently solve present colleges experimental teaching resource tired, manage a mixed and disorderly difficult problem, for the distribution of a cloud computing resources and management not perfect solution.
Shanghai Dian Ji University application " a kind of method and system based on memory source supply scheduling virtual machine quantity " patent (application number: CN201410100308), the method achieve a kind of method that arrival rate according to virtual machine creating request and system creation virtual machine service speed determine virtual machine carrying quantity, but this method only considered the relation of server and client side's Installed System Memory quantity, other such as CPU quantity and the factor such as performance, user behavior are not all considered, the decisive factor therefore for scheduling and resource allocation problem is considered perfect not.
Huawei Tech Co., Ltd has applied for " system of virtual cluster, resource allocation methods and management node " patent (application number: CN201410073555), this patent can under the performance change ratio of virtual machine more frequently situation, be adjusted to the resource of virtual machine distribution in time to meet the resource requirement of virtual machine, this scheme is all dynamic conditioning to the Resourse Distribute of all virtual machines, need real-time calculating, therefore the burden of server can be increased, requirement for server performance is higher, is not suitable for the cloud cluster of middle and small scale.
Electronic information industry Co., Ltd of the tide (Beijing) has applied for " the resources of virtual machine dispatching method in a kind of server cluster and system " patent (application number: CN201310576856), this patent is by obtaining the CPU usage of main frame in historical data, in the main frame higher to CPU usage, virtual machine moves the dynamic equilibrium realizing cloud cluster resource, although improve stability and the applicability of system to a certain extent, but it is unilateral for only considering that CPU usage is carried out scheduling of resource and distributed, other are memory size such as, cpu performances etc. are also the key factors of carrying out Resourse Distribute.
In sum, existing several technical scheme to cloud service resource management, all there are following three kinds of problems: the normative reference imperfection of (1) Resource Distribution and Schedule, only go measurement can not reflect real resource consumption situation from CPU usage or memory size.(2) dynamic of Resourse Distribute is strong, and need its resource of distributing of real-time calculating, resource consumption is large, not with reference to static information and historical data.(3) cluster management pattern is single, only relies on individual layer management mode, and its resource scheduling system can not the behavior of its Resourse Distribute of self-correction, lacks fault-tolerance and flexibility.
Summary of the invention
The object of the invention is to overcome the shortcoming of prior art and deficiency, a kind of two-level management way of mixed type desktop cloud service platform is provided.
Object of the present invention is realized by following technical scheme:
A two-level management way for mixed type desktop cloud service platform, comprises the step of following order:
S1, structure mixed type desktop cloud data center;
S2, server node send registration request to central management node;
S3, central management node receive and the registration request of processing server node;
S4, central management node receive user's request by service interface layer is unified;
S5, central management node are according to two layer scheduling method response user requests;
S6, central management node adopt two-layer moving method to maintain system load balancing according to current system mode and virtual machine running status.
Described step S1 specifically comprises the following steps:
S11, some station server nodes are connected in same shared storage device by switch, to form a territory;
S12, in the file system of shared storage device, create corresponding service catalogue and place the configuration file of domain node, and under the virtualization services hypervisor (Vmm-node) of independent research is stored in install catalogue;
S13, choose the host node of a station server node as NFS (NFS), and using the root in shared storage device as NFS share directory, other a few station server node accesses the content of shared storage device by the mode of NFS;
S14, repeat the step of S11 ~ S13, create several domain nodes, and to be linked together composition data center by local area network (LAN);
S15, by Intel Virtualization Technology, the physical resource of each domain node of data center is carried out pond, form several resource pools.
Described step S2 specifically comprises the following steps:
S21, the server node newly disposed are read by NFS mode and run the Vmm-Node program in shared storage device, and this program is run as finger daemon;
S22, virtualization services managing process collect the relevant information of domain node, and obtain the IP of this node, then calculate the Capacity value of server node, and Capacity value represents the maximum parallel virtual machine number that this server node can hold;
The data such as domain node configuration information and the Capacity value of self collected as log-on message, are sent to central management node and register by S23, virtualization services managing process.
Described step S3 specifically comprises the following steps:
S31, central management node connect with server node, and main service processes creates subprocess response registration request;
The log-on message of server node is updated in database by S32, subprocess, comprises the data of domain node belonging to the data such as the Capacity value of server node and this server node;
The resource pool information that S33, subprocess update of domain node are corresponding;
S34, central management node return the confirmation of successful registration to server node, and set up heartbeat detection by Keep-Alive mechanism with server node.
Described step S5 specifically comprises the following steps:
Virtual machine image record sheet in database is loaded in internal memory at initial phase by the scheduler module of S51, central management node;
S52, central management node are according to the configuration file of user's request dynamic generating virtual machines mirror image in internal memory;
S53, scheduler module read configuration file, configure obtain current scheduling strategy according to user;
S54, scheduler module, according to the virtual machine image record sheet of the configuration information query caching of virtual machine image in internal memory, judge whether this virtual machine creates mirror image;
If this virtual machine of S55 creates mirror image, then the domain node dispatched before directly obtaining in record as ground floor scheduling node, and jumps to step S57;
If do not create mirror image before this virtual machine of S56, then from cluster, select suitable domain node as ground floor scheduling node according to the running status of current adopted scheduling strategy and cluster, configuration file according to mirror image in this domain node creates virtual machine image in shared storage device, upgrades the virtual machine image record sheet in buffer memory and is synchronously written in database;
S57, scheduler module read virtual machine image record sheet, in the domain node that ground floor dispatching office is chosen, from this domain node, select a suitable server node as second layer scheduling node according to the running status of current scheduling strategy and server node, i.e. the current node for responding user's request;
The virtual machine configuration information dynamically generated is sent to the virtualization services managing process of the server node selected by step S57 by S58, scheduler module;
S59, server node create according to virtual machine configuration information and start virtual machine, finally complete the response to user's request, and are updated in database table by virtual machine information by virtualization services managing process.
Described step S6 specifically comprises the following steps:
The loading condition of each node of transferring module Real-time Obtaining current cluster of S61, central management node and the running state information of virtual machine;
S62, the transferring module information collected by step S61 judges the required virtual machine performing migration operation;
S62, transferring module, in conjunction with current scheduling strategy, judge to move between migration or territory in virtual machine execution domains to be migrated according to the loading condition of each node of cluster, if moved between execution domains, jump to step S65;
S63, transferring module are moved in virtual machine execution domains to be migrated, i.e. local migration, the ruuning situation of the Servers-all node of domain node belonging to virtual machine, select object migration node;
Virtual machine to be migrated is moved to destination node from source node by online moving method in the territory based on shared storage and the realization of internal memory pre-copy technology by S64, transferring module, and transition process keeps the availability of virtual machine service;
S65, transferring module are moved in virtual machine execution domains to be migrated, i.e. overall situation migration, according to the ruuning situation of the current each domain node of cluster, select the domain node of object migration, i.e. ground floor migration node;
The mirror image of virtual machine moves in the shared storage device of object domain node by the online migrating technology of memory block by S66, transferring module;
S67, transferring module, according to the ruuning situation of each server node of object domain node, select the server node of object migration, i.e. second layer migration node;
Virtual machine (vm) migration to be migrated to destination server node, destination server node is started this virtual machine by S68, transferring module, and this virtual machine of deletion on source server node.
The course of work of the present invention:
The present invention proposes a kind of novel desktop cloud framework, the basis of this framework is invented a kind of method of two-level management, has overcome the deficiency such as unicity, the limitation of virtual machine (vm) migration, the poor expandability of system scale of existing desktop cloud technical scheme individual layer scheduling.It is " centralized management, distributed deployment " that the core of this mixed type framework builds principle, and data center is made up of several " territories ", and territory to be connected to via switch by multiple stage that the server node of same shared storage device forms.Belong to the Servers-all node in same territory, in territory, manage resource concentratedly by resource pool, to improve the convenience of Resourse Distribute and the efficiency of scheduling in territory.Domain node adopts distributed deployment way, and different domain nodes can be deployed to identical or different machine room, thus improves the extensibility of system scale.This two-level management method mainly comprises two aspects, one is two layer scheduling methods, ground floor scheduling first by scheduling virtual machine to domain node, in the shared storage device corresponding to domain node, create the image file of virtual machine simultaneously, dispatch for the second layer and prepare.Second layer scheduling, by the configuration of each station server node according to domain node and operation conditions, selects suitable server node to complete the startup of virtual machine.Second aspect is the two-layer moving method of autonomous Design research and development, the loading condition enabling virtual machine coupling system current and the running status of virtual machine, virtual machine is carried out overall situation migration between Yu Nei local migration or territory by the mode of online migration, thus makes system can reach load balancing in subrange and global scope.
Compared with prior art, tool has the following advantages and beneficial effect in the present invention:
1, based on the mixed type desktop cloud server architecture created, there is the feature of " centralized management, distributed deployment ".The server node of same domain node is concentrated and is disposed, and all data are all left in shared storage device, by resource pool, resource is managed concentratedly, resource-sharing is realized in territory, make the server node in domain node more convenient when visit data, greatly reduce the cost multiple servers node in territory being carried out to load balancing, migration in territory can be made simultaneously to become simple, flexible.In addition, the region limits of physics machine room can be broken by mode domain node being carried out to distributed deployment, system scale can be expanded further.The node server forming territory can at different physical locations by the network interconnection, and independent toward each other and isolation, the operation do a domain node and variation can not affect other domain node.
2, adopt two layer scheduling methods to dispatch virtual machine, more flexible, efficient relative to individual layer dispatching method, efficiency and the practicality of scheduling virtual machine are higher.The multidate information of the static information of server performance and current system service condition is combined, the scheduling strategy of virtual machine dynamic assignment in the static allocation between the territory realizing resource and territory.
3, adopt two-layer moving method to move virtual machine, local migration or overall situation migration can be adopted targetedly according to the loading condition of each node of system, virtual machine (vm) migration be operated and becomes flexible, efficient.Shift function, can the Real-time Balancing system load when single domain transships as the miscellaneous function of dispatching patcher.Transition process all have employed online migration pattern, and can ensure that virtual machine service is not interrupted in transition process, system availability is higher.
Accompanying drawing explanation
Fig. 1 is mixed type desktop cloud server side logic Organization Chart;
Fig. 2 is mixed type desktop cloud system integrated stand composition;
Fig. 3 is system data center configuration diagram;
Fig. 4 is the process chart of mixed type desktop cloud service end two-level management method;
Fig. 5 is the communication construction that new server node adds data center;
Fig. 6 is the sequential chart that server node is registered to central management node;
Fig. 7 is the flow chart of two layer scheduling methods;
Fig. 8 is the pre-copy mechanism loop iteration procedure chart of internal memory migration;
Fig. 9 is the flow chart of virtual machine (vm) migration process.
Embodiment
Below in conjunction with embodiment and accompanying drawing, the present invention is described in further detail, but embodiments of the present invention are not limited thereto.
The inventive method designs based on mixed type desktop cloud server architecture, and whole desktop cloud service end comprises following five core components: data center (Data Center), service end central management node (Vmm-Server), scheduler module, transferring module and monitoring module.Data center is the basis of whole framework, leaves data center in all resource set, for managing resource concentratedly, provides the running environment of virtual desktop reality.And service end central management node is responsible for process and respond various user request, is undertaken alternately by communication protocol and data center, the resource of data center is managed.Scheduler module has been responsible for scheduling and the distribution function of virtual machine and virtual resource, and transferring module then completes the shift function to virtual machine and virtual resource, and auxiliary dispatching module realizes the load balancing of system.Monitoring module has been responsible for the monitoring of virtual machine running status and system cluster running status.
As shown in Figure 1, system have employed hierarchical design thought and designs, be logically four different levels according to the difference of responsibility by system divides, be bottom-uply respectively physical layer, virtualization layer (virtualization), management level and service interface layer.Physical layer comprises the hardware device of all formation resource pools such as server host, switch device, shared storage device, router device, and these hardware devices provide basic resource to upper strata jointly.Server host mainly provides computational resource and memory source, and shared storage device is responsible for providing storage resources, and switch and router device then provide Internet resources.The effect of virtualization layer is to use different Intel Virtualization Technologies to carry out pond to the resource that physical layer provides, and create computing pool, storage pool and network pool respectively, these three kinds of ponds combine and jointly form resource pool, and unification provides virtual resource to upper strata.Management level are used for managing the running environment of whole virtual machine, mainly comprise the management of management to virtual machine and resource.This part is the core of whole system, is responsible for calling of respond services interface layer, completes the process to user's request and response, plays the effect of forming a connecting link in systems in which.Service interface layer is used for providing service interface for the operation system of upper layer application, unified API is adopted to call for the operation system of upper layer application, facilitate upper strata to apply based on HDCS (Hybrid Desktop Cloud Server), namely secondary development is carried out in the service of mixed type desktop cloud.In addition, service interface layer also receives user's request for unified, encapsulates, then pass to corresponding management layer module according to the type of request and process request.
In order to realize at all levels in system between loose coupling relation, adopt the mode of interface interchange to carry out alternately between layers, lower floor by interface to serve upper layers.The then interaction mechanism that transmits of Effect-based operation between the module of every layer and module, intermodule communicates each other by Message Transmission.These two kinds of interactive modes make the mutual and specific implementation each other between layers, between module and module have nothing to do, and then substantially increase the maintainability of whole system.
The inventive method designs based on mixed type desktop cloud server architecture, in a preferred embodiment, can be applied in cloud laboratory scene.As shown in Figure 2, be applied to the embodiment in cloud laboratory, comprise cloud laboratory system client and the service end of upper-layer service logic system, and mixed type desktop cloud service end.Mixed type desktop cloud service end, as versatility desktop cloud platform, is the virtual desktop serve that cloud laboratory system provides student's computer experiment used, and cloud laboratory system is then responsible for the service logic on process upper strata.Wherein, cloud experimental teaching management system can adopt the cloud experimental teaching management system that Chinese workers IBM practice training centre laboratory is researched and developed.Student carries out computer experiment by the cloud client of cloud experimental teaching management system, and when logging on experimental situation, cloud client transmit a request to mixed type desktop cloud service end request virtual desktop serve.In an embodiment, the server node of data center can adopt Dell PowerEdge R720 server (server is equipped with 6 core CPU and 64G internal memory) or HP DL380p G8 server (server is equipped with 2*10 core CPU and 128G internal memory), and shared storage device can adopt the Storwize V3500V6.4.1 of IBM, EMC VNX5400 disk array or Hitachi HUS130, the shared memory capacity of each domain node is 10T.Embodiment adopts 2 domain nodes, and each domain node is equipped with 2 station server nodes and a shared storage device, and server node is connected to shared storage device by switch.Choose the minimum server node in IP address as NFS host node, using the root in shared storage device as share directory, other server node accesses shared storage device content by NFS mode.Two domain nodes are deployed in different machine rooms respectively, and are connected to central management node by gigabit LAN.Cloud client can be deployed in the PC of Lenovo Qitian M4600 model, adopts Pentium four CPU, the 256 MB of memory of monokaryon.Cloud client is for receiving the input information of user, and service end user instruction being sent to cloud experimental teaching management system processes, when needs use virtual desktop serve, by the service of directly calling mixed type desktop cloud service end and providing.
The present invention is based on the two-level management method of mixed type desktop cloud server architecture, its basic handling flow chart as shown in Figure 4, comprises the steps:
S1, structure mixed type desktop cloud data center.Mixed type desktop cloud data center is the basis of whole mixed type desktop cloud server architecture, the two-level management method invented also is designed and realization on this basis, deposit resource by data center is unified, and the running environment that virtual desktop is really is provided.When building data center, need the deployment place of each domain node having planned data center, the server node number that each domain node comprises needs to plan in conjunction with the size of machine room and the port number of switch.As shown in Figure 3, its building process mainly comprises the following steps the formation of data center:
S11, in same territory, some station server nodes are connected in same shared storage device by switch;
S12, in the file system of shared storage device, create corresponding service catalogue and place the configuration file of domain node, and under the virtualization services hypervisor (Vmm-node) of independent research is stored in install catalogue;
S13, choose the host node of a station server node as NFS, and using the root in shared storage device as NFS share directory, other a few station server node accesses the content of shared storage device by the mode of NFS;
S14, repeat the step of S11 ~ S13, create several domain nodes, and to be linked together composition data center by local area network (LAN);
S15, by Intel Virtualization Technology, the physical resource of each domain node of data center is carried out pond, form several resource pools;
S2, server node send registration request to management node.After building data center in step sl, the server node disposed in each machine room needs to send registration request to central management node, and application adds data center.New server node adds the communication construction of data center as shown in Figure 5, and every station server node can be connected by Vmm-Server (desktop cloud management system) service processes of virtualization services managing process and central management node when initialization.The sequential chart that server node is registered to central management node as shown in Figure 6, mainly comprises the following steps:
S21, the server node newly disposed are read by NFS mode and run the virtualization services hypervisor (Vmm-node) in shared storage device, and this program is run as finger daemon.When configuring domain node, when after the share directory configuring NFS, virtualization services hypervisor (Vmm-node) can be positioned over share store /install catalogue under, the Servers-all node of same domain node can be accessed by NFS mode and run this program;
S22, virtualization services managing process collect the relevant information of domain node, and obtain the IP of this node, then calculate the Capacity value (representing the maximum parallel virtual machine number that this server node can hold) of server node.Virtualization services managing process calculates Capacity value by the information such as processor, internal memory getting this server node;
S23, virtualization services managing process are using the data such as domain node configuration information and the Capacity value of self collected as log-on message, and being sent to central management node registers;
S3, central management node receive and the registration request of processing server node.The service processes of central management node, always at the connection request of monitoring service device node finger daemon, when receiving new connection request, will be completed the process to register flow path by subprocess.The registration request handling process of central management node to server node mainly comprises the following steps:
S31, central management node connect with server node, and main service processes creates subprocess response registration request;
The log-on message of server node is updated in database by S32, subprocess, comprises the data of domain node belonging to the data such as the Capacity value of server node and this server node.After having upgraded information, the state of this server node becomes available, and be scheduled in real time module and transferring module known;
The resource pool information that S33, subprocess update of domain node are corresponding.The physical resource of the server node newly added is carried out pond by virtualization modules by system, and correspondingly upgrades the capacity of resource pool, main computational resource and the memory source upgrading resource pool;
S34, central management node return the confirmation of successful registration to server node, and set up heartbeat detection by Keep-Alive mechanism with server node.Central management node, by setting up heartbeat detection with server node, can learn exiting of server node in real time, thus nullify server node, and the corresponding data of more new database;
S4, central management node receive user's request by service interface layer is unified.Service interface layer is used for providing service interface for the operation system of upper layer application, adopts unified API to call for the operation system of upper layer application, facilitates upper strata to apply and carries out secondary development based on mixed type desktop cloud service end.In addition, service interface layer also receives user's request for unified, encapsulates, then pass to corresponding management layer module according to the type of request and process request;
S5, central management node are according to two layer scheduling method response user requests.Designed two layer scheduling methods with realizing of the present invention are based on mixed type desktop cloud server architecture, summary, ground floor scheduling will choose domain node according to scheduling strategy from cluster, under the shared storage directory of this domain node, then create the mirror image of virtual machine.Second layer scheduling selects a suitable server node by the domain node chosen from ground floor dispatching office according to current adopted scheduling strategy, and is completed the response of asking user by this server node.The method of this two-layer scheduling, relative to individual layer scheduling mode, can improve flexibility and the efficiency of scheduling virtual machine.The handling process of two layer scheduling methods of the present invention as shown in Figure 7, mainly comprises the following steps:
Virtual machine image record sheet in database is loaded in internal memory at initial phase by the scheduler module of S51, central management node;
S52, central management node are according to the configuration file of user's request dynamic generating virtual machines mirror image in internal memory;
S53, scheduler module read configuration file, configure obtain current scheduling strategy according to user.
S54, scheduler module, according to the virtual machine image record sheet of the configuration information query caching of virtual machine image in internal memory, judge whether this virtual machine creates mirror image;
If this virtual machine of S55 creates mirror image, then the domain node dispatched before directly obtaining in record as ground floor scheduling node, and jumps to step S57;
If do not create mirror image before this virtual machine of S56, then from cluster, select suitable domain node as ground floor scheduling node according to the running status of current adopted scheduling strategy and cluster, configuration file according to mirror image in this domain node creates virtual machine image in shared storage device, upgrades the virtual machine image record sheet in buffer memory and is synchronously written in database;
S57, scheduler module read virtual machine image record sheet, in the domain node that ground floor dispatching office is chosen, from this domain node, select a suitable server node as second layer scheduling node according to the running status of current scheduling strategy and server node, i.e. the current node for responding user's request;
The virtual machine configuration information dynamically generated is sent to the virtualization services managing process of the server node selected by step S57 by S58, scheduler module;
S59, server node create according to virtual machine configuration information and start virtual machine, finally complete the response to user's request, and are updated in database table by virtualization services managing process by virtual machine information;
S6, central management node adopt two-layer moving method to maintain system load balancing according to current system mode and virtual machine running status.Two-layer moving method of the present invention can adopt local migration or overall situation migration targetedly according to the loading condition of each node of system, virtual machine (vm) migration is operated and becomes flexible, efficient.Transition process all have employed online migration pattern, its core technology is the internal memory migration technology based on pre-copy mechanism, as shown in Figure 8, can ensure that virtual machine service is not interrupted in transition process, system availability is higher for the memory copying iterative process of pre-copy mechanism.Two-layer moving method handling process of the present invention as shown in Figure 9, mainly comprises the following steps:
The loading condition of each node of transferring module Real-time Obtaining current cluster of S61, central management node and the running state information of virtual machine;
S62, the transferring module information collected by step S61 judges the required virtual machine performing migration operation;
S62, transferring module, in conjunction with current scheduling strategy, judge to move between migration or territory in virtual machine execution domains to be migrated according to the loading condition of each node of cluster, if moved between execution domains, jump to step S65;
S63, transferring module are moved in virtual machine execution domains to be migrated, i.e. local migration, the ruuning situation of the Servers-all node of domain node belonging to virtual machine, select object migration node;
Virtual machine to be migrated is moved to destination node from source node by online moving method in the territory based on shared storage and the realization of internal memory pre-copy technology by S64, transferring module, and transition process keeps the availability of virtual machine service;
S65, transferring module are moved in virtual machine execution domains to be migrated, i.e. overall situation migration, according to the ruuning situation of the current each domain node of cluster, select the domain node of object migration, i.e. ground floor migration node;
The mirror image of virtual machine moves in the shared storage device of object domain node by the online migrating technology of memory block by S66, transferring module;
S67, transferring module, according to the ruuning situation of each server node of object domain node, select the server node of object migration, i.e. second layer migration node;
Virtual machine (vm) migration to be migrated to destination server node, destination server node is started this virtual machine by S68, transferring module, and this virtual machine of deletion on source server node.
Above-described embodiment is the present invention's preferably execution mode; but embodiments of the present invention are not restricted to the described embodiments; change, the modification done under other any does not deviate from Spirit Essence of the present invention and principle, substitute, combine, simplify; all should be the substitute mode of equivalence, be included within protection scope of the present invention.
Claims (6)
1. a two-level management way for mixed type desktop cloud service platform, is characterized in that, comprises the step of following order:
S1, structure mixed type desktop cloud data center;
S2, server node send registration request to central management node;
S3, central management node receive and the registration request of processing server node;
S4, central management node receive user's request by service interface layer is unified;
S5, central management node are according to two layer scheduling method response user requests;
S6, central management node adopt two-layer moving method to maintain system load balancing according to current system mode and virtual machine running status.
2. the two-level management way of mixed type desktop cloud service platform according to claim 1, it is characterized in that, described step S1 specifically comprises the following steps:
S11, some station server nodes are connected in same shared storage device by switch, to form a territory;
S12, in the file system of shared storage device, create corresponding service catalogue and place the configuration file of domain node, and under the virtualization services hypervisor of independent research is stored in install catalogue;
S13, choose the host node of a station server node as NFS, and using the root in shared storage device as NFS share directory, other a few station server node accesses the content of shared storage device by the mode of NFS;
S14, repeat the step of S11 ~ S13, create several domain nodes, and to be linked together composition data center by local area network (LAN);
S15, by Intel Virtualization Technology, the physical resource of each domain node of data center is carried out pond, form several resource pools.
3. the two-level management way of mixed type desktop cloud service platform according to claim 1, it is characterized in that, described step S2 specifically comprises the following steps:
S21, the server node newly disposed are read by NFS mode and run virtualization services hypervisor (Vmm-Node) program in shared storage device, and this program is run as finger daemon;
S22, virtualization services managing process collect the relevant information of domain node, and obtain the IP of this node, then calculate the Capacity value of server node, and Capacity value represents the maximum parallel virtual machine number that this server node can hold;
The data such as domain node configuration information and the Capacity value of self collected as log-on message, are sent to central management node and register by S23, virtualization services managing process.
4. the two-level management way of mixed type desktop cloud service platform according to claim 1, it is characterized in that, described step S3 specifically comprises the following steps:
S31, central management node connect with server node, and main service processes creates subprocess response registration request;
The log-on message of server node is updated in database by S32, subprocess, comprises the data of domain node belonging to the data such as the Capacity value of server node and this server node;
The resource pool information that S33, subprocess update of domain node are corresponding;
S34, central management node return the confirmation of successful registration to server node, and set up heartbeat detection by Keep-Alive mechanism with server node.
5. the two-level management way of mixed type desktop cloud service platform according to claim 1, it is characterized in that, described step S5 specifically comprises the following steps:
Virtual machine image record sheet in database is loaded in internal memory at initial phase by the scheduler module of S51, central management node;
S52, central management node are according to the configuration file of user's request dynamic generating virtual machines mirror image in internal memory;
S53, scheduler module read configuration file, configure obtain current scheduling strategy according to user;
S54, scheduler module, according to the virtual machine image record sheet of the configuration information query caching of virtual machine image in internal memory, judge whether this virtual machine creates mirror image;
If this virtual machine of S55 creates mirror image, then the domain node dispatched before directly obtaining in record as ground floor scheduling node, and jumps to step S57;
If do not create mirror image before this virtual machine of S56, then from cluster, select suitable domain node as ground floor scheduling node according to the running status of current adopted scheduling strategy and cluster, configuration file according to mirror image in this domain node creates virtual machine image in shared storage device, upgrades the virtual machine image record sheet in buffer memory and is synchronously written in database;
S57, scheduler module read virtual machine image record sheet, in the domain node that ground floor dispatching office is chosen, from this domain node, select a suitable server node as second layer scheduling node according to the running status of current scheduling strategy and server node, i.e. the current node for responding user's request;
The virtual machine configuration information dynamically generated is sent to the virtualization services managing process of the server node selected by step S57 by S58, scheduler module;
S59, server node create according to virtual machine configuration information and start virtual machine, finally complete the response to user's request, and are updated in database table by virtual machine information by virtualization services managing process.
6. the two-level management way of mixed type desktop cloud service platform according to claim 1, it is characterized in that, described step S6 specifically comprises the following steps:
The loading condition of each node of transferring module Real-time Obtaining current cluster of S61, central management node and the running state information of virtual machine;
S62, the transferring module information collected by step S61 judges the required virtual machine performing migration operation;
S62, transferring module, in conjunction with current scheduling strategy, judge to move between migration or territory in virtual machine execution domains to be migrated according to the loading condition of each node of cluster, if moved between execution domains, jump to step S65;
S63, transferring module are moved in virtual machine execution domains to be migrated, i.e. local migration, the ruuning situation of the Servers-all node of domain node belonging to virtual machine, select object migration node;
Virtual machine to be migrated is moved to destination node from source node by online moving method in the territory based on shared storage and the realization of internal memory pre-copy technology by S64, transferring module, and transition process keeps the availability of virtual machine service;
S65, transferring module are moved in virtual machine execution domains to be migrated, i.e. overall situation migration, according to the ruuning situation of the current each domain node of cluster, select the domain node of object migration, i.e. ground floor migration node;
The mirror image of virtual machine moves in the shared storage device of object domain node by the online migrating technology of memory block by S66, transferring module;
S67, transferring module, according to the ruuning situation of each server node of object domain node, select the server node of object migration, i.e. second layer migration node;
Virtual machine (vm) migration to be migrated to destination server node, destination server node is started this virtual machine by S68, transferring module, and this virtual machine of deletion on source server node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410479717.9A CN104253865B (en) | 2014-09-18 | 2014-09-18 | A kind of two-level management method of mixed type desktop cloud service platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410479717.9A CN104253865B (en) | 2014-09-18 | 2014-09-18 | A kind of two-level management method of mixed type desktop cloud service platform |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104253865A true CN104253865A (en) | 2014-12-31 |
| CN104253865B CN104253865B (en) | 2017-06-06 |
Family
ID=52188388
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410479717.9A Active CN104253865B (en) | 2014-09-18 | 2014-09-18 | A kind of two-level management method of mixed type desktop cloud service platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104253865B (en) |
Cited By (31)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104811483A (en) * | 2015-04-02 | 2015-07-29 | 广东睿江科技有限公司 | Cloud system allocation method and cloud system |
| CN105187512A (en) * | 2015-08-13 | 2015-12-23 | 航天恒星科技有限公司 | Method and system for load balancing of virtual machine clusters |
| CN105227666A (en) * | 2015-10-12 | 2016-01-06 | 浪潮(北京)电子信息产业有限公司 | The whole machine cabinet management framework that a kind of facing cloud calculates |
| CN105357310A (en) * | 2015-11-20 | 2016-02-24 | 中国石油天然气股份有限公司华北油田分公司 | Energy cloud platform and data interaction method based on energy cloud platform |
| CN106294496A (en) * | 2015-06-09 | 2017-01-04 | 北京京东尚科信息技术有限公司 | A kind of data migration method based on hadoop cluster and instrument |
| CN106411581A (en) * | 2016-09-19 | 2017-02-15 | 广州云晫信息科技有限公司 | Disaster recovery and dynamic transfer system of cloud desktop system |
| CN106452892A (en) * | 2016-10-24 | 2017-02-22 | 深圳市深信服电子科技有限公司 | Virtual management method and system, and node |
| CN107070725A (en) * | 2017-05-12 | 2017-08-18 | 郑州云海信息技术有限公司 | A kind of method that server two-level management intermodule communication is shaken hands |
| CN107241384A (en) * | 2017-05-03 | 2017-10-10 | 复旦大学 | A kind of content distribution service priority scheduling of resource method based on many cloud frameworks |
| CN107423116A (en) * | 2017-07-30 | 2017-12-01 | 长沙曙通信息科技有限公司 | A kind of desktop virtualization virtual machine cluster system implementation method |
| CN107491360A (en) * | 2017-07-21 | 2017-12-19 | 山东九州信泰信息科技股份有限公司 | The method for being classified other redundant storage is carried out to record in tables of data |
| CN107967117A (en) * | 2016-10-20 | 2018-04-27 | 杭州海康威视数字技术股份有限公司 | A kind of data storage, reading, method for cleaning, device and cloud storage system |
| WO2018090605A1 (en) * | 2016-11-18 | 2018-05-24 | 中兴通讯股份有限公司 | Data center management method and system |
| CN108322537A (en) * | 2018-02-02 | 2018-07-24 | 郑州云海信息技术有限公司 | Method, apparatus, equipment and the storage medium in Cloud Server node resource pond |
| CN108650320A (en) * | 2018-05-15 | 2018-10-12 | 北京中创腾锐技术有限公司 | Isomorphism cascade device configuration file synchronous method under a kind of cluster environment |
| CN109327536A (en) * | 2018-11-12 | 2019-02-12 | 郑州源创智控有限公司 | One kind fast implementing the central controlled method of dispersing apparatus |
| CN109600274A (en) * | 2018-11-26 | 2019-04-09 | 广州微算互联信息技术有限公司 | The acquisition methods and server of control node |
| CN109597671A (en) * | 2018-12-11 | 2019-04-09 | 国云科技股份有限公司 | A kind of desktop cloud implementation method being compatible with centralized and distributed operation |
| WO2020020154A1 (en) * | 2018-07-23 | 2020-01-30 | 中兴通讯股份有限公司 | Offline management method and device for cloud desktop, and storage medium |
| CN110750331A (en) * | 2019-10-21 | 2020-02-04 | 北京华育兴业科技有限公司 | Container cluster scheduling method and platform for education desktop cloud application |
| CN111126940A (en) * | 2019-11-22 | 2020-05-08 | 泰康保险集团股份有限公司 | Service application processing method, device, equipment and computer readable storage medium |
| CN111629013A (en) * | 2019-02-27 | 2020-09-04 | 北京奇虎科技有限公司 | Method and device for managing service node and node network |
| CN111752726A (en) * | 2020-06-25 | 2020-10-09 | 武汉众邦银行股份有限公司 | Method for quickly and crossly sharing data among multiple systems |
| CN112068846A (en) * | 2020-08-07 | 2020-12-11 | 福建升腾资讯有限公司 | Application distribution method, device and medium based on dual systems |
| CN112486628A (en) * | 2020-11-20 | 2021-03-12 | 湖南麒麟信安科技股份有限公司 | Virtual machine migration method between virtual private clouds |
| CN112714903A (en) * | 2018-09-19 | 2021-04-27 | 亚马逊科技公司 | Scalable cell-based packet processing service using client-provided decision metadata |
| CN112822062A (en) * | 2019-11-15 | 2021-05-18 | 辽宁省交通高等专科学校 | Management method for desktop cloud service platform |
| CN114791855A (en) * | 2022-06-23 | 2022-07-26 | 中航金网(北京)电子商务有限公司 | Task scheduling method, device, medium, equipment and program product under cloud platform |
| CN115834573A (en) * | 2023-01-11 | 2023-03-21 | 厦门简算科技有限公司 | Cloud server scheduling method and system |
| CN116204494A (en) * | 2023-04-28 | 2023-06-02 | 深圳竹云科技股份有限公司 | Method and device for migrating active directory data, electronic equipment and storage medium |
| CN116302509A (en) * | 2023-02-21 | 2023-06-23 | 中船（浙江）海洋科技有限公司 | Cloud server dynamic load optimization method and device based on CNN-converter |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110276982A1 (en) * | 2010-05-06 | 2011-11-10 | Hitachi, Ltd. | Load Balancer and Load Balancing System |
| CN102307153A (en) * | 2011-10-14 | 2012-01-04 | 王宁 | Virtual desktop transmission device and method |
| CN102722413A (en) * | 2012-05-16 | 2012-10-10 | 上海兆民云计算科技有限公司 | Distributed resource scheduling method for desktop cloud cluster |
| CN102739771A (en) * | 2012-04-18 | 2012-10-17 | 上海和辰信息技术有限公司 | Cloud application integrated management platform and method supporting service fusion |
| CN103164283A (en) * | 2012-05-10 | 2013-06-19 | 上海兆民云计算科技有限公司 | Method and system for dynamic scheduling management of virtualized resources in virtualized desktop system |
| CN103414712A (en) * | 2013-08-05 | 2013-11-27 | 深圳市杰云科技有限公司 | Management system and method of distributed virtual desktop |
-
2014
- 2014-09-18 CN CN201410479717.9A patent/CN104253865B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110276982A1 (en) * | 2010-05-06 | 2011-11-10 | Hitachi, Ltd. | Load Balancer and Load Balancing System |
| CN102307153A (en) * | 2011-10-14 | 2012-01-04 | 王宁 | Virtual desktop transmission device and method |
| CN102739771A (en) * | 2012-04-18 | 2012-10-17 | 上海和辰信息技术有限公司 | Cloud application integrated management platform and method supporting service fusion |
| CN103164283A (en) * | 2012-05-10 | 2013-06-19 | 上海兆民云计算科技有限公司 | Method and system for dynamic scheduling management of virtualized resources in virtualized desktop system |
| CN102722413A (en) * | 2012-05-16 | 2012-10-10 | 上海兆民云计算科技有限公司 | Distributed resource scheduling method for desktop cloud cluster |
| CN103414712A (en) * | 2013-08-05 | 2013-11-27 | 深圳市杰云科技有限公司 | Management system and method of distributed virtual desktop |
Non-Patent Citations (1)
| Title |
|---|
| 张毅: "基于OpenStack的虚拟桌面云***服务端设计与实现", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
Cited By (45)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104811483B (en) * | 2015-04-02 | 2018-07-06 | 广东睿江云计算股份有限公司 | Cloud system dispositions method and cloud system |
| CN104811483A (en) * | 2015-04-02 | 2015-07-29 | 广东睿江科技有限公司 | Cloud system allocation method and cloud system |
| CN106294496A (en) * | 2015-06-09 | 2017-01-04 | 北京京东尚科信息技术有限公司 | A kind of data migration method based on hadoop cluster and instrument |
| CN106294496B (en) * | 2015-06-09 | 2020-02-07 | 北京京东尚科信息技术有限公司 | Data migration method and tool based on hadoop cluster |
| CN105187512A (en) * | 2015-08-13 | 2015-12-23 | 航天恒星科技有限公司 | Method and system for load balancing of virtual machine clusters |
| CN105227666A (en) * | 2015-10-12 | 2016-01-06 | 浪潮(北京)电子信息产业有限公司 | The whole machine cabinet management framework that a kind of facing cloud calculates |
| CN105357310A (en) * | 2015-11-20 | 2016-02-24 | 中国石油天然气股份有限公司华北油田分公司 | Energy cloud platform and data interaction method based on energy cloud platform |
| CN106411581A (en) * | 2016-09-19 | 2017-02-15 | 广州云晫信息科技有限公司 | Disaster recovery and dynamic transfer system of cloud desktop system |
| CN107967117A (en) * | 2016-10-20 | 2018-04-27 | 杭州海康威视数字技术股份有限公司 | A kind of data storage, reading, method for cleaning, device and cloud storage system |
| US11003367B2 (en) | 2016-10-20 | 2021-05-11 | Hangzhou Hikvision Digital Technology Co., Ltd. | Data storage, reading, and cleansing method and device, and cloud storage system |
| CN106452892A (en) * | 2016-10-24 | 2017-02-22 | 深圳市深信服电子科技有限公司 | Virtual management method and system, and node |
| WO2018090605A1 (en) * | 2016-11-18 | 2018-05-24 | 中兴通讯股份有限公司 | Data center management method and system |
| CN108075917A (en) * | 2016-11-18 | 2018-05-25 | 中兴通讯股份有限公司 | The management method and system of a kind of data center |
| CN107241384A (en) * | 2017-05-03 | 2017-10-10 | 复旦大学 | A kind of content distribution service priority scheduling of resource method based on many cloud frameworks |
| CN107241384B (en) * | 2017-05-03 | 2020-11-03 | 复旦大学 | Content distribution service resource optimization scheduling method based on multi-cloud architecture |
| CN107070725B (en) * | 2017-05-12 | 2019-10-01 | 郑州云海信息技术有限公司 | A kind of method that server two-level management intermodule communication is shaken hands |
| CN107070725A (en) * | 2017-05-12 | 2017-08-18 | 郑州云海信息技术有限公司 | A kind of method that server two-level management intermodule communication is shaken hands |
| CN107491360A (en) * | 2017-07-21 | 2017-12-19 | 山东九州信泰信息科技股份有限公司 | The method for being classified other redundant storage is carried out to record in tables of data |
| CN107491360B (en) * | 2017-07-21 | 2019-07-30 | 山东九州信泰信息科技股份有限公司 | The method for being classified other redundant storage is carried out to record in tables of data |
| CN107423116A (en) * | 2017-07-30 | 2017-12-01 | 长沙曙通信息科技有限公司 | A kind of desktop virtualization virtual machine cluster system implementation method |
| CN108322537A (en) * | 2018-02-02 | 2018-07-24 | 郑州云海信息技术有限公司 | Method, apparatus, equipment and the storage medium in Cloud Server node resource pond |
| CN108650320B (en) * | 2018-05-15 | 2021-05-11 | 北京中创腾锐技术有限公司 | Method for synchronizing configuration files of isomorphic cascading equipment in cluster environment |
| CN108650320A (en) * | 2018-05-15 | 2018-10-12 | 北京中创腾锐技术有限公司 | Isomorphism cascade device configuration file synchronous method under a kind of cluster environment |
| WO2020020154A1 (en) * | 2018-07-23 | 2020-01-30 | 中兴通讯股份有限公司 | Offline management method and device for cloud desktop, and storage medium |
| CN112714903A (en) * | 2018-09-19 | 2021-04-27 | 亚马逊科技公司 | Scalable cell-based packet processing service using client-provided decision metadata |
| CN109327536B (en) * | 2018-11-12 | 2020-12-25 | 郑州源创智控有限公司 | Method for quickly realizing centralized control of decentralized equipment |
| CN109327536A (en) * | 2018-11-12 | 2019-02-12 | 郑州源创智控有限公司 | One kind fast implementing the central controlled method of dispersing apparatus |
| CN109600274A (en) * | 2018-11-26 | 2019-04-09 | 广州微算互联信息技术有限公司 | The acquisition methods and server of control node |
| CN109597671A (en) * | 2018-12-11 | 2019-04-09 | 国云科技股份有限公司 | A kind of desktop cloud implementation method being compatible with centralized and distributed operation |
| CN109597671B (en) * | 2018-12-11 | 2022-06-10 | 国云科技股份有限公司 | Desktop cloud implementation method compatible with centralized and distributed operation |
| CN111629013A (en) * | 2019-02-27 | 2020-09-04 | 北京奇虎科技有限公司 | Method and device for managing service node and node network |
| CN110750331A (en) * | 2019-10-21 | 2020-02-04 | 北京华育兴业科技有限公司 | Container cluster scheduling method and platform for education desktop cloud application |
| CN112822062A (en) * | 2019-11-15 | 2021-05-18 | 辽宁省交通高等专科学校 | Management method for desktop cloud service platform |
| CN111126940A (en) * | 2019-11-22 | 2020-05-08 | 泰康保险集团股份有限公司 | Service application processing method, device, equipment and computer readable storage medium |
| CN111752726A (en) * | 2020-06-25 | 2020-10-09 | 武汉众邦银行股份有限公司 | Method for quickly and crossly sharing data among multiple systems |
| CN111752726B (en) * | 2020-06-25 | 2023-07-14 | 武汉众邦银行股份有限公司 | Method for quickly cross sharing data among multiple systems |
| CN112068846A (en) * | 2020-08-07 | 2020-12-11 | 福建升腾资讯有限公司 | Application distribution method, device and medium based on dual systems |
| CN112068846B (en) * | 2020-08-07 | 2023-11-10 | 福建升腾资讯有限公司 | Application distribution method, device and medium based on dual systems |
| CN112486628A (en) * | 2020-11-20 | 2021-03-12 | 湖南麒麟信安科技股份有限公司 | Virtual machine migration method between virtual private clouds |
| CN114791855A (en) * | 2022-06-23 | 2022-07-26 | 中航金网(北京)电子商务有限公司 | Task scheduling method, device, medium, equipment and program product under cloud platform |
| CN114791855B (en) * | 2022-06-23 | 2022-09-16 | 中航金网(北京)电子商务有限公司 | Task scheduling method, device, medium, equipment and program product under cloud platform |
| CN115834573A (en) * | 2023-01-11 | 2023-03-21 | 厦门简算科技有限公司 | Cloud server scheduling method and system |
| CN115834573B (en) * | 2023-01-11 | 2023-06-02 | 厦门简算科技有限公司 | Cloud server scheduling method and system |
| CN116302509A (en) * | 2023-02-21 | 2023-06-23 | 中船（浙江）海洋科技有限公司 | Cloud server dynamic load optimization method and device based on CNN-converter |
| CN116204494A (en) * | 2023-04-28 | 2023-06-02 | 深圳竹云科技股份有限公司 | Method and device for migrating active directory data, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104253865B (en) | 2017-06-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104253865B (en) | A kind of two-level management method of mixed type desktop cloud service platform | |
| US10700991B2 (en) | Multi-cluster resource management | |
| CN112560244B (en) | Virtual simulation experiment system and method based on Docker | |
| CN103635882B (en) | Controlling network is utilized | |
| CN101430649B (en) | Virtual computation environmental system based on virtual machine | |
| CN109313564A (en) | For supporting the server computer management system of the highly usable virtual desktop of multiple and different tenants | |
| WO2019149414A1 (en) | Method and system for migration of containers in a container orchestration platform between compute nodes | |
| CN109768892B (en) | Micro-service network security experiment system | |
| Averitt et al. | Virtual computing laboratory (VCL) | |
| CN105391652A (en) | USB equipment network sharing system and method based on USB redirection | |
| CN103957270A (en) | Cloud atomic unit delivery and deployment method and device | |
| CN105959347A (en) | All-digital collaborative simulation system communication method based on P2P network | |
| Chang et al. | Architecture design of datacenter for cloud english education platform | |
| Benali et al. | Evaluation of traffic-aware VM placement policies in distributed cloud using cloudsim | |
| CN111061617A (en) | Cloud computing-based space-based network networking simulation test system | |
| Pandey et al. | Scheduling workflow applications based on multi-source parallel data retrieval in distributed computing networks | |
| Luo et al. | From monolithic systems to a federated e-learning cloud system | |
| CN105637471B (en) | Method and apparatus for being monitored and controlling to storage environment | |
| WO2020253490A1 (en) | Resource allocation method, apparatus and device | |
| Shelar et al. | Efficient virtual machine placement with energy saving in cloud data center | |
| Bouhouch et al. | Data Migration: Cloudsim Extension | |
| López et al. | On a course on computer cluster configuration and administration | |
| Luo et al. | Constructing a virtual computer laboratory based on OpenStack | |
| CN103309719A (en) | Virtual machine management system applied to cloud computing | |
| Anton et al. | Educational services in cloud with IBM technology: A new model for open, on demand learning in higher education |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |