WO2017082757A1

WO2017082757A1 - Computer data processing system and method for communication traffic based optimization of virtual machine communication

Info

Publication number: WO2017082757A1
Application number: PCT/RU2015/000778
Authority: WO
Inventors: Alexander Sergeevich TROFIMOV; Mikhail Evgenyevich MYSOV
Original assignee: Huawei Technologies Co., Ltd.
Priority date: 2015-11-13
Filing date: 2015-11-13
Publication date: 2017-05-18
Also published as: CN108351802A; CN108351802B

Abstract

A computer data processing system is provided for virtualizing at least partially computing hardware, and being adapted to run local virtual machines, wherein the system comprises a means to provide, to a local virtual machine, a communication module and wherein the local virtual machine is configured to use the communication module for network communication, a balancer module configured to analyze a network communication between the local virtual machine and a communication partner, the balancer module being configured to enable a shared memory communication if the communication partner is a second local virtual machine of the computer data processing system and network communication traffic of the communication reaches a first threshold value.

Description

COMPUTER DATA PROCESSING SYSTEM AND METHOD FOR COMMUNICATION TRAFFIC BASED OPTIMIZATION OF VIRTUAL MACHINE COMMUNICATION

FIELD OF THE INVENTION

The present invention relates to the field of virtualization technology and network communication, especially to a computer data processing system comprising virtual machines and a method for operating a computer data processing system. In particular the computer data processing system provides a balancer module to analyze network communication and/or to initiate migration of virtual machines and/or enable shared memory communication between virtual machines.

Enabling shared memory communication between virtual machines can be implemented independently and separately from migrating virtual machines. However, preferably both techniques are combined with each other.

BACKGROUND

Virtualization technology is a fast growing field in the area of computer science and information technologies. Virtualization in general provides an abstraction layer for performing computing operations on a set of resources. For instance, virtualization can decouple computing operations from direct interaction with computing hardware. Thus, a more abstract and unified access to the underlying hardware resources is provided and mechanisms to effectively use and share hardware resources can be applied. In the context of virtualization technology, a virtual machine (VM) may be regarded as a simulation of a physical computer system using the resources provided by a host system on which the virtual machines run. The host system provides the actual, non-virtualized (hardware) resources. Of course, the host system can be a distributed computing system and access to the host system's resources can be provided by an abstraction layer.

A hypervisor (also called "virtual machine monitor"), a system which may be implemented in software, firmware and/or hardware, is provided to create and run VMs on the host system. Hence, a computer data processing system on which a hypervisor is running one or more VMs may be called a VM host.

An operation system (OS) running on the VM host provides general system and infrastructure management to the VMs and is typically referred to as the "host OS". Each VM can also be referred to as a "guest" on the host system.

Migration describes the process by which a running VM is moved from one VM host to another, typically with little disruption of providing service. This is sometimes also referred to as "live migration".

At present, several implementations of virtualization concepts are known, some of which are described in the following:

a) Para-virtualization provides a software interface that is similar, but not identical with actual hardware of a computer system. It is therefore necessary to adapt VM code (e.g. code of an OS running in a VM) to be able to be used. The beneficial effect on the one hand is that as to the adaption, VM code becomes more simplified as it is no longer based on an interpretation execution logic. Thus, para-virtualization has very good performance.

b) Hardware-assisted virtualization is a platform virtualization approach. Platform virtualization is virtualization of computers as complete hardware platforms, certain logical abstractions of their components or the functionality required to run various OS. Hardware-assisted virtualization is also known as accelerated virtualization, hardware virtual machine (HVM) or native virtualization. Hardware- assisted virtualization enables to simulate a complete hardware environment or VM, in which an unmodified guest OS can be executed in complete isolation.

Hardware-assisted virtualization uses help of hardware capabilities, primarily the host processors, but also virtualization of peripherals such as provided by an Input/Output Memory Management Unit (IOMMU, also referred to as "AMD-Vi" or "Intel VT-d") or Single Root I/O Virtualization (SR-IOV, also referred to as "Intel VT- c" or "PCI-SIG Single Root I/O virtualization"). These terms are now explained in more detail:

- IOMMU allows guest VMs to directly use peripheral devices, such as Ethernet cards, accelerated graphics cards, and hard-drive controllers, through direct memory access (DMA) and interrupt remapping. This feature can also be called Peripheral Component Interconnect (PCI) pass-through.

- SR-IOV provides a set of general Input/Output virtualization methods based on PCI Express (PCIe) native hardware. With SR-IOV enabled, e.g. network interfaces of a VM host are directly accessible to guest VMs, avoiding involvement of a hypervisor and resulting in high overall performance.

Although hardware-assisted virtualization is one of the most important technologies, several problems exist with it and require to be solved.

The presented solution preferably addresses host OS based hypervisor platforms, but is not limited to them. It can also be applied to other kinds of hypervisor platforms.

The above mentioned virtualization platforms typically can work autonomously, but are also applied in frameworks that provide infrastructure management to the virtualization platforms, in particular in large scale environments where automated provisioning of resources is appreciated. Examples are CloudStack (by the Apache Software Foundation), Eucalyptus (developed by Eucalyptus Systems Inc.), OpenStack (initially developed by Rackspace Inc. and NASA), and vCloud Director (by VMWARE Inc.).

Fig. 1 shows a schematic view of a host OS based platform according to the prior art. An exemplary host OS based platform as shown typically comprises the following components (which are described in detail below):

- VM1, VM2: VMs 1 and 2

- PVF: para- virtual frontend network device

- PVB: para- virtual backend network device

- VF: virtual function device, especially supporting SR-IOV

- NIC PF: network interface controller (physical function, especially with SR- IOV support)

- bridge/OVS: software packet switching mechanism (e.g.standard bridge or OpenVSwithch (OVS))

The host OS based platform as shown in Fig. 1 comprises a VM host, running two VMs VM1 and VM2 and a host OS that includes a hypervisor and bridging/swichting functionality. A hypervisor is present on the host OS controlling the execution of the VMs and providing networking capabilities such as software packet switching and bridging mechanisms. To facilitate network communication between VMs running on the same VM host as well as with external remote communication partners, each VM is equipped with two kinds of virtual interfaces. A para-virtual frontend network device (PVF) and virtual function device, especially supporting SR- IOV (VF). The bridge/OVS provides the corresponding networking counterparts to each VM, namely a para-virtual backend network device (PVB) and a network interface controller (physical function with SR-IOV support). The PVFs of the VMs are connected to the corresponding PVBs located in the bridge/OVS, whereas the VFs of each VM are connected to one NIC PF located in the bridge/OVS.

Network communication among VMs and external communication partners can e.g. be performed virtual interfaces that are attached to each VM. Para-virtual devices (PVF, PVB) offer very fast communication between local VMs located on the same VM host.

A hardware-assisted approach implementing SR-IOV supporting virtual function devices is very efficient for outgoing communication as data processing and interface management can be sourced out to a physical network controller. At the same time this solution has significant performance drawbacks regarding communication between VMs located on the same VM host due to additional overhead brought by the architecture and speed limits of the networking stack involved (network traffic has to pass through physical networking hardware and PCIe bus topology to reach another VM). Additionally, live migration of VMs is not supported.

From the prior art, several approaches and developments are known:

"Live Migration with Pass-through Device for Linux VM," Edwin Zhai, Gregory D. Cummings and Yaozu Dong, In Proceedings of the Linux Symposium, Ottawa, Ontario, Canada, (OLS), 2008, proposes a solution to enable live migration on KVM with one attached virtual function device that supports SR-IOV by using a total of two attached virtual network devices. The approach is being described in view of Fig. 2 that shows a VM host including two virtual machines VM1 and VM2 and KVM/QEMU running as a hypervisor. A "bonding driver" is used to union two network devices (a para-virtual frontend network device (PVF) and a SR-IOV supporting virtual function device (VF)) in a single network device. This is done by a standard Linux host OS bonding mechanism in so called active backup mode.

The combined network device is referred to as "bond-backup" in Fig. 2. The hypervisor comprises a bridging and switching mechanism (Bridge/OVS) including network devices corresponding to the VMs running on the VM host and means to facilitate network communication with remote communication partners. Two para- virtual backend network devices (PVB) are shown, each of them corresponding to one PVF of each virtual machine VM1, VM2. One network interface controller (NIC PF) (enabling a physical function with SR-IOV support) is connected with the virtual function devices (VF) on each virtual machine VM1, VM2. In a regular operating mode, a packet is sent from inside a VM to a remote communication partner outside the VM host by the "bond - backup" device by means of the virtual function device supporting SR-IOV.

Before a live migration is to be carried out, the SR-IOV supporting virtual function device is detached from the VM and the bonding mechanism automatically forwards traffic to the backup interface (para- virtual network device) during migration. Using the backup device involves high CPU utilization as the para-virtualized devices and the software implemented bridging and switching mechanism are now used. After live migration, the SR-IOV device is attached to the migrated VM again. Network traffic between a VM and a remote communication partner is then carried out by the virtual function device supporting SR-IOV again. However, the detaching and attaching operation of the virtual function device supporting SR-IOV is neither automated nor carried out efficiently.

US 8151265 B2 (and similarly US 8464259 B2, US 7945436 B2, US 20080276258 Al) discloses an approach for enabling live migration of VMs with attached virtual function devices supporting SR-IOV by using an abstract emulation layer to switch between an emulated network device and a virtual function device during a live migration process. If a user wants to execute a live migration, the system switches connection to the emulated layer automatically. However, the abstraction layer introduces additional overhead in the way how network traffic is exchanged. Moreover, network device emulation is the slowest option available for virtual networking.

US 8533713 B2 (and similarly US 8146082 B2) proposes an approach to enable live migration on VMs with attached virtual function devices supporting SR- IOV by using an abstraction layer to enable saving and restoring of network device states. If a user wants to execute a live migration process, the system automatically saves the state of a network device and transfers it to a remote VM. When the VM resumes execution on the target host, the network device state is restored. The proposed approach is located at hardware-level. As some hardware registers need to be restored significant service downtime between saving and restoring a state occurs. Additionally due to the complexity of the approach, only hardware vendors can implement the solution as available documentation does not contain all information about internal registers for restoring sequence. As some registers can only be accessed in read-only mode, some states can't be restored by simply storing values in related registers, which involves high efforts to implement the solution.

US 20130077501 Al proposes a solution to indirectly facilitate live migration of VMs with attached virtual function devices supporting SR-IOV by disclosing a transport level switching mechanism that can be used for efficient rebalancing of traffic flow in case of detaching one of the networking interfaces. Packets are sent over all available network interfaces and are reconstructed at the endpoint. If a user wants to execute a live migration process and a SR-IOV supporting virtual function device is to be detached, the system would automatically redirect traffic via other available interfaces. A drawback of this proposed solution is that not all transport level protocols support extended attributes and extended option fields for storing necessary information to transmit network communication via multiple interfaces. A second drawback is the lack of automating the described procedure. Additionally, OS system code has to be modified to enable the proposed functionality.

A solution is presented to improve communication between VMs. Specifically, a data processing system and method for operating a data processing system according to the independent claims is provided. Further aspects and embodiments are subject to the dependent claims.

SUMMARY

According to a first aspect, the invention provides a computer data processing system for virtualizing at least partially computing hardware, and being adapted to run local virtual machines, wherein the system comprises a means to provide, to a local virtual machine, a communication module and wherein the local virtual machine is configured to use the communication module for network communication, a balancer module configured to analyze a network communication between the local virtual machine and a communication partner, the balancer module being configured to enable a shared memory communication if the communication partner is a second local virtual machine of the computer data processing system and network communication traffic of the communication reaches a first threshold value.

According to a first implementation of the first aspect, the balancer module can be configured to initiate a migration of the local virtual machine and/or of the communication partner if the commumcation partner is a local virtual machine or a remote virtual machine and network communication traffic of the communication reaches a second threshold value.

According to a second implementation of the first aspect, the communication module can comprise a first communication interface and a second communication interface, wherein the balancer module can be configured to enable a communication through the first communication interface if the communication partner is the second local virtual machine and/or to enable a commumcation through the second commumcation interface if the communication partner is the remote virtual machine.

According to a third implementation of the first aspect, the first communication interface can be a para-virtualized communication device, wherein the second communication interface can be a virtual function device.

According to a fourth implementation of the first aspect, the computer data processing system can comprise a hypervisor or host OS, wherein the hypervisor or host OS can comprise a mechanism for network communication bridging or network commumcation switching, wherein the network communication bridging or network commumcation switching mechanism can comprise a backend interface for the first communication interface, and wherein the backend interface can be a para-virtual backend device.

According to a fifth implementation of the first aspect, the network communication bridging or network communication switching mechanism can be configured to operate a network interface controller driver in functional association with data transmitted via the second communication interface, and wherein the network interface controller driver can be a physical function with SR-IOV support.

According to a sixth implementation of the first aspect, the network interface controller driver can be configured to enable communication of the virtual machine with communication partners outside the computer data processing system.

According to a seventh implementation of the first aspect, the computer data processing system can comprise a monitoring module configured to monitor the computer data processing system and can comprise means to measure a CPU utilization, means to measure a memory space utilization, means to measure network utilization, means to measure resources utilization of local virtual machines, means to access, to read from and/or to write to a state memory database.

According to an eighth implementation of the first aspect, the monitoring module may be configured to detect a state of the computer data processing system and configuration changes of local resources of the computer data processing system and to record the state of the computer data processing system and the configuration changes in the state memory database.

According to an ninth implementation of the first aspect, the computer data processing system can comprise a control module configured to obtain information on configuration changes by querying the state memory database, wherein the control module can be configured to reconfigure at least one of the local virtual machines, the communication module, the first and/or the second communication interface based on the information obtained from the state memory database.

According to a tenth implementation of the first aspect, the computer data processing system can comprise or can functionally be connected to a memory module wherein the control module can be configured to allocate a portion of a shared memory space provided by the memory module for each local virtual machine, and wherein each local virtual machine can store and/or read data to/from the allocated shared memory space to send/receive data from another local virtual machine. The control module can be configured to perform the allocation or de-allocation of the shared memory space to establish shared memory connections between the local virtual machine and the second local virtual machine based on instructions from the balancer module to enable the shared memory communication.

According to an eleventh implementation of the first aspect, the balancer module can be configured to analyze a system state by querying the state memory database and/or network communication, to determine whether a balancing action or an optimization of a network communication needs to be performed, and to store the analysis result in the state memory database.

According to a twelfth implementation of the first aspect, the balancer modules can be configured to collect states of remote virtual machines from other remote balancer modules. The balancer module can be configured to synchronize the state memory database with state memory databases of the other remote balancer modules.

According to a thirteen implementation of the first aspect, the balancer module can be configured to provide respective reconfiguration information to the control module and/or can be configured to decide whether a local and/or remote virtual machine should be migrated to the computer data processing system or another computer data processing system and to enable the migration by communicating with the other remote balancer modules.

According to a fourteenth implementation of the first aspect, the balancer module can be configured to decide whether or not to migrate the local or remote virtual machine based on information stored in the state. The balancer module can be configured to grant the control module rights to perform the migration, and wherein the control module executes the migration.

According to a second aspect, the invention provides a method for operating a computer data processing system for virtualizing at least partially computing hardware, the computer data processing system being adapted to run local virtual machines, wherein the system provides, to a local virtual machine, a communication module and wherein the local virtual machine uses the communication module for network communication, wherein a balancer module analyzes a network communication between the local virtual machine and a communication partner, the balancer module enabling a shared memory communication if the communication partner is a second local virtual machine and network communication traffic of the communication reaches a first threshold value.

The method for operating a computer data processing system for virtualizing at least partially computing hardware according to the second aspect, comprises

providing, to a local virtual machine ninning on the computer data processing system, a communication module, wherein the local virtual machine uses the communication module for network communication,

analyzing, by the balancer module, a network communication between the local virtual machine and a communication partner, and

enabling a shared memory communication if the commumcation partner is a second local virtual machine and network communication traffic of the communication reaches a first threshold value.

According to a first implementation of the second aspect, by the balancer module a migration of the local virtual machine and/or of the communication partner can be initiated if the communication partner is a local virtual machine or a remote virtual machine and network communication traffic of the communication reaches a second threshold value.

According to a second implementation of the second aspect, the communication module can comprise a first communication interface and a second communication interface, and the method comprises the step of enabling, by the balancer module, a communication through the first communication interface if the communication partner is the second local virtual machine and/or enabling a communication through the second communication interface if the communication partner is the remote virtual machine. The first communication interface can be a para- virtualized communication device, wherein the second communication interface can be a virtual function device.

The computer data processing system may comprise a hypervisor or host OS, wherein the hypervisor or host OS can comprise a mechanism for network communication bridging or network communication switching, wherein the network communication bridging or network communication switching mechanism can comprise a backend interface for the first communication interface, and wherein the backend interface can be a para-virtual backend device.

According to a third implementation of the second aspect, wherein the method further comprises operating, by the network communication bridging or network communication switching mechanism, a network interface controller driver in functional association with data transmitted via the second communication interface, and wherein the network interface controller driver can be a physical function with SR-

IOV support.

According to a fourth implementation of the second aspect, further comprising enabling, at the network interface controller driver, communication of the virtual machine with communication partners outside the computer data processing system.

According to a fifth implementation of the second aspect, the method can further comprise monitoring, by a monitoring module, the computer data processing system; measuring a CPU utilization, a memory space utilization, network utilization, resources utilization of local virtual machines; accessing, reading from and/or writing to a state memory database.

According to a sixth implementation of the second aspect, the method further comprises detecting, by a momtoring module, a state of the computer data processing system and configuration changes of local resources of the computer data processing system; and recording the state of the computer data processing system and the configuration changes in the state memory database.

According to an seventh implementation of the second aspect, method can comprise obtaining, by a control module, information on configuration changes by querying the state memory database, reconfiguring, by the control module, at least one of the local virtual machines, the communication module, the first and/or the second communication interface based on the information obtained from the state memory database.

According to an eighth implementation of the second aspect, method comprises allocating, by the control module, a portion of a shared memory space provided by a memory module for each local virtual machine, and wherein each local virtual machine can store and/or read data to/from the allocated shared memory space to send/receive data from another local virtual machine, wherein memory module in comprised in the computer data processing system or is functionally connected to the computer data processing system. The method further comprising allocating or de-allocating, by the control module, the shared memory space to establish shared memory connections between the local virtual machine and the second local virtual machine based on instructions from the balancer module to enable the shared memory communication.

According to a ninth implementation of the second aspect, the method comprises analyzing, by the balancer module, a system state by querying the state memory database and/or network communication, to determine whether a balancing action or an optimization of a network communication needs to be performed, and to store the analysis result in the state memory database.

According to an tenth implementation of the second aspect, the method further comprises collecting, by the balancer modules, states of remote virtual machines from other remote balancer modules; and synchronizing, by the balancer module the state memory database with state memory databases of the other remote balancer modules.

According to a eleventh implementation of the second aspect, the method further comprises providing, by the balancer module, respective reconfiguration information to the control module and/or deciding whether a local and/or remote virtual machine should be migrated to the computer data processing system or another computer data processing system and deciding to enable the migration by communicating with the other remote balancer modules.

According to a twelfth implementation of the second aspect, the method further comprises deciding, by the balancer module, whether or not to migrate the local or remote virtual machine based on information stored in the state, and granting, by the balancer module, the control module rights to perform the migration, and executing the migration by the control module.

According to a third aspect, the invention provides a computer data processing system for virtualizing at least partially computing hardware, and being adapted to run local virtual machines, wherein the system comprises a means to provide, to a local virtual machine, a communication module and wherein the local virtual machine is configured to use the communication module for network communication, a balancer module configured to analyze a network communication between the local virtual machine and a communication partner, the balancer module being configured to initiate a migration of the local virtual machine and/or of the communication partner if the communication partner is a local virtual machine or a remote virtual machine and network communication traffic of the communication reaches a second threshold value.

According to a fourth aspect, the invention provides a method for operating a computer data processing system for virtualizing at least partially computing hardware, the computer data processing system being adapted to run local virtual machines, wherein the method comprises providing, to a local virtual machine, a communication module, wherein the local virtual machine uses the communication module for network communication; analyzing by a balancer module, a network communication between the local virtual machine and a communication partner, initiating by the balancer module a migration of the local virtual machine and/or of the communication partner if the communication partner is a local virtual machine or a remote virtual machine and network communication traffic of the communication reaches a second threshold value.

According to a fifth aspect, the invention provides a computer system comprising at least two computer data processing systems, each configured to provide, to a local virtual machine, a communication module, wherein the local virtual machine is configured to use the communication module for network communication, wherein for each computer data processing system a balancer module is provided, configured to analyze a state of at least two virtual machines, wherein the balancer modules of the computer data processing systems are configured to exchange information about a state of the at least two virtual machines, wherein at least one of the balancer modules is further configured to initiating a migration of a virtual machine to and/or from one of the computer data processing systems and/or to a third computer data processing system.

According to a sixth aspect, the invention provides a method for operating a computer system comprising at least two computer data processing systems, of which each provides, to a local virtual machine, a communication module, wherein the local virtual machine uses the communication module for network communication, wherein the method comprises analyzing by a balancer module provided for each computer data processing system a state of at least two virtual machines, exchanging, by the balancer modules of the computer data processing systems, information about a state of the at least two virtual machines, initiating, by at least one of the balancer modules a migration of a virtual machine to and/or from one of the computer data processing systems and/or to a third computer data processing system.

BRIEF DESCRIPTION OF THE DRAWINGS

The above described aspects and embodiments of the present invention will now also be discussed with reference to the figures:

Fig. 1 shows a schematic overview of a host OS based hypervisor platform according to the prior art.

Fig. 2 shows a schematic overview of another virtualization platform according to the prior art.

Fig. 3 shows a schematic overview of the first and second aspect of the present invention.

Fig. 4 exemplarily illustrates possible migration scenarios.

Fig. 5 shows a more detailed schematic overview of the first and second aspect the present invention.

Fig. 6 shows a schematic overview of a monitoring module.

Fig. 7 shows a schematic overview of a balancer module.

Fig. 8 schematically shows an interaction of several modules and systems. Fig. 9 shows a schematic overview of a decentralized decision making process.

Fig. 10 shows a schematic overview of shared memory communication between two VMs.

DETAILLED DESCRIPTION OF THE EMBODIMENTS

Generally, it has to be noted that all arrangement, devices, modules, components, models, elements, units and means and so forth described in the present application could be implemented by software or hardware elements or any kind of combination thereof. All steps which are performed by the various entities described in the present application as well as the functionality described to be performed the various entities are intended to mean that the respective entity is adapted to or configured to perform the respective steps and functionalities. Even if in the following description of the specific embodiments, a specific functionality or step to be performed by a general entity is not reflected in the description of a specific detailed element of the entity which performs the specific step or functionality, it should be clear for a skilled person that these methods and functionalities can be implemented in respective hardware or software elements, or any kind of combination thereof. Further, the method of the present invention and its various steps are embodied in the functionalities of the various described apparatus elements.

The present invention primarily focuses on an improved, efficient network architecture including decentralized resource balancing and automatic setup of shared memory communication.

The proposed solution not only decreases complexity of VM management but also aims at the following objectives: - automating live migration of VMs with attached para-virtualized devices and virtual function devices supporting SR-IOV; and/or

- decreasing CPU utilization overhead when using para-virtualized communication devices and proposing a decentralized balancing and decision-making mechanism for efficient migration automation; and/or

- providing shared memory communication, thereby improving communication of VMs running on the same VM host.

To achieve these objectives, a virtualized networking stack is optimized by introducing balancing components and optimizing communication mechanisms (by adding extensions that support network traffic exchange via shared allocated memory space). A method of automatically initiating migration of VMs and/or setting up shared memory communication between VMs running on the same VM host is introduced. Further, a way of decentralized decision-making for balancing, shared memory communication and/or VM migration is implemented. This is realized by evaluating the utilization of resources available for one or more VM hosts and VM's, synchronizing this information among balancer modules of data processing systems and applying new configurations based on a decentralized decision-making approach using the synchronized and evaluated information to the involved data processing systems.

Fig. 3 shows a general setup of the invention. In Fig. 3, a computer data processing system 100 is shown. The computer data processing system 100 virtualizes at least partially computing hardware and is adapted to run at least a first local VM 101 and a second local VM 102. While in Fig. 3, two local VMs 101, 102 are shown, the computer data processing system 100 can run more or less local VMs.

The first local VM 101 and the second local VM 102 in the following typically denote VMs running on the same computer data processing system 100, whereby a remote VM denotes a VM running on another computer data processing system that is connected to the computer data processing system 100, e.g. by means of a computer network. The computer data processing system 100 is thus also referred to as local computer data processing system 100. Computer data processing systems that are connected to the local computer data processing system 100, e.g. by means of a computer network, are consequently also called remote computer data processing systems.

The computer data processing system 100 provides to the local VMs 101, 102 communication modules 103, 104, respectively (e.g. a first communication module 103 to the first local VM 101 and a second communication module 103 to the second local VM 102). Each local VM 101, 102 can use its communication module 103/104 for any kind of network communication. Network communication is passed through the communication modules 103, 104, which can direct the communication to a physical and/or a (para-)virtual networking device. Network communication can be established between local VMs 101, 102, (as illustrated by the curved arrow in Fig. 3), but also between a local VM and a remote VM located in another computer data processing system (not shown in Fig. 3), or between a local VM and an arbitrary remote communication partner (which can be a remote server, client, or any other networking device).

The computer data processing system 100 as shown in Fig. 3 also comprises a balancer module 105. The balancer module 105 analyzes the network communication between a local VM 101, 102 and communication partners (which can be another local VM 101, 102, a remote VM or any other remote networking device). The balancer module 105 also can enable shared memory communication between the first local VM 101 and a local communication partner such as the second local VM 102 (as illustrated by the double-arrow in Fig. 3), if the network communication traffic of the network communication between the local VM and the communication partner reaches a first threshold value.

Shared memory communication in this case means communication of two local VMs 101, 102 using a same allocated shared memory space. The shared memory space can be alternately or simultaneously used by both local VMs to read and write data, and thereby send and receive data from one local VM to another local VM. As one shared memory space is used by both local VMs involved in communication, no additional central processing unit (CPU) cycles to copy data from a memory space allocated and used by the first local VM 101 to a separate memory space allocated and used by the second local VM 102 are necessary. Since no additional CPU cycles are needed for the data exchange, the shared memory communication is also called zero copy communication. This type of communication especially improves communication via para- virtual devices and CPU cycles and memory bandwidth can be saved.

Data exchanged in a communication between the two local VMs, 101, 102 neither does have to pass a local system bus (e.g. PCI or PCIe) nor a peripheral system device (e.g. a physical network interface controller). Therefore shared memory communication leads to highly increased data throughput in communication of two local VMs 101, 102.

The first threshold value for network communication traffic can be a single value which is compared to parameter values of the network communication, e.g. used network bandwidth, an amount of network bandwidth used over a certain time interval (e.g. an integral, average or median of used network bandwidth), occurrences of timeouts, package loss, interface errors and/or response time. All or some of these parameters may indicate high network load. The first threshold value may also be more complex than a single value. In fact, the first threshold value generally defines a situation in view of monitored parameters when network load is deemed to be too high and/or when communication is adversely affected. Based on monitored parameters a discrimination in respect to the first threshold value can be made and depending on the discrimination result it is decided whether an action has to be taken, i.e. whether VM migration is initiated and/or shared memory communication is enabled. Of course, also more than one threshold value can be set and depending on whether and what threshold value is reached (exceeded or surpassed) a finer granular decision can be made. The necessary data can be collected by installing monitoring probes or in the virtual or physical network interface(s) or communication module(s) used in the computer data processing system 100 and its local VMs 101, 102 or by listening to (virtual) monitoring ports installed on switching or bridging mechanisms or devices.

Hence, for example, if the first threshold value is reached, e.g. when a package loss rate increases over the set threshold, the balancer module may enable shared memory communication between the two local VMs 101, 102.

Further or alternatively, the balancer module 105 can initiate a migration of the first VM 101 and/or the second local VM 102 and/or a remote VM if network communication traffic of the network communication between the local VM and the remote VM reaches a second threshold value.

Reaching the second threshold value may be detected and defined similar to the first threshold value as described above. However, the parameters for determining whether the first threshold value is reached can be derived from the local computer data processing system 100 alone, due to the locality of the communication partners and the possibility to obtain information on the communication traffic from the host OS or hypervisor. The traffic information required for determining whether the second threshold value is reached can be obtained from sources outside the local computer data processing system 100. For example, (virtual) bridges or switches used for connecting the local computer data processing system 100 to another (remote) computer data processing system running a remote VM, i.e. a VM external to the local computer data processing system 100, can provide such information. Also the other computer data processing systems can provide such information to the local computer data processing system 100, or the balancer module 105, respectively. Generally, also other parameters like memory utilization, CPU load, a number of VMs running on a computer data processing system can be used in the comparison, discrimination and definition of the first and second threshold value.

Enabling shared memory communication and initiating a migration of local and/or remote VMs are independent steps that can, but not necessarily have to be carried out in sequence. The balancer module 105 can enable shared memory communication without initiating a migration of local and/or remote VMs and vice versa, and thus the local computer data processing system 100 may provide the one functionality without the other.

Migration of VMs can be initiated to optimize use of resources of VMs running inside or outside the local computer data processing system 100. For optimization, the following scenarios are addressed:

If communication between a local VM and a remote VM reaches a first threshold value, either the first local VM 101 running on the computer data processing system 100 can be migrated to a remote computer data processing system where the remote VM is located. Alternatively, the remote VM running on the remote computer data processing system can be migrated to the local computer data processing system 100 where the local VM 101 is running.

If the resources of neither the local computer data processing system 100, nor of the remote computer data processing system are sufficient to run two VMs after migration, both VMs can be migrated to a third computer data processing system. Hence, if network communication between a local VM 101 running on a local computer data processing system 100 and a remote VM running on a remote computer data processing system reaches a threshold, both the local VM 101 and the remote VM can be migrated to the third computer data processing system.

This is illustrated by Fig. 4. In Fig. 4a), a local VM 101 is moved from the local computer data processing system 100 to a remote computer data processing system 100', on which a remote VM 102' is running. Thus, after the migration, both VMs 101, 102' are running on the remote computer data processing system 100'.

Another example is shown in Fig. 4b). Here, the remote VM 102' is moved to the local computer data processing system 100 from the remote computer data processing system 100'. Hence, after the migration, both VMs 101, 102' are running on the local computer data processing system 100.

Fig. 4c) illustrates the situation, where the local VM 101 running on the local computer data processing system 100 and the remote VM 102' running on the remote computer data processing system 100' are both moved to a third computer data processing system 100".

Of course, the functionalities shown in Figs. 4a) to 4c) can also be combined. E.g. a VM 101 of the local computer data processing system 100 may be moved to computer data processing system 100', while at the same time a VM 102' of the remote computer data processing system 100' is migrated to the local computer data processing system 100. Another VM might be migrated or moved to and/or from the third computer data processing system 100" from/to either the local computer data processing system 100 and/or the remote computer data processing system 100'. Therefore, in case of a plurality of computer data processing systems with a plurality of VMs, the VMs might be exchanged and swapped between the computer data processing systems.

While network traffic can be analyzed to detect whether it is beneficial to carry out a migration of VMs, various information about resources used (such as utilization of CPU, memory, network, storage and other system configuration) of the involved local and remote computer data processing systems as well as the involved local and remote VMs running on the local and remote computer data processing systems can be evaluated (e.g. in respect to the first and/or second threshold value). The evaluation is performed to reach a decision on whether to initiate migration and/or enable shared memory communication.

In case that a migration is to be initiated, the results of the described resource evaluation (such as e.g. utilization of CPU, memory, network or storage) can be used to determine a beneficial migration scenario of local and/or remote VMs to local and/or remote computer data processing systems. For example a computer data processing system 100 with low CPU and memory load can be chosen as a target destination where one or more resource-intensive remote VMs can be migrated to. Especially, it should be noted that the invention is not limited to analyzing network traffic in order to decide whether it is beneficial to initiate a migration of local and/or remote VMs, as inter alia described with reference to Fig. 4, or to enable shared memory communication. A migration or shared memory communication may additionally or alternatively be initiated by evaluating any system resource sate such as a CPU state, a memory state, a network state and/or a storage state. Thus, the first threshold value or the second threshold value may also be determined by evaluating one or a combination of any system resource state such as CPU, memory, network and storage.

When evaluating resources, the same evaluation principles used for network traffic evaluation can be used (i.e. evaluating single values and/or integral, average or median values representing resource utilization or occurrences of device errors and/or response time). A monitoring module 115, as described in view of Fig. 6 below, can be used to gather the necessary information.

A decision logic of the balancer module 105, which is responsible for deciding on configuration changes such as initiating a migration or initiating shared memory communication, may employ an optimization function defined for each VM 101, 102 and/or computer data processing system 100 during the initial configuration of the VMs 101, 102 and/or the computer data processing systems 100. Each optimization function can be redefined at any time. Definition and redefinition of the optimization function can e.g. be carried out based upon user input or machine learning mechanisms/algorithms.

The optimization function can, for example, be implemented as a table of rules, where a threshold value regarding a parameter (e.g. a load value representing a state of a CPU, a network, a memory and/or a storage) can be associated with a predefined action (e.g. initiation of migration or shared memory communication). The entries stored in the table can also be used to decide when an action is to be triggered (e.g. by evaluating a system state of local and/or remote VMs and/or computer data processing systems), but also how an action is to be processed (e.g. by evaluating system states of local and/or remote VMs and/or computer data processing systems). Based thereon it can then be decided which migration scenario or shared memory scenario is most beneficial for the present situation. In view of Fig. 5, a first communication module 103, of the first local VM 101 can comprise a first communication interface 106 and a second communication interface 108 is described. A second communication module 104 of the second local VM 102 can comprise a first communication interface 107 and a second communication interface 109. The first communication interface 106, 108 as well as the second communication interface 107, 109 can handle any type of network communication with any kind of communication partner. However, it is beneficial to use the first communication interface 106, 108 for communication between the local VMs 101, 102, whereas it is advantageous to use the second communication interface 107, 109 for communication of a local VM 101, 102 with a remote communication partner, which can be a remote VM. While shared memory communication is the preferred way of communication between local VMs 101, 102, there can be scenarios where using the first communication interfaces 106, 108 is preferred over the use of shared memory communication. For example this is the case if no memory for shared memory communication is available or if a necessary bandwidth for communication between local VMs 101, 102 is so low, that the overall resource consumption of communication via the first communication interface is better compared to the shared memory communication.

In a specific implementation, the first communication interfaces 106, 108 can be para-virtualized communication devices. The second communication interfaces 107, 109 can be virtual function devices. The virtual function devices can support SR-IOV. Para-virtualized communication devices enable very fast communication between local VMs 101, 102 and provide live migration support. Virtual function devices (in particular those supporting SR-IOV) are capable of directly addressing hardware resources and are therefore best suited for outgoing communication with remote communication partners or remote VMs. A combined use of different types of communication interfaces can be beneficial, as every type of interface can be used for the kind of communication it is best suited for. Overall performance of network communication can be increased by using different types of communication interfaces, e.g. in a hybrid fashion, while at the same time negative effects typical for a certain type of communication interface can be avoided. The first and second communication module 103, 104 hence can each combine a para-virtualized device and virtual function device. A communication module employing such a combination can also be referred to as "bond-zlb".

To facilitate network communication of the local VMs 101, 102 via the first communication interfaces 106, 108 or the second communication interfaces 107, 109, the computer data processing system 100 can provide the hypervisor or host OS 110. The hypervisor or host OS 110 can include a mechanism for network communication bridging or network communication switching 111.

Network communication bridging is used to create an aggregate network from either two or more communication networks or two or more network segments. Regarding the invention, network bridging can facilitate communication of several local VMs by enabling the local VMs to use the same communication network. Network communication switching assures that data transmitted by a network is only forwarded to network devices where the forwarded data is needed for further processing. Transmitted data packets are not modified; only the headers of the data packets that determine to which network device the data is to be forwarded are modified. Network switching prevents data sent from a local VM to a remote VM from also being sent to other local VMs. The mechanism for network communication bridging or network communication switching 111 implemented by the hypervisor or host OS 110 thereby provides efficient and effective communication between local VMs 101, 102 and remote communication partners.

The network commumcation bridging or network communication switching mechanism 111 can comprise backend interfaces 112, 113 corresponding to the first communication interfaces 106, 108 of the communication modules 103, 104 of the local VMs 101, 102. These backend interfaces 112, 113 can be para-virtual backend devices.

Further the network communication bridging or network communication switching mechanism 111 is configured to operate a network interface controller driver 114 in functional association with data transmitted via the second communication interfaces 107, 109 of the communication modules 103, 104 of the local VMs 101, 102. The functional association allows to provide a virtualized network interface controller driver to the second communication interfaces 107, 109 of the communication module 103, 104 of each local VM 101, 102 capable of directly using the hardware resources of the physical network interface device used in the computer data processing system 100.

In this way, necessary processing steps needed to handle data sent to remote communication partners can be processed by the physical network hardware of the computer data processing system 100, which can save other local resources of the computer data processing system 100 (such as CPU and memory utilization). Additionally the physical network interface device(s) can provide S -IOV support. Therefore, the network interface controller driver 114 enables communication of the local VMs 101, 102 with communication partners outside the local computer data processing system 100.

As it is now described in view of Fig. 6, to observe and collect state information of the computer data processing system 100, which are necessary for decision making about migration and shared memory communication, a monitoring module 115 can be implemented. The monitoring module 115 is capable of collecting information about CPU, memory, network and storage utilization and system configuration of the computer data processing system 100. The monitoring module 115 runs on the local computer data processing system 100 and can be part of the hypervisor or host OS 110. It accesses state information of the local computer data processing system 100 as well as of each local VM 101, 102 that is running on the local computer data processing system 100.

In Fig. 6 this is illustrated by lines connecting the monitoring module 115 and the system resources of the computer data processing system 100 (exemplarily shown within the hypervisor/host OS block 110) as well as by lines connecting the monitoring module 115 and several instances of local VMs 101, 102. The monitoring module 115 is capable of detecting general network activity (such as network load or bandwidth used at an interface of a VM) but also the direction of network connections. The monitoring information obtained by the monitoring module 115 and the state information gathered by it is in particular used to decide on whether to establish shared memory communication between local VMs, as explained with reference to the first and second threshold value.

The monitoring module 115 can use polling to actively sample states and state changes especially of the above-mentioned types of resources. The polling can be implemented interrupt-driven and thus as a synchronous process with almost no processing overhead.

State and state changes detected by the monitoring module 115 can be recorded in a state memory database 116. The state memory database may be provided as a database (such as an SQL database, a Key- Value store, a NoSQL database, etc.) or simply one or more text files (e.g. a JSON-format file). In Fig. 6, the monitoring module 115 and the state memory database 116 are shown as part of the hypervisor or host OS 110. However, the monitoring module 115 and the state memory database 116 can also be operated independently from a hypervisor or host OS 110.

In Fig. 7, a more detailed view of the balancer module 105 is shown. The balancer module 105 is shown as a part of the hypervisor or host OS 110 and provides a balancer core 701, a state information unit 702, a listening unit 703 and a broadcasting unit 704.

The balancer module 105 analyzes a system state by obtaining state information from the state memory database 116 and in particular a network communication state and the state of other system resources. Based on the obtained sate information the balancer module 105 decides whether to perform an optimization of the network communication. In particular, it is decided on whether to enable shared memory communication and/or whether to initiate a migration of one or more local VMs 101, 102 and/or one or more remote VMs. The balancer module 105 therefore accesses state information of the local computer data processing system 100 and of remote computer data processing systems.

To analyze the state of the local computer data processing system 100 the balancer module 105 uses the data provided by the monitoring module 115 stored in the state memory database 116. The arrow connecting the state memory database 116 with the balancer core 701 illustrates that the balancer core 701 loads resource utilization data from the state memory database 116. The balancer module 105 can also directly monitor local network communication, e.g. the monitoring module 115 can be part of the balancer module 105.

For analyzing the state of a remote computer data processing system the balancer module 105 uses data provided by the listening unit 703. This is illustrated by the arrow connecting the listening unit 703 and the balancer core 701. The listening unit 703 can collect state information about the state of at least one other remote computer data processing system by communicating with at least one other remote balancer module of the at least one remote computer data processing system. This way, state information and information on pending decisions are shared.

Information about resource utilization, configuration and states of remote computer data processing systems also include information about the resource utilization configuration and states of remote VMs stored in the respective local state memory databases. This allows synchronization of the state information stored in the state memory database 116 with the state information stored in state memory databases of other remote computer data processing systems, respectively. This process is illustrated by the arrow connecting the balancer core 701 with the state memory database 116. The balancer core 701 can access state information of remote computer data processing systems by querying the local (e.g. synchronized) state memory database 116, or by directly querying a remote state memory database, or by forwarding a respective request to a remote balancer module.

The balancer core 701 continuously analyses the provided local and/or remote state information and decides on how to change a configuration of the computer data processing system 100 based on an evaluation of the optimization function according to the decision logic of the balancer module 105.

Once a decision is made and information about a suggested new local configuration is available, the state information unit 702 saves this information to the local state memory database 116 (illustrated by the arrow connecting the state information unit 702 and the state memory database 116). The new local configuration can reflect a planned or active VM migration to/from local and/or remote computer data processing systems and/or a reconfiguration of at least one VM. The decision and the respective configuration can also enable shared memory communication, but also general information about a configuration of the local VMs 101, 102, communication modules 103, 104 or first and/or second communication interfaces 106, 108, 107, 109 can be changed or altered. This information can also be forwarded to other remote balancer modules connected via a communication network by means of broadcasting unit 704. Broadcasting can be carried out at a start, stop and a state synchronization in the operating cycle of the balancer module 105. Communication among balancer modules connected by a communication network can be implemented by means of any transmission scheme such as unicast, multicast, anycast or broadcast.

Whenever a local and a remote VM is involved in a decision of one or more balancer modules, the decision is only accepted and executed, if the balancer module 105 and the involved remote balancer modules consider the decision to be optimal.

In Fig. 7, the balancer module 105 and the state memory database 116 are shown as part of the hypervisor or host OS 110. However, the balancer module 105 and the state memory database 116 can also be operated independently from a hypervisor or host OS 110. In a preferred embodiment, the balancer module 105 is a module separate from the hypervisor or host OS 110.

Turning back to Fig. 5, for carrying out the configuration changes decided on by the balancer module 105 and stored in the state memory database 116, the computer data processing system 100 can further comprise a control module 117. The control module 117 obtains information on how to change the configuration of the state of the local computer data processing system 100 by querying the state memory database 116. The control module 117 is capable of reconfiguring the local VMs 102, 103, the communication modules 103, 104, and/or the first and second communication interfaces 106, 108, 107, 109. Therefore, the control module 117 receives information provided by the balancer module 105, e.g. by querying the state memory database 116.

The obtained information includes reconfiguration information on how to configure the state of the computer data processing system 100 and/or the local VMs 101, 102, but also whether a migration of a local or remote VM from or to the computer data processing system 100 needs to be prepared and executed and/or whether shared memory communication needs to be established. The reconfiguration information can include instructions to control the behavior of the communication module 103, 104, the first communication interface

106, 108 and the second communication interface 107, 109 of the local VM 101, 102. The control module 117 can send detach and attach events to the first communication interface 106, 108 and the second communication interface 107, 109 according to the reconfiguration information. In a migration scenario, this e.g. allows to automate detaching the second communication interface 107, 109 from the local VM 101, 102 that is to be migrated and to enable local and remote network communication by means of the first communication interface 106, 108 while a migration process is going on. After the migration of the VM is finished, the control module 117 can again attach the second communication interface 107, 109 to the migrated VM and facilitates using the first communication interface 106, 108 and the second communication interface

107, 109 for the type of network communication it is best suited for. In a specific implementation, detaching and attaching the second communication interface 107, 109 is supported by a PCIe hot unplug/plug ability of the second communication interface 107, 109.

The control module 117 can initiate communication (e.g. by means of the local balancer module 105) with remote balancer modules to decide whether the suggested reconfiguration instruction intended for the local computer data processing system 100 is compliant with the decisions of the remote balancer modules.

The balancer module 105 can decide whether or not to perform an action, e.g. to migrate a local or remote VM or initiate shared memory communication. The rights to perform this action are then granted to the control module 117 by the balancer module 105. This can be achieved by storing and exchanging the information by means of the state memory database 116, e.g. the balancer module 105 writes information to the state memory database 116, which are then read by the control module 117. The control module 117 then executes the respective actions. The control module can only apply configuration changes to the local computer data processing system 100 and/or the local VMs 101, 102.

Fig. 8 exemplarily shows some typical actions carried out by the control module 117. The local computer data processing system 100 and a remote computer data processing system 800 are shown. The local computer data processing system 100 runs and operates two local VMs 101, 102 and further comprises the state memory database 116 and the control module 117. The balancer module 105, the monitoring module 115, the host OS 110 and other components that can belong to computer data processing systems as known from e.g. Fig. 5 are not shown.

Fig. 8 also shows a remote computer data processing system 800. The remote computer data processing system 800 operates a remote VM 801a and further comprises a state memory database 816 and a control module 817. The balancer module, the monitoring module, the host OS as well as other components of the remote computer data processing system 800 are not shown. The remote VM 801a is to be migrated to computer data processing system 100, where it is referred to as further VM 801b. Fig. 8 shows the further VM 801b, depicted in dashed lines, of the local computer data processing system 100. The dashed lines indicate that VM 801b is operated by computer data processing system 100 after a migration from the remote computer data processing system 800.

The control modules 117, 817 obtain information on configuration changes by querying the corresponding state memory databases 116, 816, respectively. The queries of the control modules 117, 817 to the state memory databases 116, 816 is illustrated by arrow connecting state memory database 116 with control module 117 and by an arrow connecting state memory database 816 with control module 817.

The information returned to the control modules 117, 817 after querying the state memory databases 116, 816 can include information as how to reconfigure states and resources of local VMs, e.g. communication modules, first and or second communication interfaces, as well as system state and resources of the computer data processing system. In particular, information to initiate migration of VMs and to enable shared memory communication of local VMs can be included. Information stored in the state memory databases 116, 816 can be gathered by means of the monitoring modules on each computer data processing system 100, 800 and can be synchronized among all state memory databases by means of the balancer modules on each computer data processing system 100, 800. A balancer module can accept or deny a request for reconfiguration of another balancer module, i.e. it can decide whether it participates in a migration initiated by the other balancer module. The information about accepting or denying a request can also be stored in the state memory databases. In Fig. 8, the control module 117 queries the state memory database 116 for state information, for example, to migrate the local VM 102 away to another computer data processing system. The arrow connecting the control module 117 with local VM 102 illustrates how configuration changes are applied to second local VM 102 and that the control module 117 executes the migration. The arrow pointing from second local VM 102 to the area outside of the computer data processing system 100 illustrates that second local VM 102 is migrated away to another computer data processing system (the system is not shown in Fig. 8). The computer data processing systems can be configured in a way, that the remote balancer module of the computer data processing system where the second local VM 102 is migrated to must have accepted the request of the local balancer module. It is also possible, that the remote balancer module has requested the migration of the second local VM 102 and the local balancer module has accepted the request. As both balancer modules agreed on the migration of second local VM 102, rights are granted to the control module 117 to execute a migration of the second local VM 102.

Fig. 8 further shows, that control module 817 queries reconfiguration information from state memory database 816 by the arrow connecting the state memory database 816 with the control module 817. The arrow connecting the control module 817 with the remote VM 801a (that is presently being operated by the remote computer data processing system 800) illustrates that configuration changes are applied to the remote VM 801a by the control module 817, and that the control module 817 executes the migration of remote VM 801a to the computer data processing system 100. To enable control module 817 to execute the migration of remote VM 801a, the balancer module of the local computer data processing system 100 and the remote computer data processing system 800 agreed on initiating the migration and stored respective reconfiguration information and rights in the state memory databases 116, 816.

The remote VM 801a previously running on the computer data processing system 800 is then, after the migration, running as further VM 801b on the computer data processing system 100. The control module 117 again queries the state memory database 116 for reconfiguration information. The arrows connecting the control module 117 with the local VM 801b and the local VM 101 indicate that configuration changes are applied by the control module 117 to the local VM 101 and the further VM 801b. The configuration changes only affect the local system state of the computer data processing system 100, approval of other balancer modules is not necessary. According to the reconfiguration information applied to the local VM 101 and the further VM 801b, also the shared memory communication can be enabled by the control module 117. This is illustrated by the dashed double headed arrow connecting the local VM 101 and the further VM 801b. The decision to enable shared memory communication between the local VM 101 and the further VM 801b can be made by the local balancer module 105 of the computer data processing system 100 before initiating the migration of the further VM 801a in accordance with the decision of the balancer module of the remote computer data processing system 800. However, the decision can also be made after the migration and based only on the communication performed and monitored between the local VM 101 and the further VM 801b by the monitoring module 115. The initiation of the migration itself is typically effected by the balancer module local to the computer data processing system (as is the decision to enable shared memory communication), e.g. by balancer module 105 for the local computer data processing system 100, while the decision to migrate a VM, e.g. the second local VM 102, can also be originally initiated by a remote balancer module.

The benefits of the decentralized approach of exchanging and evaluating balancing information are also described with reference to Fig. 9. In Fig. 9, the decentralized decision making process is shown for resource balancing. A plurality of computer data processing systems 100, 100', 100" (of course, alternatively computer data processing system 800 could be used) is sown, each of which implements or is associated with a balancer module 105, 105', 105". Network traffic among the balancer modules 105, 105', 105" can be minimized by only synchronizing differential information after an initial complete synchronization. Additionally, the system is completely redundant, and failure of a single balancer module 105, 105', 105" does not affect operation of the system 900 of computer data processing systems 100, 100', 100".

The computer data processing system 100 can further comprise or be functionally connected to a memory module 118 that can facilitate shared memory communication, as shown in Fig. 10. The computer data processing system 100 with the first and the second local VM 101, 102, the control module 117, the memory module 118, and a portion of shared memory space 119 provided by the memory module 118 is shown. The balancer module 105 and the monitoring module 115 as well as the host OS 110 and other entities of the computer data processing system 100 shown in Fig. 5 are not shown.

The control module 117 is used to apply configuration changes decided by the balancer module 105 and communicated by use of the state memory database 116. The control module 117 also can allocate a portion of shared memory space 119 provided by the memory module 118, for each local VM 101, 102. The thin double arrows connecting each local VM 101, 102 with the shared memory space 119 illustrate that each local VM 101, 102 can write and/or read data 1001, 1002 to and/or from the shared memory space 119. Simultaneous or in-place writing and/or reading of data 1001, 1002 of the first and the second local VMs 101, 102 to and/or from the shared memory space 119 thereby is rendered possible and enables a communication between the first and the second local VMs 101, 102 not requiring additional CPU cycles or CPU processing steps to copy information. In particular, no additional CPU processing steps are necessary to copy information stored in a memory space that is exclusively used by the first local VM 101 to a memory space that is exclusively used by the second local VM 102, as the same shared memory space is directly available to both local VMs 101, 102. Thereby, an efficient way (in the sense of utilization of CPU and other system resources) of network communication between the two local VMs 101, 102 is provided.

The shared memory communication between the two local VMs 101, 102 is illustrated by the double headed arrow connecting the local VM 101 and the local VM 102 in Fig. 10. The control module 117 can perform allocation of the shared memory space 119 before shared memory communication is enabled and also can de-allocate the shared memory space 119 when shared memory communication is no longer needed.

Shared memory communication between two local VMs 101, 102 can also be implemented by reading from and writing to the memory buffers of the corresponding other local VM 101, 102 that is involved in shared memory communication. The memory buffers are located in the shared memory space 119. In particular, this can be implemented by mapping the send- and receive-buffers of the corresponding local VMs 101, 102 directly to each other.

The data 1001, 1002 shown within the first and second local VM 101, 102 is on the one hand connected by a thin double arrow with the portion of shared memory space 119 and on the other hand by a double headed arrow with the corresponding data 1001, 1002 in the respective other local VM. This illustrates that the first and second local VM 101, 102 can include a module that is used to handle shared memory communication. This module can be the communication module 103, 104 and in particular the first communication interface 106, 108, as explained with reference to Fig. 5. The communication module 103, 104 or the first communication interface 106, 108 can be used to facilitate shared memory communication, but also a separate module can be implemented inside the first and second local VM 101, 102 for this purpose.

In case that the first communication interface 106, 108 is used to enable shared memory communication, the first communication interface 106, 108 can still support a regular network communication mode. Thus, it is e.g. possible for the first local VM 101 to communicate with the second local VM 102 by means of shared memory communication and simultaneously communicate with a further communication partner (which can be a local and/or remote VM) by means of the first communication interface 106, 108 in a mode of regular network communication, as e.g. implemented in a para-virtualized device.

However, regardless of whether the cornmunication module 103, 104 is used for shared memory communication, the control module 117 can signal to the communication modules 103, 104, when shared memory communication is to be enabled or disabled, so that the communication module can adapt the configuration of the first and or the second communication interfaces 106, 108, 107, 109 accordingly.

In the following section, typical workflows of operations performed by the computer data processing system 100 are described. Each of the following lists of sequences describes a typical operating scenario.

1. Starting the computer data processing system 100 (VMs are not running yet):

- The balancer module 105, the monitoring module 115 and the control module 117 are loaded. At this point, the state memory database 116 can be empty. - The balancer module 105 notifies other remote balancer modules by broadcasting a message that the balancer module 105 is running (the balancer modules may also be organized in a self-organizing network topology, e.g. a topology that is based on distributed hash tables (DHT), and/or, for example, a repository that facilitates initial connection of the balancer modules may be provided).

- Other remote balancer modules that are present on other remote computer data processing systems and connected to the remote balancer module 105 via a computer network reply to the balancer module 105 that they are running (the reply can be sent by user datagram protocol (UDPP) in unicast mode).

- The information gathered from the replying remote balancer modules is stored in the state memory database 116.

2. Starting regular working operation:

- The computer data processing system 100 takes up regular operation.

- An arbitrary number of local VMs are started.

- The momtoring module 115 scans state of resources of the computer data processing system 100 and of the VMs running on the computer data processing system 100 and stores the monitored information in the state memory database 116.

- The balancer module 105 keeps track of the changing monitoring results stored in the state memory database 116 and analyses them.

- The balancer module 105 shares state information of the local computer data processing system 100 and of the VMs running on the local computer data processing system 100 as well as the results of the analysis of the monitoring information stored in the state memory database 116 by the monitoring module

115 with other remote balancer modules running on other remote computer data processing systems.

- The balancer module 105 synchronizes the local state memory database 116 with other remote state memory databases by collecting feedback from other remote balancer modules. - The above described steps are periodically repeated in a background process. 3a. Making decisions to initiate balancing actions involving exclusively local VMs 101, 102:

- The balancer module 105 identifies that a threshold value regarding network traffic is exceeded on at least one local VM 101.

- The balancer module 105 decides to enable shared memory communication between the involved local VMs 101, 102 and stores this decision in the state memory database 116.

- The control module 117 queries the decision of the balancer module 105 from the state memory database 116 and applies the new configuration to the involved local VMs 101, 102. To apply the new configuration, the control module 117 can allocate a portion of shared memory space 119 provided by the memory module 118 and enable shared memory communication before signalling to the communication modules 103, 104 that shared memory communication is used for communication between the local VMs 101, 102.

3b. Making decisions to initiate balancing actions involving non-local VMs:

- The balancer module 105 identifies that a threshold value regarding network traffic is reached concerning traffic between one local and one remote VMs or two remote VMs.

- The balancer module 105 decides to improve communication between the VMs, e.g. to enable shared memory communication between two VMs.

- To achieve this, at least one of the VMs has to be migrated to a computer data processing system running the other VM that is involved in communication.

- The balancer module 105 decides to migrate the VM running on the computer data processing system that is suffering from higher resource utilization (for example by evaluating CPU, memory or storage load) to a computer data processing system that is subject to a lower resource utilization. It is possible that when the VMs are both running on a computer data processing system of which each is subject to high resource utilization, both VMs are migrated to a third computer data processing system with low resource utilization.

- The balancer module 105 notifies the other remote balancer modules about this decision (e.g. by means of the state memory database 116). - The remote balancer modules involved in the decision of the balancer module 105 can grant or reject the decision based on their own analysis of a current computer data processing system state. The result of the decision can again be shared among the balancer modules by means of the state memory databases. - The control module 117 initiates migration according to the decision of the balancer modules (which the control module 117 received from the state memory database 116) and enables shared memory communication of the now local VMs after the migration is completed.

In this, steps 3a. or 3b. can be executed exclusively following steps 1. and 2. or they can be executed in combination. Of course, step 3b. may also be executed before step 3a. and step 3a. and/or 3b. may be executed repeatedly.

The next section describes how shared memory communication can be automatically be set-up and stopped again:

a. Automatically setting up shared memory communication:

- Initially two or more local VMs 101, 102 are running on a same computer data processing system 100.

- The balancer module 105 decides that shared memory communication between two local VMs 101, 102 is necessary. The decision can be based on analysis of traffic between the two local VMs 101, 102 but also on a preceding migration, which already was initiated because of high network traffic between the now local VMs 101, 102.

- The balancer module 105 notifies the control module 117 by means of the state memory database 116 about its decision.

- The control module 117 allocates a portion of shared memory space 119 provided by the memory module 118 for each local VM 101, 102 and enables shared memory communication between the two local VMs 101, 102.

- The communication modules 103, 104 are notified that now shared memory communication is used for communication between the two local VMs 101, 102

b. Automatically stopping shared memory communication:

- Shared memory communication can be stopped if communication between two local VMs 101, 102 does no longer require high-bandwidth so that another way of network communication can be used. Or if migration of a local VM 101, 102 to a remote computer data processing system is requested by a balancer module.

- Initially there are two local VMs 101, 102 running on the same computer data processing system 100 that are using shared memory communication.

- The balancer module 105 decides that communication via shared memory communication is no longer needed and signals the control module 117 to reconfigure communication between the two local VMs 101, 102.

- Information between the balancer module 105 and the control module 117 is exchanged by means of the state memory database 116.

- The control module 117 configures the communication modules 103, 104 of the local VMs 101, 102 to use another communication interface than the shared memory communication.

- The control module 117 safely de-allocates the shared memory space 119 provided by the memory module 118.

The invention has been described in conjunction with various embodiments herein. However, other variations to the enclosed embodiments can be understood and effected by those skilled in the art and practicing the claimed invention, from a study of the drawings, the disclosure and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. A computer program may be stored/distributed on a suitable medium, such as an optical storage medium or a solid- state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the internet or other wired or wireless telecommunication systems.

Claims

1. Computer data processing system (100) for virtualizing at least partially computing hardware, and being adapted to run local virtual machines (101, 102), wherein the system comprises:

-a means to provide, to a local virtual machine (101, 102), a communication module (103, 104) and wherein the local virtual machine (102, 103) is configured to use the communication module (103, 104) for network communication,

-a balancer module (105) configured to analyze a network commumcation between the local virtual machine (101) and a communication partner, the balancer module (105) being configured to enable a shared memory communication if the communication partner is a second local virtual machine (102) of the computer data processing system (100) and network communication traffic of the communication reaches a first threshold value.

2. Computer data processing system (100) according to claim 1, wherein the balancer module (105) is configured to initiate a migration of the local virtual machine (101, 102) and/or of the communication partner if the communication partner is a local virtual machine (101, 102) or a remote virtual machine and network communication traffic of the communication reaches a second threshold value.

3. Computer data processing system according to claim 1 or 2, wherein the communication module (103, 104) comprises a first communication interface (106, 108) and a second commumcation interface (107, 109), wherein the balancer module (105) is configured to enable a communication through the first communication interface (106, 108) if the communication partner is the second local virtual machine (102) and/or to enable a communication through the second communication interface (107, 109) if the communication partner is the remote virtual machine, wherein the first communication interface (106, 108) is a para-virtualized communication device, and wherein the second communication interface (107, 109) is a virtual function device.

4. Computer data processing system (100) according to any one of the preceding claims, wherein the computer data processing system (100) comprises a hypervisor or host OS (110), and wherein the hypervisor or host OS (110) comprises a mechanism for network communication bridging or network communication switching (111), and wherein the network communication bridging or network communication switching mechanism (111) comprises a backend interface (112, 113) for the first communication interface, and wherein the backend interface (112, 113) is a para- virtual backend device.

5. Computer data processing system (100) according to claim 5, wherein the network communication bridging or network communication switching mechanism (111) is configured to operate a network interface controller driver (114) in functional association with data transmitted via the second communication interface (107, 109), and wherein the network interface controller driver (114) is a physical function with SR-IOV support.

6. Computer data processing system (100) according to claim 6, wherein the network interface controller driver (114) is configured to enable communication of the local virtual machine (101, 102) with communication partners outside the computer data processing system (100).

7. Computer data processing system (100) according to any one of the preceding claims, wherein the computer data processing system (100) comprises a monitoring module (115) configured to monitor the computer data processing system (100) and comprises means to measure a CPU utilization, means to measure a memory space utilization, means to measure network utilization, means to measure resources utilization of local virtual machines, means to access, to read from and/or to write to a state memory database.

8. Computer data processing system (100) according to claim 7, wherein the monitoring module (115) is configured to detect a state of the computer data processing system (100) and configuration changes of local resources of the computer data processing system (100) and to record the state of the computer data processing system (100) and the configuration changes in the state memory database (116).

9. Computer data processing system (100) according to any one of the preceding claims, wherein the computer data processing system (100) comprises a control module (117) configured to obtain information on configuration changes by querying the state memory database (116), wherein the control module (117) is configured to reconfigure at least one of the local virtual machines (101, 102), the communication module (103, 104), the first and/or the second communication interface (106, 108, 107, 109) based on the information obtained from the state memory database (116).

10. Computer data processing system (100) according to any one of the preceding claims, wherein the computer data processing system (100) comprises or is functionally connected to a memory module (118) and wherein the control module (117) is configured to allocate a portion of a shared memory space (119) provided by the memory module (118) for each local virtual machine (101, 102), and wherein each local virtual machine (101, 102) is configured to store and/or read data to/from the allocated shared memory space (119) to send/receive data from another local virtual machine (101, 102), and wherein the control module (117) is configured to perform the allocation or de-allocation of the shared memory space (119) to establish shared memory connections between the local virtual machine (101) and the second local virtual machine (102) based on instructions from the balancer module (105) to enable the shared memory communication.

11. Computer data processing system (100) according to any one of the preceding claims, wherein the balancer module (105) is configured to analyze a system state by querying the state memory database (116) and/or network communication, to determine whether a balancing action or an optimization of a network communication needs to be performed, and to store the analysis result in the state memory database (116).

12. Computer data processing system (100) according to any one of the preceding claims, wherein the balancer modules are configured to collect states of remote virtual machines from other remote balancer modules, and/or wherein the balancer module (105) is configured to synchronize the state memory database (116) with state memory databases of the other remote balancer modules.

13. Computer data processing system (100) according to any one of the preceding claims, wherein the balancer module (105) is configured to provide respective reconfiguration information to the control module (117) and/or is configured to decide whether a local and/or remote virtual machine should be migrated to the computer data processing system (100) or another computer data processing system and to enable the migration by communicating with the other remote balancer modules.

14. Computer data processing system (100) according to any one of the preceding claims, wherein the balancer module (105) is configured to decide whether or not to migrate the local or remote virtual machine based on information stored in the state memory database (116), and wherein the balancer module (105) is configured to grant the control module (117) rights to perform the migration, and wherein the control module (117) is configured to execute the migration.

15. Method for operating a computer data processing system (100) for virtualizing at least partially computing hardware, the computer data processing system being adapted to run local virtual machines (101, 102), wherein the system

-provides, to a local virtual machine (101, 102), a communication module (103, 104) and wherein the local virtual machine (101, 102) uses the communication module (103, 104) for network communication, wherein

-a balancer module (105) analyzes a network communication between the local virtual machine (101) and a communication partner, the balancer module (105) enabling a shared memory communication if the communication partner is a second local virtual machine (102) and network communication traffic of the communication reaches a first threshold value.