WO2017067391A1 - 虚拟机的数据共享方法及装置 - Google Patents
虚拟机的数据共享方法及装置 Download PDFInfo
- Publication number
- WO2017067391A1 WO2017067391A1 PCT/CN2016/101631 CN2016101631W WO2017067391A1 WO 2017067391 A1 WO2017067391 A1 WO 2017067391A1 CN 2016101631 W CN2016101631 W CN 2016101631W WO 2017067391 A1 WO2017067391 A1 WO 2017067391A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- shared
- virtual machine
- data
- address information
- packet
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 80
- 238000000151 deposition Methods 0.000 claims description 8
- 239000000284 extract Substances 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 11
- 238000004891 communication Methods 0.000 description 9
- 230000003993 interaction Effects 0.000 description 7
- 230000005540 biological transmission Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 5
- 238000012546 transfer Methods 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 2
- 238000002360 preparation method Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 1
- 238000005538 encapsulation Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 239000000047 product Substances 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/40—Support for services or applications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
Definitions
- the present disclosure relates to the field of communications technologies, for example, to a data sharing method and apparatus for a virtual machine.
- NFV Network Function Virtualization
- Network function virtualization needs to provide high-speed data forwarding, including data forwarding between server physical network cards and virtual machines. Data forwarding between virtual NICs.
- the forwarding function of the virtual network in the server is generally implemented by a virtual bridge or an Open Virtual Switch (OVS), but both the virtual bridge and the OVS are running in the kernel mode, and the virtual machines are located in the user. State, and when communicating between virtual machines, data needs to be forwarded via virtual bridge or OVS, and data needs to be copied multiple times when forwarding between user mode and kernel mode. Therefore, in this case, Sending data from one virtual machine to another requires multiple copies to forward data, resulting in inefficient data sharing between virtual machines.
- OVS Open Virtual Switch
- the present disclosure proposes a data sharing method and device for a virtual machine, which solves the technical problem of low data sharing efficiency between virtual machines.
- a data sharing method for a virtual machine includes:
- the first virtual machine stores the first to-be-shared data into the shared memory
- the first virtual machine acquires storage address information and a destination physical address of the first data to be shared;
- the first virtual machine encapsulates the storage address information and the destination physical address into a first to-be-shared message
- the first virtual machine stores the first to-be-shared message in a sending queue corresponding to the first virtual machine
- the first virtual machine stores the first to-be-shared packet in the receiving queue corresponding to the destination physical address, so that the second virtual machine corresponding to the destination physical address is shared according to the first to be shared.
- the first address to be shared is obtained by the storage address information in the packet, and the first virtual machine and the second virtual machine are running in the same host.
- the present disclosure also provides a data sharing device for a virtual machine, including:
- a storage module configured to store the first to-be-shared data in the shared memory
- a first acquiring module configured to acquire storage address information and a destination physical address of the first data to be shared
- a first encapsulating module configured to encapsulate the storage address information and the destination physical address into a first to-be-shared message
- the depositing module is configured to store the first to-be-shared message in a sending queue corresponding to the first virtual machine
- the depositing module stores the first to-be-shared packet in the receiving queue corresponding to the destination physical address, so that the second virtual machine corresponding to the destination physical address is based on the first to-be-shared packet.
- the storage address information in the text obtains the first to-be-shared data, and the data sharing device and the second virtual machine run in the same host.
- the present disclosure also proposes a non-transitory computer readable storage medium storing computer executable instructions for performing the data sharing method of the virtual machine described above.
- the first virtual machine first stores the first to-be-shared data in the shared memory, and then acquires the storage address information and the destination physical address of the first to-be-shared data, and The storage address information and the destination physical address are encapsulated into a first to-be-shared packet, and finally the first to-be-shared packet is stored in a sending queue corresponding to the first virtual machine, and the first The second virtual machine corresponding to the destination physical address obtains the first to-be-served according to the storage address information in the first to-be-shared packet, where the to-be-shared packet is stored in the receiving queue corresponding to the destination physical address.
- the technical solution of the present disclosure does not need to copy and forward data through a virtual bridge or a virtual switch.
- data is stored in a memory sharing area, and the storage is performed.
- the storage address information of the data is notified to other virtual machines, so that other virtual machines acquire the data in the shared memory, thereby improving the virtual machine.
- Data sharing efficiency is not need to copy and forward data through a virtual bridge or a virtual switch.
- FIG. 1 is a schematic flowchart of a first embodiment of a data sharing method of a virtual machine according to the present disclosure
- FIG. 2 is a schematic diagram of shared memory between virtual machines of the present disclosure
- FIG. 3 is a schematic diagram of a format of a message of the present disclosure.
- FIG. 4 is a schematic flowchart of a second embodiment of a data sharing method of a virtual machine according to the present disclosure
- FIG. 5 is a schematic flowchart of a third embodiment of a data sharing method of a virtual machine according to the present disclosure
- FIG. 6 is a schematic diagram of functional modules of a first embodiment of a data sharing device of a virtual machine according to the present disclosure
- FIG. 7 is a schematic diagram of functional modules of a second embodiment of a data sharing device of a virtual machine according to the present disclosure.
- FIG. 8 is a schematic diagram of functional modules of a third embodiment of a data sharing device of a virtual machine according to the present disclosure.
- FIG. 9 is a structural block diagram of an electronic device according to an embodiment of the present disclosure.
- the present disclosure provides a data sharing method of a virtual machine.
- FIG. 1 is a schematic flowchart diagram of a first embodiment of a data sharing method for a virtual machine according to the present disclosure.
- This embodiment provides a data sharing method for a virtual machine, where the data sharing method of the virtual machine includes:
- step S10 the first virtual machine stores the first to-be-shared data in the shared memory.
- the application scenario is to share data between multiple virtual machines in the same host, and when data sharing is performed on multiple virtual machines in the same host, in order to reduce operations such as copying and forwarding data
- the memory sharing technology realizes the control state transfer between the virtual machines and the interaction of the network data, that is, the shared memory is set in the host, and the data to be shared is stored in the shared memory, and the shared memory is used to complete the virtual machine. Data interaction and control state information interaction.
- the shared memory includes a common data sharing area, where the data sharing area is used for storing and receiving messages between virtual machines, and the shared memory further includes a control state shared storage area, and the control state shared storage.
- the area is mainly used to exchange message descriptors and status information between the virtual machine and the data forwarding process in the host (ie, I/O (Input/Output) process), and from the perspective of the virtual machine, it
- the transmission control state description area and the receiving control status description area from the host I/O to the virtual machine are subdivided into a virtual machine to host data forwarding process.
- the data forwarding process should include one or more I/O threads.
- the number of network planes supported by the host is determined, and one plane corresponds to one I/O thread, for example, all of a host machine. If the virtual machine has only two network planes, the control plane and the data plane, it can start only two I/O threads, one thread handles the data transmission and reception of the data plane, and one thread is used for processing control. Face data forwarding.
- Step S20 The first virtual machine acquires storage address information and a destination physical address of the first data to be shared.
- Step S30 The first virtual machine encapsulates the storage address information and the destination physical address into a first to-be-shared packet.
- the storage address information and the destination physical address of the first to-be-shared data are acquired first, and then the obtained location is obtained.
- the storage address information and the destination physical address are encapsulated into a first to-be-shared message.
- the to-be-shared message includes: CSD (Control-Status Descriptors), BD (buffer Descriptors), and buffer description.
- the CSD descriptor includes size message size, header header pointer, tail tail pointer, Flags flag bit
- BD includes Flags BD tag, len length, index index, buf_ptr message physical address, etc., where size defines each The number of BDs of the message, when the header and tail are used to send and receive data, the first and last index numbers of the BDs with the data are identified, and each data packet occupies one BD, and the BD is mainly used to describe the buffer address for storing and receiving data. The length of the data sent and received. Flags defines some function flags of CSD, which are used to store some feature flags of data messages, and index is used to identify the index of BD. The most important part of BD is buf_ptr and len, buf_ptr is used to store the physical address of the buffer for sending and receiving data, and len is the data payload length of the buffer.
- step S40 the first virtual machine stores the first to-be-shared message in the sending queue corresponding to the first virtual machine.
- the first virtual machine stores the first to-be-shared packet in the receiving queue corresponding to the destination physical address, so that the second virtual machine corresponding to the destination physical address is shared according to the first to be shared.
- the first address to be shared is obtained by the storage address information in the packet, and the first virtual machine and the second virtual machine are running in the same host.
- the first virtual machine invokes a data forwarding process to store the to-be-shared message in a sending queue corresponding to the first virtual machine, and the first step is performed by the data forwarding process.
- a to-be-shared message is stored in the receiving queue corresponding to the destination physical address.
- the data forwarding process runs in a user state, that is, by calling the data forwarding process to perform data sharing between virtual machines, data copying and forwarding can be avoided, thereby improving data sharing efficiency, and a better understanding of the implementation.
- the application scenario is as follows:
- the virtual network card driver of the first virtual machine first applies for the control state descriptor CSD from the shared memory area, obtains the BD descriptor by the CSD, and then applies for the pkt_buf address. Converting the pkt_buf address into the virtual address of the first virtual machine, and preparing the first to-be-shared data to be sent, that is, copying the first to-be-shared data into the pkt_buf address, and filling in the len (the buffer) The data payload length is), and then the physical address of the pkt_buf is filled in the buf_ptr field in the BD descriptor, encapsulated into the first to-be-shared message, and the preparation for sending the data is completed.
- the BD of the receiving queue is processed by calling the network card driver by calling the data forwarding process, and the host physics pointed to by the buf_ptr of the BD is first acquired. The address is converted into a virtual machine address of the second virtual machine. Finally, the first to-be-shared message is obtained to complete communication between the virtual machines.
- the data forwarding process guards the sending queue of the virtual network card of the virtual machine, that is, the first to-be-shared packet corresponding to the first virtual machine, and the data forwarding process sequentially takes the tail from multiple queues.
- the tag starts to process the packet, obtains the buf_prt address in the BD, and converts it into a virtual address of the data forwarding process, and obtains the destination physical address of the first to-be-shared packet.
- the pre-stored forwarding table is searched according to the destination physical address.
- the receiving queue corresponding to the second virtual machine is extracted, and parameters such as buf_prt, flags, and len of the sending queue are directly copied to the buf_ptr, flags, and len fields of the receiving queue, and finally the receiving queue with the added parameter is sent to the receiving queue.
- the second virtual machine corresponding to the physical address is described.
- the I/O process After the I/O process receives the first to-be-shared packet from the sending queue of the virtual machine, the I/O process searches the forwarding table according to the destination physical address, finds the receiving queue of the first to-be-shared packet, and the first The to-be-shared packet is sent to the receiving queue, so that the second virtual machine corresponding to the destination physical address obtains the first to-be-shared data according to the storage address information in the first to-be-shared packet.
- the first virtual machine first stores the first to-be-shared data in the shared memory, and then acquires the storage address information and the destination physical address of the first to-be-shared data, and
- the storage address information and the destination physical address are encapsulated into a first to-be-shared packet, and the first to-be-shared packet is stored in the sending queue corresponding to the first virtual machine, and the first to-be-sent
- the shared message is stored in the receiving queue corresponding to the destination physical address, and the second virtual machine corresponding to the destination physical address is configured to obtain the first to be shared according to the storage address information in the first to-be-shared packet.
- Data is different from data sharing between virtual machines in the related art.
- the present disclosure does not need to copy and forward data through a virtual bridge or a virtual switch.
- data is stored in a memory shared area, and a storage address of the stored data is stored.
- Information is sent to other virtual machines for other virtual machines to obtain the data in shared memory, thereby improving data sharing between virtual machines effectiveness.
- a second embodiment of the data sharing method of the virtual machine in this embodiment, referring to FIG. 4, after the step S20, the data sharing method of the virtual machine may further include:
- step S50 when the destination physical address acquisition fails, the first virtual machine encapsulates the storage address information into a first to-be-shared packet.
- step S40 the first virtual machine stores the first to-be-shared message in the sending queue corresponding to the first virtual machine.
- the first virtual machine may send a broadcast first to be shared message, or may not find a queue in the forwarding table in the first to-be-shared message.
- the shared packet is processed in a special way. That is, a first packet to be shared needs to be sent to each virtual machine. Then, the virtual machine needs to apply for pkt_buf for the receiving queues of other multiple virtual machines, and copy the message payload to all pkt_buf, copy len and flags to the corresponding fields of the BD descriptor, and convert pkt_buf into the host physical address and Fill in the buf_ptr field.
- the destination physical address of the virtual machine to be shared is not added to the first to-be-shared packet.
- the storage address information is directly encapsulated into the first to be shared. And sharing the packet, and invoking the data forwarding process, to save the first to-be-shared packet into the sending queue corresponding to the first virtual machine, that is, the data forwarding process cannot be the first to-be-shared packet.
- the data forwarding process sending the extracted storage address information to other virtual machines in the host corresponding to the first virtual machine, that is, the data forwarding process will be the first
- the to-be-shared message is stored in the receiving queues corresponding to the other plurality of virtual machines, so that the other plurality of virtual machines obtain the first to-be-shared data according to the storage address information in the first to-be-shared packet, to complete the first The virtual machine shares data with other virtual machines.
- a third embodiment of the data sharing method of the virtual machine of the present disclosure is proposed based on the first embodiment.
- the data sharing method of the virtual machine may further include:
- step S60 the first virtual machine acquires the second to-be-shared packet stored in the receiving queue of the first virtual machine in real time or periodically.
- Step S70 The first virtual machine extracts storage address information in the second to-be-shared packet when the second to-be-shared packet is obtained.
- Step S80 The first virtual machine acquires second to-be-shared data corresponding to the storage address information in the shared memory according to the storage address information.
- the first virtual machine acquires the storage in the receiving queue in the first virtual machine.
- the method of the second to-be-shared message includes: 1) timing acquisition, that is, the manner in which the user presets the first virtual machine to obtain the to-be-shared packet in the host machine is obtained periodically, such as every 10 minutes. Whether there is a message to be shared in the queue. 2) real-time acquisition, that is, the manner in which the user presets the first virtual machine to obtain the to-be-shared message in the host machine in advance is acquired in real time, and then, as long as there is a to-be-shared in the receiving queue of the first virtual machine The first virtual machine can detect and obtain the to-be-shared message.
- the first virtual machine extracts the storage address information in the second to-be-shared packet, and obtains the shared memory according to the storage address information.
- the second to-be-shared data corresponding to the address information is stored to complete data sharing between the virtual machines.
- At least one of the first to-be-shared data and the second to-be-shared data is generated by multiple data packets, that is, the first virtual machine is stored in the memory sharing.
- the stored data is generated by multiple data packets, and the data forwarding process may acquire multiple at least one of the first to-be-shared packet and the second to-be-shared packet.
- Multiple parameters corresponding to the data packet enable multiple data packets to be called at one time, which reduces the average time of each data packet interaction, thereby improving the efficiency of data sharing between virtual machines.
- a larger packet length can also be used to send and receive data.
- the length of the MTU (Maximum Transmission Unit) of the general network is 1500 bytes, and a larger MTU can be used in practical applications to reduce each The cost of the packet, which requires that the two ends of the packet to be shared and received must use the same MTU. If the MTUs of the two ends do not match, the packets may be fragmented and cannot communicate normally.
- MTU Maximum Transmission Unit
- the forwarding table is generated or updated according to the source physical address of the packet, that is, a correspondence table between the destination physical address and the virtual machine receiving queue is generated or updated, that is, the I/O process maintains one for each I/O thread. Forwarding table, the correspondence between the physical address of the forwarding table maintenance destination and the virtual machine receiving queue.
- the present disclosure optionally provides a data sharing device for a virtual machine.
- FIG. 6 is a schematic diagram of functional modules of a first embodiment of a communication device of a virtual machine according to the present disclosure.
- the functional block diagram shown in FIG. 6 is merely an exemplary diagram of an alternative embodiment.
- the functional modules of the data sharing device of the virtual machine shown in FIG. 6 can be easily performed by those skilled in the art.
- a supplement to the new functional modules; the name of each functional module is a custom name, which is only used to assist in understanding the multiple program function blocks of the virtual machine's data sharing device.
- This embodiment provides a data sharing device for a virtual machine, where the data sharing device of the virtual machine includes:
- the storage module 10 is configured to store the first to-be-shared data into the shared memory.
- the application scenario is to share data between multiple virtual machines in the same host, and when data sharing is performed on multiple virtual machines in the same host, in order to reduce operations such as copying and forwarding data
- the memory sharing technology realizes the control state transfer between the virtual machines and the interaction of the network data, that is, the shared memory is set in the host, and the storage module 10 stores the data to be shared into the shared memory, where the shared memory is used.
- the data interaction between the virtual machines and the control state information are completed.
- the shared memory includes a common data sharing area, where the data sharing area is used to store and receive messages between virtual machines.
- the shared memory also includes a control state shared storage area, and the control state shared storage area is mainly used for exchanging message descriptors and status information between the virtual machine and the data forwarding process (ie, I/O process) in the host, and from the virtual From the perspective of the machine, it is subdivided into the transmission control state description area of the virtual machine to the host data forwarding process and the connection from the host I/O to the virtual machine. Control state description area.
- the data forwarding process should include one or more data I/O threads. Generally, the number of network planes supported by the host is determined, and one plane corresponds to one I/O thread, for example, a host machine. All virtual machines can only start two I/O threads if there are only two control planes and data planes. One thread handles the data transmission and reception of the data plane, and one thread handles the data forwarding of the control plane.
- the first obtaining module 20 is configured to acquire storage address information and a destination physical address of the first data to be shared.
- the first encapsulating module 30 is configured to encapsulate the storage address information and the destination physical address into a first to-be-shared message.
- the first acquiring module 20 first obtains the storage address information and the destination physical address of the first to-be-shared data. Then, the first encapsulating module 30 encapsulates the obtained storage address information and the destination physical address into a first to-be-shared message.
- the to-be-shared message includes: CSD (Control-Status Descriptors).
- BD buffer Descriptors
- CSD descriptors include size message size, header header pointer, tail tail pointer, Flags flag bit
- BD includes Flags BD tag, len length, index index, buf_ptr The physical address of the message, etc., where size defines the number of BDs for each message, and the header and tail are set to the first and last index numbers of the BDs that identify the data when the data is sent and received, and each packet occupies one BD.
- BD is mainly used to describe the buffer address for storing and receiving data and the length of data to be sent and received.
- Flags defines some function flags of CSD, which are used to store some feature flags of data messages, and index is used to identify the index of BD. The most important part of BD is buf_ptr and len, buf_ptr is used to store the physical address of the buffer for sending and receiving data, and len is the buffer. The data payload length of the zone.
- the depositing module 40 is configured to store the first to-be-shared message in a sending queue corresponding to the first virtual machine.
- the depositing module stores the first to-be-shared packet in the receiving queue corresponding to the destination physical address, so that the second virtual machine corresponding to the destination physical address is based on the first to-be-shared packet.
- the storage address information in the text obtains the first to-be-shared data, and the data sharing device and the second virtual machine run in the same host.
- the first virtual machine invokes a data forwarding process to store the to-be-shared message in a sending queue corresponding to the first virtual machine, and the first step is performed by the data forwarding process.
- the to-be-shared packet is stored in the receiving queue corresponding to the destination physical address, and the data forwarding process runs in a user state, that is, by calling the data forwarding process to perform data sharing between virtual machines, data can be avoided. Copy and forward, which improves the efficiency of data sharing.
- the first obtaining module 20 requests a control state descriptor CSD from a shared memory area, obtains a BD descriptor from the CSD, and then applies a pkt_buf address, and converts the pkt_buf address into a virtual address of the first virtual machine, and Preparing the first to-be-shared data to be sent, that is, copying the first to-be-shared data into the pkt_buf address, and filling in len (the data payload length of the buffer), and then filling the physical address of the pkt_buf into the BD description.
- the first encapsulating module 30 is encapsulated into the first to-be-shared message, and the preparation for transmitting data is completed.
- the BD of the receiving queue is processed by calling the network card driver by calling the data forwarding process, and the host physics pointed to by the buf_ptr of the BD is first acquired. The address is converted into a virtual machine address of the second virtual machine. Finally, the first to-be-shared message is obtained to complete communication between the virtual machines.
- the data forwarding process guards the sending queue of the virtual network card of the virtual machine, that is, the first to-be-shared packet corresponding to the first virtual machine, and the data forwarding process sequentially takes the tail from multiple queues.
- the tag starts to process the packet, obtains the buf_prt address in the BD, and converts it into a virtual address of the data forwarding process, and obtains the destination physical address of the first to-be-shared packet.
- the pre-stored forwarding table is searched according to the destination physical address.
- the receiving queue corresponding to the second virtual machine is extracted, and parameters such as buf_prt, flags, and len of the sending queue are directly copied to the buf_ptr, flags, and len fields of the receiving queue, and finally the receiving queue to which the parameter is added is sent to The second virtual machine corresponding to the destination physical address.
- the I/O process After the I/O process receives the first to-be-shared packet sent from the sending queue of the virtual machine, Determining a forwarding table of the destination physical address, finding a receiving queue of the first to-be-shared packet, and sending the first to-be-shared packet to the receiving queue, where the second virtual machine corresponding to the destination physical address is used Acquiring the first to-be-shared data according to the storage address information in the first to-be-shared packet.
- the first virtual machine first stores the first to-be-shared data in the shared memory, and then acquires the storage address information and the destination physical address of the first to-be-shared data, and The storage address information and the destination physical address are encapsulated into a first to-be-shared packet, and the first to-be-shared packet is stored in the sending queue corresponding to the first virtual machine, and the first to-be-sent The shared message is stored in the receiving queue corresponding to the destination physical address, and the second virtual machine corresponding to the destination physical address is configured to obtain the first to be shared according to the storage address information in the first to-be-shared packet.
- the technical solution of the present disclosure does not need to copy and forward data through the virtual bridge or the virtual switch.
- the data is stored in the memory sharing area, and the data is stored.
- the storage address information is notified to other virtual machines for other virtual machines to acquire the data in the shared memory, thereby improving the relationship between the virtual machines. According to the efficiency of sharing.
- a second embodiment of the data sharing device of the virtual machine of the present disclosure is proposed based on the first embodiment.
- the data of the virtual machine is used.
- Shared devices include:
- the second encapsulating module 50 is configured to encapsulate the storage address information into a first to-be-shared packet when the destination physical address acquisition fails.
- the depositing module 40 is configured to store the first to-be-shared message in a sending queue corresponding to the first virtual machine.
- the data sharing device may send a broadcast to the first to be shared message, or may not find the queue in the forwarding table in the first to-be-shared message.
- the packet is processed in a special manner. That is, a first packet to be shared needs to be sent to each virtual machine. Then, the second encapsulating module 50 needs to apply for pkt_buf for the receiving queues of other multiple virtual machines, and copy the packet payload to all pkt_bufs, copy len and flags to the corresponding fields of the BD descriptor, and convert the pkt_buf into a host. Physical address and fill in the buf_ptr field.
- the second encapsulation module 50 directly stores the storage.
- the address information is encapsulated into a first to-be-shared message, and the storage module 40 invokes a data forwarding process to store the first to-be-shared message into the first virtual machine pair. If the data forwarding process is unable to obtain the destination physical address from the first to-be-shared packet, the data forwarding process sends the extracted storage address information to the first virtual machine.
- the data forwarding process stores the first to-be-shared message in a receiving queue corresponding to the other plurality of virtual machines, so that the other multiple virtual machines are in accordance with the
- the storage address information in the first to-be-shared packet acquires the first to-be-shared data to complete data sharing between the first virtual machine and other multiple virtual machines.
- a third embodiment of the data sharing device of the virtual machine of the present disclosure is proposed based on the first embodiment.
- the data of the virtual machine is used.
- the shared device also includes:
- the second obtaining module 60 is configured to acquire the second to-be-shared message stored in the receiving queue in real time or periodically.
- the extracting module 70 is configured to extract the storage address information in the second to-be-shared packet when the second to-be-shared packet is obtained.
- the second obtaining module 60 is configured to acquire, according to the storage address information, second to-be-shared data corresponding to the storage address information in the shared memory.
- the manner in which the second obtaining module 60 acquires the to-be-shared message stored in the receiving queue in the first virtual machine includes: 1) timing acquisition, that is, the user is in the host machine beforehand. Presetting the manner in which the first virtual machine obtains the to-be-shared packet is a timing acquisition, such as detecting whether there is a message to be shared in the received queue every 10 minutes. 2) real-time acquisition, that is, the manner in which the user presets the first virtual machine to obtain the to-be-shared message in the host machine in advance is acquired in real time, and then, as long as there is a to-be-shared in the receiving queue of the first virtual machine The message, the first virtual machine can be detected.
- the extracting module 70 extracts the storage address information in the second to-be-shared packet, and the second acquiring module 60 is configured according to the The second address to be shared corresponding to the storage address information in the shared memory is obtained by storing the address information, so as to complete data sharing between the virtual machines.
- At least one of the first to-be-shared data and the second to-be-shared data is generated by multiple data packets, that is, the first virtual machine is stored in the memory sharing.
- the stored data is generated by multiple data packets.
- the data forwarding process detects at least one of the first and second to-be-shared packets, the data forwarding process may obtain multiple data packets. Multiple parameters, enabling multiple calls to be made at once, reducing the average time for each packet interaction This improves the efficiency of data sharing between virtual machines.
- a larger packet length can also be used to send and receive data.
- the length of the MTU (Maximum Transmission Unit) of the general network is 1500 bytes, and a larger MTU can be used in practical applications to reduce each The cost of the packet, which requires that the two ends of the packet to be shared and received must use the same MTU. If the MTUs of the two ends do not match, the packets may be fragmented and cannot communicate normally.
- MTU Maximum Transmission Unit
- the forwarding table is generated or updated according to the source physical address of the packet, that is, a correspondence table between the destination physical address and the virtual machine receiving queue is generated or updated, that is, the I/O process maintains one for each I/O thread. Forwarding table, the correspondence between the physical address of the forwarding table maintenance destination and the virtual machine receiving queue.
- the foregoing embodiment method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be through hardware, but in many cases, the former is optional.
- Implementation Based on such understanding, the technical solution of the present disclosure, which is essential or contributes to the related art, may be embodied in the form of a software product stored in a storage medium (such as ROM/RAM, disk, CD-ROM).
- the instructions include a number of instructions for causing a terminal device (which may be a cell phone, computer, server, air conditioner, or network device, etc.) to perform the methods described in various embodiments of the present disclosure.
- embodiments of the present disclosure also provide a non-transitory computer readable storage medium storing computer executable instructions for executing a data sharing method of the above virtual machine.
- FIG. 9 is a structural block diagram of an electronic device according to an embodiment of the present disclosure.
- the electronic device provided by the embodiment of the present disclosure may include a processor 81 and a memory 83, and may further include a communication interface 82 and a bus 84.
- the processor 81, the communication interface 82, and the memory 83 can complete communication with each other through the bus 84.
- Communication interface 82 can be used for information transfer.
- the processor 81 can call the logic instructions in the memory 83 to execute the data sharing method of the virtual machine of the above embodiment.
- the logic instructions in the memory 83 described above may be implemented in the form of a software functional unit and sold or used as a stand-alone product, and may be stored in a computer readable storage medium.
- the technical solution of the present disclosure may be embodied in the form of a software product stored in a storage medium, including a plurality of instructions for causing a computer device (which may be a personal computer, a server, or a network) The device or the like) performs all or part of the steps of the method described in various embodiments of the present disclosure.
- the foregoing storage medium may be a non-transitory storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and the like.
- a medium that can store program code, or a transitory storage medium including: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and the like.
- the embodiment of the present disclosure implements that the second virtual machine corresponding to the destination physical address acquires the first to-be-shared data according to the storage address information in the first to-be-shared packet, and performs data between the virtual machine in the related art.
- the technical solution of the present disclosure does not need to copy and forward data through a virtual bridge or a virtual switch.
- data is stored in a memory sharing area, and the storage address information of the stored data is notified to other virtual machines for Other virtual machines acquire the data in shared memory, thereby improving the efficiency of data sharing between virtual machines.
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Multimedia (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
一种虚拟机的数据共享方法,第一虚拟机将第一待共享数据存储到共享内存中;所述第一虚拟机获取所述第一待共享数据的存储地址信息以及目的物理地址;所述第一虚拟机将所述存储地址信息以及所述目的物理地址封装为第一待共享报文;以及所述第一虚拟机将所述第一待共享报文存入所述第一虚拟机对应的发送列队中。
Description
本公开涉及通信技术领域,例如涉及一种虚拟机的数据共享方法及装置。
随着通信技术的快速发展,NFV(Network Function Virtualization,网络功能虚拟化)技术也日益成熟,网络功能虚拟化需要提供高速的数据转发功能,其中,包括服务器物理网卡间的数据转发以及虚拟机之间的虚拟网卡的数据转发。
在服务器中虚拟网络的转发功能一般通过虚拟网桥,或OVS(Open Virtual Switch,虚拟交换机)来实现,但是无论是虚拟网桥还是OVS,都是运行在内核态,而虚拟机都是位于用户态,而虚拟机之间进行通讯时,需要经由虚拟网桥或是OVS进行数据的转发,而数据在用户态和内核态之间转发时,需要经过多次的拷贝,因此,这种情况下将数据从一个虚拟机发送到另一个虚拟机,需要经过多次拷贝才能进行数据的转发,导致虚拟机之间的数据共享效率低。
发明内容
本公开提出了一种虚拟机的数据共享方法及装置,解决了虚拟机之间的数据共享效率低的技术问题。
本公开提供的一种虚拟机的数据共享方法,包括:
第一虚拟机将第一待共享数据存储到共享内存中;
所述第一虚拟机获取所述第一待共享数据的存储地址信息以及目的物理地址;
所述第一虚拟机将所述存储地址信息以及所述目的物理地址封装为第一待共享报文;以及
所述第一虚拟机将所述第一待共享报文存入所述第一虚拟机对应的发送列队中;
其中,所述第一虚拟机将所述第一待共享报文存储至所述目的物理地址对应的接收列队中,以供所述目的物理地址对应的第二虚拟机根据所述第一待共享报文中的存储地址信息获取所述第一待共享数据,所述第一虚拟机以及所述第二虚拟机运行于同一个宿主机内。
本公开还提出一种虚拟机的数据共享装置,包括:
存储模块,设置为将第一待共享数据存储到共享内存中;
第一获取模块,设置为获取所述第一待共享数据的存储地址信息以及目的物理地址;
第一封装模块,设置为将所述存储地址信息以及所述目的物理地址封装为第一待共享报文;以及
存入模块,设置为将所述第一待共享报文存入所述第一虚拟机对应的发送列队中;
其中,所述存入模块将所述第一待共享报文存储至所述目的物理地址对应的接收列队中,以供所述目的物理地址对应的第二虚拟机根据所述第一待共享报文中的存储地址信息获取所述第一待共享数据,所述数据共享装置以及所述第二虚拟机运行于同一个宿主机内。
此外,本公开还提出了一种非暂态计算机可读存储介质,存储有计算机可执行指令,所述计算机可执行指令用于执行上述虚拟机的数据共享方法。
本公开提出的虚拟机的数据共享方法及装置,第一虚拟机先将第一待共享数据存储到共享内存中,然后获取所述第一待共享数据的存储地址信息以及目的物理地址,并将所述存储地址信息以及所述目的物理地址封装为第一待共享报文,最后将所述第一待共享报文存入所述第一虚拟机对应的发送列队中,并将所述第一待共享报文存储至所述目的物理地址对应的接收列队中,实现了所述目的物理地址对应的第二虚拟机根据所述第一待共享报文中的存储地址信息获取所述第一待共享数据,与相关技术中虚拟机之间进行数据共享时,本公开的技术方案不需要经由虚拟网桥或是虚拟交换机进行数据的拷贝转发,本方案中在内存共享区存储数据,并将存储数据的存储地址信息通知给其它虚拟机,以供其它虚拟机在共享内存中获取所述数据,从而提高了虚拟机之间的数据共享的效率。
附图概述
图1为本公开虚拟机的数据共享方法第一实施例的流程示意图;
图2为本公开虚拟机之间共享内存的示意图;
图3为本公开报文格式示意图;
图4为本公开虚拟机的数据共享方法第二实施例的流程示意图;
图5为本公开虚拟机的数据共享方法第三实施例的流程示意图;
图6为本公开虚拟机的数据共享装置第一实施例的功能模块示意图;
图7为本公开虚拟机的数据共享装置第二实施例的功能模块示意图;
图8为本公开虚拟机的数据共享装置第三实施例的功能模块示意图;
图9为本公开实施例提供的电子设备的结构框图。
本公开目的实现、功能特点及优点将结合实施例,参照附图进行说明。
此处所描述的具体实施例仅仅用以解释本公开。在不冲突的情况下,以下实施例及实施例中的特征可以任意组合。
本公开提供了一种虚拟机的数据共享方法。
参照图1,图1为本公开虚拟机的数据共享方法第一实施例的流程示意图。
本实施例提出一种虚拟机的数据共享方法,所述虚拟机的数据共享方法包括:
步骤S10,第一虚拟机将第一待共享数据存储到共享内存中。
在本实施例中,应用场景是同一个宿主机中的多个虚拟机进行数据共享,而在同一个宿主机中的多个虚拟机进行数据共享时,为了减小数据拷贝转发等操作,可以通过内存共享技术实现虚拟机之间的控制状态传递和网络数据的交互,即在宿主机中设置共享内存,并将待共享数据存储到共享内存中,所述共享内存用于完成虚拟机之间的数据交互和控制状态信息交互。参照图2,所述共享内存包含一个公共的数据共享区,所述数据共享区用于存放虚拟机之间的收发报文,所述共享内存还包括一个控制状态共享存储区,控制状态共享存储区主要用于虚拟机与宿主机中的数据转发进程(即I/O(Input/Output,输入输出)进程)之间交换报文描述符及状态信息,而从虚拟机的角度来看,它细分为虚拟机到宿主机数据转发进程的发送控制状态描述区和从宿主机I/O到虚拟机的接收控制状态描述区。可选地,所述数据转发进程应该包括一个或多个I/O线程,一般情况下,由宿主机支持的网络平面数确定,一个平面对应一个I/O线程,例如:一个宿主机的所有虚拟机如果只有控制面和数据面两个网络平面,就可以只启动两个I/O线程,一个线程处理数据面的数据收发,一个线程用于处理控制
面的数据转发。
步骤S20,所述第一虚拟机获取所述第一待共享数据的存储地址信息以及目的物理地址。
步骤S30,所述第一虚拟机将所述存储地址信息以及所述目的物理地址封装为第一待共享报文。
在本实施例中,所述第一虚拟机在共享内存中存储所述第一待共享数据后,先获取所述第一待共享数据的存储地址信息以及目的物理地址,然后,将获取的所述存储地址信息以及所述目的物理地址封装为第一待共享报文,参照图3,待共享报文包括:CSD(Control-Status Descriptors,控制状态描述符),BD(buffer Descriptors,缓冲区描述符),CSD描述符包括size报文大小、header头指针,tail尾指针、Flags标志位,BD包括Flags BD标签,len长度,index索引、buf_ptr报文物理地址等等,其中,size定义了每个报文的BD数,header和tail用于收发数据时,标识有数据的BD的首尾索引号,而每个数据包会占用一个BD,而BD主要用于描述存放收发数据的缓冲区地址及收发数据的长度。Flags定义了CSD的一些功能标志,用于存放数据报文的一些特征标志,index用于标识BD的索引。BD中最重要的部分为buf_ptr和len,buf_ptr用于存放收发数据的缓冲区的物理地址,len为该缓冲区的数据净荷长度。
步骤S40,所述第一虚拟机将所述第一待共享报文存入所述第一虚拟机对应的发送列队中。
其中,所述第一虚拟机将所述第一待共享报文存储至所述目的物理地址对应的接收列队中,以供所述目的物理地址对应的第二虚拟机根据所述第一待共享报文中的存储地址信息获取所述第一待共享数据,所述第一虚拟机以及所述第二虚拟机运行于同一个宿主机内。
在本实施例中,所述第一虚拟机调用数据转发进程,以将所述待共享报文存入所述第一虚拟机对应的发送列队中,并且通过所述数据转发进程将所述第一待共享报文存储至所述目的物理地址对应的接收列队中。所述数据转发进程运行于用户态,即通过调用所述数据转发进程进行虚拟机之间的数据共享,可避免了数据的拷贝转发,从而提高了数据共享的效率,为更好的理解本实施例中,举例应用场景如下:
首先,在第一虚拟机中,第一虚拟机的虚拟网卡驱动首先从共享内存区申请控制状态描述符CSD,由CSD获得BD描述符,然后再申请pkt_buf地址,
并将pkt_buf地址转换成所述第一虚拟机的虚拟地址,并准备要发送的第一待共享数据,也就是将所述第一待共享数据拷贝到pkt_buf地址中,并填写len(该缓冲区的数据净荷长度),然后将pkt_buf的物理地址填写到BD描述符中的buf_ptr字段,封装为第一待共享报文,完成发送数据的准备工作。对于第二虚拟机,在所述接收列队中存有第一待共享报文时,通过调用所述数据转发进程,以调用网卡驱动程序处理接收队列的BD,首先获取BD的buf_ptr指向的主机物理地址,并转换成第二虚拟机的虚拟机地址,最后,获取所述第一待共享报文以完成虚拟机之间的通讯。
在宿主机的所述数据转发进程中,数据转发进程守护虚拟机的虚拟网卡的发送队列,即所述第一虚拟机对应的第一待共享报文,数据转发进程依次从多个队列的tail标签开始处理报文,获取BD中的buf_prt地址,并转换成数据转发进程的虚拟地址,并获取所述第一待共享报文的目的物理地址,此时,根据目的物理地址查找预存的转发表,从而提取第二虚拟机对应的接收队列,并将发送队列的buf_prt、flags、len等参数直接复制到接收队列的buf_ptr、flags、len字段,最后将添加了参数的所述接收队列发送给所述目的物理地址对应的第二虚拟机。即I/O进程从虚拟机的发送队列收到发送的第一待共享报文后,根据目的物理地址查找转发表,找到所述第一待共享报文的接收队列,并将所述第一待共享报文发送到所述接收队列,以供所述目的物理地址对应的第二虚拟机根据第一待共享报文中的存储地址信息获取第一待共享数据。
本实施例提出的虚拟机的数据共享方法,第一虚拟机先将第一待共享数据存储到共享内存中,然后获取所述第一待共享数据的存储地址信息以及目的物理地址,并将所述存储地址信息以及所述目的物理地址封装为第一待共享报文,最后将所述第一待共享报文存入所述第一虚拟机对应的发送列队中,并将所述第一待共享报文存储至所述目的物理地址对应的接收列队中,实现了所述目的物理地址对应的第二虚拟机根据所述第一待共享报文中的存储地址信息获取所述第一待共享数据,与相关技术中虚拟机之间进行数据共享不同,本公开不需要经由虚拟网桥或是虚拟交换机进行数据的拷贝转发,本方案中在内存共享区存储数据,并将存储数据的存储地址信息通知给其它虚拟机,以供其它虚拟机在共享内存中获取所述数据,从而提高了虚拟机之间的数据共享的效率。
可选地,为了提高虚拟机数据共享的灵活性,基于第一实施例提出本公开
虚拟机的数据共享方法的第二实施例,在本实施例中,参照图4,所述步骤S20之后,所述虚拟机的数据共享方法还可以是包括:
步骤S50,在所述目的物理地址获取失败时,所述第一虚拟机将所述存储地址信息封装为第一待共享报文。
步骤S40,所述第一虚拟机将所述第一待共享报文存入所述第一虚拟机对应的发送列队中。
在本实施例中,所述第一虚拟机有可能会发送广播第一待共享报文,或当在第一待共享报文在转发表中无法找到队列,此时,对所述第一待共享报文进行特殊处理,即此时需要向每个虚拟机发送一份第一待共享报文。那么,所述虚拟机需要为其它多个虚拟机的接收队列申请pkt_buf,并复制报文净荷到所有的pkt_buf,复制len及flags到BD描述符相应字段,同时将pkt_buf转换成主机物理地址并填写到buf_ptr字段。也就是说第一虚拟机在存储数据时,没有将待共享的虚拟机的目的物理地址添加到所述第一待共享报文中,此时,直接将所述存储地址信息封装为第一待共享报文,并调用数据转发进程,以将所述第一待共享报文存入所述第一虚拟机对应的发送列队中,即所述数据转发进程无法从所述第一待共享报文中获取目的物理地址,那么所述数据转发进程将提取的所述存储地址信息发送给所述第一虚拟机对应的宿主机内的其它多个虚拟机,即所述数据转发进程将所述第一待共享报文存储至其它多个虚拟机对应的接收列队中,以供其它多个虚拟机根据所述第一待共享报文中的存储地址信息获取第一待共享数据,以完成第一虚拟机与其它多个虚拟机的数据共享。
可选地,为了提高虚拟机数据共享的灵活性,基于第一实施例提出本公开虚拟机的数据共享方法的第三实施例,在本实施例中,参照图5,所述步骤S40之后,所述虚拟机的数据共享方法还可以是包括:
步骤S60,所述第一虚拟机实时或定时获取所述第一虚拟机的接收队列中存储的第二待共享报文。
步骤S70,在获取到所述第二待共享报文时,所述第一虚拟机提取所述第二待共享报文中的存储地址信息。
步骤S80,所述第一虚拟机根据所述存储地址信息获取所述共享内存中所述存储地址信息对应的第二待共享数据。
在本实施例中,所述第一虚拟机获取所述第一虚拟机中的接收队列中存储
的第二待共享报文的方式包括:1)定时获取,即用户事先在所述宿主机中预设所述第一虚拟机获取待共享报文的方式为定时获取,如每隔10分钟检测接收列队中是否有待共享报文。2)实时获取,即用户事先在所述宿主机中预设所述第一虚拟机获取待共享报文的方式为实时获取,那么,只要在所述第一虚拟机的接收列队中存有待共享报文,所述第一虚拟机即可到检测并获取待共享报文。
在获取到所述第二待共享报文时,所述第一虚拟机提取所述第二待共享报文中的存储地址信息,根据所述存储地址信息即可获取所述共享内存中所述存储地址信息对应的第二待共享数据,以完成虚拟机之间的数据共享。
可选地,为了提高虚拟机数据共享的效率,所述第一待共享数据和所述第二待共享数据中的至少一个由多个数据包生成,即第一虚拟机在内存共享中存储待共享数据时,存储的数据是由多个数据包生成的,那么,数据转发进程在检测到所述第一待共享报文和第二待共享报文中的至少一个时,即可获取多个数据包对应的多个参数,实现了一次性调用多个数据包,减小了每个数据包交互的平均时间,从而提高了虚拟机之间数据共享的效率。可选地,还可以采用较大的包长来收发数据,例如一般网络的MTU(Maximum Transmission Unit,最大传输单元)的长度是1500字节,在实际应用中可以采用更大的MTU来减少每包的成本,它要求待共享报文收发的两端必须采用相同的MTU,如果两端的MTU不匹配,可能导致报文碎片从而无法正常通讯。
可选地,根据报文的源物理地址生成或更新转发表,即生成或更新目的物理地址与虚拟机接收队列的对应关系表,也就是说I/O进程为每个I/O线程维护一个转发表,转发表维护目的物理地址与虚拟机接收队列的对应关系。
本公开可选提供一种虚拟机的数据共享装置。
参照图6,图6为本公开虚拟机的通讯装置第一实施例的功能模块示意图。
对本领域的技术人员来说,图6所示功能模块图仅仅是一个可选实施例的示例图,本领域的技术人员围绕图6所示的虚拟机的数据共享装置的功能模块,可轻易进行新的功能模块的补充;每个功能模块的名称是自定义名称,仅用于辅助理解该虚拟机的数据共享装置的多个程序功能块,
本实施例提出一种虚拟机的数据共享装置,所述虚拟机的数据共享装置包括:
存储模块10,设置为将第一待共享数据存储到共享内存中。
在本实施例中,应用场景是同一个宿主机中的多个虚拟机进行数据共享,而在同一个宿主机中的多个虚拟机进行数据共享时,为了减小数据拷贝转发等操作,可以是通过内存共享技术实现虚拟机之间的控制状态传递和网络数据的交互,即在宿主机中设置共享内存,所述存储模块10将待共享数据存储到共享内存中,所述共享内存用于完成虚拟机之间的数据交互和控制状态信息交互,参照图2,所述共享内存包含一个公共的数据共享区,所述数据共享区用于存放虚拟机间之间的收发报文,所述共享内存还包括一个控制状态共享存储区,控制状态共享存储区主要用于虚拟机与宿主机中的数据转发进程(即I/O进程)之间交换报文描述符及状态信息,而从虚拟机的角度来看,它细分为虚拟机到宿主机数据转发进程的发送控制状态描述区和从宿主机I/O到虚拟机的接收控制状态描述区。可选地,所述数据转发进程应该包括一个或多个数据I/O线程,一般情况下,由宿主机支持的网络平面数确定,一个平面对应一个I/O线程,例如:一个宿主机的所有虚拟机如果只有控制面和数据面两个网络平面,就可以只启动两个I/O线程,一个线程处理数据面的数据收发,一个线程用于处理控制面的数据转发。
第一获取模块20,设置为获取所述第一待共享数据的存储地址信息以及目的物理地址。
第一封装模块30,设置为将所述存储地址信息以及所述目的物理地址封装为第一待共享报文。
在本实施例中,所述存储模块10在共享内存中存储所述第一待共享数据后,所述第一获取模块20先获取所述第一待共享数据的存储地址信息以及目的物理地址,然后,所述第一封装模块30将获取的所述存储地址信息以及所述目的物理地址封装为第一待共享报文,参照图3,待共享报文包括:CSD(Control-Status Descriptors,控制状态描述符),BD(buffer Descriptors,缓冲区描述符),CSD描述符包括size报文大小、header头指针,tail尾指针、Flags标志位,BD包括Flags BD标签,len长度,index索引、buf_ptr报文物理地址等等,其中,size定义了每个报文的BD数,header,tail设置为收发数据时,标识有数据的BD的首尾索引号,而每个数据包会占用一个BD,而BD主要用于描述存放收发数据的缓冲区地址及收发数据的长度。Flags定义了CSD的一些功能标志,用于存放数据报文的一些特征标志,index用于标识BD的索引。BD中最重要的部分为buf_ptr和len,buf_ptr用于存放收发数据的缓冲区的物理地址,len为该缓冲
区的数据净荷长度。
存入模块40,设置为将所述第一待共享报文存入所述第一虚拟机对应的发送列队中。
其中,所述存入模块将所述第一待共享报文存储至所述目的物理地址对应的接收列队中,以供所述目的物理地址对应的第二虚拟机根据所述第一待共享报文中的存储地址信息获取所述第一待共享数据,所述数据共享装置以及所述第二虚拟机运行于同一个宿主机内。
在本实施例中,所述第一虚拟机调用数据转发进程,以将所述待共享报文存入所述第一虚拟机对应的发送列队中,并且通过所述数据转发进程将所述第一待共享报文存储至所述目的物理地址对应的接收列队中,所述数据转发进程运行于用户态,即通过调用所述数据转发进程进行虚拟机之间的数据共享,可避免了数据的拷贝转发,从而提高了数据共享的效率,为更好的理解本实施例中,举例应用场景如下:
首先,所述第一获取模块20从共享内存区申请控制状态描述符CSD,由CSD获得BD描述符,然后再申请pkt_buf地址,并将pkt_buf地址转换成所述第一虚拟机的虚拟地址,并准备要发送的第一待共享数据,也就是将所述第一待共享数据拷贝到pkt_buf地址中,并填写len(该缓冲区的数据净荷长度),然后将pkt_buf的物理地址填写到BD描述符中的buf_ptr字段,第一封装模块30封装为第一待共享报文,完成发送数据的准备工作。对于第二虚拟机,在所述接收列队中存有第一待共享报文时,通过调用所述数据转发进程,以调用网卡驱动程序处理接收队列的BD,首先获取BD的buf_ptr指向的主机物理地址,并转换成第二虚拟机的虚拟机地址,最后,获取所述第一待共享报文以完成了虚拟机之间的通讯。
在宿主机的所述数据转发进程中,数据转发进程守护虚拟机的虚拟网卡的发送队列,即所述第一虚拟机对应的第一待共享报文,数据转发进程依次从多个队列的tail标签开始处理报文,获取BD中的buf_prt地址,并转换成数据转发进程的虚拟地址,并获取所述第一待共享报文的目的物理地址,此时,根据目的物理地址查找预存的转发表,从而提取第二虚拟机对应的的接收队列,并将发送队列的buf_prt、flags、len等参数直接复制到接收队列的buf_ptr、flags、len字段,最后将添加了参数的所述接收队列发送给所述目的物理地址对应的第二虚拟机。即I/O进程从虚拟机的发送队列收到发送的第一待共享报文后,根据
目的物理地址查找转发表,找到所述第一待共享报文的接收队列,并将所述第一待共享报文发送到所述接收队列,以供所述目的物理地址对应的第二虚拟机根据第一待共享报文中的存储地址信息获取第一待共享数据。
本实施例提出的虚拟机的数据共享装置,第一虚拟机先将第一待共享数据存储到共享内存中,然后获取所述第一待共享数据的存储地址信息以及目的物理地址,并将所述存储地址信息以及所述目的物理地址封装为第一待共享报文,最后将所述第一待共享报文存入所述第一虚拟机对应的发送列队中,并将所述第一待共享报文存储至所述目的物理地址对应的接收列队中,实现了所述目的物理地址对应的第二虚拟机根据所述第一待共享报文中的存储地址信息获取所述第一待共享数据,与相关技术中虚拟机之间进行数据共享时,本公开的技术方案不需要经由虚拟网桥或是虚拟交换机进行数据的拷贝转发,本方案中在内存共享区存储数据,并将存储数据的存储地址信息通知给其它虚拟机,以供其它虚拟机在共享内存中获取所述数据,从而提高了虚拟机之间的数据共享的效率。
可选地,为了提高虚拟机数据共享的灵活性,基于第一实施例提出本公开虚拟机的数据共享装置的第二实施例,在本实施例中,参照图7,所述虚拟机的数据共享装置包括:
第二封装模块50,设置为在所述目的物理地址获取失败时,将所述存储地址信息封装为第一待共享报文。
所述存入模块40,是设置为将所述第一待共享报文存入所述第一虚拟机对应的发送列队中。
在本实施例中,所述数据共享装置有可能会发送广播第一待共享报文,或当在第一待共享报文在转发表中无法找到队列,此时,对所述第一待共享报文进行特殊处理,即此时需要向每个虚拟机发送一份第一待共享报文。那么,所述第二封装模块50需要为其它多个虚拟机的接收队列申请pkt_buf,并复制报文净荷到所有的pkt_buf,复制len及flags到BD描述符相应字段,同时将pkt_buf转换成主机物理地址并填写到buf_ptr字段。也就是说第一虚拟机在存储数据时,没有将待共享的虚拟机的目的物理地址添加到所述第一待共享报文中,此时,所述第二封装模块50直接将所述存储地址信息封装为第一待共享报文,所述存入模块40调用数据转发进程,以将所述第一待共享报文存入所述第一虚拟机对
应的发送列队中,即所述数据转发进程无法从所述第一待共享报文中获取目的物理地址,那么所述数据转发进程将提取的所述存储地址信息发送给所述第一虚拟机对应的宿主机内的其它多个虚拟机,即所述数据转发进程将所述第一待共享报文存储至其它多个虚拟机对应的接收列队中,以供其它多个虚拟机根据所述第一待共享报文中的存储地址信息获取第一待共享数据,以完成第一虚拟机与其它多个虚拟机的数据共享。
可选地,为了提高虚拟机数据共享的灵活性,基于第一实施例提出本公开虚拟机的数据共享装置的第三实施例,在本实施例中,参照图8,所述虚拟机的数据共享装置还包括:
第二获取模块60,设置为实时或定时获取接收队列中存储的第二待共享报文。
提取模块70,设置为在获取到所述第二待共享报文时,提取所述第二待共享报文中的存储地址信息。
所述第二获取模块60,是设置为根据所述存储地址信息获取所述共享内存中所述存储地址信息对应的第二待共享数据。
在本实施例中,所述第二获取模块60获取所述第一虚拟机中的接收队列中存储的待共享报文的方式包括:,1)定时获取,即用户事先在所述宿主机中预设所述第一虚拟机获取待共享报文的方式为定时获取,如每隔10分钟检测接收列队中是否有待共享报文。2)实时获取,即用户事先在所述宿主机中预设所述第一虚拟机获取待共享报文的方式为实时获取,那么,只要在所述第一虚拟机的接收列队中存有待共享报文,所述第一虚拟机即可到检测。
在所述第二获取模块60获取到所述第二待共享报文时,所述提取模块70提取所述第二待共享报文中的存储地址信息,所述第二获取模块60根据所述存储地址信息即可获取所述共享内存中所述存储地址信息对应的第二待共享数据,以完成虚拟机之间的数据共享。
可选地,为了提高虚拟机数据共享的效率,所述第一待共享数据和所述第二待共享数据中的至少一个由多个数据包生成,即第一虚拟机在内存共享中存储待共享数据时,存储的数据是由多个数据包生成的,那么,数据转发进程在检测到所述第一和第二待共享报文中的至少一个时,即可获取多个数据包对应的多个参数,实现了一次性调用多个数据包,减小了每个数据包交互的平均时
间,从而提高了虚拟机之间数据共享的效率。可选地,还可以采用较大的包长来收发数据,例如一般网络的MTU(Maximum Transmission Unit,最大传输单元)的长度是1500字节,在实际应用中可以采用更大的MTU来减少每包的成本,它要求待共享报文收发的两端必须采用相同的MTU,如果两端的MTU不匹配,可能导致报文碎片从而无法正常通讯。
可选地,根据报文的源物理地址生成或更新转发表,即生成或更新目的物理地址与虚拟机接收队列的对应关系表,也就是说I/O进程为每个I/O线程维护一个转发表,转发表维护目的物理地址与虚拟机接收队列的对应关系。
需要说明的是,在本文中,术语“包括”、“包含”或者任何其它变体意在涵盖非排他性的包含,从而使得包括一系列要素的过程、方法、物品或者装置不仅包括那些要素,而且还包括没有明确列出的其它要素,或者是还包括为这种过程、方法、物品或者装置所固有的要素。在没有更多限制的情况下,由语句“包括一个……”限定的要素,并不排除在包括该要素的过程、方法、物品或者装置中还存在另外的相同要素。
上述本公开实施例序号仅仅为了描述,不代表实施例的优劣。
通过以上的实施方式的描述,本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现,当然也可以通过硬件,但很多情况下前者是可选的实施方式。基于这样的理解,本公开的技术方案本质上或者说对相关技术做出贡献的部分可以以软件产品的形式体现出来,该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中,包括若干指令用以使得一台终端设备(可以是手机,计算机,服务器,空调器,或者网络设备等)执行本公开多个实施例所述的方法。
此外,本公开实施例还提出了一种非暂态计算机可读存储介质,存储有计算机可执行指令,所述计算机可执行指令用于执行上述虚拟机的数据共享方法。
图9为本公开实施例提供的电子设备的结构框图。本公开实施例提供的电子设备可以是包括:处理器(processor)81和存储器(memory)83,还可以包括通信接口(Communications Interface)82和总线84。其中,处理器81、通信接口82、存储器83可以通过总线84完成相互间的通信。通信接口82可以用于信息传输。
处理器81可以调用存储器83中的逻辑指令,以执行上述实施例的虚拟机的数据共享方法。
此外,上述的存储器83中的逻辑指令可以通过软件功能单元的形式实现并作为独立的产品销售或使用时,可以存储在一个计算机可读取存储介质中。基于这样的理解,本公开的技术方案可以以软件产品的形式体现出来,该计算机软件产品存储在一个存储介质中,包括若干指令用以使得一台计算机设备(可以是个人计算机,服务器,或者网络设备等)执行本公开多个实施例所述方法的全部或部分步骤。而前述的存储介质可以是非暂态存储介质,包括:U盘、移动硬盘、只读存储器(ROM,Read-Only Memory)、随机存取存储器(RAM,Random Access Memory)、磁碟或者光盘等多种可以存储程序代码的介质,也可以是暂态存储介质。
本公开实施例实现了所述目的物理地址对应的第二虚拟机根据所述第一待共享报文中的存储地址信息获取所述第一待共享数据,与相关技术中虚拟机之间进行数据共享时,本公开的技术方案不需要经由虚拟网桥或是虚拟交换机进行数据的拷贝转发,本方案中在内存共享区存储数据,并将存储数据的存储地址信息通知给其它虚拟机,以供其它虚拟机在共享内存中获取所述数据,从而提高了虚拟机之间的数据共享的效率。
Claims (9)
- 一种虚拟机的数据共享方法,包括:第一虚拟机将第一待共享数据存储到共享内存中;所述第一虚拟机获取所述第一待共享数据的存储地址信息以及目的物理地址;所述第一虚拟机将所述存储地址信息以及所述目的物理地址封装为第一待共享报文;以及所述第一虚拟机将所述第一待共享报文存入所述第一虚拟机对应的发送列队中;其中,所述第一虚拟机将所述第一待共享报文存储至所述目的物理地址对应的接收列队中,以供所述目的物理地址对应的第二虚拟机根据所述第一待共享报文中的存储地址信息获取所述第一待共享数据,所述第一虚拟机以及所述第二虚拟机运行于同一个宿主机内。
- 如权利要求1所述的虚拟机的数据共享方法,其中,所述第一虚拟机获取所述第一待共享数据的存储地址信息以及目的物理地址的步骤之后,所述虚拟机的数据共享方法还包括:在所述目的物理地址获取失败时,所述第一虚拟机将所述存储地址信息封装为第一待共享报文;所述第一虚拟机将所述第一待共享报文存入所述第一虚拟机对应的发送列队中。
- 如权利要求1所述的虚拟机的数据共享方法,其中,所述第一虚拟机将所述第一待共享报文存入所述第一虚拟机对应的发送列队中的步骤之后,执行以下步骤:所述第一虚拟机实时或定时获取所述第一虚拟机中的接收队列中存储的第二待共享报文;在获取到所述第二待共享报文时,所述第一虚拟机提取所述第二待共享报文中的存储地址信息;所述第一虚拟机根据所述存储地址信息获取所述共享内存中所述存储地址信息对应的第二待共享数据。
- 如权利要求1-3任一项所述的虚拟机的数据共享方法,其中,所述第一待共享数据和所述第二待共享数据中至少一个由多个数据包生成。
- 一种虚拟机的数据共享装置,包括:存储模块,设置为将第一待共享数据存储到共享内存中;第一获取模块,设置为获取所述第一待共享数据的存储地址信息以及目的物理地址;第一封装模块,设置为将所述存储地址信息以及所述目的物理地址封装为第一待共享报文;以及存入模块,设置为将所述第一待共享报文存入所述第一虚拟机对应的发送列队中;其中,所述存入模块将所述第一待共享报文存储至所述目的物理地址对应的接收列队中,以供所述目的物理地址对应的第二虚拟机根据所述第一待共享报文中的存储地址信息获取所述第一待共享数据,所述数据共享装置以及所述第二虚拟机运行于同一个宿主机内。
- 如权利要求5所述的虚拟机的数据共享装置,还包括:第二封装模块,设置为在所述目的物理地址获取失败时,将所述存储地址信息封装为第一待共享报文;以及所述存入模块,是设置为将所述第一待共享报文存入所述第一虚拟机对应的发送列队中。
- 如权利要求5所述的虚拟机的数据共享装置,还包括:第二获取模块,设置为实时或定时获取接收队列中存储的第二待共享报文;提取模块,设置为在获取到所述第二待共享报文时,提取所述第二待共享报文中的存储地址信息;所述第二获取模块,是设置为根据所述存储地址信息获取所述共享内存中所述存储地址信息对应的第二待共享数据。
- 如权利要求5-7任一项所述的虚拟机的数据共享装置,其中,所述第一待共享数据和所述第二待共享数据中的至少一个由多个数据包生成。
- 一种非暂态计算机可读存储介质,存储有计算机可执行指令,所述计算机可执行指令用于执行权利要求1-4任一项所述的方法。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201510695398.X | 2015-10-22 | ||
CN201510695398.XA CN106612306A (zh) | 2015-10-22 | 2015-10-22 | 虚拟机的数据共享方法及装置 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2017067391A1 true WO2017067391A1 (zh) | 2017-04-27 |
Family
ID=58556722
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2016/101631 WO2017067391A1 (zh) | 2015-10-22 | 2016-10-10 | 虚拟机的数据共享方法及装置 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN106612306A (zh) |
WO (1) | WO2017067391A1 (zh) |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110912733A (zh) * | 2019-10-31 | 2020-03-24 | 北京浪潮数据技术有限公司 | 一种裸机拓扑关系确定方法、***、设备及计算机介质 |
CN112905304A (zh) * | 2021-03-08 | 2021-06-04 | 深信服科技股份有限公司 | 一种虚拟机间通信方法、装置、物理主机及介质 |
CN114138371A (zh) * | 2021-12-02 | 2022-03-04 | 北京天融信网络安全技术有限公司 | 配置动态加载方法、装置及计算机设备、存储介质 |
CN114697215A (zh) * | 2022-03-31 | 2022-07-01 | 西安超越申泰信息科技有限公司 | 一种虚拟化网络性能提升的方法、***、设备及介质 |
CN115002028A (zh) * | 2022-04-30 | 2022-09-02 | 济南浪潮数据技术有限公司 | 一种报文处理方法、装置及介质 |
CN115334156A (zh) * | 2021-04-26 | 2022-11-11 | 深信服科技股份有限公司 | 报文的处理方法、装置、设备、存储介质 |
CN115580644A (zh) * | 2022-12-07 | 2023-01-06 | 北京小米移动软件有限公司 | 宿主机内客户***间的通讯方法、装置、设备及存储介质 |
Families Citing this family (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108573162A (zh) * | 2017-05-31 | 2018-09-25 | 北京金山云网络技术有限公司 | 数据拷贝***、方法及装置 |
CN109460373B (zh) * | 2017-09-06 | 2022-08-26 | 阿里巴巴集团控股有限公司 | 一种数据共享方法、终端设备和存储介质 |
WO2019056380A1 (zh) * | 2017-09-25 | 2019-03-28 | 华为技术有限公司 | 一种数据访问的方法和装置 |
CN107678835B (zh) * | 2017-09-30 | 2020-05-08 | 东软集团股份有限公司 | 一种数据传输方法及*** |
CN107995129B (zh) * | 2017-11-30 | 2021-12-17 | 锐捷网络股份有限公司 | 一种nfv报文转发方法和装置 |
CN110554977A (zh) * | 2018-05-30 | 2019-12-10 | 阿里巴巴集团控股有限公司 | 数据缓存方法、数据处理方法、计算机设备、存储介质 |
CN111107663B (zh) * | 2018-10-26 | 2022-09-27 | 中兴通讯股份有限公司 | 数据的传输方法及装置、存储介质、电子装置 |
CN109379269A (zh) * | 2018-11-29 | 2019-02-22 | 南京中孚信息技术有限公司 | 虚拟交换设备数据转发方法、装置和计算机设备 |
CN111796902B (zh) * | 2019-04-08 | 2024-03-19 | 维塔科技(北京)有限公司 | 切换共享内存区的方法和装置、存储介质和电子设备 |
CN110109873B (zh) * | 2019-05-08 | 2023-04-07 | 重庆大学 | 一种用于消息队列的文件管理方法 |
CN110489150A (zh) * | 2019-08-26 | 2019-11-22 | 深信服科技股份有限公司 | 一种虚拟机热升级方法、装置及其相关设备 |
CN111130863B (zh) * | 2019-12-12 | 2023-07-14 | 北京浪潮数据技术有限公司 | 一种虚拟网卡管理方法、***、设备及计算机存储介质 |
CN113206807B (zh) * | 2020-01-31 | 2022-12-09 | 伊姆西Ip控股有限责任公司 | 用于信息处理的方法、电子设备和计算机存储介质 |
CN111797497B (zh) * | 2020-05-21 | 2024-05-17 | 中国电力科学研究院有限公司 | 一种用于电磁暂态并行仿真的通讯方法及*** |
CN114095458B (zh) * | 2020-08-07 | 2023-03-24 | 华为技术有限公司 | 一种数据处理方法及装置 |
CN112241307A (zh) * | 2020-10-21 | 2021-01-19 | 海光信息技术股份有限公司 | 一种虚拟机创建方法、装置及相关设备 |
CN112769716B (zh) * | 2021-01-12 | 2022-03-18 | 烽火通信科技股份有限公司 | 基于混合型虚拟网桥的虚拟交换机的数据转发方法及装置 |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101819564A (zh) * | 2009-02-26 | 2010-09-01 | 国际商业机器公司 | 协助在虚拟机之间进行通信的方法和装置 |
CN101847105A (zh) * | 2009-03-26 | 2010-09-29 | 联想(北京)有限公司 | 一种计算机及多操作***共享内存的方法 |
CN103559075A (zh) * | 2013-10-30 | 2014-02-05 | 华为技术有限公司 | 一种数据传输方法、装置和***及内存装置 |
US20150026681A1 (en) * | 2013-06-28 | 2015-01-22 | Huawei Technologies Co., Ltd. | Virtual Switching Method, Related Apparatus, and Computer System |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101667144B (zh) * | 2009-09-29 | 2013-02-13 | 北京航空航天大学 | 一种基于共享内存的虚拟机通信方法 |
US8533711B2 (en) * | 2009-11-30 | 2013-09-10 | Red Hat Israel, Ltd. | Method and system for adjusting a selection algorithm for selecting a candidate host with a highest memory sharing history value with a target virtual machine from amongst a set of host machines that have a standard deviation of memory sharing history with the virtual machine below a threshold amount |
-
2015
- 2015-10-22 CN CN201510695398.XA patent/CN106612306A/zh active Pending
-
2016
- 2016-10-10 WO PCT/CN2016/101631 patent/WO2017067391A1/zh active Application Filing
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101819564A (zh) * | 2009-02-26 | 2010-09-01 | 国际商业机器公司 | 协助在虚拟机之间进行通信的方法和装置 |
CN101847105A (zh) * | 2009-03-26 | 2010-09-29 | 联想(北京)有限公司 | 一种计算机及多操作***共享内存的方法 |
US20150026681A1 (en) * | 2013-06-28 | 2015-01-22 | Huawei Technologies Co., Ltd. | Virtual Switching Method, Related Apparatus, and Computer System |
CN103559075A (zh) * | 2013-10-30 | 2014-02-05 | 华为技术有限公司 | 一种数据传输方法、装置和***及内存装置 |
Cited By (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110912733A (zh) * | 2019-10-31 | 2020-03-24 | 北京浪潮数据技术有限公司 | 一种裸机拓扑关系确定方法、***、设备及计算机介质 |
CN110912733B (zh) * | 2019-10-31 | 2022-11-04 | 北京浪潮数据技术有限公司 | 一种裸机拓扑关系确定方法、***、设备及计算机介质 |
CN112905304A (zh) * | 2021-03-08 | 2021-06-04 | 深信服科技股份有限公司 | 一种虚拟机间通信方法、装置、物理主机及介质 |
CN115334156A (zh) * | 2021-04-26 | 2022-11-11 | 深信服科技股份有限公司 | 报文的处理方法、装置、设备、存储介质 |
CN114138371A (zh) * | 2021-12-02 | 2022-03-04 | 北京天融信网络安全技术有限公司 | 配置动态加载方法、装置及计算机设备、存储介质 |
CN114138371B (zh) * | 2021-12-02 | 2024-02-06 | 北京天融信网络安全技术有限公司 | 配置动态加载方法、装置及计算机设备、存储介质 |
CN114697215A (zh) * | 2022-03-31 | 2022-07-01 | 西安超越申泰信息科技有限公司 | 一种虚拟化网络性能提升的方法、***、设备及介质 |
CN115002028A (zh) * | 2022-04-30 | 2022-09-02 | 济南浪潮数据技术有限公司 | 一种报文处理方法、装置及介质 |
CN115002028B (zh) * | 2022-04-30 | 2024-02-13 | 济南浪潮数据技术有限公司 | 一种报文处理方法、装置及介质 |
CN115580644A (zh) * | 2022-12-07 | 2023-01-06 | 北京小米移动软件有限公司 | 宿主机内客户***间的通讯方法、装置、设备及存储介质 |
Also Published As
Publication number | Publication date |
---|---|
CN106612306A (zh) | 2017-05-03 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2017067391A1 (zh) | 虚拟机的数据共享方法及装置 | |
CN107515775B (zh) | 一种数据传输方法及装置 | |
CN113326228B (zh) | 基于远程直接数据存储的报文转发方法、装置及设备 | |
CN106034084B (zh) | 一种数据传输方法及装置 | |
WO2018028590A1 (zh) | 报文传输 | |
EP3211852A1 (en) | Ssh protocol-based session parsing method and system | |
WO2016179968A1 (zh) | 一种队列管理方法、装置及存储介质 | |
WO2017000593A1 (zh) | 报文处理方法及装置 | |
US20190044879A1 (en) | Technologies for reordering network packets on egress | |
CN107360268B (zh) | 一种数据包处理方法、装置及设备 | |
CN113891396B (zh) | 数据包的处理方法、装置、计算机设备和存储介质 | |
WO2015027806A1 (zh) | 一种内存数据的读写处理方法和装置 | |
US10693785B2 (en) | Method and system for forwarding data, virtual load balancer, and readable storage medium | |
CN109547162B (zh) | 基于两套单向边界的数据通信方法 | |
US20210359952A1 (en) | Technologies for protocol-agnostic network packet segmentation | |
CN110167197B (zh) | Gtp下行数据传输优化方法及装置 | |
CN105610730B (zh) | Cpu与网络设备之间的消息交互方法及*** | |
WO2017114378A1 (zh) | Ip数据包的转发 | |
US8170042B2 (en) | Transmit-side scaler and method for processing outgoing information packets using thread-based queues | |
US10230647B2 (en) | Data packet processing method and device | |
CN109417507A (zh) | 部分延迟的报文访问 | |
CN105939293A (zh) | 一种skb回收方法及装置 | |
WO2018068655A1 (zh) | 一种许可管理方法和*** | |
CN100579075C (zh) | 一种快速响应icmp回送请求报文的方法 | |
WO2015154393A1 (zh) | 业务节点能力处理方法、装置、业务分类器及业务控制器 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 16856824 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 16856824 Country of ref document: EP Kind code of ref document: A1 |