CN106354543A - NUMA memory page migration method based on virtual machine and host machine memory address conversion - Google Patents
NUMA memory page migration method based on virtual machine and host machine memory address conversion Download PDFInfo
- Publication number
- CN106354543A CN106354543A CN201610657507.3A CN201610657507A CN106354543A CN 106354543 A CN106354543 A CN 106354543A CN 201610657507 A CN201610657507 A CN 201610657507A CN 106354543 A CN106354543 A CN 106354543A
- Authority
- CN
- China
- Prior art keywords
- application program
- numa
- virtual machine
- described application
- address
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/10—Address translation
- G06F12/1009—Address translation using page tables, e.g. page table structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Memory System Of A Hierarchy Structure (AREA)
Abstract
The invention discloses an NUMA memory page migration method based on virtual machine and host machine memory address conversion. The method includes the steps that an application program in a virtual machine is monitored in real time through a VMM, and a set (HPA) of addresses of hot pages of the application program and information of NUMA affinity of the application program are obtained; when it is found that the NUMA affinity of the application program is reduced to be lower than a threshold, the application program is marked as a migration-needing state; an NUMA node where a VCPU operated by the application program is located is obtained through the VMM to serve as a target node, GPA->HVA address conversion is conducted on the hot pages of the application program to obtain a virtual address of a host machine memory page needing migration, then the virtual address, obtained after conversion, of the host machine memory page and the target node serve as parameter, a calling system calls move_pages to migrate the memory page to the target node. The method solves the problem that in the virtual machine environment, memory page migration cannot be conducted on programs running in the virtual machine.
Description
Technical field
The present invention relates to computer system virtualization field, more particularly, to one kind are based on virtual machine and host memory address
The numa internal memory of conversion moves a page method.
Background technology
Intel Virtualization Technology can turn to one or more virtual computer system by virtual for physical computer system.Often
Individual virtual computer system (referred to as client computer or virtual machine) is owned by the virtual hardware of oneself (as cpu, internal memory and equipment
Deng), an independent virtual machine performing environment to be provided.And the real physical computer system running virtual machine then becomes host
Machine.Fault-tolerance and high resource utilization that Intel Virtualization Technology is had due to it and be widely used in cloud computing and high-performance calculation etc.
Field.
In virtualized environment, virtual machine monitor (virtual machine management, vmm) is to be present in firmly
Software management layers between part and operating system, it is mainly responsible for real physical resource, such as cpu, internal memory and i/o
Equipment etc., and use abstract for the hardware resource of bottom for virtual machine for corresponding virtual device interface.
Internal memory virtualization is one of basis of Intel Virtualization Technology.The purpose of internal memory virtualization be to virtual machine provide one from
Zero beginning and continuous memory headroom is and it is necessary to will be effectively isolated, dispatch and shared drive between each virtual machine
Resource.In order to reach this purpose, vmm introduces one layer of new address space, i.e. guest physical address space (guest
Physical address, gpa), this address space is not real physical address space, it be host virtually
Location space is in a mapping in client address space.As shown in figure 1, for client computer, guest physical address space is all
It is zero-based continuation address space, but for host, the physical address space of client computer is not necessarily continuous
, guest physical address space is possible to be mapped in several discontinuous host address sections.Introduce client computer thing
Reason address space after, be the formation of from application program place virtual machine virtual address (guest virtual address,
Gva) arrive virtual machine physical address (guest physical address, gpa), then from virtual machine physical address gpa to host
Machine virtual address (host virtual address, hva), then from hva to host physical address (host physical
Address, hpa) three layer address conversion.
In kvm, the mapping that one kvm_memory_slot data structure of kvm records each address section is closed
System, this data structure contains the initial client computer page frame number (guest frame number, gfn) of this mapping range corresponding,
The page number of mapping and initial host virtual address.Then kvm just can realize to guest physical address to place
Conversion between main frame virtual address, namely first corresponding mapping range is found according to guest physical address, then basis
This guest physical address can be obtained by its corresponding host virtual address in the side-play amount of this mapping range.And then lead to again
Cross the page table of host and also can achieve the conversion between host physical address for the guest physical address, namely gpa to hpa
Conversion, as shown in Figure 2.
The framework that existing main flow polycaryon processor adopts is nonuniformity internal storage access (non-uniform memory
Access, abbreviation numa) framework.The feature of numa framework be system associated hardware resource (as internal memory, cpu) be divided into many
Individual node.In numa framework, processor accesses the speed than the internal memory accessing remote node for the speed of the internal memory of local node
Degree is fast.As shown in Fig. 3 the schematic diagram of the numa IA frame serverPC with 8 numa nodes of physical platform.Due to not
The distance between processor (cpu) and each internal memory with position is different, and the distance between each processor is also different, therefore
The time that different processor accesses each internal memory is also different.
Existing mainly include process migration and internal memory migration with regard to numa framework performance tuning method.Wherein, internal memory moves
Move and refer to all move to the data of application program on the cpu node at program place, so can allow between cpu and internal memory
Affinity reaches optimum thus improving performance.However, existing research is concentrated mainly on non-virtualized environment, that is, carry out numa tune
Excellent to as if host application program.For the application program running in virtual machine, due to lacking host physics
The information of framework and corresponding numa tuning method, its performance potential must be affected.Further, since traditional internal memory migration side
All pages to the program of liking of method migration, this will certainly lead to very big expense.But, each application program is in fortune
The page (referred to as " hot page ") often accessing during row is often little, therefore the internal memory beyond the migration hot page of application program
Page does not have much effects to performance boost.
Therefore, this invention address that exploitation is under numa framework, moved based on the fine-grained internal memory of kvm memory address conversion
Page method, can carry out ground by the method after obtaining the gva of virutal machine memory page of the program operating in virtual machine
Location is converted into hva and the physical machine page of this program is migrated.
Content of the invention
In view of the drawbacks described above of prior art, the technical problem to be solved is to provide one kind in numa framework
Under based on virtual machine and host memory address conversion page moving method, make in the numa of the application program in virtual machine
Deposit tuning to be possibly realized, and opening of internal memory tuning is substantially reduced by the hot page that address resolution only migrates virtual machine application
Pin.
In order to realize foregoing invention purpose, changed the invention provides a kind of based on virtual machine and host memory address
Numa internal memory moves a page method, comprises the following steps:
Step 1, the application program passing through in vmm monitor in real time virtual machine, obtain the hot page address of described application program
Set and the numa affinity information of described application program；
Step 2, when find described application program numa affinity drop below threshold value when, then by described application program
It is masked as needing the state of migration；
Step 3, to being masked as needing the described application program of transition state, described application program is obtained by vmm and runs
The numa node that is located of vcpu as destination node, and virtual machine physical address is carried out to place to the hot page of described application program
The conversion of main frame virtual address, by the hot page migration of described application program to destination node；Then branch to step 1 to continue to void
Plan machine application program is monitored.
Further, the numa affinity information of described application program is configured to calculate described application program operation
The distance of numa node that is located with described application data of the numa node that is located of vcpu obtaining.
Further, the hot page migration of described application program is configured to the place that will obtain after conversion to destination node
The virtual address of host memory page and destination node are realized as parameter call systemic-function move_pages.
This method compared with prior art, at least has a following beneficial technique effect:
(1) expense of numa internal memory tuning is greatly reduced by only migrating the hot page of application program, lifts numa internal memory
The performance benefits that tuning is brought.
(2) by the page of virtual machine application being carried out with the address conversion of gpa to host hva by virtual machine
In the page of application program correspond on the corresponding page of host, make to carry out numa to the application program in virtual machine
Internal memory migration is possibly realized.
Technique effect below with reference to design, concrete structure and generation to the present invention for the accompanying drawing is described further, with
It is fully understood from the purpose of the present invention, feature and effect.
Brief description
Fig. 1 is that the new virtual machine physical address space of the internal memory virtualization introducing of a preferred embodiment of the present invention shows
It is intended to；
Fig. 2 be the present invention the kvm of a preferred embodiment in from virtual address gva to host physical address hpa
Address conversion schematic diagram；
Fig. 3 is that the Non Uniform Memory Access of the physical platform of a preferred embodiment of the present invention accesses showing of (numa) framework
It is intended to；
Fig. 4 is that the numa internal memory of a preferred embodiment of the present invention moves the schematic flow sheet of page.
Specific embodiment
Below in conjunction with the accompanying drawings embodiments of the invention are elaborated, the present embodiment is being front with technical solution of the present invention
Put and implemented, following present detailed embodiment and specific operating process, but protection scope of the present invention does not limit
In following embodiments.
As shown in figure 4, of the present invention move page side based on the numa internal memory of virtual machine and the conversion of host memory address
Method comprises the following steps:
Step 1, monitor the operation information of application program in virtual machine by vmm.For current loads in virtual machine relatively
High application, by monitoring apply internal storage access obtain the hot page of application.And by calculating what application program ran
The distance of the numa node that the numa node that vcpu is located is located with application data carrys out the numa affinity of calculation procedure.
Step 2, when monitoring application program numa affinity be less than certain threshold value when (this threshold value is related to platform), mark
This application program of will is to need to carry out internal memory migration state.
Step 3, to being masked as the program that needs to carry out transition state, obtaining its vcpu place numa node by vmm is
Destination node t.Then, for the hot page of this application program, because the address of page hot in virtual machine is virtual machine physical address
Gpa, so need to be converted into virtual address hva of corresponding host page.The main generation of corresponding address conversion function
Code is as follows:
After the address conversion of the hot page of the application program in virtual machine is hva, by destination node t and hot accordingly
Page moves a page system as parameter call internal memory and calls move_pages, by the hot page migration of application program to corresponding destination node
On.To the program of migration in need carry out after internal memory moves page, returning to the application program that step 1 continues monitoring virtual machine.
In sum, the technical program at least have the advantages that by by the object of numa internal memory migration from journey
All internal memories of sequence are reduced to the hot page of internal memory that program often accesses, and significantly reduce the expense of numa internal memory tuning, lifting
The performance benefits that numa internal memory tuning is brought.Further, by introducing the address conversion letter of virtual machine gpa to host hva
Number, makes the numa internal memory tuning of application program under virtual environment be possibly realized.
The preferred embodiment of the present invention described in detail above.It should be appreciated that the ordinary skill of this area need not be created
The property made work just can make many modifications and variations according to the design of the present invention.Therefore, all technical staff in the art
Pass through the available technology of logical analysis, reasoning, or a limited experiment under this invention's idea on the basis of existing technology
Scheme, all should be in the protection domain being defined in the patent claims.
Claims (3)
1. a kind of based on virtual machine and host memory address conversion numa internal memory move page method it is characterised in that include with
Lower step:
Step 1, the application program passing through in vmm monitor in real time virtual machine, obtain the set of the hot page address of described application program
Numa affinity information with described application program；
Step 2, when find described application program numa affinity drop below threshold value when, then by described application program mark
For needing the state of migration；
Step 3, to being masked as needing the described application program of transition state, obtain what described application program ran by vmm
The numa node that vcpu is located is as destination node, and carries out virtual machine physical address to host to the hot page of described application program
The conversion of machine virtual address, by the hot page migration of described application program to destination node；Then branch to step 1 to continue to virtual
Machine application program is monitored.
2. as claimed in claim 1 a page method is moved based on the numa internal memory of virtual machine and the conversion of host memory address, it is special
Levy and be, the numa affinity information of described application program is configured to calculate the vcpu place that described application program runs
The distance of numa node that is located with described application data of numa node obtaining.
3. as claimed in claim 1 a page method is moved based on the numa internal memory of virtual machine and the conversion of host memory address, it is special
Levy and be, the hot page migration of described application program is configured to the host page that will obtain after conversion to destination node
Virtual address and destination node as parameter call systemic-function move_pages realize.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610657507.3A CN106354543B (en) | 2016-08-11 | 2016-08-11 | A kind of NUMA memory converted based on virtual machine and host memory address moves a page method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610657507.3A CN106354543B (en) | 2016-08-11 | 2016-08-11 | A kind of NUMA memory converted based on virtual machine and host memory address moves a page method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106354543A true CN106354543A (en) | 2017-01-25 |
| CN106354543B CN106354543B (en) | 2019-11-26 |
Family
ID=57843792
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610657507.3A Active CN106354543B (en) | 2016-08-11 | 2016-08-11 | A kind of NUMA memory converted based on virtual machine and host memory address moves a page method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106354543B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107038061A (en) * | 2017-04-14 | 2017-08-11 | 上海交通大学 | A kind of high-efficiency network I/O processing method based on NUMA and hardware ancillary technique |
| CN107168771A (en) * | 2017-04-24 | 2017-09-15 | 上海交通大学 | A kind of scheduling virtual machine device and method under Non Uniform Memory Access access architectures |
| WO2018214482A1 (en) * | 2017-05-24 | 2018-11-29 | 华为技术有限公司 | Method and apparatus for monitoring virtual machines |
| CN112099799A (en) * | 2020-09-21 | 2020-12-18 | 天津飞腾信息技术有限公司 | NUMA-aware multi-copy optimization method and system for SMP system read-only code segments |
| CN114442928A (en) * | 2021-12-23 | 2022-05-06 | 苏州浪潮智能科技有限公司 | Method and device for realizing cold and hot data migration between DRAM and PMEM |
| CN115033339A (en) * | 2022-05-09 | 2022-09-09 | 阿里巴巴（中国）有限公司 | Address mapping method, device, equipment and storage medium |
| WO2023020141A1 (en) * | 2021-08-19 | 2023-02-23 | 北京字节跳动网络技术有限公司 | Cloud service control method and apparatus, and storage medium and electronic device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103942087A (en) * | 2014-03-31 | 2014-07-23 | 华为技术有限公司 | Virtual machine thermal migration method, related device and cluster computing system |
| CN104199718A (en) * | 2014-08-22 | 2014-12-10 | 上海交通大学 | Dispatching method of virtual processor based on NUMA high-performance network cache resource affinity |
| CN104615480A (en) * | 2015-02-04 | 2015-05-13 | 上海交通大学 | Virtual processor scheduling method based on NUMA high-performance network processor loads |
-
2016
- 2016-08-11 CN CN201610657507.3A patent/CN106354543B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103942087A (en) * | 2014-03-31 | 2014-07-23 | 华为技术有限公司 | Virtual machine thermal migration method, related device and cluster computing system |
| CN104199718A (en) * | 2014-08-22 | 2014-12-10 | 上海交通大学 | Dispatching method of virtual processor based on NUMA high-performance network cache resource affinity |
| CN104615480A (en) * | 2015-02-04 | 2015-05-13 | 上海交通大学 | Virtual processor scheduling method based on NUMA high-performance network processor loads |
Non-Patent Citations (2)
| Title |
|---|
| 丁圣阁 等: "半虚拟化I_O模型的KVM虚拟机域间通信优化方法", 《计算机科学与探索》 * |
| 马汝辉: "基于多核的虚拟化技术研究", 《中国博士学位论文全文数据库 信息科技辑》 * |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107038061A (en) * | 2017-04-14 | 2017-08-11 | 上海交通大学 | A kind of high-efficiency network I/O processing method based on NUMA and hardware ancillary technique |
| CN107038061B (en) * | 2017-04-14 | 2019-07-05 | 上海交通大学 | A kind of high-efficiency network I/O processing method based on NUMA and hardware ancillary technique |
| CN107168771A (en) * | 2017-04-24 | 2017-09-15 | 上海交通大学 | A kind of scheduling virtual machine device and method under Non Uniform Memory Access access architectures |
| WO2018196296A1 (en) * | 2017-04-24 | 2018-11-01 | 上海交通大学 | Virtual machine scheduling device and method under non-uniform memory access architecture |
| US11204798B2 (en) | 2017-04-24 | 2021-12-21 | Shanghai Jiao Tong University | Apparatus and method for virtual machine scheduling in non-uniform memory access architecture |
| WO2018214482A1 (en) * | 2017-05-24 | 2018-11-29 | 华为技术有限公司 | Method and apparatus for monitoring virtual machines |
| CN108958879A (en) * | 2017-05-24 | 2018-12-07 | 华为技术有限公司 | A kind of monitoring method and device of virtual machine |
| CN112099799A (en) * | 2020-09-21 | 2020-12-18 | 天津飞腾信息技术有限公司 | NUMA-aware multi-copy optimization method and system for SMP system read-only code segments |
| WO2023020141A1 (en) * | 2021-08-19 | 2023-02-23 | 北京字节跳动网络技术有限公司 | Cloud service control method and apparatus, and storage medium and electronic device |
| CN114442928A (en) * | 2021-12-23 | 2022-05-06 | 苏州浪潮智能科技有限公司 | Method and device for realizing cold and hot data migration between DRAM and PMEM |
| CN114442928B (en) * | 2021-12-23 | 2023-08-08 | 苏州浪潮智能科技有限公司 | Method and device for realizing cold and hot data migration between DRAM and PMEM |
| CN115033339A (en) * | 2022-05-09 | 2022-09-09 | 阿里巴巴（中国）有限公司 | Address mapping method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106354543B (en) | 2019-11-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106354543A (en) | NUMA memory page migration method based on virtual machine and host machine memory address conversion | |
| CN103902885B (en) | Towards multi-security level(MSL) virtual desktop system secure virtual machine shielding system and method | |
| US20140223122A1 (en) | Managing virtual machine placement in a virtualized computing environment | |
| US9563458B2 (en) | Offloading and parallelizing translation table operations | |
| US11221868B2 (en) | Security enhanced hypervisor userspace notifications | |
| KR101441188B1 (en) | Techniques for memory de-duplication in a virtual system | |
| CN102567217B (en) | MIPS platform-oriented memory virtualization method | |
| CN104199718B (en) | A kind of dispatching method of the virtual processor based on NUMA high performance network cache resources affinity | |
| CN102339229A (en) | Virtualization method based on operating system layer | |
| CN101751284A (en) | I/O resource scheduling method for distributed virtual machine monitor | |
| US9201823B2 (en) | Pessimistic interrupt affinity for devices | |
| CN104615480A (en) | Virtual processor scheduling method based on NUMA high-performance network processor loads | |
| US20190266000A1 (en) | Guest intermediate address tagging for guest applications | |
| US10776020B2 (en) | Memory protection in virtualized computer systems using shadow page tables | |
| CN107168771A (en) | A kind of scheduling virtual machine device and method under Non Uniform Memory Access access architectures | |
| US20200341797A1 (en) | Virtual machine memory migration facilitated by persistent memory devices | |
| DE112018005404T5 (en) | SIMPLIFY ACCESSING A STORAGE'S LOCALITY DOMAIN INFORMATION | |
| CN106095576A (en) | Under virtualization multi-core environment, nonuniformity I/O accesses resources of virtual machine moving method | |
| CN101425046A (en) | Method for implementing distributed I/O resource virtualizing technique | |
| CN104503825A (en) | Mixed type equipment virtualization method based on KVM (Kernel-based Virtual Machine) | |
| Gouda et al. | Virtualization approaches in cloud computing | |
| US11074094B2 (en) | Securing hypercall support for user space processes in virtual machines | |
| CN104750623A (en) | Memory virtualization method and memory virtualization device | |
| US11586454B2 (en) | Selective memory deduplication for virtual machines | |
| US10789174B2 (en) | Guest intermediate address translation for virtual machines |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |