CN104111896A - Virtual memory management method and virtual memory management device for mass data processing - Google Patents
Virtual memory management method and virtual memory management device for mass data processing Download PDFInfo
- Publication number
- CN104111896A CN104111896A CN201410368134.9A CN201410368134A CN104111896A CN 104111896 A CN104111896 A CN 104111896A CN 201410368134 A CN201410368134 A CN 201410368134A CN 104111896 A CN104111896 A CN 104111896A
- Authority
- CN
- China
- Prior art keywords
- memory
- memory allocation
- allocation unit
- data
- virtual
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000015654 memory Effects 0.000 title claims abstract description 567
- 238000007726 management method Methods 0.000 title claims abstract description 71
- 238000012545 processing Methods 0.000 title claims abstract description 28
- 238000000034 method Methods 0.000 claims abstract description 62
- 230000008569 process Effects 0.000 claims description 37
- 238000013507 mapping Methods 0.000 claims description 27
- 238000000151 deposition Methods 0.000 claims description 25
- 238000003860 storage Methods 0.000 claims description 19
- 230000008878 coupling Effects 0.000 claims description 9
- 238000010168 coupling process Methods 0.000 claims description 9
- 238000005859 coupling reaction Methods 0.000 claims description 9
- 238000009826 distribution Methods 0.000 claims description 7
- 230000007306 turnover Effects 0.000 claims description 7
- 230000003139 buffering effect Effects 0.000 claims description 3
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 230000003750 conditioning effect Effects 0.000 claims description 3
- 238000006467 substitution reaction Methods 0.000 claims description 3
- 230000006870 function Effects 0.000 claims description 2
- 238000005457 optimization Methods 0.000 abstract description 2
- 239000002131 composite material Substances 0.000 abstract 1
- 230000003247 decreasing effect Effects 0.000 abstract 1
- 238000005516 engineering process Methods 0.000 description 4
- 230000011218 segmentation Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 239000011888 foil Substances 0.000 description 1
- 239000012634 fragment Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000007595 memory recall Effects 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Landscapes
- Memory System (AREA)
Abstract
The invention discloses a virtual memory management method and a virtual memory management device for mass data processing, belongs to the technical field of computer system optimization and is mainly applied to one-step real-time processing of computer mass data in quantity higher than that of available physical memory. When memory allocation reaches an assigned critical value, the most suitable physical memory is found to substitute for disc virtual memory according to memory block access correlation composite index and memory allocation application size. By a memory allocation method based on size matching, a front segment of an allocation unit is allocated to an object having high requirement for memory while a rear segment is allocated to an object having low requirement for memory; thus transform matching degree of the virtual memory is increased while data movement amount is reduced. By the virtual memory management method, during allocation of mass memories which are of capacity ranging from 100KB to 10MB and of the total quantity higher than the available physical memory, Thrashing in scheduling of the virtual memory is decreased, scheduling is accurate and predictable to some degree, and system performance is improved remarkably.
Description
Technical field
The invention belongs to computer system optimizing technical field, be specifically related to a kind of virtual memory management method and device thereof of large Data processing.
Background technology
Although computer hardware development rapidly, memory size increases very fast, and from KB level to MB level, then to GB level, a rank increases a thousand fold, and demand is also in continuous growth.Entered now the epoch of large data processing, more memory configurations can not meet all application demands.On the other hand, jumbo memory configurations also means that higher hardware drops into, and people always wish to configure general computing machine also can process a large amount of data.
Capacity and the cost of relative internal memory, disk has low, the capacious feature of cost, has just occurred virtual memory technology under such background.Virtual memory refers to and utilizes a part of hard-disc storage space to use as internal memory, to increase total used amount of ram.After physical memory is finished, operating system selects the partial content in internal memory to write on the swap file of virtual memory.
Because the access speed of calculator memory is far away higher than the read or write speed of hard disk, virtual memory technology has only solved the problem of physical memory deficiency.When using virtual memory, the performance of computing machine obviously declines.Therefore virtual memory technology always wishes to find best virtual memory dispatching algorithm, uses less virtual memory as far as possible, and the data replacement that as far as possible in advance next step may be used is called in physical memory.
The key issue that virtual memory management is main, the one, scheduling: determine which data should be transferred in physical memory; The 2nd, address mapping: reference address is for conversion into physical memory addresses or empty internal memory physical address; The 3rd, replace: determine which program and data should be transferred out main memory (physical memory); The 4th, upgrade: the consistance that guarantee main memory and auxiliary depositing (virtual memory) data.Wherein replacement problem and scheduling problem are unified in the lump processes as scheduling problem.
Virtual memory scheduling mode has 3 kinds of paging type, segmentation, segmentation paging system.Paging is logic and physical address space to be all divided into the page of fixed size.Physical memory is by page serial number, and each program space independently addressing has the page number order of oneself, by dispatching each page of virtual store Program, can discretely pack page locations different in physical memory into.Segmentation is that the logical organization of follow procedure is divided address space, and the length of section is random, and allows to extend.These two kinds of methods are combined and just form segmentation paging system scheduling.
In some cases, operating system is busy with from physical memory to virtual memory, calling in, recalling page, and cpu resource takies very large, and the internal memory operation stand-by period is very long, is commonly referred to (Thrashing) phenomenon of jolting.
Operating system can recall to a shared internal memory of current inactive process in virtual memory conventionally, and the active process operating is placed in physical memory as far as possible.When but the internal memory taking when a process has just surpassed physical memory capacity, operating system virtual memory adjustment method just has limitation.The speed ability that we often run into application software in this case significantly declines, and even occurs seemingly-dead machine phenomenon.
Another problem is that operating system is considered general issue more, when physical memory exhausts substantially, just starts scheduling and uses virtual memory, if Memory Allocation demand is large subsequently, is accompanied by a large amount of virtual memory scheduling, and speed ability exponentially declines.
The large Data processing of one class needs a large amount of storage allocations, and the internal memory of every sub-distribution is larger, and such as 100KB is to 10MB, and a lot of memory size of distributing is identical or close, such as processing onesize mass data.Conventionally when distributing total amount to approach or surpassing in available computer physics internal memory, operating system is enabled virtual memory management.But operating system virtual memory management can not optimization be processed this situation, virtual memory dispatching efficiency is low, frequently call in, recall, and data mobile is too much etc., can cause the large data processing of this class in the situation that data volume is very large, speed is very slow, even cannot process.
Summary of the invention
The object of the invention is to for addressing the above problem, a kind of virtual memory management method and device thereof of large Data processing are provided, to optimize the management of large Data processing virtual memory, reduce (Thrashing) phenomenon of jolting in virtual memory scheduling, strengthen virtual memory scheduling accuracy, foresight, thus elevator system performance.
Virtual memory management method of the present invention is characterised in that virtual memory management and running wherein at least comprise:
1, Memory Allocation Single Component Management: the internal memory operation of base conditioning and management be take Memory Allocation unit as unit, Memory Allocation unit adopts size pairing distribution method; During large Memory Allocation, arrange allocation units, allocation units size mUnitSize=applies for big or small RequestSize * 125%; Leading portion BigMem distributes large data in district, and back segment LitMem district distributes small data; During little Memory Allocation, preferentially in the allocation units that arranged, look for LitMem free area, if find allocation units to align backward, if find, do not arrange to be assigned to little Memory Allocation district.
2, the associated aggregative index of Memory Allocation unit access: reflection Memory Allocation unit preferentially recalls to the index of virtual memory, n allocation units are accessed associated aggregative index: Xn=(current time-last access time) * Fa * Fm/Fn
2, wherein chronomere is millisecond, and Fa is the Memory Allocation cell-average access frequency, and Fm is the maximum access in the Memory Allocation unit frequency, and Fn is n the allocation units access frequency; When Xn>Xa, n Memory Allocation unit can recall to virtual memory, and wherein Xa is the associated aggregative index mean value of access.
3, virtual memory scheduling match substitution:
(1) memory block larger in Memory Allocation unit is only called in or is recalled in virtual memory scheduling, to reduce scheduled for executing number of times;
(2) when less memory block needs in Memory Allocation unit, in physical memory, move, with a small amount of internal storage data, move and increase virtual memory scheduling conversion matching rate;
(3) virtual memory scheduling replacement ratio compared with time, as long as source and target Memory Allocation cell size is the 75% close convertible object of coupling that just belongs to; In the situation that the large a lot of size of data of Data processing are close, virtual memory scheduling accuracy rate is high, efficiency is high like this;
(4) in physical memory, can recall by accessing associated aggregative index and decide;
(5) always keep necessary free physical memory space, be called minimum turnover spare area MinSpareMem, guarantee that virtual memory scheduling process does not affect system and normally moves.
The inventive method can realize according to the following steps:
1, initialization: to larger and continuous physical memory of operating system application, create or open virtual memory image file at disk, file size is 4G-32G, determines according to application configuration.Notice that this file is not the PageFile.sys in windows.
2, little Memory Allocation: carry out little Memory Allocation when application distributes the internal memory that is less than BitMemSize, it is empty in the Memory Allocation unit having arranged, searching L_MemArea and available size is greater than the unit of applying for size RequestSize, find record to return, do not find in Ze little Memory Allocation district and distribute.
3, create Memory Allocation unit: according to Memory Allocation application size RequestSize, physical memory in management is set up Memory Allocation unit the record that a size is RequestSize * 125%, when if free physical memory is less than MinSpareMem, setting up procedure 9.
4, large Memory Allocation: when distributing the internal memory that is greater than BitMemSize, application carries out large Memory Allocation, if there is the BigMem district of Memory Allocation unit to discharge, and size to fit, directly select, record and return; Otherwise, if when free physical memory is less than minimum turnover spare memory MinSpareMem, starts and recall task, perform step 3 simultaneously, then new unit is returned to applicant.
5, little core position is adjusted: when Memory Allocation units match is processed, because do not consider whether little memory field has data, if Memory Allocation unit is reused, and when in new use, BigMemSize can cover little memory field, little memory field need to move, moving method is execution step 2, and copies data, and the initialization of then little memory field is standby.
6, general memory distributes: according to application memory size, select respectively to perform step 2 and step 4.
7, internal storage data access: according to address mapping table, determine that institute's visit data is whether in physical memory, directly access in physical memory, otherwise select to carry out the task that recalls, then carry out and call in task visit data.
8, Memory Allocation units match is processed: source Memory Allocation cell size is that 1/2～1 times of target Memory Allocation unit is that the match is successful; No while then detecting the little Memory Allocation district of target Memory Allocation unit have data, when the large memory field data of Memory Allocation unit, source move on to target Memory Allocation unit, little internal storage data that whether can coverage goal Memory Allocation unit, if can cover, performs step 5.
9, Memory Allocation unit recalls: according to the associated aggregative index of Memory Allocation unit access and recall size requirement, select the most suitable Memory Allocation unit recalling, can select to recall several Memory Allocation unit of asking that associated aggregative index is equal; If there are data in the little Memory Allocation district of the Memory Allocation unit being transferred out, perform step 5.
10, call in Memory Allocation unit: first perform step 8, if do not find the Memory Allocation unit of coupling,, perform step 3, by object content in virtual memory copy to find or newly assigned Memory Allocation unit in, record scheduler mapping table.
Apparatus of the present invention comprise:
Access interface module: comprise physics memory management and virtual memory management initialization, Memory Allocation, internal memory release, read data, write data, be with buffering read data, be with buffer write data method, virtual memory scheduling is carried out automatically, be not reflected to interface module, this module is calculated the associated aggregative index of Memory Allocation unit access simultaneously, in Memory Allocation and access process, automatically calculate, during for virtual memory scheduling decision-making which preferentially recall.
Memory Allocation administration module: manage this and install operable physical memory space and the auxiliary virtual memory space of depositing, carry out Memory Allocation, release, management address mapping table, starts and recalls task management module if desired, or startup is called in executive routine, recalled executive routine; Memory Allocation is managed with Memory Allocation unit structural form of the present invention, and Memory Allocation comprises large memory block and distributes and little Memory Allocation two lines, with large internal memory, is assigned as preferentially, and in order to adapt to large Memory Allocation, little memory field LitMem can move.
Call in execution module: carry out and to call in allocation units to the process of physical memory from virtual memory, be wherein included in and in physical memory, search or distribute a Memory Allocation unit matching, scheduler mapping table, calls in data from auxiliary depositing.
Recall task management module: record and execution with the form of a separate threads recall task, reduce virtual memory scheduling process does not affect main thread operation as far as possible.
Recall execution module: carry out to recall a Memory Allocation unit to the process of virtual memory from physical memory, comprise that the data that detect and process LitMem district in moving target, Mobile data are to virtual memory, scheduler mapping table, the Memory Allocation unit step after initialization is shifted out.
Initialization module: start this device and enter the initial work that normal operating conditions is carried out, comprise physical memory, virtual memory, address mapping table that device is managed; Loading configuration and operating statistic information is configured device modules.
Beneficial effect of the present invention: large processing procedure related data amount of Data processing often surpasses free physical memory amount, and the vague generalization virtual memory management method that operating system provides has solved capacity problem, but efficiency is not high.Method and apparatus of the present invention is at large Data processing, effectively reduce (Thrashing) phenomenon of jolting in virtual memory scheduling, strengthen virtual memory scheduling accuracy, foresight, in speed ability and the stability of using the large Data processing system of virtual memory, have substantive raising.In actual software exploitation, when physical memory is not enough, apply for fast bulk memory, it is all to distribute unsuccessfully that operating system is returned, rather than can become the distribution of work after virtual memory scheduling, and application system also just can not continue Processing tasks like this.The present invention has solved this problem effectively, when physical memory is not enough, uses virtual memory, is only that speed is slightly slow.
Accompanying drawing explanation
Fig. 1 Memory Allocation process flow diagram that comprises virtual scheduling management of the present invention;
Fig. 2 Memory Allocation of the present invention unit and virtual memory debugging unit structural drawing;
Fig. 3 physical memory distribution state of the present invention schematic diagram;
Fig. 4 internal storage data browsing process figure that comprises virtual scheduling management of the present invention;
Fig. 5 virtual memory data of the present invention recall (depositing to auxiliary from physical memory) treatment scheme;
Fig. 6 virtual memory data of the present invention are called in (from the auxiliary physical memory of depositing) treatment scheme;
Fig. 7 apparatus of the present invention module forms structural representation.
Embodiment
Below in conjunction with accompanying drawing, the technical scheme of embodiments of the present invention is carried out to detailed, complete elaboration.
The invention provides a kind of virtual memory schedule management method, it at least comprises: Memory Allocation Single Component Management method, virtual memory scheduling match substitution method, the associated aggregative index of Memory Allocation unit access.Set forth groundwork flow process below.
As shown in Figure 1, in virtual memory management method of the present invention, the Memory Allocation flow process that comprises virtual scheduling management:
S100, when data handling utility application size is RequestSize internal memory, enter this flow process, comprising:
Whether S101 judgement application memory size, belong to large internal memory category, if the capable S102 of imperial palace counter foil, otherwise carry out S110.In the present invention, large internal memory is the larger internal memory of data handling utility application assignment sizes, for example big or small 100KB.Otherwise little internal memory refers to the internal memory of applying for that assignment sizes is less, for example, be less than 100KB.The separatrix size of large internal memory represents with BigMemSize.Emphasis of the present invention, for large Data processing, distributes the feature of large internal memory in a large number, and the emphasis of virtual memory scheduling is also for large internal memory.
S102 detects when whether time Memory Allocation starts virtual memory recalls program.When MaxAvailPhyMem – AllocatedMem<MinSpareMem, need to start and recall task S103, otherwise carry out S105.Wherein MaxAvailPhyMem is the physical memory size that apparatus of the present invention are managed, the physical memory size of AllocatedMem for having distributed, and MinSpareMem is minimum turnover spare physical memory size.
S103 carries out the Memory Allocation units match search program that meets the condition of recalling, if find the suitable Memory Allocation unit recalling, execution recalls program S104, otherwise carries out traditional virtual dispatching method S108, recalls the Memory Allocation unit that one section of maximum duration is not accessed.The condition that wherein recalls is decided by the associated aggregative index of Memory Allocation unit access, and this index illustrates below.
S105 carries out the Memory Allocation unit search program of large Memory Allocation district's free time, finds and carries out S106 allocation scheme, and no finding carried out S107.Wherein the large Memory Allocation district He little Memory Allocation district of Memory Allocation unit sees that Fig. 2 illustrates.
S107 arranges new Memory Allocation unit MAU, and MAU.mUnitSize=RequestSize * 125%, puts MAU.BigAreaSize=RequestSize, MAU.LitAreaSize=RequestSize/4.
Whether S110 searches medium and small Memory Allocation district, Memory Allocation unit the unit of suitable not use, if there is record distribute and return successfully; Do not find and carry out S111, from Fig. 2, the little Memory Allocation of S206 district arranges Memory Allocation.MAU.mUnitSize-MAU.BigAreaSize=RequestSize～RequestSize * 2 of Memory Allocation unit MAU, are coupling.
As Fig. 2, virtual memory management method of the present invention comprises Memory Allocation Single Component Management method:
The internal memory operation of base conditioning of the present invention and management be take Memory Allocation unit as unit, and Memory Allocation unit adopts size pairing distribution method, as shown in Figure 2.During large Memory Allocation, arrange allocation units, allocation units size mUnitSize=applies for big or small RequestSize * 125%; Leading portion BigMem distributes large data in district, and back segment LitMem district distributes small data; During little Memory Allocation, preferentially in the allocation units that arranged, look for LitMem free area, if find allocation units to align backward, if find, do not arrange to be assigned to little Memory Allocation district.
Wherein the minimum value of Memory Allocation unit is determined by configuration parameter, is designated as BigMemSize.The Memory Allocation that is greater than BigMemSize adopts large internal memory allocation flow, is less than BigMemSize and adopts little Memory Allocation flow process.
In addition, in Memory Allocation unit, between BigMem and LitMem district, conventionally leave a blank, by less distribution, waste, obtain good performance.In Memory Allocation units match, displacement, mainly, to consider BigMem, leave a blank and can expand, compress, if LitMem has data also can move to other position.Therefore the initial value of Memory Allocation unit MAU is MAU.BigAreaSize=MAU.mUnitSize * 75%, after processes such as dispatching, replace, redistribute, the range of control of MAU.BigAreaSize is MAU.mUnitSize * 50%～MAU.mUnitSize * 100%, and the range of control of MAU.LitAreaSize is 0～MAU.mUnitSize * 50%.The part scope of leaving a blank in same Memory Allocation unit is 0～MAU.mUnitSize * 50%.
As Fig. 4, internal storage data browsing process S400 in virtual memory management method of the present invention:
Whether S402, the data of accessing according to address mapping table judgement in auxiliary depositing (virtual memory), if carry out S404, otherwise carry out normal browsing process.Address mapping table only records the data such as Memory Allocation element address, Memory Allocation address, Memory Allocation unit access information, and this instructions is not to this in-depth explanation.
If S404 recalls task and does not carry out, first carry out and recall task S406, guarantee to have enough physical memories space can call in the data that need access.
S408 carries out and calls in treatment scheme S600, from auxiliary depositing, calls in the Memory Allocation unit that will access.
Alternatively, continuous several Memory Allocation unit that the associated aggregative index of Memory Allocation unit access equates can join the task of calling in, and in the process of system operation, the Memory Allocation unit may have access to is subsequently called on backstage.
As Fig. 5, in virtual memory management method of the present invention, Memory Allocation unit recalls flow process S500, from physical memory, a Memory Allocation unit is recalled to auxiliary depositing (virtual memory):
Whether S502, the detection medium and small memory field L_MemArea in Memory Allocation unit use, if used, carry out S504, and little internal memory is moved to other position.
S506 copies auxiliary depositing (virtual memory) to Memory Allocation unit, change address mapping table.Wherein the auxiliary virtual memory storage file of depositing creates or opens when initialization.In order to reduce management complexity, the space that this document predistribution is larger, 4G～32G for example, to improve access speed and reduce disk fragments.When every secondary program starts operation, pointer sets to 0, and deposits backward and recalls data subsequently in the mode of increment storage.
As Fig. 6, in virtual memory management method of the present invention, flow process S600 is called in out in Memory Allocation unit, from auxiliary depositing (virtual memory), calls in and needs the Memory Allocation unit of access or access subsequently probability greatly and the Memory Allocation unit of Data Position in auxiliary depositing.Calling in cell size is mUnitSizeA.
In the Memory Allocation unit MAU of S602 in physical memory, search matching unit, MAU.BigMem is not used, and mUnitSizeA=MAU.mUnitSize/2～MAU.mUnitSize.Find and carry out S606, do not find and carry out S604.
S604 arranges a new Memory Allocation unit MAU, MAU.mUnitSize=mUnitSizeA, and operation steps is shown in that Fig. 1 illustrates in detail.
S606 detects the Memory Allocation unit MAU finding, if MAU.LitMem is meaningful, moves on to other position, MAU initialization.
S608 copies required data of calling in from auxiliary depositing (virtual memory) and finds or newly assigned Memory Allocation unit to physical memory, scheduler mapping table.
In virtual memory management method of the present invention, the associated aggregative index of Memory Allocation unit access is to judge that Memory Allocation unit preferentially recalls the foundation of selection, is also the foundation that prediction virtual memory is called in advance and recalled, and comprising:
N allocation units are accessed associated aggregative index: Xn=(current time-last access time) * Fa * Fm/Fn
2.Wherein chronomere is millisecond, and Fa is the Memory Allocation cell-average access frequency, and Fm is the maximum access in the Memory Allocation unit frequency, and Fn is n the allocation units access frequency.When Xn>Xa, n Memory Allocation unit can recall to virtual memory, and wherein Xa is the associated aggregative index mean value of access.
Wherein last access time, the access times that millisecond is unit are take in internal storage access unit record.When carrying out internal storage data access program S400, upgrade above-mentioned each value.Initialization Xn=0 while creating Memory Allocation unit, Fn=1.
The invention provides a kind of virtual memory management device based on said method, comprising:
Access interface module: for system applies provides the function of Memory Allocation, data access.Comprise physics memory management and virtual memory management initialization, specify size and the virtual memory original document size of using physical memory; Memory Allocation, internal memory release, read data, write the methods such as data, band buffering read data, band buffer write data.Virtual memory scheduling is carried out automatically, is not reflected to interface module.This module is calculated the associated aggregative index of Memory Allocation unit access simultaneously, in Memory Allocation and access process, automatically calculate, during for virtual memory scheduling decision-making which preferentially recall.
Memory Allocation administration module: manage this and install operable physical memory space and the auxiliary virtual memory space of depositing, carry out Memory Allocation, release, management address mapping table, when physical memory is less, start and recall task management module, or startup is called in executive routine, recalled executive routine.Memory Allocation is managed with Memory Allocation unit structural form of the present invention, as Fig. 2.Memory Allocation comprises large memory block and distributes and little Memory Allocation two lines, with large internal memory, is assigned as preferentially, and in order to adapt to large Memory Allocation, little memory field LitMem can move.
Call in execution module: carry out and call in allocation units to the process of physical memory from virtual memory (auxiliary depositing).Wherein be included in and in physical memory, search or distribute a Memory Allocation unit matching, scheduler mapping table, calls in data etc. from auxiliary depositing.
Recall task management module: when device starts, create a separate threads and be used for specially carrying out the task that recalls, from physical memory, recall obsolete Memory Allocation of some long periods unit, this module mainly comprises and recalls task queue management and call the task that recalls of carrying out.Separate threads mode by running background is carried out the task of recalling, and reduce virtual memory scheduling process does not affect main thread operation as far as possible.
Recall execution module: carry out to recall a Memory Allocation unit to the process of virtual memory (auxiliary depositing) from physical memory, comprise the data that detect and process LitMem district in moving target, Mobile data to auxiliaryly depositing, scheduler mapping table, the steps such as Memory Allocation unit after initialization is shifted out.
Initialization module: start apparatus of the present invention and enter the necessary initial work that normal operating conditions is carried out, comprise physical memory, auxiliary depositing (virtual memory), address mapping table that device is managed, create and recall task management thread; Loading configuration and operating statistic information is configured device modules.
Virtual memory management device of the present invention is worked according to the following steps:
Step 1: initialization: to larger and continuous physical memory of operating system application, by this device management, at disk, create or open virtual memory image file, file size is about 4G-32G, according to application configuration and available secondary, deposit capacity and determine, this file is not the PageFile.sys in windows.The setting principle of virtual memory image file size have subsequently more speak more bright.As Fig. 2, initialization physical memory structure and address mapping table, used in order to Memory Allocation and access; Establishment recalls task management thread.
Step 2: little Memory Allocation: as Fig. 2, carry out little Memory Allocation when application distributes the internal memory that is less than BitMemSize.In the Memory Allocation unit having arranged, search L_MemArea for empty and available size are greater than the unit of applying for size RequestSize, find record to return, do not find in Ze little Memory Allocation district and distribute.
Step 3: create Memory Allocation unit: as Fig. 2, according to Memory Allocation application size RequestSize, set up at the physical memory of this device management Memory Allocation unit the record that a size is RequestSize * 125%.When if free physical memory is less than MinSpareMem, setting up procedure 9.
Step 4: large Memory Allocation: carry out large Memory Allocation when application distributes the internal memory that is greater than BitMemSize.If there is the BigMem district of Memory Allocation unit to discharge, and size to fit, directly select, record and return.Otherwise, if when free physical memory is less than minimum turnover spare memory MinSpareMem, start the task that recalls.Perform step 3 simultaneously, then new unit is returned to applicant.
Step 5: little core position is adjusted: when Memory Allocation units match is processed, because do not consider whether little memory field has data, if Memory Allocation unit is reused, and when in new use, BigMemSize can cover little memory field, little memory field need to move.Moving method is execution step 2, and copies data, and the initialization of then little memory field is standby.
Step 6: general memory distributes.As Fig. 1, according to application memory size, select respectively to perform step 2 and step 4.
Step 7: internal storage data access: as Fig. 4, according to address mapping table, determine that institute's visit data is whether in physical memory, directly access in physical memory, otherwise select to carry out the task that recalls, then carry out and call in task visit data.During internal storage data access, upgrade last access time and the access times of Memory Allocation unit, this is the significant data that calculates the associated aggregative index of access.
Step 8: Memory Allocation units match is processed: source Memory Allocation cell size is that 1/2～1 times of target Memory Allocation unit is that the match is successful.No while then detecting the little Memory Allocation district of target Memory Allocation unit have data, when the large memory field data of Memory Allocation unit, source move on to target Memory Allocation unit, little internal storage data that whether can coverage goal Memory Allocation unit, if can cover, performs step 5.
Step 9: Memory Allocation unit recalls: as Fig. 5, according to the associated aggregative index of Memory Allocation unit access and recall size requirement, select the most suitable Memory Allocation unit recalling, can select to recall several Memory Allocation unit of asking that associated aggregative index is equal.If there are data in the little Memory Allocation district of the Memory Allocation unit being transferred out, perform step 5.
Step 10: call in Memory Allocation unit: as Fig. 6, first perform step 8, if do not find the Memory Allocation unit of coupling,, perform step 3, by object content in auxiliary depositing (virtual memory) copy to find or newly assigned Memory Allocation unit in, record scheduler mapping table.
Step 11: calculate the associated aggregative index of Memory Allocation unit access: when recalling Memory Allocation unit and search coupling, calculate the associated aggregative index of current access, the associated aggregative index Xn=of access of each Memory Allocation unit (current time-last access time) * Fa * Fm/Fn2, sees above in detail interview and asks associated aggregative index computing method.
More than virtual memory schedule management method and the device thereof for large data processing that the present invention proposes, below situation that we are applied before and after the present invention describe:
Application testing computing machine of the present invention is mini-computer, memory configurations is 2G-32G, one time real-time processing data is about 2000-10000 frame image data, every frame data size is about 512KB～25MB, according to large data processing needs, the processing of each data need to be deposited raw data, processing middle data and display buffer data, is about 2.5 times of former size of data; Reality is once processed and is needed internal memory to be about 6GB-32GB in real time, BigMemSize=512KB.
The auxiliary virtual memory file size of depositing of the present invention configures by following principle:
(A), as far as possible with hyperdisk, for example solid state hard disc, SAS dish are for preferential;
(B) if biconditional operation system at a dish, except virtual memory file, also need to guarantee that residual capacity is not less than 1/10 of total disk size;
(C) meeting under above-mentioned condition, in 4GB～32GB scope, selecting larger value that virtual memory file size is set.
The virtual memory management or some the published relevant virtual memory management methods that adopt operating system to provide, when inferior real-time processing data amount is greater than available physical physical memory, lower than direct speed, system from disk read data, there is waiting as long for not response etc. in common Memory Allocation failure, data access speed, adopt after the inventive method and device pin, the situation that there will not be Memory Allocation failure, data access speed is not less than reads disk speed, and forward scheduling foresight increases simultaneously.
The foregoing is only one embodiment of the present of invention, can not limit the present invention with this, all within the scope of thought and method of the present invention, any modification, the improvement made, within all should being included in the scope of protection of the invention.
Claims (4)
1. a virtual memory management method for large Data processing, is characterized in that virtual memory management and running wherein at least comprise:
(1), Memory Allocation Single Component Management: the internal memory operation of base conditioning and management be take Memory Allocation unit as unit, and Memory Allocation unit adopts size pairing distribution method; During large Memory Allocation, arrange allocation units, allocation units size mUnitSize=applies for big or small RequestSize * 125%; Leading portion BigMem distributes large data in district, and back segment LitMem district distributes small data; During little Memory Allocation, preferentially in the allocation units that arranged, look for LitMem free area, if find allocation units to align backward, if find, do not arrange to be assigned to little Memory Allocation district;
(2), the associated aggregative index of Memory Allocation unit access: reflection Memory Allocation unit preferentially recalls to the index of virtual memory, and n allocation units are accessed associated aggregative index: Xn=(current time-last access time) * Fa * Fm/Fn
2, wherein chronomere is millisecond, and Fa is the Memory Allocation cell-average access frequency, and Fm is the maximum access in the Memory Allocation unit frequency, and Fn is n the allocation units access frequency; When Xn>Xa, n Memory Allocation unit can recall to virtual memory, and wherein Xa is the associated aggregative index mean value of access;
(3), virtual memory scheduling match substitution:
(a) memory block larger in Memory Allocation unit is only called in or is recalled in virtual memory scheduling, to reduce scheduled for executing number of times;
(b) when less memory block needs in Memory Allocation unit, in physical memory, move, with a small amount of internal storage data, move and increase virtual memory scheduling conversion matching rate;
(c) virtual memory scheduling replacement ratio compared with time, as long as source and target Memory Allocation cell size is the 75% close convertible object of coupling that just belongs to; In the situation that the large a lot of size of data of Data processing are close, virtual memory scheduling accuracy rate is high, efficiency is high like this;
(d) in physical memory, can recall by accessing associated aggregative index and decide;
(e) always keep necessary free physical memory space, be called minimum turnover spare area MinSpareMem, guarantee that virtual memory scheduling process does not affect system and normally moves.
2. the virtual memory management method of large Data processing as claimed in claim 1, is characterized in that virtual memory management and running realize according to the following steps:
A, initialization: to larger and continuous physical memory of operating system application, at disk, create or open virtual memory image file, file size is 4G-32G, determine according to application configuration.Notice that this file is not the PageFile.sys in windows.
B, little Memory Allocation: when application distributes the internal memory that is less than BitMemSize, carry out little Memory Allocation, it is empty in the Memory Allocation unit having arranged, searching L_MemArea and available size is greater than the unit of applying for size RequestSize, find record to return, do not find in Ze little Memory Allocation district and distribute.
C, establishment Memory Allocation unit: according to Memory Allocation application size RequestSize, physical memory in management is set up Memory Allocation unit the record that a size is RequestSize * 125%, when if free physical memory is less than MinSpareMem, setting up procedure 9.
D, large Memory Allocation: when distributing the internal memory that is greater than BitMemSize, application carries out large Memory Allocation, if there is the BigMem district of Memory Allocation unit to discharge, and size to fit, directly select, record and return; Otherwise, if when free physical memory is less than minimum turnover spare memory MinSpareMem, starts and recall task, perform step 3 simultaneously, then new unit is returned to applicant.
E, little core position are adjusted: when Memory Allocation units match is processed, because do not consider whether little memory field has data, if Memory Allocation unit is reused, and when in new use, BigMemSize can cover little memory field, little memory field need to move, moving method is execution step 2, and copies data, and the initialization of then little memory field is standby.
F, general memory distribute: according to application memory size, select respectively to perform step 2 and step 4.
The access of g, internal storage data: according to address mapping table, determine that institute's visit data is whether in physical memory, directly access in physical memory, otherwise select to carry out the task that recalls, then carry out and call in task visit data.
H, Memory Allocation units match process: source Memory Allocation cell size is that 1/2～1 times of target Memory Allocation unit is that the match is successful; No while then detecting the little Memory Allocation district of target Memory Allocation unit have data, when the large memory field data of Memory Allocation unit, source move on to target Memory Allocation unit, little internal storage data that whether can coverage goal Memory Allocation unit, if can cover, performs step 5.
I, Memory Allocation unit recall: according to the associated aggregative index of Memory Allocation unit access and recall size requirement, select the most suitable Memory Allocation unit recalling, can select to recall several Memory Allocation unit of asking that associated aggregative index is equal; If there are data in the little Memory Allocation district of the Memory Allocation unit being transferred out, perform step 5.
Call in j, Memory Allocation unit: first perform step 8, if do not find the Memory Allocation unit of coupling,, perform step 3, by object content in virtual memory copy to find or newly assigned Memory Allocation unit in, record scheduler mapping table.
3. a virtual memory management device for large Data processing, is characterized in that comprising:
Access interface module: for system applies provides the function of Memory Allocation, data access, comprise physics memory management and virtual memory management initialization, specify size and the virtual memory original document size of using physical memory; Memory Allocation, internal memory release, read data, write data, band buffering read data, band buffer write data; Virtual memory scheduling is carried out automatically, is not reflected to interface module; This module is calculated the associated aggregative index of Memory Allocation unit access simultaneously, in Memory Allocation and access process, automatically calculate, during for virtual memory scheduling decision-making which preferentially recall;
Memory Allocation administration module: manage this and install operable physical memory space and the auxiliary virtual memory space of depositing, carry out Memory Allocation, release, management address mapping table, when physical memory is less, start and recall task management module, or startup is called in executive routine, recalled executive routine; Memory Allocation is managed with Memory Allocation unit structural form of the present invention; Memory Allocation comprises large memory block and distributes and little Memory Allocation two lines, with large internal memory, is assigned as preferentially, and in order to adapt to large Memory Allocation, little memory field LitMem can move;
Call in execution module: carry out and call in allocation units to the process of physical memory from virtual memory; Wherein be included in and in physical memory, search or distribute a Memory Allocation unit matching, scheduler mapping table, calls in data from auxiliary depositing;
Recall task management module: when device starts, create a separate threads and be used for specially carrying out the task that recalls, from physical memory, recall obsolete Memory Allocation of some long periods unit, this module mainly comprises and recalls task queue management and call the task that recalls of carrying out; Separate threads mode by running background is carried out the task of recalling, and reduce virtual memory scheduling process does not affect main thread operation as far as possible;
Recall execution module: carry out to recall a Memory Allocation unit to the process of virtual memory from physical memory, comprise the data that detect and process LitMem district in moving target, Mobile data to auxiliaryly depositing, scheduler mapping table, the Memory Allocation unit after initialization is shifted out;
Initialization module: start apparatus of the present invention and enter the necessary initial work that normal operating conditions is carried out, comprise physical memory, virtual memory, address mapping table that device is managed, create and recall task management thread; Loading configuration and operating statistic information is configured device modules.
4. the virtual memory management device of saying as claim 3, is characterized in that working according to the following steps:
A, initialization: to larger and continuous physical memory of operating system application, by this device management, at disk, create or open virtual memory image file, file size is 4G-32G, according to application configuration and available secondary, depositing capacity determines, initialization physical memory structure and address mapping table, used in order to Memory Allocation and access; Establishment recalls task management thread;
B, little Memory Allocation: when application distributes the internal memory that is less than BitMemSize, carry out little Memory Allocation, it is empty in the Memory Allocation unit having arranged, searching L_MemArea and available size is greater than the unit of applying for size RequestSize, find record to return, do not find in Ze little Memory Allocation district and distribute;
C, establishment Memory Allocation unit: according to Memory Allocation application size RequestSize, at the physical memory of this device management, set up Memory Allocation unit the record that a size is RequestSize * 125%.When if free physical memory is less than MinSpareMem, setting up procedure 9;
D, large Memory Allocation: when application distributes the internal memory that is greater than BitMemSize, carry out large Memory Allocation; If there is the BigMem district of Memory Allocation unit to discharge, and size to fit, directly select, record and return; Otherwise, if when free physical memory is less than minimum turnover spare memory MinSpareMem, starts and recall task, perform step 3 simultaneously, then new unit is returned to applicant;
E, little core position are adjusted: when Memory Allocation units match is processed, because do not consider whether little memory field has data, if Memory Allocation unit is reused, and when in new use, BigMemSize can cover little memory field, little memory field need to move; Moving method is execution step 2, and copies data, and the initialization of then little memory field is standby;
F, general memory distribute.: according to application memory size, select respectively to perform step 2 and step 4;
The access of g, internal storage data: according to address mapping table, determine that institute's visit data is whether in physical memory, directly access in physical memory, otherwise select to carry out the task that recalls, then carry out and call in task visit data; During internal storage data access, upgrade last access time and the access times of Memory Allocation unit;
H, Memory Allocation units match process: source Memory Allocation cell size is that 1/2～1 times of target Memory Allocation unit is that the match is successful; No while then detecting the little Memory Allocation district of target Memory Allocation unit have data, when the large memory field data of Memory Allocation unit, source move on to target Memory Allocation unit, little internal storage data that whether can coverage goal Memory Allocation unit, if can cover, performs step 5;
I, Memory Allocation unit recall: according to the associated aggregative index of Memory Allocation unit access and recall size requirement, select the most suitable Memory Allocation unit recalling, can select to recall several Memory Allocation unit of asking that associated aggregative index is equal; If there are data in the little Memory Allocation district of the Memory Allocation unit being transferred out, perform step 5;
Call in j, Memory Allocation unit: first perform step 8, if do not find the Memory Allocation unit of coupling,, perform step 3, by object content in virtual memory copy to find or newly assigned Memory Allocation unit in, record scheduler mapping table;
K, calculate the associated aggregative index of Memory Allocation unit access: when recalling Memory Allocation unit and search coupling, calculate the associated aggregative index of current access, the associated aggregative index Xn=of access of each Memory Allocation unit (current time-last access time) * Fa * Fm/Fn2, wherein Xa is the associated aggregative index mean value of access.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410368134.9A CN104111896B (en) | 2014-07-30 | 2014-07-30 | Virtual memory management method and its device in big data processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410368134.9A CN104111896B (en) | 2014-07-30 | 2014-07-30 | Virtual memory management method and its device in big data processing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104111896A true CN104111896A (en) | 2014-10-22 |
| CN104111896B CN104111896B (en) | 2017-07-14 |
Family
ID=51708694
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410368134.9A Expired - Fee Related CN104111896B (en) | 2014-07-30 | 2014-07-30 | Virtual memory management method and its device in big data processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104111896B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105518741A (en) * | 2014-12-23 | 2016-04-20 | 英特尔公司 | Device and method used for managing virtual graphic processing unit |
| CN105824737A (en) * | 2016-03-31 | 2016-08-03 | 华中科技大学 | Memory data set replacing system and replacing method for big data processing system |
| CN106294190A (en) * | 2015-05-25 | 2017-01-04 | 中兴通讯股份有限公司 | A kind of memory space management and device |
| CN106339250A (en) * | 2016-08-19 | 2017-01-18 | 郭笃刚 | Management method of virtual memory of computer |
| CN111666227A (en) * | 2020-06-16 | 2020-09-15 | 北京紫光展锐通信技术有限公司 | Page bump protection method and device for memory recovery of operating system |
| CN115422554A (en) * | 2022-10-25 | 2022-12-02 | 支付宝(杭州)信息技术有限公司 | Request processing method, compiling method and trusted computing system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101676883A (en) * | 2008-09-17 | 2010-03-24 | 英业达股份有限公司 | Memory allocation and management method |
| CN102156675A (en) * | 2010-02-12 | 2011-08-17 | 中兴通讯股份有限公司 | Method and device for allocating memory |
| WO2013095639A1 (en) * | 2011-12-23 | 2013-06-27 | Intel Corporation | Utility and lifetime based cache replacement policy |
-
2014
- 2014-07-30 CN CN201410368134.9A patent/CN104111896B/en not_active Expired - Fee Related
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101676883A (en) * | 2008-09-17 | 2010-03-24 | 英业达股份有限公司 | Memory allocation and management method |
| CN102156675A (en) * | 2010-02-12 | 2011-08-17 | 中兴通讯股份有限公司 | Method and device for allocating memory |
| WO2013095639A1 (en) * | 2011-12-23 | 2013-06-27 | Intel Corporation | Utility and lifetime based cache replacement policy |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9824026B2 (en) | 2014-12-23 | 2017-11-21 | Intel Corporation | Apparatus and method for managing a virtual graphics processor unit (VGPU) |
| WO2016101149A1 (en) * | 2014-12-23 | 2016-06-30 | Intel Corporation | Apparatus and method for managing virtual graphics processor unit |
| US10565127B2 (en) | 2014-12-23 | 2020-02-18 | Intel Corporation | Apparatus and method for managing a virtual graphics processor unit (VGPU) |
| CN105518741A (en) * | 2014-12-23 | 2016-04-20 | 英特尔公司 | Device and method used for managing virtual graphic processing unit |
| CN105518741B (en) * | 2014-12-23 | 2019-04-09 | 英特尔公司 | Device and method for managing virtual pattern processor unit |
| CN106294190A (en) * | 2015-05-25 | 2017-01-04 | 中兴通讯股份有限公司 | A kind of memory space management and device |
| CN106294190B (en) * | 2015-05-25 | 2020-10-16 | 中兴通讯股份有限公司 | Storage space management method and device |
| CN105824737B (en) * | 2016-03-31 | 2018-08-17 | 华中科技大学 | Memory data set exchange system and method for replacing for big data processing system |
| CN105824737A (en) * | 2016-03-31 | 2016-08-03 | 华中科技大学 | Memory data set replacing system and replacing method for big data processing system |
| CN106339250A (en) * | 2016-08-19 | 2017-01-18 | 郭笃刚 | Management method of virtual memory of computer |
| CN106339250B (en) * | 2016-08-19 | 2019-09-27 | 郭笃刚 | A kind of management method of computer virtual memory |
| CN111666227A (en) * | 2020-06-16 | 2020-09-15 | 北京紫光展锐通信技术有限公司 | Page bump protection method and device for memory recovery of operating system |
| CN111666227B (en) * | 2020-06-16 | 2021-09-07 | 北京紫光展锐通信技术有限公司 | Page bump protection method and device for memory recovery of operating system |
| CN115422554A (en) * | 2022-10-25 | 2022-12-02 | 支付宝(杭州)信息技术有限公司 | Request processing method, compiling method and trusted computing system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104111896B (en) | 2017-07-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104111896A (en) | Virtual memory management method and virtual memory management device for mass data processing | |
| CN103049397B (en) | A kind of solid state hard disc inner buffer management method based on phase transition storage and system | |
| US10049040B2 (en) | Just in time garbage collection | |
| CN103995855B (en) | The method and apparatus of data storage | |
| US10860494B2 (en) | Flushing pages from solid-state storage device | |
| CN107148613A (en) | Current consumption profile based on storage facilities determines the adjustment of the spare space to being not useable in the storage facilities of user | |
| CN107423234A (en) | Multicomputer system and caching sharing method | |
| CN106547612A (en) | A kind of multi-task processing method and device | |
| KR102538126B1 (en) | Tail latency aware foreground garbage collection algorithm | |
| CN106484330A (en) | A kind of hybrid magnetic disc individual-layer data optimization method and device | |
| CN103608782A (en) | Selective data storage in LSB and MSB pages | |
| CN101373445B (en) | Method and apparatus for scheduling memory | |
| CN114546296B (en) | ZNS solid state disk-based full flash memory system and address mapping method | |
| CN100383792C (en) | Buffer data base data organization method | |
| CN103902475B (en) | Solid state disk concurrent access method and device based on queue management mechanism | |
| CN102073464A (en) | Method for creating allocation-on-demand incremental volume | |
| CN109783398A (en) | One kind is based on related perception page-level FTL solid state hard disk performance optimization method | |
| CN101968755A (en) | Application load change adaptive snapshot generating method | |
| CN103455443A (en) | Buffer management method and device | |
| CN104598394A (en) | Data caching method and system capable of conducting dynamic distribution | |
| CN109960471A (en) | Date storage method, device, equipment and storage medium | |
| CN101866320B (en) | Data management method and flash memory storage system and controller using the same | |
| CN102681792B (en) | Solid-state disk memory partition method | |
| CN107590287A (en) | A kind of file system caching of page write-back method, system, device and storage medium | |
| CN113778662A (en) | Memory recovery method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20170714 |