CN103077127B - A kind of method and apparatus of specified data migrating objects - Google Patents
A kind of method and apparatus of specified data migrating objects Download PDFInfo
- Publication number
- CN103077127B CN103077127B CN201210590184.2A CN201210590184A CN103077127B CN 103077127 B CN103077127 B CN 103077127B CN 201210590184 A CN201210590184 A CN 201210590184A CN 103077127 B CN103077127 B CN 103077127B
- Authority
- CN
- China
- Prior art keywords
- data
- screening
- migration
- storage
- threshold value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention is applicable to field of Internet communication, a kind of method of specified data migrating objects is provided, it comprises that each storage hierarchy exists the threshold value having set, and by the temperature value of valuation data and described threshold value comparison, carries out local screening and obtains local result of screening; By the result of described local screening, again screen in conjunction with the remaining space of each storage hierarchy, obtain described Data Migration object; After described Data Migration finishes, upgrade described threshold value according to migration results. The present invention utilizes local screening and overall situation screening to combine, first in each storage hierarchy from row filter, again the data object screening by part is further screened, in conjunction with the remaining space of storages at different levels, the object of specified data migration, the dynamic change that thereby the object that can make migration is the least possible and can adaptive system load, can also make full use of the storage resources of high-quality.
Description
Technical field
The invention belongs to field of Internet communication, relate in particular to a kind of method and dress of specified data migrating objectsPut.
Background technology
Along with the explosive increase of data, the cluster of storage mass data arises at the historic moment. Because store in clusterData volume very large, what have reaches PB level, synchronization may have a large amount of data accesses. Therefore,Whether the deposit position of data is reasonable, directly has influence on the access delay of data.
Data Migration, refers to the moving process of data between different storage mediums. Under normal circumstances, numberThe number of times occurring according to migration is all fewer, and object is also how relevant with system upgrade or data backup. But dividingData Migration in level storage system, the number of times of generation is many, its objective is the position in order to adjust data,Make the access performance of all data in system reach optimum. In hierarchical stor, conventionally can be situated between according to storageThe performance of matter is by its layering, and storage hierarchy is higher, configures highlyer, and access performance is better.
Complete Data Migration, the first step will be selected the object of migration, and second step is source and the order of selecting migrationMark, final step is only real data mobile. This is the process of then carrying out of first making a plan, itsMiddle first two steps are exactly the process of making a plan, and directly concern the effect of whole Data Migration. Select migration rightElephant according to being a numerical value, this numerical value is the direct reflection to data access " temperature ", numerical value is higher,Temperature is higher. And the target of Data Migration is exactly, data high to " temperature " are retained in to higher accumulation layerInferior upper, the data that " temperature " is low are retained in lower storage hierarchy.
Migration needs consumes resources, comprises hardware resource and Internet resources, and this consumption meeting is to system bandCarry out certain negative effect, therefore the data volume of migration should be more few better. For this reason, a lot of migration strategys are just establishedDetermine threshold value, only had the temperature value of data to exceed threshold value, just can move, be no more than threshold value, can not moveMove, arrange like this, really make Data Migration amount reduce a lot, but arranging of threshold value is more difficult. CauseFor larger variation may occur the load in system, make the temperature of data that very large variation also occur,If threshold value does not make corresponding changes, be just difficult to adapt to, therefore needing has dynamic threshold value. Different threshold valuesMethod to set up inconsistent, some threshold values are relative value, for example the data temperature value on lower storage level justAs long as the temperature value of 60% data in a high storage hierarchy thereon, these data just can be moved above, this thresholdValue is a kind of dynamic threshold value really, if but the data in higher storage level are all high temperatures, butBe that its remaining space also has a lot, data below cannot be moved into. Do not make full use of the storage money of high-qualitySource. Therefore, when screening migrating objects, must consider the remaining space of storage level, could protect like thisThe reasonable utilization of card storage resources. In addition, also some migration strategy, in screening migrating objects, usesBe " rule set ", as long as migrating objects met rule, just move this " rule set "Need very professional personage to set, and for single application scenarios, be difficult to adapt to the negative of dynamic changeCarry.
Summary of the invention
The embodiment of the present invention provides a kind of method and apparatus of specified data migrating objects, is intended to solve current trueThe method of given data migrating objects does not make full use of the storage resources of high-quality, makes to also have very at remaining spaceWhen many, data below can not be moved into. Also there is the problem that is difficult to the load that adapts to dynamic change simultaneously.
For this reason, the embodiment of the present invention provides following technical scheme:
A device for specified data migrating objects, comprising:
Local screening object select module, for choosing valuation data all in each storage hierarchy, asThe object of local screening;
Local screening module, is connected with part screening object select module, and its each storage hierarchy existsThe threshold value having set, for by the temperature value of described valuation data and described threshold value comparison, carries out partScreening obtains the result of local screening;
Overall situation screening module, is connected with part screening module, for the result that described part is screened, knotThe remaining space that closes each storage hierarchy screens again, filters out the data object that does not need migration, obtainsThe object of described Data Migration;
More new module, after finishing, upgrades described threshold value according to migration results for described Data Migration.
The embodiment of the present invention also provides a kind of method of specified data migrating objects, comprises the following steps:
A: each storage hierarchy exists the threshold value having set, by the temperature value of valuation data with described inThreshold value comparison, carries out local screening and obtains local result of screening;
B: by the result of described local screening, again screen in conjunction with the remaining space of each storage hierarchy,Obtain described Data Migration object;
C: after described Data Migration finishes, upgrade described threshold value according to migration results.
Compared with prior art, embodiments of the invention tool has the following advantages:
The present invention utilizes local screening and overall situation screening to combine, first in each storage hierarchy from row filter,Again the result of screening by part is further screened, in conjunction with the remaining space of storages at different levels, determine numberAccording to the object of migration, thus can make the object of migration the least possible and can adaptive system load dynamicallyChange, can also make full use of the storage resources of high-quality.
Brief description of the drawings
Fig. 1 is the method flow diagram of the specified data migrating objects that provides of the embodiment of the present invention;
Fig. 2 is the structure chart of the device of the specified data migrating objects that provides of the embodiment of the present invention.
Detailed description of the invention
In order to make object of the present invention, technical scheme and advantage clearer, below in conjunction with accompanying drawing and realityExecute example, the present invention is further elaborated. Should be appreciated that described herein is only of the present inventionPart embodiment, instead of whole embodiment. Based on the embodiment in the present invention, the common skill in this areaThe every other embodiment that art personnel obtain under the prerequisite of not making creative work, belongs to thisThe scope of bright protection.
Fig. 1 is the method flow diagram of the specified data migrating objects that provides of the embodiment of the present invention, for the ease of sayingBright, only show the part relevant to the embodiment of the present invention.
As shown in Figure 1, the method comprises the following steps:
Step 101, there is the threshold value having set in each storage hierarchy, by the temperature value of valuation dataWith described threshold value comparison, carry out local screening and obtain local result of screening.
Be understandable that, while carrying out part screening, choose valuation data all in each storage hierarchy,As the object of part screening. Concrete, in hierarchical stor, there are valuation data, each valuationData have temperature value to answer in contrast, and described temperature value can be big or small.
In the present embodiment, described hierarchical stor is divided at least three levels, is first from top to bottom successivelyLayer, the second layer, the 3rd layer, be equivalent to a pyramidal structure, and can between any two levelsCarry out Data Migration, each storage hierarchy can be toward other storage hierarchy migration datas. Wherein, firstLayer, toward the second layer or the 3rd layer of migration, moves under being called; The 3rd layer is moved toward the second layer or ground floor, is calledMove.
Be understandable that, the hierarchical stor of three levels always has the direction of 6 Data Migrations.
Concrete, described threshold value is a nonnegative real number, can be big or small.
Described threshold value comprises upper limit threshold and lower threshold, preferred, first upper limit threshold is set as to 0,Lower threshold is set as positive infinity.
Concrete, temperature value is greater than to upper limit threshold or is less than the described valuation data of lower threshold, asThe result of local screening. The data that are less than upper limit threshold by temperature value or are greater than lower threshold screen out.
Step 102, by the result of described local screening, carries out again in conjunction with the remaining space of each storage hierarchyScreening, filters out the data object that does not need migration, obtains described Data Migration object.
Concrete, the result that described local screening is obtained, arranges from big to small according to the size of its temperature value,Match with storage hierarchy, then in storage hierarchy, attempt depositing.
Concrete, if can hold described data in described storage hierarchy, deposit successfully;
Concrete, if can not hold described data, described data attempt depositing to next stage storage hierarchyPut, until described data are all deposited successfully.
In the present embodiment, it is in the situation that not considering the original memory location of data that described trial is deposited, a rootAccording to the remaining space of its temperature value and storage hierarchys at different levels, determine its deposit position because having. Deposit if originalStorage space put with because of have to deposit level consistent, described data do not need migration; If original memory location withBecause of have to deposit level inconsistent, because the deposit position that has is for moving target.
Remaining space refers to the difference between the best usage space of described storage hierarchy and actual usage space.
The ratio of the best usage space usage spaces whole with it of one-level storage is less than 1, is generally 80%,The ratio of the usage space usage spaces whole with it of all the other storage hierarchys can be adjusted according to actual conditions.
Step 103, after described Data Migration finishes, upgrades described threshold value according to migration results.
Preferably, after described Data Migration finishes, on being updated in this storage hierarchy, moves described upper limit thresholdData in minimum temperature value, the maximum temperature in the data that lower threshold moves under being updated in this storage levelValue.
Preferably, if not, move or under move generation, the threshold value that maintains this storage hierarchy is constant.
Based on identical design, the embodiment of the present invention also provides a kind of device of specified data migrating objects, asShown in Fig. 2, this device comprises:
Local screening object select module 201, for choosing valuation data all in each storage hierarchy,As the object of part screening.
Concrete, in system, there are the data of valuation, each valuation data have temperature value in contrastShould, described temperature value can be big or small.
Local screening module 202, is connected with part screening object select module 201, its each accumulation layerThe inferior threshold value having set that all exists, for by the temperature value of described valuation data and described threshold value comparison,Carry out local screening and obtain local result of screening.
Be understandable that, while carrying out part screening, choose valuation data all in each storage hierarchy,As the object of part screening. Concrete, in system, there are valuation data, each valuation data haveTemperature value is answered in contrast, and described temperature value can be big or small.
Concrete, threshold value is a nonnegative real number, can be big or small.
Described threshold value comprises upper limit threshold and lower threshold, preferred, first upper limit threshold is set as to 0,Lower threshold is set as positive infinity.
Concrete, temperature value is greater than to upper limit threshold or is less than the described valuation data of lower threshold, asThe result of local screening. The data that are less than upper limit threshold by temperature value or are greater than lower threshold screen out.
Overall screening module 203, is connected with described local screening module 202, for described part is sievedThe result of choosing, screens again in conjunction with the remaining space of each storage hierarchy, filters out the number that does not need migrationAccording to object, obtain the object of described Data Migration.
Concrete, the result that described local screening is obtained, arranges from big to small according to the size of its temperature value,Match with storage hierarchy, then in storage hierarchy, attempt depositing.
Concrete, if can hold described data in described storage hierarchy, deposit successfully;
Concrete, if can not hold described data, described data attempt depositing to next stage storage hierarchyPut, until described data are all deposited successfully.
In the present embodiment, it is in the situation that not considering the original memory location of data that described trial is deposited, a rootAccording to the remaining space of its temperature value and storage hierarchys at different levels, determine its deposit position because having. Deposit if originalStorage space put with because of have to deposit level consistent, described data do not need migration; If original memory location withBecause of have to deposit level inconsistent, because the deposit position that has is for moving target.
Remaining space refers to the difference between the best usage space of described storage hierarchy and actual usage space.
The ratio of the best usage space usage spaces whole with it of one-level storage is less than 1, is generally 80%,The ratio of the usage space usage spaces whole with it of all the other storage hierarchys can be adjusted according to actual conditions.
More new module 204, after finishing, upgrades described threshold value according to migration results for described Data Migration.
Preferably, after described Data Migration finishes, on being updated in this storage hierarchy, moves described upper limit thresholdData in minimum temperature value, the maximum temperature in the data that lower threshold moves under being updated in this storage levelValue.
Preferably, if not, move or under move generation, the threshold value that maintains this storage hierarchy is constant.
The local screening of embodiment of the present invention utilization and overall situation screening combine, first in each storage hierarchy voluntarilyScreening, then the result of screening by part is further screened, in conjunction with the remaining space of storages at different levels,The object of specified data migration, thus can make the object of migration the least possible and can adaptive system loadDynamic change, can also make full use of the storage resources of high-quality.
It will be appreciated by those skilled in the art that module in the device in embodiment can according to embodiment describe intoRow is distributed in the device of embodiment, also can carry out respective change be positioned at be different from one of the present embodiment orIn multiple devices. The module of above-described embodiment can be merged into a module, also can further split into manyIndividual submodule.
Through the above description of the embodiments, those skilled in the art can be well understood to the present inventionThe mode that can add essential general hardware platform by software realizes, and can certainly pass through hardware, but veryUnder susceptible condition, the former is better embodiment. Based on such understanding, technical scheme of the present invention in essenceThe part in other words prior art being contributed can embody with the form of software product, this computerSoftware product is stored in a storage medium, comprises that some instructions are in order to make a station terminal equipment (passableMobile phone, personal computer, server, or the network equipment etc.) carry out described in each embodiment of the present inventionMethod.
The above is only the preferred embodiment of the present invention, it should be pointed out that common for the artTechnical staff, under the premise without departing from the principles of the invention, can also make some improvements and modifications,These improvements and modifications also should be looked protection scope of the present invention.
Claims (4)
1. a method for specified data migrating objects, is characterized in that, comprises the following steps:
A: each storage hierarchy exists the threshold value having set, by the temperature value of valuation data and described thresholdValue relatively, is carried out the local result that obtains local screening of screening; Described steps A comprises:
A1: described threshold value comprises upper limit threshold and lower threshold, first upper limit threshold is set as to 0, lower limitThreshold is positive infinity, described threshold value be can be big or small nonnegative real number;
A2: temperature value is greater than to upper limit threshold or is less than the described valuation data of lower threshold, as describedThe result of local screening;
B: by the result of described local screening, again screen in conjunction with the remaining space of each storage hierarchy, obtainObtain described Data Migration object;
Described step B comprises:
B1: the data that described local screening is obtained, arrange from big to small according to its temperature value, then toward depositingReservoir time upper trial deposited; It is in the situation that not considering the original memory location of data that described trial is deposited, onlyAccording to the remaining space of its temperature value and storage hierarchys at different levels, determine its deposit position because having, if originalMemory location with because of have to deposit level consistent, described data do not need migration; If original memory locationWith because of have to deposit level inconsistent, because the deposit position having is migration target;
B2: if can hold described data in described storage hierarchy, deposit successfully;
B3: if can not hold described data, described data attempt depositing to next stage storage hierarchy,Until described data are all deposited successfully;
C: after described Data Migration finishes, upgrade described threshold value according to migration results;
Described step C comprises:
C1: after described Data Migration finishes, the number moving on described upper limit threshold is updated in this storage hierarchyAccording in minimum temperature value, the maximum temperature value in the data that lower threshold moves under being updated in this storage level;
C2: if not, move or under move generation, the threshold value that maintains this storage hierarchy is constant.
2. the method for specified data migrating objects as claimed in claim 1, is characterized in that, described residueSpace is the difference between the best usage space of described storage hierarchy and actual usage space.
3. the method for specified data migrating objects as claimed in claim 1, is characterized in that, one-level storageThe ratio of best usage space usage spaces whole with it be less than 1.
4. a device for specified data migrating objects, is characterized in that, comprising:
Local screening object select module, for choosing valuation data all in each storage hierarchy, asThe object of local screening;
Local screening module, is connected with part screening object select module, and its each storage hierarchy existsThe threshold value having set, for by the temperature value of described valuation data and described threshold value comparison, carries out partScreening obtains the result of local screening; Particularly, described threshold value comprises upper limit threshold and lower threshold, firstUpper limit threshold is set as to 0, and lower threshold is set as positive infinity, described threshold value be can be big or small non-Negative real number; Temperature value is greater than to upper limit threshold or is less than the described valuation data of lower threshold, as describedThe result of local screening;
The data that described local screening module obtains described local screening, arrange from big to small according to its temperature valueRow are then attempted depositing in storage hierarchy; If can hold described data in described storage hierarchy, depositPut successfully; If can not hold described data, described data attempt depositing to next stage storage hierarchy,Until described data are all deposited successfully; It is the situation not considering the original memory location of data that described trial is depositedUnder, only according to the remaining space of its temperature value and storage hierarchys at different levels, determine its deposit position because having, asThe original memory location of fruit with because of have to deposit level consistent, described data do not need to move; Deposit if originalStorage space put with because of have to deposit level inconsistent, because the deposit position having is migration target; Overall situation screening mouldPiece, is connected with part screening module, for by the result of described local screening, in conjunction with each storage hierarchyRemaining space screens again, filters out the data object that does not need migration, obtains described Data MigrationObject;
More new module, after finishing, upgrades described threshold value according to migration results for described Data Migration; SpecificallyGround, after described Data Migration finishes, in the data of moving on described upper limit threshold is updated in this storage hierarchyMinimum temperature value, the maximum temperature value in the data that lower threshold moves under being updated in this storage level;
If not, move or under move generation, the threshold value that maintains this storage hierarchy is constant.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210590184.2A CN103077127B (en) | 2012-12-29 | 2012-12-29 | A kind of method and apparatus of specified data migrating objects |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210590184.2A CN103077127B (en) | 2012-12-29 | 2012-12-29 | A kind of method and apparatus of specified data migrating objects |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103077127A CN103077127A (en) | 2013-05-01 |
| CN103077127B true CN103077127B (en) | 2016-05-11 |
Family
ID=48153659
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210590184.2A Active CN103077127B (en) | 2012-12-29 | 2012-12-29 | A kind of method and apparatus of specified data migrating objects |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103077127B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106658353A (en) * | 2016-09-14 | 2017-05-10 | 广东欧珀移动通信有限公司 | Data migration method and mobile equipment |
| CN106507269B (en) * | 2016-09-14 | 2020-02-07 | Oppo广东移动通信有限公司 | Data migration method and terminal equipment |
| CN108874311B (en) * | 2018-05-29 | 2022-02-08 | 北京盛和大地数据科技有限公司 | Data migration method and device in converged storage system |
| CN108810140B (en) * | 2018-06-12 | 2021-09-28 | 湘潭大学 | High-performance hierarchical storage optimization method based on dynamic threshold adjustment in cloud storage system |
| CN116668380B (en) * | 2023-07-28 | 2023-10-03 | 北京中科网芯科技有限公司 | Message processing method and device of convergence diverter equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007280324A (en) * | 2006-04-12 | 2007-10-25 | Hitachi Ltd | Computer system, management computer, and virtual storage device |
| CN102117248A (en) * | 2011-03-09 | 2011-07-06 | 浪潮(北京)电子信息产业有限公司 | Caching system and method for caching data in caching system |
| CN102122303A (en) * | 2011-03-15 | 2011-07-13 | 浪潮(北京)电子信息产业有限公司 | Method for data migration, service system and sever equipment |
-
2012
- 2012-12-29 CN CN201210590184.2A patent/CN103077127B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2007280324A (en) * | 2006-04-12 | 2007-10-25 | Hitachi Ltd | Computer system, management computer, and virtual storage device |
| CN102117248A (en) * | 2011-03-09 | 2011-07-06 | 浪潮(北京)电子信息产业有限公司 | Caching system and method for caching data in caching system |
| CN102122303A (en) * | 2011-03-15 | 2011-07-13 | 浪潮(北京)电子信息产业有限公司 | Method for data migration, service system and sever equipment |
Non-Patent Citations (1)
| Title |
|---|
| 海量存储***中数据分级存储的关键技术研究;宋丽娜;《中国优秀硕士学位论文全文数据库 信息科技辑》;20120715(第7期);第35-38页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103077127A (en) | 2013-05-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103077127B (en) | A kind of method and apparatus of specified data migrating objects | |
| US20220027189A1 (en) | System and Method for Optimizing Placements of Virtual Machines on Hypervisor Hosts | |
| CN104484233B (en) | A kind of resource allocation methods | |
| CN103399781B (en) | Cloud Server and virtual machine management method thereof | |
| CN109144699A (en) | Distributed task dispatching method, apparatus and system | |
| CN106095531B (en) | A kind of dispatching method of virtual machine loaded based on grade and physical machine in cloud platform | |
| CN110249310A (en) | The resource management for virtual machine in cloud computing system | |
| CN105468435A (en) | NFV dynamic resource distribution method | |
| CN106201661A (en) | Method and apparatus for elastic telescopic cluster virtual machine | |
| CN104679594B (en) | A kind of middleware distributed computing method | |
| CN104461914A (en) | Automatic simplified-configured self-adaptation optimization method | |
| CN103780646A (en) | Cloud resource scheduling method and system | |
| CN106569898A (en) | Resource distribution method and mobile terminal | |
| CN106357791A (en) | Method, device and system for processing services | |
| CN103778222A (en) | File storage method and system for distributed file system | |
| Pujol et al. | Scaling online social networks without pains | |
| CN107943423A (en) | The management method and computer-readable recording medium of storage resource in cloud system | |
| CN103713852A (en) | Method for processing information, service platform and electronic devices | |
| CN110297959A (en) | Date storage method, device, storage medium and Edge Server | |
| CN106384292A (en) | System and method for collecting relation among social network users | |
| CN109359205A (en) | A kind of remote sensing image cutting method and equipment based on geographical grid | |
| CN102779058A (en) | Interface data loading device and interface data loading method | |
| CN108459926A (en) | Data remote backup method, apparatus and computer-readable medium | |
| CN106874108A (en) | Thin cloud is minimized in mobile cloud computing use number technology | |
| CN104683480A (en) | Distribution type calculation method based on applications |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20230330 Address after: 518000 A-301, office building, Shenzhen Institute of advanced technology, No. 1068, Xue Yuan Avenue, Shenzhen University Town, Shenzhen, Guangdong, Nanshan District, China Patentee after: Shenzhen shen-tech advanced Cci Capital Ltd. Address before: 1068 No. 518055 Guangdong city in Shenzhen Province, Nanshan District City Xili University School Avenue Patentee before: SHENZHEN INSTITUTES OF ADVANCED TECHNOLOGY |
|
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20230424 Address after: 518000 b402, blocks a and B, Nanshan medical device Industrial Park, No. 1019, Nanhai Avenue, Yanshan community, merchants street, Nanshan District, Shenzhen, Guangdong Patentee after: Shenzhen hongzhituoxin venture capital enterprise (L.P.) Address before: 518000 A-301, office building, Shenzhen Institute of advanced technology, No. 1068, Xue Yuan Avenue, Shenzhen University Town, Shenzhen, Guangdong, Nanshan District, China Patentee before: Shenzhen shen-tech advanced Cci Capital Ltd. |