CN103140840B - The method and device of data management - Google Patents
The method and device of data management Download PDFInfo
- Publication number
- CN103140840B CN103140840B CN201180001900.2A CN201180001900A CN103140840B CN 103140840 B CN103140840 B CN 103140840B CN 201180001900 A CN201180001900 A CN 201180001900A CN 103140840 B CN103140840 B CN 103140840B
- Authority
- CN
- China
- Prior art keywords
- block
- storage pool
- record
- index
- key
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000000034 method Methods 0.000 title claims abstract description 59
- 238000013523 data management Methods 0.000 title claims abstract description 18
- 238000012217 deletion Methods 0.000 claims abstract description 25
- 230000037430 deletion Effects 0.000 claims abstract description 25
- 230000009471 action Effects 0.000 claims abstract description 21
- 239000000203 mixture Substances 0.000 claims abstract description 4
- 238000000151 deposition Methods 0.000 claims description 17
- 238000002372 labelling Methods 0.000 claims description 4
- 230000006399 behavior Effects 0.000 claims description 3
- 238000007726 management method Methods 0.000 abstract description 7
- 239000012634 fragment Substances 0.000 abstract description 5
- 230000008569 process Effects 0.000 description 11
- 230000008859 change Effects 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000001174 ascending effect Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 235000013399 edible fruits Nutrition 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000000630 rising effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0613—Improving I/O performance in relation to throughput
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0644—Management of space entities, e.g. partitions, extents, pools
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Signal Processing For Digital Recording And Reproducing (AREA)
Abstract
A kind of method and device of data management.Described method is equal by the block size being logically divided in multiple storage pool, and same storage pool by disk；Inlet porting array of indexes in internal memory, represents the information of each storage pool by this entrance array of indexes；By the Key in key-value pair Key Value, the pond index of the storage pool marked off in described disk, and the block index composition tlv triple of block in storage pool, leave in internal memory；Position a certain data record storage position in described disk by entrance array of indexes set in internal memory and the triplet information deposited, and carry out reading and writing or the deletion action of these data according to this storage position.Just can effectively be managed mass data by this technical scheme, without carrying out disk fragments arrangement, make disk utilization to be greatly improved, and be greatly improved index and read the speed of K V, thus the storage of efficient data and management operation are provided.
Description
Technical field
The present invention relates to technical field of data administration, particularly relate to the method and device of a kind of data management.
Background technology
At present, along with computer technology and the development of network technology, data message amount is the growth of geometrical progression,
The data of the mankind reach unprecedented scale, and storage and management to these ultra-large data have become as
Industry a major challenge.Traditional relevant database is when managing this ultra-large data, it is impossible to provide
Efficient disk read-write operation, it is impossible to meet good autgmentability and high availability.Under this background,
Various non-relational database, the appearance of particularly key-value pair K-V (Key-Value) data base Store DB
Well solve the problems referred to above.
In prior art, Key-Value storage has become as the one of industry and commonly uses storage mode, is employed
Increasing local, particularly in cloud storage；These storage engines have plenty of additional Append only
Pattern, has plenty of the pattern that can directly revise, and these technical schemes be directed to backstage data compression or
Person's defragmentation, is required for carrying out irregular global data and moves and improve disk utilization, and this
During compression or defragmentation, all the performance of data base can be caused the biggest impact, it is impossible to provide
Efficient disk read-write operation.
Summary of the invention
It is an object of the invention to provide the method and device of a kind of data management, it is possible to effective management magnanimity number
According to, without carrying out disk fragments arrangement, make disk utilization to be greatly improved, and be greatly improved
Index and the speed of reading K-V, thus the storage of efficient data and management operation are provided.
A kind of method of data management, described method includes:
Block disk being divided in multiple storage pool, and same storage pool equal in magnitude；
Inlet porting array of indexes in internal memory, described entrance array of indexes is used for representing the letter of each storage pool
Breath；
When the write operation of the record received for key-value pair key-value type, by set in internal memory
The entrance array of indexes put positions the data corresponding for described value storage position in described disk, and root
The write operation of described data is carried out according to described storage position.
A kind of method of data management, described method includes:
The block size being divided in multiple storage pool, and same storage pool by disk is equal；
Inlet porting array of indexes in internal memory, described entrance array of indexes is used for representing the letter of each storage pool
Breath；
In internal memory, storage has the tlv triple of the record correspondence of key-value pair key-value type, described tlv triple bag
Include the Key of described record, and store pond index and the block index of the storage pool of described record；
When application program is read or deletes the record of described key-value pair Key-Value type, according to described record
Key search in internal memory correspondence tlv triple；
When carrying out read operation, then index according to the pond in described tlv triple and find the storage pool depositing this record
Original position, and calculate block according to the size of block in described storage pool and the index of the block in described tlv triple
Side-play amount in described storage pool；Described record is obtained at magnetic according to described original position and described side-play amount
Storage position in dish, reads the Value of described record from described storage position；
When carrying out deletion action, from internal memory, delete the described tlv triple found, and according to described ternary
The bitmap block of the block depositing described record is labeled as the free time by pond index and the block index of group.
A kind of device of data management, described device includes:
Storage pool division unit, is arranged in disk, for disk is divided into multiple storage pool and same
Block in storage pool equal in magnitude；
Storage pool information arranges unit, is arranged in internal memory, for inlet porting array of indexes, is entered by this
Mouth array of indexes represents the information of each storage pool；
Data write operation unit, for writing behaviour when receive record for key-value pair key-value type
When making, entrance array of indexes set in unit is set by described storage pool information and positions described value
Corresponding data storage position in described disk, and carry out writing of described data according to described storage position
Operation.
A kind of device of data management, described device includes:
Storage pool division unit, is arranged in disk, for disk is divided into multiple storage pool and same
Block size in storage pool is equal；
Storage pool information arranges unit, is arranged in internal memory, for inlet porting array of indexes, is entered by this
Mouth array of indexes represents the information of each storage pool；
Triple store unit, is arranged in internal memory, for storing the record of key-value pair key-value type
Corresponding tlv triple, described tlv triple includes the Key of described record, and stores the storage pool of described record
Pond index and block index；
Tlv triple acquisition module, for when application program is read or deletes described key-value pair Key-Value type
During record, in internal memory, search the tlv triple of correspondence according to the Key of described record；
Data reading operation module, for when carrying out read operation, indexes according to the pond in described tlv triple and finds
Deposit the original position of the storage pool of this record, and according to the size of block in described storage pool and described tlv triple
In block index calculate block side-play amount in described storage pool；According to described original position and described skew
Measure described record storage position in disk, read the Value of described record from described storage position；
Data deletion operation module, for when carrying out deletion action, deletes described in finding from internal memory
Tlv triple, and index the bitmap block mark of the block depositing described record with block according to the pond index of described tlv triple
It is designated as the free time.
By above-mentioned provided technical scheme it can be seen that described method is multiple by being logically divided into by disk
Block size in storage pool, and same storage pool is equal；Inlet porting array of indexes in internal memory, by this
Entrance array of indexes represents the information of each storage pool；By the Key in key-value pair Key-Value, described
The pond index of the storage pool marked off in disk, and the block index composition tlv triple of block in storage pool, deposit
It is placed in internal memory；Certain is positioned by entrance array of indexes set in internal memory and the triplet information deposited
One data record storage position in described disk, and the reading and writing of these data are carried out according to this storage position
Or deletion action.Mass data just can be effectively managed, without carrying out disk by this technical scheme
Defragmentation, makes disk utilization to be greatly improved, and is greatly improved index and reads the speed of K-V,
Thus the storage of efficient data and management operation are provided.
Accompanying drawing explanation
Fig. 1 is the schematic flow sheet of data managing method described in the embodiment of the present invention；
Fig. 2 is another schematic flow sheet of data managing method described in the embodiment of the present invention；
Fig. 3 is the model schematic described in present example shown by method；
Fig. 4 is the schematic flow sheet of write operation in the lifted instantiation of the embodiment of the present invention；
Fig. 5 is the schematic flow sheet of read operation in the lifted instantiation of the embodiment of the present invention；
Fig. 6 is the schematic flow sheet of deletion action in the lifted instantiation of the embodiment of the present invention；
Fig. 7 is the structural representation of data administrator described in the embodiment of the present invention；
The structural representation of the another kind of data administrator that Fig. 8 is provided by the embodiment of the present invention.
Detailed description of the invention
Embodiment of the present invention provides the method and device of a kind of data management, it is possible to effectively manage magnanimity
Data, without carrying out disk fragments arrangement, make disk utilization to be greatly improved, and greatly carry
High index and the speed of reading K-V, thus the storage of efficient data and management operation are provided.
Below according to accompanying drawing, the embodiment of the present invention is described in detail, is illustrated in figure 1 the embodiment of the present invention
The schematic flow sheet of described method, described method includes:
Step 11: the block size being divided in multiple storage pool, and same storage pool by disk is equal.
In this step, when carrying out data management, first by disk by being logically divided into multiple storage pool,
And block size in same storage pool is equal, such as, disk can be divided into 2048 storage pools, Chi Zhong
The size of block is followed successively by 512B, 1K, 1.5K, 2K, 2.5K ... 1M, say, that 2048 marked off
Individual storage pool has the block pond of 512B, has the block pond of 1K, has the block pond of 1.5K, by that analogy, the most permissible
Multiple for 512B.
Step 12: inlet porting array of indexes in internal memory, described entrance array of indexes is used for representing that each is deposited
The information of reservoir.
In this step, after the division according to above-mentioned steps 11, inlet porting array of indexes in internal memory,
Represented that by this entrance array of indexes the information of each storage pool, described entrance array of indexes include: storage
The pond index in pond, block big in the bitmap block of each block, the total length of storage pool and storage pool in storage pool
Little.Wherein, the total length of storage pool is optional.
The pond index of above-mentioned storage pool points to each storage pool, and above-mentioned bitmap block is used for representing in storage pool each
The service condition of block, especially by idle to described bitmap block mark or take and realize, such as, can pass through
0,1 sequence identifies, and is designated 0 expression idle, if being designated when certain block is taken up by data during initialization
1。
Step 13: when the write operation of the record received for key-value pair key-value type, by interior
Deposit set entrance array of indexes and position the data corresponding for described value storage position in described disk
Put, and carry out the write operation of described data according to described storage position.
In this step, after the operation of above-mentioned steps, it is possible to by entrance set in internal memory
Array of indexes positions a certain data record storage position in described disk, and enters according to this storage position
The write operation of these data of row.
Here, data can identify with the record of a key-value pair Key-Value type, a note
The Key of record is unique in whole data base, can find this record according to this Key, namely should
Data.Such as key-value pair (key1, " hello, world "), by searching key1 in data base,
Just can find Value: " hello, world ".
Specifically, when application program writes the record of a key-value pair Key-Value type, according to Value
Size obtain respective stored pond pond index (for example: write a key-value pair Key-Value type
Record, the size of its value is 1000B, if disk file is divided into 512B, 1024B, 1.5K......
1M, totally 2048 storage pools, then the size of Value rounds up (multiples of the 512 of ＞=value), 1000B
Round up just for 1024B, then this record should be stored in the storage pool that block size is 1024B,
Thus further determine that the pond index that the storage pool of this 1024B is corresponding).
Then, the entrance array of indexes in the respective stored pond arranged in internal memory is obtained according to this pond index, from this
Entrance array of indexes obtains the bitmap block information of this storage pool, finds the free block in storage pool, and obtain
The block index that this record is to be deposited；According to this block index, the size of block and storage pool original position in storage pool
Calculate acquisition this record storage position in described disk, such as, if depositing in entrance array of indexes
The original position of the storage pool this record is 200000, obtains bitmap block information according to block index, finds the
One idle block (if being the 10th, this block index is 10), then deposit the storage position of this record
TargetPos=200000+10*1024=210240, i.e. represents the storage in disk of this record
Position, then carry out data write operation, this record is stored in described storage position.
It addition, above-mentioned after the size according to Value obtains the pond index in respective stored pond, also include meter
Calculate the process of the original position of storage pool corresponding to described pond index, record each storage pool in disk
Putting in order, described entrance array of indexes also includes the total length of storage pool, calculates described pond index correspondence
The original position of storage pool specifically includes: all before the storage pool corresponding by being arranged in described pond index are deposited
The total length sum of reservoir adds 1 and is the original position that described pond indexes the storage pool of correspondence.
Above-mentioned storage pool original position refers to each storage pool logical place in disk file, in system
Starting the original position of first storage pool when initializing is 0, and the original position of second storage pool is first
Individual storage pool is taken up space+1 in disk file, records each storage pool putting in order in disk, so
Rear the like the original position obtaining all storage pools, will be arranged in the storage pool that described pond index is corresponding
The total length sum of all storage pools before adds 1 and is the start bit that described pond indexes the storage pool of correspondence
Put.On the other hand, when entrance array of indexes does not include the total length of storage pool, it is also possible to pass through storage pool
In the quantity of block and the size of block be calculated total size of storage pool.
It addition, after data write operation success, described method also includes: the block position deposited by described record
The labelling of figure is revised as taking；By the key of described record, the pond index and the block that point to this record position index
Leave in internal memory as a tlv triple, follow-up can quickly position certain note by this tlv triple to facilitate
The position of record, deletes or search operation accordingly.
It addition, method described in the embodiment of the present invention can also carry out read operation, when application program reads a key assignments
During to the record of Key-Value type, in internal memory, search tlv triple file according to the Key of this record, as
Fruit is not found, then it represents that without this record in data base, read operation terminates；Otherwise find a tlv triple literary composition
Part key, pond index and block index；The original position of the storage pool depositing this record is found according to this pond index,
And calculate block side-play amount in this storage pool according to block index；Side-play amount according to being calculated is somebody's turn to do
Record storage position in disk, and from this storage position, take out the Value of this record.
It addition, method described in the embodiment of the present invention can also carry out deletion action, when application program deletes one
During the record of key-value pair Key-Value type, in internal memory, search tlv triple file according to the Key of this record,
Without finding, then deletion action terminates；Otherwise return a tlv triple file key, pond index and block
Index；From internal memory, delete this tlv triple file, and the bitmap block labelling of this record will be deposited according to block index
For the free time.
The embodiment of the present invention additionally provides another kind of data managing method, is illustrated in figure 2 the embodiment of the present invention
Another schematic flow sheet of described data managing method, Fig. 2 includes:
Step 21: the block size being divided in multiple storage pool, and same storage pool by disk is equal.
By this entrance array of indexes, step 22: inlet porting array of indexes in internal memory, represents that each is deposited
The information of reservoir.
Step 23: in internal memory, storage has the tlv triple of the record correspondence of key-value pair key-value type, described
Tlv triple includes the Key of described record, and stores pond index and the block index of the storage pool of described record.
Step 24: when application program is read or deletes the record of a key-value pair Key-Value type, searches
And obtain tlv triple.
In this step, when application program is read or deletes the record of a key-value pair Key-Value type,
Key according to this record searches tlv triple in internal memory, without finding, then it represents that without this in data base
Record, reads or deletion action terminates；Tlv triple key, pond index and block is otherwise found to index.
Step 25: if carrying out read operation, then index to find according to the pond in described tlv triple and deposit this record
The original position of storage pool, and according to the size of block in described storage pool and the index meter of the block in described tlv triple
Calculate block side-play amount in described storage pool；Described note is obtained according to described original position and described side-play amount
Record storage position in disk, reads the Value of described record from described storage position.
Step 26: if carrying out deletion action, then delete this tlv triple from internal memory, and according to described tlv triple
Pond index and block index the bitmap block of the block depositing described record is labeled as the free time.
Technical scheme by said method embodiment, it becomes possible to effectively manage mass data, without
Carry out disk fragments arrangement, make disk utilization to be greatly improved, and be greatly improved index and read K-V
Speed, thus provide efficient data storage and management operation.
Being described in detail technique scheme with concrete example below, this example presses disk space
Being logically divided into multiple storage pool, the block size in same pond is equal；Specifically disk can be drawn
Being divided into 2048 storage pools, in pond, the size of block is followed successively by 512B, 1K, 1.5K, 2K, 2.5K ... 1M；
It is illustrated in figure 3 the model schematic shown by method described in this example.
Depositing an entrance array of indexes in internal memory, the element of this entrance array of indexes can pass through a knot
Structure body struct idx_bpool realizes, and is used for representing that the information of some storage pool, this entrance index
The specifying information of the element of array is as shown in the table:
Below concrete to write, to read and deletion action is described, it is illustrated in figure 4 the flow process of write operation
Schematic diagram, in this example, length to the value of all operations rounds up the multiple into 512B, at Fig. 4
In:
When application program writes the record of key-value pair (key, a value) type,
Step 1: application program writes (key, a value) class record, proceeds by write operation；
Step 2: use standard library function to obtain the length of value, round up be 512 multiple, then
Obtain pond index according to the value obtained, i.e. find the storage pool depositing this record；
Such as, if maximum storage block is 1M, the record of a 1M300B to be deposited, then it can be stored in 2
In the 1M memory block of individual free time, i.e. first memory block deposits front 1M data, and second memory block is deposited remaining
300B data, and this memory block remainder space can not deposit other data again.
It addition, the most multiple storage pools store, then the pond of multiple storage pool is found to index.
Step 3: according to the bitmap of this storage pool, finds first idle block in pond (i.e. to find in bitmap
First is idle position), also obtain the block index that this record is to be deposited simultaneously.
Here, bitmap is a kind of data structure, is used for representing the service condition of block in storage pool, it is achieved form
Can be a string 0,1 sequence, such as, be 0 entirely when bitmap is initialized, represent that the state of this block is idle；
If certain block is taken up by data, just in bitmap relevant position, it is labeled as 1；If data are deleted in certain block,
It is labeled as 0 in bitmap relevant position.
Such as one bitmap 0,000 0000, it can represent the service condition of 8 blocks in storage pool；When in pond
First block and second block storage data after, bitmap relevant position is set to 1, and ascending the throne attempts to change is 1100
0000；When in pond the data of second block be deleted after, bitmap relevant position is set to 0, ascend the throne attempt to change into
1000 0000。
Step 4: calculate block side-play amount offset in this storage pool according to block index.
Wherein, in storage pool, size size of each block is stored in internal memory set entrance index number
Group, its form of expression is structure struct idx_bpool, therefrom can directly obtain；Calculating process is lifted
Under such as:
If block index is 3, then block side-play amount in storage pool is 3*size.
The position targetLocation=that this record is deposited in disk this storage pool original position+3*
size。
If it addition, there is the pond of multiple storage pool to index, the block index under the most each storage pool individually obtains,
And calculate the position deposited in disk respectively.
Step 5: proceed by input and output I/O operation, is stored directly in record in storage pool, stores position
It is set in above-mentioned steps 4 targetLocation calculated；
It addition, the most multiple storage pools store, then according to the storage position calculated respectively, will note
Record is respectively stored in above-mentioned position, the most above-mentioned record depositing a 1M300B, then can it be stored in
In 2 idle 1M memory blocks, i.e. first memory block deposits front 1M data, and second memory block is deposited surplus
Under 300B data.
Step 6: if this IO failure, the most directly return false；
Step 7: if this I/O operation success, then revise metadata.
The process of concrete modification metadata is: relevant position in bitmap is put 1, and i.e. this block of labelling is for using shape
State；The pond index of key and this record position of sensing, block indexes, as tlv triple (key, a pond
Index, block indexes) be stored in a file；This document mapped enter in internal memory, can be fast by it
The position of certain, the location record of speed, the conveniently operation such as deletion or lookup.
Step 8: return the result of this I/O operation, if writing successfully, returning true, otherwise returning
false；
Step 9: return the result of write operation, is write as merit and is returned true, otherwise false, and this write operation is tied
Bundle.
It is described above the flow process of write operation, introduces the flow process of read operation and deletion action the most again, read behaviour
Make and deletion action can occur after writing or before, this is not limited by the present invention, such as Fig. 5
It show the schematic flow sheet of read operation, when application program reads key-value pair (key, a value) type
During record, as shown in Figure 5:
Step 1: application program reads (key, a value) record, starts read operation；
Step 2: search this key in tlv triple file, without finding, then it represents that in data base
Without this record, returning a NULL, this read operation terminates；Otherwise find tlv triple (key, a pond
Index, block indexes)；
Step 3: find original position SET_POS depositing storage pool according to pond index；According to block index meter
Calculate block side-play amount offset in this storage pool.
During wherein in storage pool, size size of each block is stored in structure struct idx_bpool,
Can directly obtain；Calculating process is exemplified below:
If block index is 3, then block side-play amount in storage pool is 3*s ize.
The position that this record is deposited in disk is targetLocation=SET_POS+3*size.
Step 4: the storage position calculated according to step 3, i.e. targetLocation, directly from storage pool
In take out value；
Step 5: operate successfully if taking value, then return this record, otherwise return false；
Step 6: return the result of read operation, successfully return a record, otherwise return false；
Be illustrated in figure 6 the schematic flow sheet of deletion action, when application program delete key-value pair (key,
During record value), as shown in Figure 6:
Step 1: application program deletes (key, a value) record, starts deletion action；
Step 2: according to key, searches corresponding tlv triple inside tlv triple file, without finding,
Then returning false, deletion action terminates；Otherwise return this tlv triple (key, pond indexes, and block indexes),
And from file, delete this tlv triple.
Step 3: obtain corresponding structure struct from entrance array of indexes according to pond index
idx_bpool；
Step 4: index according to block, the relevant position inside bitmap is labeled as 0, represents that this block is idle,
Can be used to deposit new record；
Step 5: returning deletion action result, if successfully returning true, otherwise returning false；This is deleted
Division operation terminates.
By above-mentioned reading and writing and the process of deletion action, the data managing method described in the present embodiment
Can effectively manage mass data, without carrying out disk fragments arrangement, make the disk utilization can be big
Width improves, and is greatly improved index and reads the speed of K-V, thus provides the storage of efficient data and pipe
Reason operation.
The embodiment of the present invention additionally provides the device of a kind of data management, is illustrated in figure 7 the embodiment of the present invention
The structural representation of described device, described device includes:
Storage pool division unit 71, is arranged in disk, for disk being divided into multiple storage pool, and with
Block in one storage pool equal in magnitude；The process that implements is as described in above method embodiment.
Storage pool information arranges unit 72, is arranged in internal memory, for inlet porting array of indexes, by this
Entrance array of indexes represents the information of each storage pool；The process of implementing is shown in above method embodiment institute
State.
Data write operation unit 73, for when receiving writing of the record for key-value pair key-value type
During operation, entrance array of indexes location set in unit 72 is set by described storage pool information described
The data corresponding for value storage position in described disk, and carry out described number according to described storage position
According to write operation, the process that implements is as described in above method embodiment.
It addition, during implementing, described data write operation module 73 specifically also includes:
Pond index acquisition module 731, for writing the record of a key-value pair Key-Value type when application program
Time, the pond index in respective stored pond is obtained according to the length value of Value；
Block message acquisition module 732, for entering according to the respective stored pond arranged in the index acquisition internal memory of this pond
Mouth array of indexes, obtains the bitmap block information of this storage pool from this entrance array of indexes, finds in storage pool
Free block, and obtain this record to be deposited block index；
Storage position computation module 733, for rising according to size and the storage pool of block in this block index, storage pool
Beginning position calculates acquisition this record storage position in described disk；
Record memory module 734, is used for carrying out write operation, the record of this Key-Value type is stored in institute
State in storage position.
The embodiment of the present invention additionally provides the device of another kind of data management, is illustrated in figure 8 provided another
A kind of structural representation of data administrator, described device includes:
Storage pool division unit 81, is arranged in disk, for disk being divided into multiple storage pool, and with
Block size in one storage pool is equal.
Storage pool information arranges unit 82, is arranged in internal memory, for inlet porting array of indexes, by this
Entrance array of indexes represents the information of each storage pool.
Triple store unit 83, is arranged in internal memory, for storing the note of key-value pair key-value type
The tlv triple that record is corresponding, described tlv triple includes the Key of described record, and stores the storage of described record
The pond index in pond and block index.
Tlv triple acquisition module 84, for when application program is read or deletes described key-value pair Key-Value type
Record time, according to the Key of described record search in internal memory correspondence tlv triple.
Data reading operation module 85, for when carrying out read operation, indexes according to the pond in described tlv triple and looks for
To the original position of the storage pool depositing this record, and according to the size of block in described storage pool and described ternary
Block index in group calculates block side-play amount in described storage pool；According to described original position and described partially
Move and measure described record storage position in disk, read described record from described storage position
Value。
Data deletion operation module 86, for when carrying out deletion action, deletes the institute found from internal memory
State tlv triple, and the pond index and block according to described tlv triple indexes the bitmap block of the block depositing described record
It is labeled as the free time.
It should be noted that in said apparatus embodiment, included unit or module are according to merit
Logic can carry out dividing, but be not limited to above-mentioned division, as long as being capable of corresponding function；
It addition, the specific name of each functional unit is also only to facilitate mutually distinguish, it is not limited to the present invention
Protection domain.
It addition, one of ordinary skill in the art will appreciate that realize in above-described embodiment method all or part of
Step can be by program and completes to instruct relevant hardware, and corresponding program can be stored in a kind of calculating
In machine readable storage medium storing program for executing, storage medium mentioned above can be read only memory, disk or CD etc..
In sum, the embodiment of the present invention can effectively manage mass data, and without carrying out, disk is broken
Sheet arranges, and makes disk utilization to be greatly improved, and is greatly improved index and reads the speed of K-V,
Thus the storage of efficient data and management operation are provided.
The above, the only present invention preferably detailed description of the invention, but protection scope of the present invention not office
Be limited to this, any those familiar with the art in the technical scope that the embodiment of the present invention discloses,
The change that can readily occur in or replacement, all should contain within protection scope of the present invention.Therefore, the present invention
Protection domain should be as the criterion with scope of the claims.
Claims (14)
1. the method for a data management, it is characterised in that described method includes:
Block disk being divided in multiple storage pool, and same storage pool equal in magnitude；
Inlet porting array of indexes in internal memory, described entrance array of indexes is used for representing the letter of each storage pool
Breath；
When the write operation of the record received for key-value pair key-value type, by set in internal memory
Entrance array of indexes position data corresponding to described value storage position in described disk, and according to
Described storage position carries out the write operation of described data；
Wherein, the described data corresponding by the described value in entrance array of indexes location set in internal memory exist
Storage position in described disk, and the write operation of described data is carried out according to described storage position, specifically wrap
Include:
When the record of key-value pair Key-Value type is write disk, obtain phase according to the size of Value
The pond answering storage pool indexes；
Obtain the entrance array of indexes in the respective stored pond arranged in internal memory according to described pond index, from described enter
Mouth array of indexes obtains the bitmap block of described storage pool, determines for storing described note according to described bitmap block
The free block of record, obtains the block index of described free block；
Institute is calculated according to the size of block and the original position of storage pool in described piece of index, described storage pool
State record storage position in described disk；
Carry out write operation, described record is stored in described storage position.
Method the most according to claim 1, it is characterised in that described entrance array of indexes includes: deposit
The pond index of reservoir, the bitmap block of each block and the size of block in storage pool in storage pool.
Method the most according to claim 2, it is characterised in that described bitmap block is used for representing storage pool
In the service condition of each block.
Method the most according to claim 1, it is characterised in that obtain phase in the size according to Value
After answering the pond index of storage pool, described method also includes calculating the initial of storage pool corresponding to described pond index
Position.
Method the most according to claim 4, it is characterised in that record each storage pool in disk
Putting in order, described entrance array of indexes also includes the total length of storage pool,
The original position of the storage pool that described calculating described pond index is corresponding includes: will be arranged in described pond index
The total length sum of the corresponding all storage pools before storage pool adds 1 and is the storage that described pond index is corresponding
The original position in pond.
Method the most according to claim 1, it is characterised in that after data write operation success, described
Method also includes:
It is revised as taking by the bitmap block labelling depositing the block of described record；
Pond index and the block index composition tlv triple of by the key of described record, pointing to described record position leave in
In internal memory.
Method the most according to claim 6, it is characterised in that described method also includes carrying out data
Read operation, specifically includes:
When reading the record of key-value pair Key-Value type, look in internal memory according to the Key of described record
Look for the tlv triple of correspondence；
Pond index according to the described tlv triple found obtains the start bit of the storage pool depositing described record
Put, and calculate block deposit described according to the size of block in described storage pool and the index of the block in described tlv triple
Side-play amount in reservoir；
Described record storage position in disk is obtained, from institute according to described original position and described side-play amount
State storage position and read the Value of described record.
Method the most according to claim 6, it is characterised in that described method also includes carrying out data
Deletion action, specifically includes:
When deleting the record of key-value pair Key-Value type, look in internal memory according to the Key of described record
Look for the tlv triple of correspondence；
From internal memory, delete this tlv triple, and the pond index and block index according to described tlv triple will be deposited described
The bitmap block of the block of record is labeled as the free time.
9. according to one of them described method of claim 1-6, it is characterised in that to described key-value pair
Before value in Key-Value carries out write operation, the length of this value being rounded up is 512 words
The multiple of joint.
10. the method for a data management, it is characterised in that described method includes:
The block size being divided in multiple storage pool, and same storage pool by disk is equal；
Inlet porting array of indexes in internal memory, described entrance array of indexes is used for representing the letter of each storage pool
Breath；
In internal memory, storage has the tlv triple of the record correspondence of key-value pair key-value type, and described tlv triple includes
The Key of described record, and store pond index and the block index of the storage pool of described record；
When application program is read or deletes the record of described key-value pair Key-Value type, according to described record
Key search in internal memory correspondence tlv triple；
When carrying out read operation, then index according to the pond in described tlv triple and find the storage pool depositing this record
Original position, and calculate block according to the size of block in described storage pool and the index of the block in described tlv triple
Side-play amount in described storage pool；Described record is obtained at magnetic according to described original position and described side-play amount
Storage position in dish, reads the Value of described record from described storage position；
When carrying out deletion action, from internal memory, delete the described tlv triple found, and according to described ternary
The bitmap block of the block depositing described record is labeled as the free time by pond index and the block index of group.
11. methods as claimed in claim 10, it is characterised in that described entrance array of indexes includes: deposit
The pond index of reservoir, the bitmap block of each block and the size of block in storage pool in storage pool.
12. methods according to claim 10, it is characterised in that described bitmap block is used for representing storage
The service condition of each block in pond.
The device of 13. 1 kinds of data managements, it is characterised in that described device includes:
Storage pool division unit, is arranged in disk, for disk is divided into multiple storage pool and same
Block in storage pool equal in magnitude；
Storage pool information arranges unit, is arranged in internal memory, for inlet porting array of indexes, is entered by this
Mouth array of indexes represents the information of each storage pool；
Data write operation unit, for when the write operation of the record received for key-value pair key-value type
Time, entrance array of indexes set in unit is set by described storage pool information and positions described value pair
The data answered storage position in described disk, and write behaviour according to what described storage position carried out described data
Make；
Wherein, described data write operation unit includes:
Pond index acquisition module, is used for when the record of key-value pair Key-Value type is write disk, root
The pond index in respective stored pond is obtained according to the size of Value；
Block message acquisition module, for entering according to the respective stored pond arranged in the index acquisition internal memory of described pond
Mouth array of indexes, obtains the bitmap block of described storage pool, according to described piece of position from described entrance array of indexes
Figure determines the free block for storing described record, obtains the block index of described free block；
Storage position computation module, for according to the size of block in described piece of index, described storage pool and storage
The original position in pond calculates described record storage position in described disk；
Record memory module, is used for carrying out write operation, is stored in described storage position by described record.
The device of 14. 1 kinds of data managements, it is characterised in that described device includes:
Storage pool division unit, is arranged in disk, for disk is divided into multiple storage pool and same
Block size in storage pool is equal；
Storage pool information arranges unit, is arranged in internal memory, for inlet porting array of indexes, is entered by this
Mouth array of indexes represents the information of each storage pool；
Triple store unit, is arranged in internal memory, right for storing recording of key-value pair key-value type
The tlv triple answered, described tlv triple includes the Key of described record, and store the storage pool of described record
Pond index and block index；
Tlv triple acquisition module, for when application program is read or deletes described key-value pair Key-Value type
During record, in internal memory, search the tlv triple of correspondence according to the Key of described record；
Data reading operation module, for when carrying out read operation, indexes according to the pond in described tlv triple and finds
Deposit the original position of the storage pool of this record, and according to the size of block in described storage pool and described tlv triple
In block index calculate block side-play amount in described storage pool；According to described original position and described skew
Measure described record storage position in disk, read the Value of described record from described storage position；
Data deletion operation module, for when carrying out deletion action, deletes described in finding from internal memory
Tlv triple, and index the bitmap block mark of the block depositing described record with block according to the pond index of described tlv triple
It is designated as the free time.
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2011/080446 WO2013044502A1 (en) | 2011-09-30 | 2011-09-30 | Data management method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103140840A CN103140840A (en) | 2013-06-05 |
| CN103140840B true CN103140840B (en) | 2016-08-03 |
Family
ID=47994167
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201180001900.2A Expired - Fee Related CN103140840B (en) | 2011-09-30 | 2011-09-30 | The method and device of data management |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN103140840B (en) |
| WO (1) | WO2013044502A1 (en) |
Families Citing this family (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104182508B (en) * | 2014-08-19 | 2018-10-30 | 华为技术有限公司 | A kind of data processing method and data processing equipment |
| CN104360819B (en) * | 2014-09-29 | 2018-04-27 | 华为技术有限公司 | Method for writing data and device |
| CN106294603B (en) * | 2016-07-29 | 2019-07-23 | 北京奇虎科技有限公司 | File memory method and device |
| CN106708427B (en) * | 2016-11-17 | 2019-05-10 | 华中科技大学 | A kind of storage method suitable for key-value pair data |
| CN108614734B (en) * | 2016-12-13 | 2022-08-12 | 迈普通信技术股份有限公司 | Security parameter index management method and device |
| CN107783732A (en) * | 2017-10-30 | 2018-03-09 | 郑州云海信息技术有限公司 | A kind of data read-write method, system, equipment and computer-readable storage medium |
| CN110908996B (en) * | 2018-09-18 | 2024-07-16 | 北京京东尚科信息技术有限公司 | Data processing method and device |
| CN109933291B (en) * | 2019-03-20 | 2022-05-06 | 浪潮商用机器有限公司 | SRAM data processing method, device, equipment and storage medium |
| CN118132286A (en) * | 2020-08-05 | 2024-06-04 | 北京图森未来科技有限公司 | Method, device and system for communication between multiple processes |
| CN112214468B (en) * | 2020-10-18 | 2023-01-06 | 苏州浪潮智能科技有限公司 | Small file acceleration method, device, equipment and medium for distributed storage system |
| CN113961153B (en) * | 2021-12-21 | 2022-11-04 | 杭州趣链科技有限公司 | Method and device for writing index data into disk and terminal equipment |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1432922A (en) * | 2001-12-27 | 2003-07-30 | 日本电气株式会社 | Memory pool managing method and system for efficient use of memory |
| CN101997918A (en) * | 2010-11-11 | 2011-03-30 | 清华大学 | Method for allocating mass storage resources according to needs in heterogeneous SAN (Storage Area Network) environment |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6748401B2 (en) * | 2001-10-11 | 2004-06-08 | International Business Machines Corporation | Method and system for dynamically managing hash pool data structures |
-
2011
- 2011-09-30 WO PCT/CN2011/080446 patent/WO2013044502A1/en active Application Filing
- 2011-09-30 CN CN201180001900.2A patent/CN103140840B/en not_active Expired - Fee Related
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1432922A (en) * | 2001-12-27 | 2003-07-30 | 日本电气株式会社 | Memory pool managing method and system for efficient use of memory |
| CN101997918A (en) * | 2010-11-11 | 2011-03-30 | 清华大学 | Method for allocating mass storage resources according to needs in heterogeneous SAN (Storage Area Network) environment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103140840A (en) | 2013-06-05 |
| WO2013044502A1 (en) | 2013-04-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103140840B (en) | The method and device of data management | |
| CN103605805B (en) | A kind of storage method of magnanimity time series data | |
| CN102663086B (en) | Method for retrieving data block indexes | |
| CN102521406B (en) | Distributed query method and system for complex task of querying massive structured data | |
| CN102629269B (en) | Searching and storing method for embedded database | |
| CN102521405B (en) | Massive structured data storage and query methods and systems supporting high-speed loading | |
| CN107918612B (en) | The implementation method and device of key assignments memory system data structure | |
| CN103729303B (en) | A kind of data write of Flash and read method | |
| CN109471905B (en) | Block chain indexing method supporting time range and attribute range compound query | |
| CN102810092B (en) | data read-write method and system | |
| CN102831222A (en) | Differential compression method based on data de-duplication | |
| CN104424219B (en) | A kind of management method and device of data file | |
| CN103324763B (en) | Presenting method for tree-form data structure of mobile phone terminal | |
| CN103823865A (en) | Database primary memory indexing method | |
| CN103186617B (en) | A kind of method and apparatus storing data | |
| CN102779138B (en) | The hard disk access method of real time data | |
| CN103701469B (en) | A kind of compression and storage method of large-scale graph data | |
| CN103488710B (en) | The non-fixed-length data method of efficient storage in big data page | |
| CN106599040A (en) | Layered indexing method and search method for cloud storage | |
| CN103229164A (en) | Data access method and device | |
| CN104035956A (en) | Time-series data storage method based on distributive column storage | |
| CN102169491B (en) | Dynamic detection method for multi-data concentrated and repeated records | |
| CN103678158B (en) | A kind of data layout optimization method and system | |
| CN105159616A (en) | Disk space management method and device | |
| CN104050057B (en) | Historical sensed data duplicate removal fragment eliminating method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20170515 Address after: 510640 Guangdong City, Tianhe District Province, No. five, road, public education building, unit 371-1, unit 2401 Patentee after: Guangdong Gaohang Intellectual Property Operation Co., Ltd. Address before: 518129 Bantian HUAWEI headquarters office building, Longgang District, Guangdong, Shenzhen Patentee before: Huawei Technologies Co., Ltd. |
|
| TR01 | Transfer of patent right | ||
| CB03 | Change of inventor or designer information | ||
| CB03 | Change of inventor or designer information |
Inventor after: Li Shaohua Inventor before: Wang Daohui |
|
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20170911 Address after: 054000 No. 185, Jia Jia Cun, Da Tun Tun, Renxian County, Hebei, Xingtai Patentee after: Li Shaohua Address before: 510640 Guangdong City, Tianhe District Province, No. five, road, public education building, unit 371-1, unit 2401 Patentee before: Guangdong Gaohang Intellectual Property Operation Co., Ltd. |
|
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20160803 Termination date: 20170930 |