Summary of the invention
It is an object of the invention to provide the method and device of a kind of data management, it is possible to effective management magnanimity number
According to, without carrying out disk fragments arrangement, make disk utilization to be greatly improved, and be greatly improved
Index and the speed of reading K-V, thus the storage of efficient data and management operation are provided.
A kind of method of data management, described method includes:
Block disk being divided in multiple storage pool, and same storage pool equal in magnitude;
Inlet porting array of indexes in internal memory, described entrance array of indexes is used for representing the letter of each storage pool
Breath;
When the write operation of the record received for key-value pair key-value type, by set in internal memory
The entrance array of indexes put positions the data corresponding for described value storage position in described disk, and root
The write operation of described data is carried out according to described storage position.
A kind of method of data management, described method includes:
The block size being divided in multiple storage pool, and same storage pool by disk is equal;
Inlet porting array of indexes in internal memory, described entrance array of indexes is used for representing the letter of each storage pool
Breath;
In internal memory, storage has the tlv triple of the record correspondence of key-value pair key-value type, described tlv triple bag
Include the Key of described record, and store pond index and the block index of the storage pool of described record;
When application program is read or deletes the record of described key-value pair Key-Value type, according to described record
Key search in internal memory correspondence tlv triple;
When carrying out read operation, then index according to the pond in described tlv triple and find the storage pool depositing this record
Original position, and calculate block according to the size of block in described storage pool and the index of the block in described tlv triple
Side-play amount in described storage pool;Described record is obtained at magnetic according to described original position and described side-play amount
Storage position in dish, reads the Value of described record from described storage position;
When carrying out deletion action, from internal memory, delete the described tlv triple found, and according to described ternary
The bitmap block of the block depositing described record is labeled as the free time by pond index and the block index of group.
A kind of device of data management, described device includes:
Storage pool division unit, is arranged in disk, for disk is divided into multiple storage pool and same
Block in storage pool equal in magnitude;
Storage pool information arranges unit, is arranged in internal memory, for inlet porting array of indexes, is entered by this
Mouth array of indexes represents the information of each storage pool;
Data write operation unit, for writing behaviour when receive record for key-value pair key-value type
When making, entrance array of indexes set in unit is set by described storage pool information and positions described value
Corresponding data storage position in described disk, and carry out writing of described data according to described storage position
Operation.
A kind of device of data management, described device includes:
Storage pool division unit, is arranged in disk, for disk is divided into multiple storage pool and same
Block size in storage pool is equal;
Storage pool information arranges unit, is arranged in internal memory, for inlet porting array of indexes, is entered by this
Mouth array of indexes represents the information of each storage pool;
Triple store unit, is arranged in internal memory, for storing the record of key-value pair key-value type
Corresponding tlv triple, described tlv triple includes the Key of described record, and stores the storage pool of described record
Pond index and block index;
Tlv triple acquisition module, for when application program is read or deletes described key-value pair Key-Value type
During record, in internal memory, search the tlv triple of correspondence according to the Key of described record;
Data reading operation module, for when carrying out read operation, indexes according to the pond in described tlv triple and finds
Deposit the original position of the storage pool of this record, and according to the size of block in described storage pool and described tlv triple
In block index calculate block side-play amount in described storage pool;According to described original position and described skew
Measure described record storage position in disk, read the Value of described record from described storage position;
Data deletion operation module, for when carrying out deletion action, deletes described in finding from internal memory
Tlv triple, and index the bitmap block mark of the block depositing described record with block according to the pond index of described tlv triple
It is designated as the free time.
By above-mentioned provided technical scheme it can be seen that described method is multiple by being logically divided into by disk
Block size in storage pool, and same storage pool is equal;Inlet porting array of indexes in internal memory, by this
Entrance array of indexes represents the information of each storage pool;By the Key in key-value pair Key-Value, described
The pond index of the storage pool marked off in disk, and the block index composition tlv triple of block in storage pool, deposit
It is placed in internal memory;Certain is positioned by entrance array of indexes set in internal memory and the triplet information deposited
One data record storage position in described disk, and the reading and writing of these data are carried out according to this storage position
Or deletion action.Mass data just can be effectively managed, without carrying out disk by this technical scheme
Defragmentation, makes disk utilization to be greatly improved, and is greatly improved index and reads the speed of K-V,
Thus the storage of efficient data and management operation are provided.
Detailed description of the invention
Embodiment of the present invention provides the method and device of a kind of data management, it is possible to effectively manage magnanimity
Data, without carrying out disk fragments arrangement, make disk utilization to be greatly improved, and greatly carry
High index and the speed of reading K-V, thus the storage of efficient data and management operation are provided.
Below according to accompanying drawing, the embodiment of the present invention is described in detail, is illustrated in figure 1 the embodiment of the present invention
The schematic flow sheet of described method, described method includes:
Step 11: the block size being divided in multiple storage pool, and same storage pool by disk is equal.
In this step, when carrying out data management, first by disk by being logically divided into multiple storage pool,
And block size in same storage pool is equal, such as, disk can be divided into 2048 storage pools, Chi Zhong
The size of block is followed successively by 512B, 1K, 1.5K, 2K, 2.5K ... 1M, say, that 2048 marked off
Individual storage pool has the block pond of 512B, has the block pond of 1K, has the block pond of 1.5K, by that analogy, the most permissible
Multiple for 512B.
Step 12: inlet porting array of indexes in internal memory, described entrance array of indexes is used for representing that each is deposited
The information of reservoir.
In this step, after the division according to above-mentioned steps 11, inlet porting array of indexes in internal memory,
Represented that by this entrance array of indexes the information of each storage pool, described entrance array of indexes include: storage
The pond index in pond, block big in the bitmap block of each block, the total length of storage pool and storage pool in storage pool
Little.Wherein, the total length of storage pool is optional.
The pond index of above-mentioned storage pool points to each storage pool, and above-mentioned bitmap block is used for representing in storage pool each
The service condition of block, especially by idle to described bitmap block mark or take and realize, such as, can pass through
0,1 sequence identifies, and is designated 0 expression idle, if being designated when certain block is taken up by data during initialization
1。
Step 13: when the write operation of the record received for key-value pair key-value type, by interior
Deposit set entrance array of indexes and position the data corresponding for described value storage position in described disk
Put, and carry out the write operation of described data according to described storage position.
In this step, after the operation of above-mentioned steps, it is possible to by entrance set in internal memory
Array of indexes positions a certain data record storage position in described disk, and enters according to this storage position
The write operation of these data of row.
Here, data can identify with the record of a key-value pair Key-Value type, a note
The Key of record is unique in whole data base, can find this record according to this Key, namely should
Data.Such as key-value pair (key1, " hello, world "), by searching key1 in data base,
Just can find Value: " hello, world ".
Specifically, when application program writes the record of a key-value pair Key-Value type, according to Value
Size obtain respective stored pond pond index (for example: write a key-value pair Key-Value type
Record, the size of its value is 1000B, if disk file is divided into 512B, 1024B, 1.5K......
1M, totally 2048 storage pools, then the size of Value rounds up (multiples of the 512 of >=value), 1000B
Round up just for 1024B, then this record should be stored in the storage pool that block size is 1024B,
Thus further determine that the pond index that the storage pool of this 1024B is corresponding).
Then, the entrance array of indexes in the respective stored pond arranged in internal memory is obtained according to this pond index, from this
Entrance array of indexes obtains the bitmap block information of this storage pool, finds the free block in storage pool, and obtain
The block index that this record is to be deposited;According to this block index, the size of block and storage pool original position in storage pool
Calculate acquisition this record storage position in described disk, such as, if depositing in entrance array of indexes
The original position of the storage pool this record is 200000, obtains bitmap block information according to block index, finds the
One idle block (if being the 10th, this block index is 10), then deposit the storage position of this record
TargetPos=200000+10*1024=210240, i.e. represents the storage in disk of this record
Position, then carry out data write operation, this record is stored in described storage position.
It addition, above-mentioned after the size according to Value obtains the pond index in respective stored pond, also include meter
Calculate the process of the original position of storage pool corresponding to described pond index, record each storage pool in disk
Putting in order, described entrance array of indexes also includes the total length of storage pool, calculates described pond index correspondence
The original position of storage pool specifically includes: all before the storage pool corresponding by being arranged in described pond index are deposited
The total length sum of reservoir adds 1 and is the original position that described pond indexes the storage pool of correspondence.
Above-mentioned storage pool original position refers to each storage pool logical place in disk file, in system
Starting the original position of first storage pool when initializing is 0, and the original position of second storage pool is first
Individual storage pool is taken up space+1 in disk file, records each storage pool putting in order in disk, so
Rear the like the original position obtaining all storage pools, will be arranged in the storage pool that described pond index is corresponding
The total length sum of all storage pools before adds 1 and is the start bit that described pond indexes the storage pool of correspondence
Put.On the other hand, when entrance array of indexes does not include the total length of storage pool, it is also possible to pass through storage pool
In the quantity of block and the size of block be calculated total size of storage pool.
It addition, after data write operation success, described method also includes: the block position deposited by described record
The labelling of figure is revised as taking;By the key of described record, the pond index and the block that point to this record position index
Leave in internal memory as a tlv triple, follow-up can quickly position certain note by this tlv triple to facilitate
The position of record, deletes or search operation accordingly.
It addition, method described in the embodiment of the present invention can also carry out read operation, when application program reads a key assignments
During to the record of Key-Value type, in internal memory, search tlv triple file according to the Key of this record, as
Fruit is not found, then it represents that without this record in data base, read operation terminates;Otherwise find a tlv triple literary composition
Part key, pond index and block index;The original position of the storage pool depositing this record is found according to this pond index,
And calculate block side-play amount in this storage pool according to block index;Side-play amount according to being calculated is somebody's turn to do
Record storage position in disk, and from this storage position, take out the Value of this record.
It addition, method described in the embodiment of the present invention can also carry out deletion action, when application program deletes one
During the record of key-value pair Key-Value type, in internal memory, search tlv triple file according to the Key of this record,
Without finding, then deletion action terminates;Otherwise return a tlv triple file key, pond index and block
Index;From internal memory, delete this tlv triple file, and the bitmap block labelling of this record will be deposited according to block index
For the free time.
The embodiment of the present invention additionally provides another kind of data managing method, is illustrated in figure 2 the embodiment of the present invention
Another schematic flow sheet of described data managing method, Fig. 2 includes:
Step 21: the block size being divided in multiple storage pool, and same storage pool by disk is equal.
By this entrance array of indexes, step 22: inlet porting array of indexes in internal memory, represents that each is deposited
The information of reservoir.
Step 23: in internal memory, storage has the tlv triple of the record correspondence of key-value pair key-value type, described
Tlv triple includes the Key of described record, and stores pond index and the block index of the storage pool of described record.
Step 24: when application program is read or deletes the record of a key-value pair Key-Value type, searches
And obtain tlv triple.
In this step, when application program is read or deletes the record of a key-value pair Key-Value type,
Key according to this record searches tlv triple in internal memory, without finding, then it represents that without this in data base
Record, reads or deletion action terminates;Tlv triple key, pond index and block is otherwise found to index.
Step 25: if carrying out read operation, then index to find according to the pond in described tlv triple and deposit this record
The original position of storage pool, and according to the size of block in described storage pool and the index meter of the block in described tlv triple
Calculate block side-play amount in described storage pool;Described note is obtained according to described original position and described side-play amount
Record storage position in disk, reads the Value of described record from described storage position.
Step 26: if carrying out deletion action, then delete this tlv triple from internal memory, and according to described tlv triple
Pond index and block index the bitmap block of the block depositing described record is labeled as the free time.
Technical scheme by said method embodiment, it becomes possible to effectively manage mass data, without
Carry out disk fragments arrangement, make disk utilization to be greatly improved, and be greatly improved index and read K-V
Speed, thus provide efficient data storage and management operation.
Being described in detail technique scheme with concrete example below, this example presses disk space
Being logically divided into multiple storage pool, the block size in same pond is equal;Specifically disk can be drawn
Being divided into 2048 storage pools, in pond, the size of block is followed successively by 512B, 1K, 1.5K, 2K, 2.5K ... 1M;
It is illustrated in figure 3 the model schematic shown by method described in this example.
Depositing an entrance array of indexes in internal memory, the element of this entrance array of indexes can pass through a knot
Structure body struct idx_bpool realizes, and is used for representing that the information of some storage pool, this entrance index
The specifying information of the element of array is as shown in the table:
Below concrete to write, to read and deletion action is described, it is illustrated in figure 4 the flow process of write operation
Schematic diagram, in this example, length to the value of all operations rounds up the multiple into 512B, at Fig. 4
In:
When application program writes the record of key-value pair (key, a value) type,
Step 1: application program writes (key, a value) class record, proceeds by write operation;
Step 2: use standard library function to obtain the length of value, round up be 512 multiple, then
Obtain pond index according to the value obtained, i.e. find the storage pool depositing this record;
Such as, if maximum storage block is 1M, the record of a 1M300B to be deposited, then it can be stored in 2
In the 1M memory block of individual free time, i.e. first memory block deposits front 1M data, and second memory block is deposited remaining
300B data, and this memory block remainder space can not deposit other data again.
It addition, the most multiple storage pools store, then the pond of multiple storage pool is found to index.
Step 3: according to the bitmap of this storage pool, finds first idle block in pond (i.e. to find in bitmap
First is idle position), also obtain the block index that this record is to be deposited simultaneously.
Here, bitmap is a kind of data structure, is used for representing the service condition of block in storage pool, it is achieved form
Can be a string 0,1 sequence, such as, be 0 entirely when bitmap is initialized, represent that the state of this block is idle;
If certain block is taken up by data, just in bitmap relevant position, it is labeled as 1;If data are deleted in certain block,
It is labeled as 0 in bitmap relevant position.
Such as one bitmap 0,000 0000, it can represent the service condition of 8 blocks in storage pool;When in pond
First block and second block storage data after, bitmap relevant position is set to 1, and ascending the throne attempts to change is 1100
0000;When in pond the data of second block be deleted after, bitmap relevant position is set to 0, ascend the throne attempt to change into
1000 0000。
Step 4: calculate block side-play amount offset in this storage pool according to block index.
Wherein, in storage pool, size size of each block is stored in internal memory set entrance index number
Group, its form of expression is structure struct idx_bpool, therefrom can directly obtain;Calculating process is lifted
Under such as:
If block index is 3, then block side-play amount in storage pool is 3*size.
The position targetLocation=that this record is deposited in disk this storage pool original position+3*
size。
If it addition, there is the pond of multiple storage pool to index, the block index under the most each storage pool individually obtains,
And calculate the position deposited in disk respectively.
Step 5: proceed by input and output I/O operation, is stored directly in record in storage pool, stores position
It is set in above-mentioned steps 4 targetLocation calculated;
It addition, the most multiple storage pools store, then according to the storage position calculated respectively, will note
Record is respectively stored in above-mentioned position, the most above-mentioned record depositing a 1M300B, then can it be stored in
In 2 idle 1M memory blocks, i.e. first memory block deposits front 1M data, and second memory block is deposited surplus
Under 300B data.
Step 6: if this IO failure, the most directly return false;
Step 7: if this I/O operation success, then revise metadata.
The process of concrete modification metadata is: relevant position in bitmap is put 1, and i.e. this block of labelling is for using shape
State;The pond index of key and this record position of sensing, block indexes, as tlv triple (key, a pond
Index, block indexes) be stored in a file;This document mapped enter in internal memory, can be fast by it
The position of certain, the location record of speed, the conveniently operation such as deletion or lookup.
Step 8: return the result of this I/O operation, if writing successfully, returning true, otherwise returning
false;
Step 9: return the result of write operation, is write as merit and is returned true, otherwise false, and this write operation is tied
Bundle.
It is described above the flow process of write operation, introduces the flow process of read operation and deletion action the most again, read behaviour
Make and deletion action can occur after writing or before, this is not limited by the present invention, such as Fig. 5
It show the schematic flow sheet of read operation, when application program reads key-value pair (key, a value) type
During record, as shown in Figure 5:
Step 1: application program reads (key, a value) record, starts read operation;
Step 2: search this key in tlv triple file, without finding, then it represents that in data base
Without this record, returning a NULL, this read operation terminates;Otherwise find tlv triple (key, a pond
Index, block indexes);
Step 3: find original position SET_POS depositing storage pool according to pond index;According to block index meter
Calculate block side-play amount offset in this storage pool.
During wherein in storage pool, size size of each block is stored in structure struct idx_bpool,
Can directly obtain;Calculating process is exemplified below:
If block index is 3, then block side-play amount in storage pool is 3*s ize.
The position that this record is deposited in disk is targetLocation=SET_POS+3*size.
Step 4: the storage position calculated according to step 3, i.e. targetLocation, directly from storage pool
In take out value;
Step 5: operate successfully if taking value, then return this record, otherwise return false;
Step 6: return the result of read operation, successfully return a record, otherwise return false;
Be illustrated in figure 6 the schematic flow sheet of deletion action, when application program delete key-value pair (key,
During record value), as shown in Figure 6:
Step 1: application program deletes (key, a value) record, starts deletion action;
Step 2: according to key, searches corresponding tlv triple inside tlv triple file, without finding,
Then returning false, deletion action terminates;Otherwise return this tlv triple (key, pond indexes, and block indexes),
And from file, delete this tlv triple.
Step 3: obtain corresponding structure struct from entrance array of indexes according to pond index
idx_bpool;
Step 4: index according to block, the relevant position inside bitmap is labeled as 0, represents that this block is idle,
Can be used to deposit new record;
Step 5: returning deletion action result, if successfully returning true, otherwise returning false;This is deleted
Division operation terminates.
By above-mentioned reading and writing and the process of deletion action, the data managing method described in the present embodiment
Can effectively manage mass data, without carrying out disk fragments arrangement, make the disk utilization can be big
Width improves, and is greatly improved index and reads the speed of K-V, thus provides the storage of efficient data and pipe
Reason operation.
The embodiment of the present invention additionally provides the device of a kind of data management, is illustrated in figure 7 the embodiment of the present invention
The structural representation of described device, described device includes:
Storage pool division unit 71, is arranged in disk, for disk being divided into multiple storage pool, and with
Block in one storage pool equal in magnitude;The process that implements is as described in above method embodiment.
Storage pool information arranges unit 72, is arranged in internal memory, for inlet porting array of indexes, by this
Entrance array of indexes represents the information of each storage pool;The process of implementing is shown in above method embodiment institute
State.
Data write operation unit 73, for when receiving writing of the record for key-value pair key-value type
During operation, entrance array of indexes location set in unit 72 is set by described storage pool information described
The data corresponding for value storage position in described disk, and carry out described number according to described storage position
According to write operation, the process that implements is as described in above method embodiment.
It addition, during implementing, described data write operation module 73 specifically also includes:
Pond index acquisition module 731, for writing the record of a key-value pair Key-Value type when application program
Time, the pond index in respective stored pond is obtained according to the length value of Value;
Block message acquisition module 732, for entering according to the respective stored pond arranged in the index acquisition internal memory of this pond
Mouth array of indexes, obtains the bitmap block information of this storage pool from this entrance array of indexes, finds in storage pool
Free block, and obtain this record to be deposited block index;
Storage position computation module 733, for rising according to size and the storage pool of block in this block index, storage pool
Beginning position calculates acquisition this record storage position in described disk;
Record memory module 734, is used for carrying out write operation, the record of this Key-Value type is stored in institute
State in storage position.
The embodiment of the present invention additionally provides the device of another kind of data management, is illustrated in figure 8 provided another
A kind of structural representation of data administrator, described device includes:
Storage pool division unit 81, is arranged in disk, for disk being divided into multiple storage pool, and with
Block size in one storage pool is equal.
Storage pool information arranges unit 82, is arranged in internal memory, for inlet porting array of indexes, by this
Entrance array of indexes represents the information of each storage pool.
Triple store unit 83, is arranged in internal memory, for storing the note of key-value pair key-value type
The tlv triple that record is corresponding, described tlv triple includes the Key of described record, and stores the storage of described record
The pond index in pond and block index.
Tlv triple acquisition module 84, for when application program is read or deletes described key-value pair Key-Value type
Record time, according to the Key of described record search in internal memory correspondence tlv triple.
Data reading operation module 85, for when carrying out read operation, indexes according to the pond in described tlv triple and looks for
To the original position of the storage pool depositing this record, and according to the size of block in described storage pool and described ternary
Block index in group calculates block side-play amount in described storage pool;According to described original position and described partially
Move and measure described record storage position in disk, read described record from described storage position
Value。
Data deletion operation module 86, for when carrying out deletion action, deletes the institute found from internal memory
State tlv triple, and the pond index and block according to described tlv triple indexes the bitmap block of the block depositing described record
It is labeled as the free time.
It should be noted that in said apparatus embodiment, included unit or module are according to merit
Logic can carry out dividing, but be not limited to above-mentioned division, as long as being capable of corresponding function;
It addition, the specific name of each functional unit is also only to facilitate mutually distinguish, it is not limited to the present invention
Protection domain.
It addition, one of ordinary skill in the art will appreciate that realize in above-described embodiment method all or part of
Step can be by program and completes to instruct relevant hardware, and corresponding program can be stored in a kind of calculating
In machine readable storage medium storing program for executing, storage medium mentioned above can be read only memory, disk or CD etc..
In sum, the embodiment of the present invention can effectively manage mass data, and without carrying out, disk is broken
Sheet arranges, and makes disk utilization to be greatly improved, and is greatly improved index and reads the speed of K-V,
Thus the storage of efficient data and management operation are provided.
The above, the only present invention preferably detailed description of the invention, but protection scope of the present invention not office
Be limited to this, any those familiar with the art in the technical scope that the embodiment of the present invention discloses,
The change that can readily occur in or replacement, all should contain within protection scope of the present invention.Therefore, the present invention
Protection domain should be as the criterion with scope of the claims.