CN102567434A - Data block processing method - Google Patents
Data block processing method Download PDFInfo
- Publication number
- CN102567434A CN102567434A CN2011100363181A CN201110036318A CN102567434A CN 102567434 A CN102567434 A CN 102567434A CN 2011100363181 A CN2011100363181 A CN 2011100363181A CN 201110036318 A CN201110036318 A CN 201110036318A CN 102567434 A CN102567434 A CN 102567434A
- Authority
- CN
- China
- Prior art keywords
- data block
- data
- state
- information
- index
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data block processing method, which includes: creating a data block in internal storage; distributing unique data block identification information of the data block, marking the data block state of the data block as the internal storage stage and updating a second index; receiving to-be-written data items by the data block and updating a first index according to the key words and the storage position of the data block; marking the state of the data block as the written-in state, updating the second index, and writing the data block into a solid-state storage unit; marking the state of the data block as the solid-state storage state after the data block is written into the solid-state storage unit, releasing internal buffering space occupied by the data block and updating the second index to record file descriptors of the solid-state storage unit. By the data block processing method, the data block comprising a plurality of data items is managed by matching of different state marks and internal indexes, high-performance read-write operation can be compatible, and the requirements of high-performance random inquiry and updating of data are met.
Description
Technical field
The present invention relates to technical field of data processing, particularly a kind of data block disposal route.
Background technology
High speed development along with the internet; People's information source has obtained abundant greatly, and the information obtain manner also changes thereupon, also brings challenges when this brings opportunity to the mankind; Become in Web information under the prerequisite of how much radix growths; How can carry out data search fast and accurately, to search requirement, be one of direction of technical field of data processing research.
In data search; Search engine spider obtains increasing application, and spider is an auto-programming of search engine, and its effect is the webpage that grasps on the internet; Set up index data base, make the user can in search engine, search the webpage of related web site.
In the practical implementation process; Spider will grasp a large amount of web site urls every day; Before grasping, all need obtain information such as the IP address of waiting to grasp the website and robots, these information can not be real-time from interconnected online enquiries, and can only be through inner domain name server (DNS) inquiry.
But because data quantity stored is in continuous increase among the DNS, so inquiry velocity also can be slack-off thereupon, can not satisfy the demand of fast query.And, when the data among the DNS are upgraded, also can increase the workload of DNS, this has also influenced the speed of inquiry.Equally, also can run into similar problem in web database and other key word-data values (Key-value) database in real time.
How can better data being inquired about and upgrade, satisfy high performance read-write service, is one of direction of technical field of data processing research.
Summary of the invention
Technical matters to be solved by this invention provides a kind of data block disposal route, to support the high-performance read-write operation, satisfy to the high-performance random challenge of data with upgrade demand.
The technical scheme that the present invention adopts for the technical solution problem provides a kind of data block disposal route, comprising: a. creates data block in internal memory, stores first index and second index in the wherein said internal memory; B. be the unique data block identifying information of said data block allocations; And be internal storage state with the data block status indication of said data block; Upgrade said second index; To write down the data block information of said data block, said data block information comprises said data block identifying information, said data block state, data block length information and internal memory pointer; C. utilize said data block to receive data item to be written; Said first index is upgraded in key word and memory location according to said data item, to write down the mapping relations of said key word and said data block identifying information, data item offset information and data item length information; D. said data block write full after, be write state with the data block status indication of said data block, upgrade said second index, writing down the said data block information of said data block, and said data block is written in the solid-state memory; E. after said data block is written to said solid-state memory; With the data block status indication of said data block is the solid-state storage state; Discharge the memory buffer space that said data block takies; And upgrade said second index, to write down the said data block information of said data block, said data block information further comprises the solid-state memory filec descriptor.
The preferred embodiment one of according to the present invention; In said step c; Upgrade said second index, to write down the said data block information of said data block, said data block information further comprises effective data item number, the effective bitmap of data item, internal storage data piece capacity.
The preferred embodiment one of according to the present invention when said data item is carried out tag delete, carries out mark to the corresponding position in the effective bitmap of said data item in said second index.
The preferred embodiment one of according to the present invention in said steps d, judges through the comparative result of said internal storage data piece capacity and said data block length information whether said data block is write full.
The preferred embodiment one of according to the present invention; In said step c; When the valid data item number in the said data block is lower than threshold value; With the data block status indication of said data block is the reconstruction state, subsequently the valid data item in the said data block is dumped in the new data block that is labeled as internal storage state, and after unloading is accomplished, is the deletion state with the data block status indication of said data block.
The preferred embodiment one of according to the present invention; In said steps d; When the valid data item number in the said data block is lower than threshold value; With the data block status indication of said data block is the reconstruction state, subsequently the valid data item in the said data block is dumped in the new data block that is labeled as internal storage state, and after unloading is accomplished, is the deletion state with the data block status indication of said data block.
The preferred embodiment one of according to the present invention; In said step e; When the valid data item number in the said data block is lower than threshold value; With the data block status indication of said data block is reading state, said data block is read in the said internal memory, and after reading completion, be the reconstruction state with the data block status indication of said data block; Subsequently the valid data item in the said data block is dumped in the new data block that is labeled as internal storage state, and after unloading is accomplished, be the deletion state the data block status indication of said data block.
The preferred embodiment one of according to the present invention, said disposal route further comprises: the f. judge mark is whether the said data block of deletion state is cited, and does not then delete said data block if be cited.
The preferred embodiment one of according to the present invention in said step e, is written to said data block in the said solid-state memory with controlled rates.
Can find out by above technical scheme; Data block disposal route provided by the invention utilizes the different state mark to cooperate the internal memory index that the data block that comprises a plurality of data item is managed; Can support the high-performance read-write operation, satisfy the high-performance random challenge of data and the demand of renewal.
Description of drawings
Fig. 1 is the schematic flow sheet of the database control method in the embodiment of the invention;
Fig. 2 is the storage medium of the database in the embodiment of the invention and the synoptic diagram of memory contents;
Fig. 3 is first index and the data structure synoptic diagram of data block in the embodiment of the invention;
Fig. 4 is the second indexed data structural representation in the embodiment of the invention;
Fig. 5 is the structural representation of the Hash container in the embodiment of the invention;
Fig. 6 is the synoptic diagram of the data block life cycle management process in the embodiment of the invention;
Fig. 7 is the schematic flow sheet of the database read extract operation in the embodiment of the invention;
Fig. 8 is the schematic flow sheet that the database in the embodiment of the invention writes operation;
Fig. 9 is the idiographic flow synoptic diagram of the step S809 among Fig. 8;
Figure 10 is the schematic flow sheet that the database of the embodiment of the invention is derived operation;
Figure 11 is the structural representation of the database control device in the embodiment of the invention.
Embodiment
Below in conjunction with accompanying drawing and embodiment the present invention is elaborated.
See also Fig. 1, Fig. 1 is the schematic flow sheet of the database control method in the embodiment of the invention.In the present embodiment, the control method of database mainly comprises following step:
Step S101 receives operation information.
Step S102 is according to the index information of operation information at the index inquiry corresponding data piece that is arranged in internal memory.
Step S103 carries out corresponding operating according to operation information and index information to the corresponding data piece.
In the present invention, operation information can comprise the key word of concrete operations instruction and pending data item or pending data item.The concrete operations instruction can comprise read operation instruction, write operation instruction, derive operational order etc., and the specific operation process that various operation informations are corresponding will be described below.
See also Fig. 2, Fig. 2 is the storage medium of the database in the embodiment of the invention and the synoptic diagram of memory contents.In the present embodiment, the storage medium of database comprises internal memory and solid-state memory.Memory contents comprises index and data block.Wherein, data block is arranged in internal memory and solid-state memory according to the different conditions selectivity.Specifically, when data block is arranged in internal memory, can be described as the internal storage data piece, when data block is arranged in solid-state memory, can be described as the solid-state memory data block.Wherein, index is arranged in the internal memory, and index comprises first index and second index.
See also Fig. 3-4, Fig. 3 is first index and the data structure synoptic diagram of data block in the embodiment of the invention.Fig. 4 is the second indexed data structural representation in the embodiment of the invention.
See also Fig. 3, in the present embodiment, first index is used for the mapping relations of the positional information of recording key and corresponding data item.Positional information mainly comprises following information: data block identifying information, data item offset information and data item length information.
Wherein, the data block identifying information is used to write down the ID of the affiliated data block of corresponding data item, and the data item offset information is used for writing down the side-play amount of corresponding data item in data block, and the data item length information is used to write down the length of corresponding data item.
See also Fig. 3, the data block identifying information of second index completion from positional information is to the mapping of data block information.See also Fig. 4; In the present embodiment, data block information mainly comprises: data block identifying information, data block length information, data item sum, valid data item number, the effective bitmap of data item, data block state, internal storage data piece capacity, internal memory pointer, solid-state memory filec descriptor and reference count.
Wherein, the data block identifying information is used for the ID of recording data blocks, in the present embodiment, and unique ID of each data block allocations.Data block length information is used for the size of recording data blocks.The data item sum is used for total number of the data item in the recording data blocks.Valid data item number is used for the number of the valid data item in the recording data blocks, and promptly the data item sum deducts the number of the data item that is labeled deletion.The effective bitmap of data item is used for the effective status of record data items, and wherein, each (bit) represents a data item, if put 1 expression effectively, puts 0 expression and is labeled deletion.The data block state is used for the state of recording data blocks, is mainly used in to judge that the corresponding data piece is arranged in internal memory or is arranged in solid-state memory, hereinafter will describe various data block states in detail.Internal storage data piece capacity is used for writing down the internal storage data piece size of stored data items, is mainly used in to cooperate with data block length information to judge whether the internal storage data piece is full.The internal memory pointer is used for the memory location of recording data blocks at internal memory, and when data block was not arranged in internal memory, it was invalid then should to be worth.The solid-state memory filec descriptor is used for the memory location of recording data blocks at solid-state memory, and when data block was not arranged in solid-state memory, it was invalid then should to be worth.Reference count is used for the state of quoting of recording data blocks, is used for the life cycle of management data block.
In above-mentioned information, data block identifying information, data block length information, data item sum, valid data item number, the effective bitmap of data item are the core data item of data block, are called the metamessage of data block.
Please continue to consult Fig. 3, a plurality of data item of storage in data block, each data item stored has following content: data item sequence number, key length, data value length, key word and data value.
Wherein, the data item sequence number is the sequence number of data item in data block, is used to search the effective bitmap of data item.Key length is used for the length of recording key.Data value length is used for the length of record data value.Key word is the binary string of expression key word.Data value is the binary string of presentation data value.
In the present invention, first index and second index can be through the various algorithms realizations in this area, for example hash algorithms.
One embodiment of the present invention provides a kind of Hash container of indirect addressing, has saved the space of index, has improved the service efficiency of internal memory.To be example with first index below, be described in detail.
See also Fig. 5, in the present embodiment, first index comprises a hash table, and this hash table comprises a plurality of hash.Each hash stored one Hash node pointer.In the present embodiment, the Hash node pointer is predetermined bite (for example, 4 bytes); Wherein preceding pre-determined bit (for example; Preceding 9) be used to identify the Hash node data piece that the Hash node belongs to, back pre-determined bit (for example, back 23) is used to identify the Hash node in the inner skew of Hash node data piece.
Specifically, the identifying information of all Hash node data pieces is recorded in the list of identification information, and can inquire the identifying information of correspondence according to the preceding pre-determined bit of Hash node pointer from the correspondence position of list of identification information.In the present embodiment, the maximum quantity of the identifying information that can store of list of identification information is 2
9=512.
In addition, the Hash node is stored in the corresponding Hash node data piece, and each Hash node (for example takies predetermined bite; 20 bytes), (for example comprise key word (for example, 8 bytes), data block identifying information respectively; 2 bytes), the data item length information (for example; 2 bytes), data item offset information (for example, 4 bytes) and next Hash node pointer (for example, 4 bytes).
In the use of above-mentioned Hash container, at first the key word that receives is carried out Hash operation and from the hash table, confirm corresponding hash, and from hash, obtain the Hash node pointer.Subsequently; Utilize the identifying information of preceding pre-determined bit Hash node data piece of definite correspondence from list of identification information of Hash node pointer; And from the corresponding Hash node data piece of identifying information, obtain corresponding Hash node as side-play amount according to the back pre-determined bit of Hash node pointer; And then obtain the data item positional information relevant, for example data block identifying information, data item offset information and data item length information with this key word.
The maximum quantity of the Hash node that can store in each Hash node data piece in the present embodiment, is 2
23=8388608, so the maximum amount of data that each Hash container can be supported is 512 * 8,388,608,=42 hundred million, well satisfied name server (Domain Name Server, demand DNS).
In addition, in this Hash container, idle Hash node is used idle linked list maintenance.When data were deleted, corresponding Hash node also can be recovered.Idle Hash node conspires to create one through next the Hash node pointer in the Hash node and reclaims chained list.When receiving new data, the preferential pointer that reclaims in the chained list that uses.Therefore, the Hash node data in the Hash node data piece is always compact, in website quantity is under 300,000,000 the situation, to save as 300M * 20bytes=6Gbytes in taking.
See also Fig. 6, Fig. 6 is the synoptic diagram of the data block life cycle management process in the embodiment of the invention.
In the present embodiment, at first, in internal memory, create data block.After creating data block, be the unique data block identifying information (ID) of this data block allocations, and be " internal memory " the data block status indication of this data block.Subsequently, upgrade second index, to write down the data block information of this data block, for example data block identifying information, data block length information, data block state and internal memory pointer etc.Wherein, database has only a data block to be in " internal memory " state at any one time at most.
The data block that is labeled as " internal memory " can receive data item to be written, and this data item is appended to the data block end.Subsequently, upgrade second index, to write down effective data item number, the effective bitmap of data item, internal storage data piece capacity etc.Simultaneously; First index is upgraded in key word and memory location according to data item to be written; The mapping relations of this key word of record and data block identifying information, data item offset information and data item length information in first index can be according to this keyword query to corresponding data item so that follow-up.In case behind the data item writing data blocks, just cannot change, only allow this data item is read and tag delete.When this data item is carried out tag delete, in second index, mark is carried out in the corresponding position in the effective bitmap of data item of the data block information of this data block, for example with correspondence position 0.
Behind the continuous writing data blocks of data item, can judge whether this data block is write full through the comparative result of internal storage data piece capacity and data block length information.If it is full that data block is write, be " writing " with the data block status indication of this data block, and upgrade second index, total with record data items.Subsequently, this data block is written in the solid-state memory.Preferably, in ablation process, the data item of this data block in the internal memory (for example, 5MB/S) is written in the solid-state memory, has prevented effectively reading performance is caused excessive influence with controlled rates.Simultaneously, in internal memory, create the new data block of data block status indication again, to receive the follow-up data item that writes for " internal memory ".
After the data item of this data block of " writing " state all is written to solid-state memory, is " solid-state storage " with the data block status indication of this data block, and discharges the memory buffer space that this data block originally took.Subsequently, upgrade second index, to write down the data block information of this data block, solid-state memory filec descriptor for example.
If the cavity that is in the data block of " internal memory " state and " writing " state is too many, that is to say that the number of the valid data item in the data block is lower than threshold value, then the data block status indication with this data block is " reconstruction ".Subsequently, the interior valid data item of data block that will be labeled as " reconstruction " dumps in the new data block that is labeled as " internal memory ", to realize removing and the reconstruction to the invalid data items in the data block of " reconstruction ".After rebuilding completion, the data block status indication that will be labeled as the data block of " reconstruction " is " deletion ".At this moment, judged whether that according to the reference count of this data block other threads quote this data block,, then deleted this data block if there are not other threads to quote this data block.If there are other threads to quote this data block, then keep this data block, after other threads are used to complete, remove operation by reference and discharge this reference count, again this data block is deleted.
If the cavity that is in the data block of " solid-state storage " state is too many, then the data block status indication with this data block is " reading ", and this data block is read in the internal memory.After reading completion, be " reconstruction " with the data block status indication of this data block, and the valid data item of this data block is dumped in the data block that is labeled as " internal memory ", to realize removing and reconstruction to the interior invalid data items of the data block of " reconstructions ".After rebuilding completion, the data block status indication that will be labeled as the data block of " reconstruction " is " deletion ".At this moment, judged whether that according to the reference count of this data block other threads quote this data block,, then deleted this data block if there are not other threads to quote this data block.
Specifically, if the data block state is " internal memory ", " writing " and " reconstruction ", represent that then this data block is arranged in internal memory.If the data block state is " solid-state storage " and " reading ", represent that then this data block is arranged in solid-state memory.See through aforesaid operations, effectively management data block life cycle.
To combine specific embodiment below, describe various operating process of the present invention.
See also Fig. 7, Fig. 7 is the schematic flow sheet of the database read extract operation in the embodiment of the invention.
In step S701, receive the key word of read operation instruction and data item to be read.
In step S702, utilize this key word from the first search index corresponding position information.If do not inquire, then carry out step S708; If inquire, then carry out step S703.
In the present embodiment, positional information comprises data block identifying information, data item offset information and data item length information.Concrete query script is described in detail hereinbefore, repeats no more at this.
In step S703, utilize data block identifying information in this positional information from the corresponding data block information of second search index.In the present embodiment, data block information comprises data block state, internal memory pointer and solid-state memory filec descriptor etc.
In step S704, be arranged in internal memory or be positioned at solid-state memory according to data block condition judgement corresponding data piece, if the corresponding data piece is arranged in internal memory, then carry out step S705, if the corresponding data piece is arranged in solid-state memory, then carry out step S706.
Described like preceding text, if the data block state is " internal memory ", " writing " and " reconstruction ", represent that then this data block is arranged in internal memory.If the data block state is " solid-state storage " and " reading ", represent that then this data block is arranged in solid-state memory.
In step S705, in internal memory, read the corresponding data item according to internal memory pointer, data item offset information and data item length information.
In step S706, in solid-state memory, read the corresponding data item according to solid-state memory filec descriptor, data item offset information and data item length information.
In step S707, judge whether the key word of corresponding data item is consistent with the key word of data item to be read, if inconsistent, then carry out step S708; If consistent, then carry out step S709.
In step S708, judge that data item to be read does not exist.
In step S709, judge and to read success, and with the data value of the corresponding data item data value as data item to be read.
See also Fig. 8, Fig. 8 writes the schematic flow sheet of operation for the database of the embodiment of the invention.
In step S801, receive the key word and the data value of write operation instruction and data item to be written.
In step S802, utilize this key word from the first search index corresponding position information.If inquire, then carry out step S803; If do not inquire, then carry out step S809.
In the present embodiment, positional information comprises data block identifying information, data item offset information and data item length information.Concrete query script is described in detail hereinbefore, repeats no more at this.
In step S803, utilize data block identifying information in this positional information from the corresponding data block information of second search index.In the present embodiment, data block information comprises data block state, internal memory pointer and solid-state memory filec descriptor etc.
In step S804, be arranged in internal memory according to data block condition judgement corresponding data piece and still be arranged in solid-state memory, if the corresponding data piece is arranged in internal memory, then carry out step S805, if the corresponding data piece is arranged in solid-state memory, then carry out step S806.
Described like preceding text, if the data block state is " internal memory ", " writing " and " reconstruction ", represent that then this data block is arranged in internal memory.If the data block state is " solid-state storage " and " reading ", represent that then this data block is arranged in solid-state memory.
In step S805, in internal memory, read the corresponding data item according to internal memory pointer, data item offset information and data item length information.
In step S806, in solid-state memory, read the corresponding data item according to solid-state memory filec descriptor, data item offset information and data item length information;
In step S807, judge whether the data value of corresponding data item is consistent with the data value of data item to be written:, then write success if consistent; If inconsistent, then carry out step S808;
In step S808, under the corresponding data item, in the effective bitmap of the data item of data block this corresponding data item sign is deleted.Specifically, in the effective bitmap of data item with correspondence position 0.
In step S809, data item to be written is write the data block that is used to receive data item to be written that is arranged in internal memory, promptly be in the data block of " internal memory " state.Subsequently, upgrade first index and second index to write down above-mentioned ablation process.
See also Fig. 9, Fig. 9 is the idiographic flow synoptic diagram of the step S809 among Fig. 8.
In step S901, whether the data block of judging being in the internal memory " internal memory " state has been write full, if write fullly, then carries out step S902, if write fullly, then carries out step S903.Specifically, can confirm whether this data block is write full through the comparative result between data block length information and the internal storage data piece capacity.
In step S902, directly data item to be written is written to the end of data block, and upgrades first index and second index.
In step S903, be " writing " state with the data block status indication of this data block.Subsequently, get into step S904 and step S905.
In step S904, this data block is write solid-state memory, further upgrade first index and second index simultaneously.After writing completion, change the data block state of this data block into " solid-state storage " state, in internal memory, discharge this data block simultaneously.
In step S905, create the new data block of " internal memory " state that is labeled as, and data item to be written is written to the end of new data block, and upgrade first index and second index.
By the way, step S904 and step S905 can walk abreast and carry out, and have realized that thus read-write separates.In addition, the data item that is in the data block of " writing " state writes solid-state memory with controlled rates, such as controlled rates is 5MB/S.The present invention is through being provided with controlled rates, the influence that when having prevented effectively to write data reading performance has been caused.
See also Figure 10, Figure 10 is the schematic flow sheet that the database of the embodiment of the invention is derived operation.
In step S1001, receiving derives operational order.
In step S1002, internal memory and solid-state memory are locked, wherein, under locking state, forbid internal memory and solid-state memory are made amendment.
In step S1003, from second index of internal memory, duplicate the corresponding metamessage of data to be exported piece.In the present embodiment, the data to be exported piece can be the data block of all data blocks or predetermined quantity.As indicated above, metamessage comprises data block identifying information, data block length information, data item sum, valid data item number and the effective bitmap of data item.
In step S1004, the invoking marks operation is carried out in the reference count of data to be exported piece, deleted to avoid the data to be exported piece.For example, reference count is added one or add particular step size.
In step S1005, internal memory and solid-state memory are carried out release, wherein, under released state, allow internal memory and solid-state memory are made amendment.Because the doubling time of metamessage is very short, the follow-up derivation of data block can be carried out on the backstage, has avoided influencing reading and write operation of data block.
In step S1006, metamessage is write meta-information file.
In step S1007, read the data to be exported piece according to metamessage from internal memory or solid-state memory, and generate the derivation index according to the key word of data to be exported piece.
In step S1008, will derive index and write the derivation index file.
In step S1009, with data to be exported piece writing data blocks file.In the present embodiment, meta-information file, derivation index file and data block file can be created when receiving the derivation operational order, also can be in other suitably establishments constantly arbitrarily.
In step S1010, the releasing operation is quoted in the reference count of data to be exported piece.For example, reference count is subtracted one or subtract particular step size.At this moment,, and do not have other threads to quote this data block, then delete this data block if data block is in " deletion " state.
See also Figure 11, Figure 11 is the structural representation of the database control device of the embodiment of the invention.This control device comprises operation information receiver module 1101, index information enquiry module 1102 and data block processing module 1103.
Wherein, operation information receiver module 1101 is used to receive aforesaid operations information.
Index information enquiry module 1102 is used for according to the index information of aforesaid operations information at the index inquiry corresponding data piece that is positioned at internal memory.In the present embodiment, the corresponding data piece comprises a plurality of data item, and each data item comprises key word and data value, and corresponding data piece alternative is arranged in internal memory and solid-state memory.
Data block processing module 1103 is used for according to aforesaid operations information and index information the corresponding data piece being carried out corresponding operating.
In the practical implementation process, index comprises the first above-mentioned index and second index.Index information enquiry module 1102 is according to first search index and key word corresponding position information, and positional information comprises data block identifying information, data item offset information and data item length information.The data block information that index information enquiry module 1102 is further corresponding with the data block identifying information according to second search index, data block information comprises data block state, internal memory pointer and solid-state memory filec descriptor etc.
See also the description of preceding text about the detailed operation process of database control device, repeat no more here.
Can find out by above technical scheme; The control method of database of the present invention and device are stored a plurality of data item with the data block form; And data block selection property is stored in internal memory and the solid-state storage according to different conditions; Further cooperate the internal memory index, can support the high-performance read-write operation, satisfied to the high-performance random challenge of data with upgrade demand.
In the above-described embodiments, only the present invention has been carried out exemplary description, but those skilled in the art can carry out various modifications to the present invention after reading present patent application under the situation that does not break away from the spirit and scope of the present invention.
Claims (9)
1. a data block disposal route is characterized in that, the treating method comprises:
A. in internal memory, create data block, store first index and second index in the wherein said internal memory;
B. be the unique data block identifying information of said data block allocations; And be internal storage state with the data block status indication of said data block; Upgrade said second index; To write down the data block information of said data block, said data block information comprises said data block identifying information, said data block state, data block length information and internal memory pointer;
C. utilize said data block to receive data item to be written; Said first index is upgraded in key word and memory location according to said data item, to write down the mapping relations of said key word and said data block identifying information, data item offset information and data item length information;
D. said data block write full after, be write state with the data block status indication of said data block, upgrade said second index, writing down the said data block information of said data block, and said data block is written in the solid-state memory;
E. after said data block is written to said solid-state memory; With the data block status indication of said data block is the solid-state storage state; Discharge the memory buffer space that said data block takies; And upgrade said second index, to write down the said data block information of said data block, said data block information further comprises the solid-state memory filec descriptor.
2. the method for claim 1; It is characterized in that, in said step c, upgrade said second index; To write down the said data block information of said data block, said data block information further comprises effective data item number, the effective bitmap of data item, internal storage data piece capacity.
3. method as claimed in claim 2 is characterized in that, when said data item is carried out tag delete, in said second index, mark is carried out in the corresponding position in the effective bitmap of said data item.
4. method as claimed in claim 2 is characterized in that, in said steps d, judges through the comparative result of said internal storage data piece capacity and said data block length information whether said data block is write full.
5. method as claimed in claim 2; It is characterized in that; In said step c, when the valid data item number in the said data block is lower than threshold value, be the reconstruction state with the data block status indication of said data block; Subsequently the valid data item in the said data block is dumped in the new data block that is labeled as internal storage state, and after unloading is accomplished, be the deletion state the data block status indication of said data block.
6. method as claimed in claim 2; It is characterized in that; In said steps d, when the valid data item number in the said data block is lower than threshold value, be the reconstruction state with the data block status indication of said data block; Subsequently the valid data item in the said data block is dumped in the new data block that is labeled as internal storage state, and after unloading is accomplished, be the deletion state the data block status indication of said data block.
7. method as claimed in claim 2; It is characterized in that; In said step e, when the valid data item number in the said data block is lower than threshold value, be reading state with the data block status indication of said data block; Said data block is read in the said internal memory; And after reading completion, be the reconstruction state with the data block status indication of said data block, subsequently the valid data item in the said data block is dumped in the new data block that is labeled as internal storage state, and after unloading is accomplished, be the deletion state the data block status indication of said data block.
8. like any described method of claim 5-7, it is characterized in that said disposal route further comprises:
F. judge mark is whether the said data block of deletion state is cited, and does not then delete said data block if be cited.
9. the method for claim 1 is characterized in that, in said step e, said data block is written in the said solid-state memory with controlled rates.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110036318 CN102567434B (en) | 2010-12-31 | 2010-12-31 | Data block processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110036318 CN102567434B (en) | 2010-12-31 | 2010-12-31 | Data block processing method |
Related Parent Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201010619673 Division CN102567415B (en) | 2010-12-31 | 2010-12-31 | Control method and device of database |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102567434A true CN102567434A (en) | 2012-07-11 |
| CN102567434B CN102567434B (en) | 2013-04-10 |
Family
ID=46412863
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201110036318 Active CN102567434B (en) | 2010-12-31 | 2010-12-31 | Data block processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102567434B (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103403709A (en) * | 2012-11-15 | 2013-11-20 | 华为技术有限公司 | Method, device and system for data reading and writing |
| CN103810246A (en) * | 2013-12-27 | 2014-05-21 | 北京天融信软件有限公司 | Index building method and device and index query method and device |
| CN103853766A (en) * | 2012-12-03 | 2014-06-11 | 中国科学院计算技术研究所 | Online processing method and system oriented to streamed data |
| CN105408883A (en) * | 2013-04-30 | 2016-03-16 | 惠普发展公司，有限责任合伙企业 | Database table column annotation |
| CN105426408A (en) * | 2015-11-02 | 2016-03-23 | 北京锐安科技有限公司 | Multi-index data processing method and apparatus |
| CN102968498B (en) * | 2012-12-05 | 2016-08-10 | 华为技术有限公司 | Data processing method and device |
| CN106453674A (en) * | 2016-09-09 | 2017-02-22 | 中国互联网络信息中心 | DNS multistage domain name query method |
| CN106484770A (en) * | 2016-09-09 | 2017-03-08 | 中国互联网络信息中心 | A kind of processing method of DNS incremental area data file |
| CN106528650A (en) * | 2016-10-14 | 2017-03-22 | 努比亚技术有限公司 | Resource query method and terminal |
| CN108628969A (en) * | 2018-04-24 | 2018-10-09 | 咪咕文化科技有限公司 | Spatial keyword indexing method and platform and storage medium |
| CN108829740A (en) * | 2018-05-23 | 2018-11-16 | 摇了购（武汉）电子商务有限公司 | Date storage method and device |
| CN109885558A (en) * | 2019-01-21 | 2019-06-14 | 南京国通智能科技有限公司 | A kind of data mode configuration method, system and storage medium |
| CN110059087A (en) * | 2019-03-26 | 2019-07-26 | 阿里巴巴集团控股有限公司 | Data attribute identification method, device and equipment in a kind of piece of chain type account book |
| CN111124313A (en) * | 2019-12-24 | 2020-05-08 | 宁波三星医疗电气股份有限公司 | Data reading and writing method and device for power acquisition terminal and electronic equipment |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH02287875A (en) * | 1989-04-28 | 1990-11-27 | Fuji Xerox Co Ltd | Filing system in computer system |
| JPH08235040A (en) * | 1995-02-27 | 1996-09-13 | Hitachi Ltd | Data file management system |
| CN1270363A (en) * | 1999-04-09 | 2000-10-18 | 英业达股份有限公司 | Database management method |
| CN1538324A (en) * | 2003-04-16 | 2004-10-20 | ��Ϊ��������˾ | Method of raising officiency of data processing |
| CN1645365A (en) * | 2004-12-20 | 2005-07-27 | 浪潮电子信息产业股份有限公司 | Method for decreasing DBMS field storing redundance |
| CN1908952A (en) * | 2005-07-21 | 2007-02-07 | 富士施乐株式会社 | Information processing server, information processing system, information processing method and storage medium storing information processing program |
-
2010
- 2010-12-31 CN CN 201110036318 patent/CN102567434B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH02287875A (en) * | 1989-04-28 | 1990-11-27 | Fuji Xerox Co Ltd | Filing system in computer system |
| JPH08235040A (en) * | 1995-02-27 | 1996-09-13 | Hitachi Ltd | Data file management system |
| CN1270363A (en) * | 1999-04-09 | 2000-10-18 | 英业达股份有限公司 | Database management method |
| CN1538324A (en) * | 2003-04-16 | 2004-10-20 | ��Ϊ��������˾ | Method of raising officiency of data processing |
| CN1645365A (en) * | 2004-12-20 | 2005-07-27 | 浪潮电子信息产业股份有限公司 | Method for decreasing DBMS field storing redundance |
| CN1908952A (en) * | 2005-07-21 | 2007-02-07 | 富士施乐株式会社 | Information processing server, information processing system, information processing method and storage medium storing information processing program |
Cited By (26)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103403709A (en) * | 2012-11-15 | 2013-11-20 | 华为技术有限公司 | Method, device and system for data reading and writing |
| CN103403709B (en) * | 2012-11-15 | 2016-10-05 | 华为技术有限公司 | A kind of methods, devices and systems of reading and writing data |
| CN103853766A (en) * | 2012-12-03 | 2014-06-11 | 中国科学院计算技术研究所 | Online processing method and system oriented to streamed data |
| CN103853766B (en) * | 2012-12-03 | 2017-04-05 | 中国科学院计算技术研究所 | A kind of on-line processing method and system towards stream data |
| CN102968498B (en) * | 2012-12-05 | 2016-08-10 | 华为技术有限公司 | Data processing method and device |
| CN105408883B (en) * | 2013-04-30 | 2018-09-11 | 安提特软件有限责任公司 | Database table column is explained |
| CN105408883A (en) * | 2013-04-30 | 2016-03-16 | 惠普发展公司，有限责任合伙企业 | Database table column annotation |
| US10248620B2 (en) | 2013-04-30 | 2019-04-02 | Entit Software Llc | Database table column annotation |
| CN103810246B (en) * | 2013-12-27 | 2017-10-13 | 北京天融信软件有限公司 | A kind of index creation method and apparatus and search index method and apparatus |
| CN103810246A (en) * | 2013-12-27 | 2014-05-21 | 北京天融信软件有限公司 | Index building method and device and index query method and device |
| CN105426408B (en) * | 2015-11-02 | 2019-03-08 | 北京锐安科技有限公司 | A kind of data processing method and device of more indexes |
| CN105426408A (en) * | 2015-11-02 | 2016-03-23 | 北京锐安科技有限公司 | Multi-index data processing method and apparatus |
| CN106484770B (en) * | 2016-09-09 | 2019-08-06 | 中国互联网络信息中心 | A kind of processing method of DNS incremental area data file |
| CN106453674A (en) * | 2016-09-09 | 2017-02-22 | 中国互联网络信息中心 | DNS multistage domain name query method |
| CN106453674B (en) * | 2016-09-09 | 2019-05-17 | 中国互联网络信息中心 | A kind of DNS multistage domain name inquiry method |
| CN106484770A (en) * | 2016-09-09 | 2017-03-08 | 中国互联网络信息中心 | A kind of processing method of DNS incremental area data file |
| CN106528650B (en) * | 2016-10-14 | 2019-06-21 | 努比亚技术有限公司 | A kind of resource query method and terminal |
| CN106528650A (en) * | 2016-10-14 | 2017-03-22 | 努比亚技术有限公司 | Resource query method and terminal |
| CN108628969A (en) * | 2018-04-24 | 2018-10-09 | 咪咕文化科技有限公司 | Spatial keyword indexing method and platform and storage medium |
| CN108628969B (en) * | 2018-04-24 | 2022-05-06 | 咪咕文化科技有限公司 | Spatial keyword indexing method and platform and storage medium |
| CN108829740A (en) * | 2018-05-23 | 2018-11-16 | 摇了购（武汉）电子商务有限公司 | Date storage method and device |
| CN108829740B (en) * | 2018-05-23 | 2021-11-02 | 摇了购（武汉）电子商务有限公司 | Data storage method and device |
| CN109885558A (en) * | 2019-01-21 | 2019-06-14 | 南京国通智能科技有限公司 | A kind of data mode configuration method, system and storage medium |
| CN109885558B (en) * | 2019-01-21 | 2024-06-04 | 南京国通智能科技有限公司 | Data state configuration method, system and storage medium |
| CN110059087A (en) * | 2019-03-26 | 2019-07-26 | 阿里巴巴集团控股有限公司 | Data attribute identification method, device and equipment in a kind of piece of chain type account book |
| CN111124313A (en) * | 2019-12-24 | 2020-05-08 | 宁波三星医疗电气股份有限公司 | Data reading and writing method and device for power acquisition terminal and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102567434B (en) | 2013-04-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102567434B (en) | Data block processing method | |
| CN102541968A (en) | Indexing method | |
| CN102567415B (en) | Control method and device of database | |
| US9858303B2 (en) | In-memory latch-free index structure | |
| JP5996088B2 (en) | Cryptographic hash database | |
| CN104246764B (en) | The method and apparatus for placing record in non-homogeneous access memory using non-homogeneous hash function | |
| CN103080910B (en) | Storage system | |
| JP6362316B2 (en) | Method, system and computer program product for hybrid table implementation using buffer pool as resident in-memory storage for memory resident data | |
| CN110119425A (en) | Solid state drive, distributed data-storage system and the method using key assignments storage | |
| CN103164490B (en) | A kind of efficient storage implementation method of not fixed-length data and device | |
| CN105474196A (en) | System for managing integrated data object and method therefor | |
| CN106663047A (en) | Systems and methods for oprtimized signature comparisons and data replication | |
| CN104424219B (en) | A kind of management method and device of data file | |
| JP2017504924A (en) | Content-based organization of the file system | |
| CN102292720A (en) | Method and apparatus for managing data objects of a data storage system | |
| CN103186617B (en) | A kind of method and apparatus storing data | |
| CN101419571A (en) | Method for storing configuration parameter in NOR FLASH based on Hash arithmetic | |
| CN103577513A (en) | Systems and/or methods for caching xml information sets with delayed node instantiation | |
| WO2015093026A1 (en) | Write information storage device, method, and recording medium | |
| CN103631937A (en) | Method, device and system for establishing column storage indexes | |
| CN103617097A (en) | File recovery method and file recovery device | |
| CN109407985B (en) | Data management method and related device | |
| CN112131140A (en) | SSD-based key value separation storage method supporting efficient storage space management | |
| CN110352410A (en) | Track the access module and preextraction index node of index node | |
| US8239427B2 (en) | Disk layout method for object-based storage devices |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |