CN114996275A - Key value storage method based on multi-tree conversion mechanism - Google Patents
Key value storage method based on multi-tree conversion mechanism Download PDFInfo
- Publication number
- CN114996275A CN114996275A CN202210711424.3A CN202210711424A CN114996275A CN 114996275 A CN114996275 A CN 114996275A CN 202210711424 A CN202210711424 A CN 202210711424A CN 114996275 A CN114996275 A CN 114996275A
- Authority
- CN
- China
- Prior art keywords
- tree
- key value
- value data
- data
- key
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 27
- 230000007246 mechanism Effects 0.000 title claims abstract description 19
- 238000006243 chemical reaction Methods 0.000 title claims abstract description 13
- 238000005192 partition Methods 0.000 claims abstract description 42
- 230000002085 persistent effect Effects 0.000 claims abstract description 11
- 230000002688 persistence Effects 0.000 claims abstract description 10
- 230000009466 transformation Effects 0.000 claims description 8
- 238000003780 insertion Methods 0.000 claims description 3
- 230000037431 insertion Effects 0.000 claims description 3
- 230000003321 amplification Effects 0.000 abstract description 11
- 238000003199 nucleic acid amplification method Methods 0.000 abstract description 11
- 230000006872 improvement Effects 0.000 abstract description 4
- 239000010410 layer Substances 0.000 description 55
- 239000011229 interlayer Substances 0.000 description 5
- 238000013461 design Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 2
- VYZAMTAEIAYCRO-UHFFFAOYSA-N Chromium Chemical compound [Cr] VYZAMTAEIAYCRO-UHFFFAOYSA-N 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000002356 single layer Substances 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2246—Trees, e.g. B+trees
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a key value storage method based on a multi-tree conversion mechanism, which specifically comprises the following steps: for the written key value data, firstly storing the key value data into a write skip list, and converting the key value data into a read-only skip list to be inserted into a B + tree in the memory device after the size of the key value data reaches the limit; when the size of the B + tree reaches a certain limit, traversing the key value data according to a heat strategy, and persisting the key value data with low heat to a cold tree 0 layer in the external storage device; if the number of the key value data files in the cold tree layer 0 reaches the size limit, triggering the partition operation of the layer 0; when the key value data in the B + tree execute persistence operation, the 0-layer partition only receives the key value data which accords with a set range; if the key value data in the specific range in the cold tree reaches a certain heat degree, transferring the key value data to a hot tree of the external storage device; while key-value data of low heat in the hot tree will be transferred to the cold tree. The method provided by the invention reduces the read amplification by using the heat strategy, ensures the write-in performance and realizes the integral improvement of the performance of the key value storage system.
Description
Technical Field
The invention relates to the field of computer storage, in particular to a key value storage method based on a multi-tree conversion mechanism.
Background
With the advent of the big data era, the data storage volume is continuously increased, the data characteristics are obviously changed, and the ratio of the unstructured data to the unstructured data reaches 80% of the total data. Under the trend of data change, key value storage is widely applied, and has no clear limitation on a data structure and high expandability. Some excellent key-value storage products are currently available and widely used, such as Leveldb used by Chrome browser, facebook further proposing improved Rocksdb based on Leveldb, Redis and Memcached used by Twitter, and the like. The data structures used by the current key value storage mainly include a hash structure, a B tree and a log structure merged tree (LSM tree), and the three structures respectively have different advantages and disadvantages.
Since LSM tree based key-value storage systems have good write performance and support range scanning, a number of related products have emerged. However, due to the inherent structural features of the LSM tree structure (hierarchy in external memory), maintenance and use of this structure can negatively impact key-value store performance. Taking Leveldb as an example, firstly, the source of write amplification of Leveldb is mainly data rewriting in the interlayer merging operation. For example, when the size of each layer in the Leveldb is 10 times that of the previous layer, and a file in the i layer is merged into the i +1 layer, in the worst case, 10 files in the i +1 layer need to be read, and the data are written back to the i +1 layer after merging and sorting, and the write amplification factor is as high as 10, in this case, an SST file is moved from the L0 layer to the L6 layer, and the write amplification factor is up to 50. Secondly, the source of read amplification of Leveldb is mainly two-fold. On one hand, when searching for a key value pair, the level db may need to search in multiple hierarchies, and in the worst case, 8 files need to be searched in L0, and one file needs to be searched in each layer of the L1-L6 layers, and a total of 14 files need to be searched; on the other hand, when a key value pair is searched in an SST file, the Leveldb needs to read a plurality of metadata blocks of the file, and the actually read data comprises an index block, a bloom filter block and a data block. For example, when looking for a 1KB key-value pair, Leveldb needs to read a 16KB index block, a 4KB filter block, and a 4KB data block, for a total of 24KB of data. In the worst case, 14 SST files need to be read, so the read amplification can be as high as 336.
Disclosure of Invention
The main purpose of the present invention is to overcome the above-mentioned defects in the prior art, and to provide a key-value storage method based on a multi-tree conversion mechanism, which uses a hot policy to reduce read amplification, and at the same time, ensures write performance, and realizes overall improvement of the performance of the key-value storage system.
The invention adopts the following technical scheme:
a key value storage method based on a multi-tree conversion mechanism comprises the following steps:
for written key value data, firstly storing the key value data into a write jump table, and converting the key value data into a read-only jump table to be inserted into a B + tree in a memory device after the size of the key value data reaches the limit; traversing the key value data when the size of the B + tree reaches a certain limit;
judging according to the heat strategy, persisting the key value data with low heat to the 0 layer of the cold tree in the external storage device, and continuously storing the key value data with high heat in the B + tree;
if the number of the key value data files in the layer 0 of the cold tree reaches the size limit, triggering the partition operation of the layer 0, equally dividing the partition operation of the layer 0 according to the existing key value data range in the layer, and if the number of the key value data files in a certain partition in the layer 0 reaches the size limit, triggering the partition operation again;
when key value data with lower heat in the B + tree execute persistence operation, the 0-layer partition only receives the key value data which accords with the set range; if the key value data in the range in the cold tree reach a certain hot degree after being judged by the hot degree strategy, transferring the key value data in the range to a hot tree of the external memory device, and using a B + tree in the internal memory device to index the key value data in the hot tree;
when the size of the hot tree reaches a certain limit, the key value data is judged according to the heat strategy, the key value data in one or more low-heat ranges are transferred to the cold tree partition, and the indexes of the key value data are deleted from the B + tree.
Specifically, the heat strategy is specifically as follows:
when the key value data in the read-only skip list is inserted into the B + tree structure, an insertion time field is stored and used for judging the heat condition of the key value data;
judging whether the key value data needs to be reserved in a B + tree or not according to the time field recorded in the node, and if the key value data does not need to be reserved, persisting the key value data in a cold tree partition of the external storage device;
for key value data in the external storage device, a global heat table is used for recording operands of the key value data, and when the merging operation in the cold tree is executed, if the operand of a range unit positioned in the cold tree reaches a threshold value, all the key value data in the range unit are transferred to the hot tree;
when the hot tree reaches a certain size limit, a global heat table is used for carrying out heat judgment on key value data in the hot tree, whether the key value data in the range need to be reserved in the hot tree is judged according to the range heat condition of the key value data, and if the key value data in the range do not need to be reserved, the key value data in the hot tree are transferred to the cold tree.
Specifically, the method further comprises a data reading step, specifically:
s11: firstly, searching target data in a cache region, and if the target data is found, going to step S12; if not, go to step S13;
s12: the system finishes the reading request of the user and sends the read target data to the user;
s13: searching the target key and the value type in the B + tree, if the target key and the value type are found, the step S12 is executed, and if the target key and the value type are found, the step S14 is executed; if not, go to step S15;
s14: reading the target data in the hot tree of the external memory device according to the address obtained in the previous step, and going to step S12;
s15: searching target data in a cold tree of the external storage device from top to bottom according to layers, and if the target data is found, going to step S12; if not, go to step S16;
s16: the system completes the reading request of the user, and sends the result to the user without finding the target data.
Specifically, the method further comprises a data writing step, specifically:
s21: firstly, writing the data into a jump table structure, and if the size limit is reached, going to step S22;
s22: converting the skip list into a read-only skip list, and creating a new write skip list structure for data writing;
s23: if the size of the B + tree does not reach the limit, go to step S25; if the limit is reached, go to step S24;
s24: traversing the key value data in the B + tree, and persisting the key value data with lower heat into the cold tree of the external storage device to the step S25;
s25: storing the key value data and the current time in the read-only skip list structure into a B + tree structure;
s26: the system completes the writing request of the user and sends the information of successful writing to the user.
Specifically, the storage structure of the application is specifically:
writing in a jump table, a read-only jump table, a B + tree, and a cold tree structure and a hot tree structure in a disk; the cold tree structure is a 3-layer log structure merged tree which comprises a 0-layer log structure, a 1-layer log structure and a 2-layer log structure, and the 0-layer log structure has a partition structure; the hot tree structure is a single-level log-structured merged tree.
As can be seen from the above description of the present invention, compared with the prior art, the present invention has the following advantages:
(1) the invention provides a key value storage method based on a multi-tree conversion mechanism, which specifically comprises the following steps: for written key value data, firstly storing the key value data into a write jump table, and converting the key value data into a read-only jump table to be inserted into a B + tree in a memory device after the size of the key value data reaches the limit; traversing the key value data when the size of the B + tree reaches a certain limit; judging according to the heat strategy, persisting the key value data with low heat to the 0 layer of the cold tree in the external storage device, and continuously storing the key value data with high heat in the B + tree; if the number of the key value data files in the layer 0 of the cold tree reaches the size limit, triggering the partition operation of the layer 0, equally dividing the partition operation of the layer 0 according to the existing key value data range in the layer, and if the number of the key value data files in a certain partition in the layer 0 reaches the size limit, triggering the partition operation again; when key value data with lower heat in the B + tree execute persistence operation, the 0-layer partition only receives the key value data which accords with the set range; if the key value data in the range in the cold tree reach a certain hot degree after being judged by the hot degree strategy, transferring the key value data in the range to a hot tree of the external memory device, and using a B + tree in the internal memory device to index the key value data in the hot tree; when the size of the hot tree reaches a certain limit, the key value data is judged according to the heat strategy, the key value data in one or more low-heat ranges are transferred to the cold tree partition, and the indexes of the key value data are deleted from the B + tree. The method provided by the invention reduces the read amplification by using the heat strategy, ensures the write-in performance and realizes the integral improvement of the performance of the key value storage system.
(2) On one hand, the persistence operation is determined through the heat strategy, the hot data is temporarily stored in the memory, and the extra IO (input/output) overhead caused by reading the hot data from the external memory device is avoided; on the other hand, the key value data frequently accessed in the external memory device is indexed by using the global heat judgment strategy, the reading process of the key value data is accelerated, and the reading performance is improved.
(3) The invention uses the B + tree in the memory to store the address information in the hot tree, and can index all effective data in the hot tree, thereby reducing the reading times of the hot data, and the B + tree can ensure the searching performance in a certain range. By reducing the level of the cold tree and the design of the 0-layer partition, the reading times of cold data are reduced, and the reading amplification is reduced.
(4) The invention uses the B + tree to store the hot data, and can avoid multiple persistence operations caused when the key value data executes the updating operation. The inter-layer merge operation of a conventional LSM tree enables space reclamation but enables the same data to be written repeatedly. The conversion between the cold tree and the hot tree is used for replacing the traditional interlayer merging operation, so that the reading performance can be improved, and the data writing amount can be reduced.
Drawings
FIG. 1 is a system structure diagram of a key value storage method based on a multi-tree transformation mechanism;
FIG. 2 is a data reading flow diagram of a key value storage method based on a multi-tree transformation mechanism;
fig. 3 is a data writing flow chart of a key value storage method based on a multi-tree conversion mechanism.
Detailed Description
The invention is further described below by means of specific embodiments.
Fig. 1 is a system structure diagram of a key value storage method based on a multi-tree conversion mechanism according to an embodiment of the present invention; the applied storage structure is specifically as follows:
writing a jump table, a read-only jump table, a B + tree, and a cold tree structure and a hot tree structure in a disk; the cold tree structure is a 3-layer log structure merged tree which comprises a 0-layer log structure, a 1-layer log structure and a 2-layer log structure, and the 0-layer log structure has a partition structure; the hot tree structure is a single-layer log structure merged tree;
the storage method specifically comprises the following steps: firstly, storing the data into a write skip list, and converting the data into a read-only skip list to be inserted into a B + tree in a memory device after the size of the data reaches a limit; when the size of the B + tree reaches a certain limit, traversing key value data in the B + tree according to a heat strategy to judge;
key value data with lower heat degree are persisted to the 0 layer of a cold tree in the external storage device, and key value data with higher heat degree are continuously stored in a B + tree;
if the number of the key value data files in the layer 0 of the cold tree reaches the size limit, triggering data merging operation and layer 0 partitioning operation; the 0-layer partition operation is equally divided according to the existing key value data range in the layer; if the number of the key value data files of a certain partition in the layer 0 reaches the size limit, triggering the data merging operation and the partition operation again;
when the key value data with lower degree in the B + tree executes persistence operation, the 0-layer partition only receives the key value data which accords with the range of the 0-layer partition. If the key value data in a certain range in the cold tree reaches a certain hot degree through the hot degree strategy judgment, all the key value data in the range are transferred to the hot tree of the external memory device, and the B + tree insert key and the address information in the internal memory device are used for indexing the key value data in the hot tree.
When the size of the hot tree reaches a certain limit, traversing and judging the key value data in the hot tree according to a heat strategy, transferring the key value data in one or more low-heat ranges to the cold tree partition, and deleting indexes of the data from the B + tree;
specifically, the heat strategy is specifically as follows:
when the key value data in the read-only skip list is inserted into the B + tree structure, an insertion time field is stored and used for judging the heat condition of the key value data;
judging whether the key value data needs to be reserved in a B + tree or not according to the time field recorded in the node, and if the key value data does not need to be reserved, persisting the key value data in a cold tree partition of the external storage device;
for key value data in the external storage device, a global heat table is used for recording operands of the key value data, and when merging operation in a cold tree is executed, if the operand of a range unit positioned in the cold tree reaches a threshold value, all the key value data in the range unit are transferred to a hot tree;
when the hot tree reaches a certain size limit, a global heat table is used for carrying out heat judgment on key value data in the hot tree, whether the key value data in the range need to be reserved in the hot tree is judged according to the range heat condition of the key value data, and if the key value data in the range do not need to be reserved, the key value data in the hot tree are transferred to the cold tree.
Fig. 2 is a data reading flow chart of a key value storage method based on a multi-tree transformation mechanism.
101. The user sends a read request to the system, to 102.
102. And searching target data in the cache region, if the target data is found, acquiring the target data to 103, otherwise, acquiring the target data to 104.
103. And returning the query result to the user.
104. And searching target data in the B + tree, if the target data is not found, obtaining 105, otherwise obtaining 106.
105. And searching data layer by layer from top to bottom in the cold tree, if the data is found, acquiring target data to 103, otherwise, returning that the result does not exist to 103.
106. And judging whether the value type is address information or not, if not, acquiring target data, and going to 103, otherwise, going to 107.
107. And acquiring the target data according to the address information found in the previous step to 103.
Fig. 3 is a data writing flow chart of a key value storage method based on a multi-tree transformation mechanism.
201. A write request sent by the user to the system, to 202.
202. And judging whether the write jump table is full, if not, going to 211, and if not, going to 203.
203. The write jump table is converted to a read-only jump table, to 204.
204. And judging whether the size of the cache region is full, if not, going to 205, otherwise, going to 206.
205. A new write skip list is created, to 202.
206. And judging whether the size of the B + tree reaches the limit, if so, going to 207, and if not, going to 208.
207. And judging whether the number of the files in the partition of the cold tree layer 0 reaches the limit, if so, going to 209, and otherwise, going to 210.
208. The read-only skip list is inserted into the B + tree, to 204.
209. Merge and partition operations are performed on level 0 of the cold tree, through 210.
210. And (4) persisting the low-heat data in the B + tree to a cold tree according to a heat strategy, and going to 206.
211. Write data into the write skip table, through 212.
212. And returning the writing result to the user.
(1) The invention provides a key value storage method based on a multi-tree conversion mechanism, which specifically comprises the following steps: for written key value data, firstly storing the key value data into a write jump table, and converting the key value data into a read-only jump table to be inserted into a B + tree in a memory device after the size of the key value data reaches the limit; traversing the key value data when the size of the B + tree reaches a certain limit; judging according to the heat strategy, persisting the key value data with low heat to the 0 layer of the cold tree in the external storage device, and continuously storing the key value data with high heat in the B + tree; if the number of the key value data files in the layer 0 of the cold tree reaches the size limit, triggering the partition operation of the layer 0, equally dividing the partition operation of the layer 0 according to the existing key value data range in the layer, and if the number of the key value data files in a certain partition in the layer 0 reaches the size limit, triggering the partition operation again; when key value data with lower heat in the B + tree execute persistence operation, the 0-layer partition only receives the key value data which accords with the set range; if the key value data in the range in the cold tree reaches a certain hot degree after being judged by the hot degree strategy, transferring the key value data in the range to a hot tree of the external memory device, and using a B + tree in the internal memory device to index the key value data in the hot tree; when the size of the hot tree reaches a certain limit, the key value data is judged according to the heat strategy, the key value data in one or more low-heat ranges are transferred to the cold tree partition, and the indexes of the key value data are deleted from the B + tree. The method provided by the invention reduces the read amplification by using the heat strategy, ensures the write-in performance and realizes the integral improvement of the performance of the key value storage system.
(2) On one hand, the persistence operation is determined through the heat strategy, the hot data is temporarily stored in the memory, and the extra IO (input/output) overhead caused by reading the hot data from the external memory device is avoided; on the other hand, the key value data frequently accessed in the external memory device is indexed by using the global heat judgment strategy, the reading process of the key value data is accelerated, and the reading performance is improved.
(3) The invention uses the B + tree in the memory to store the address information in the hot tree, and can index all effective data in the hot tree, thereby reducing the reading times of the hot data, and the B + tree can ensure the searching performance in a certain range. By reducing the level of the cold tree and the design of the 0-layer partition, the reading times of cold data are reduced, and the reading amplification is reduced.
(4) The invention uses the B + tree to store the hot data, and can avoid multiple persistence operations caused when the key value data executes the updating operation. The inter-layer merge operation of a conventional LSM tree enables space reclamation but enables the same data to be written repeatedly. The conversion between the cold tree and the hot tree is used for replacing the traditional interlayer merging operation, so that the reading performance can be improved, and the data writing amount can be reduced.
The above description is only an embodiment of the present invention, but the design concept of the present invention is not limited thereto, and any insubstantial modifications made by using the design concept should fall within the scope of infringing the present invention.
Claims (5)
1. A key value storage method based on a multi-tree conversion mechanism is characterized by comprising the following steps:
for written key value data, firstly storing the key value data into a write jump table, and converting the key value data into a read-only jump table to be inserted into a B + tree in a memory device after the size of the key value data reaches the limit; traversing the key value data when the size of the B + tree reaches a certain limit;
judging according to the heat strategy, persisting the key-value data with low heat to the 0 layer of the cold tree in the external storage device, and keeping the key-value data with high heat in the B + tree;
if the number of the key value data files in the layer 0 of the cold tree reaches the size limit, triggering the partition operation of the layer 0, equally dividing the partition operation of the layer 0 according to the existing key value data range in the layer, and if the number of the key value data files in a certain partition in the layer 0 reaches the size limit, triggering the partition operation again;
when key value data with lower heat in the B + tree execute persistence operation, the 0-layer partition only receives the key value data which accords with the set range; if the key value data in the range in the cold tree reaches a certain hot degree after being judged by the hot degree strategy, transferring the key value data in the range to a hot tree of the external memory device, and using a B + tree in the internal memory device to index the key value data in the hot tree;
when the size of the hot tree reaches a certain limit, the key value data is judged according to the heat strategy, the key value data in one or more low-heat ranges are transferred to the cold tree partition, and the indexes of the key value data are deleted from the B + tree.
2. The key-value storage method based on multi-tree transformation mechanism as claimed in claim 1, wherein the hot policy is specifically:
when the key value data in the read-only skip list is inserted into the B + tree structure, an insertion time field is stored and used for judging the heat condition of the key value data;
judging whether the key value data needs to be reserved in a B + tree or not according to the time field recorded in the node, and if the key value data does not need to be reserved, persisting the key value data in a cold tree partition of the external storage device;
for key value data in the external storage device, a global heat table is used for recording operands of the key value data, and when merging operation in a cold tree is executed, if the operand of a range unit positioned in the cold tree reaches a threshold value, all the key value data in the range unit are transferred to a hot tree;
when the hot tree reaches a certain size limit, a global heat table is used for carrying out heat judgment on key value data in the hot tree, whether the key value data in the range need to be reserved in the hot tree is judged according to the range heat condition of the key value data, and if the key value data in the range do not need to be reserved, the key value data in the hot tree are transferred to the cold tree.
3. The key-value storage method based on the multi-tree transformation mechanism according to claim 1 or 2, further comprising a data reading step, specifically:
s11: firstly, searching target data in a cache region, and if the target data is found, going to step S12; if not, go to step S13;
s12: the system finishes the reading request of the user and sends the read target data to the user;
s13: searching the target key and the value type in the B + tree, if the target key and the value type are found, the step S12 is executed, and if the target key and the value type are found, the step S14 is executed; if not, go to step S15;
s14: reading the target data in the hot tree of the external memory device according to the address obtained in the previous step, and going to step S12;
s15: searching target data from top to bottom according to layers in a cold tree of the external storage device, and if the target data is found, going to step S12; if not, go to step S16;
s16: the system completes the reading request of the user, and sends the result to the user without finding the target data.
4. The key-value storage method based on the multi-tree transformation mechanism according to claim 1 or 2, further comprising a data writing step, specifically:
s21: firstly, writing the data into a skip list structure, and if the size limit is reached, going to step S22;
s22: converting the skip list into a read-only skip list, and creating a new write skip list structure for data writing;
s23: if the size of the B + tree does not reach the limit, go to step S25; if the limit is reached, go to step S24;
s24: traversing the key value data in the B + tree, and persisting the key value data with lower heat into the cold tree of the external storage device to the step S25;
s25: storing the key value data and the current time in the read-only skip list structure into a B + tree structure;
s26: the system completes the writing request of the user and sends the information of successful writing to the user.
5. The key-value storage method based on multi-tree transformation mechanism according to claim 1 or 2, wherein the applied storage structure is specifically:
writing in a jump table, a read-only jump table, a B + tree, and a cold tree structure and a hot tree structure in a disk; the cold tree structure is a 3-layer log structure merged tree which comprises a 0-layer log structure, a 1-layer log structure and a 2-layer log structure, and the 0-layer log structure has a partition structure; the hot tree structure is a single-level log-structured merged tree.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210711424.3A CN114996275A (en) | 2022-06-22 | 2022-06-22 | Key value storage method based on multi-tree conversion mechanism |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210711424.3A CN114996275A (en) | 2022-06-22 | 2022-06-22 | Key value storage method based on multi-tree conversion mechanism |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114996275A true CN114996275A (en) | 2022-09-02 |
Family
ID=83036517
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210711424.3A Pending CN114996275A (en) | 2022-06-22 | 2022-06-22 | Key value storage method based on multi-tree conversion mechanism |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114996275A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116149550A (en) * | 2022-12-29 | 2023-05-23 | 北京趋动智能科技有限公司 | Data management method, device, storage medium and equipment |
| CN118092812A (en) * | 2024-04-18 | 2024-05-28 | 华侨大学 | Key value storage and read-write method based on memory table index and iterator reduction mechanism |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180089244A1 (en) * | 2016-09-26 | 2018-03-29 | Vmware, Inc. | Key-value stores implemented using fragmented log-structured merge trees |
| CN111026329A (en) * | 2019-11-18 | 2020-04-17 | 华中科技大学 | Key value storage system based on host management tile record disk and data processing method |
| CN112100293A (en) * | 2020-09-23 | 2020-12-18 | 腾讯科技（深圳）有限公司 | Data processing method, data access method, data processing device, data access device and computer equipment |
| CN112486994A (en) * | 2020-11-30 | 2021-03-12 | 武汉大学 | Method for quickly reading data of key value storage based on log structure merging tree |
| CN113626431A (en) * | 2021-07-28 | 2021-11-09 | 浪潮云信息技术股份公司 | LSM tree-based key value separation storage method and system for delaying garbage recovery |
| CN113760171A (en) * | 2020-06-01 | 2021-12-07 | 华为技术有限公司 | Metadata storage method and device |
| CN113821171A (en) * | 2021-09-01 | 2021-12-21 | 浪潮云信息技术股份公司 | Key value storage method based on hash table and LSM tree |
| CN114416646A (en) * | 2022-01-20 | 2022-04-29 | 上海妃鱼网络科技有限公司 | Data processing method and device of hierarchical storage system |
-
2022
- 2022-06-22 CN CN202210711424.3A patent/CN114996275A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180089244A1 (en) * | 2016-09-26 | 2018-03-29 | Vmware, Inc. | Key-value stores implemented using fragmented log-structured merge trees |
| CN111026329A (en) * | 2019-11-18 | 2020-04-17 | 华中科技大学 | Key value storage system based on host management tile record disk and data processing method |
| CN113760171A (en) * | 2020-06-01 | 2021-12-07 | 华为技术有限公司 | Metadata storage method and device |
| CN112100293A (en) * | 2020-09-23 | 2020-12-18 | 腾讯科技（深圳）有限公司 | Data processing method, data access method, data processing device, data access device and computer equipment |
| CN112486994A (en) * | 2020-11-30 | 2021-03-12 | 武汉大学 | Method for quickly reading data of key value storage based on log structure merging tree |
| CN113626431A (en) * | 2021-07-28 | 2021-11-09 | 浪潮云信息技术股份公司 | LSM tree-based key value separation storage method and system for delaying garbage recovery |
| CN113821171A (en) * | 2021-09-01 | 2021-12-21 | 浪潮云信息技术股份公司 | Key value storage method based on hash table and LSM tree |
| CN114416646A (en) * | 2022-01-20 | 2022-04-29 | 上海妃鱼网络科技有限公司 | Data processing method and device of hierarchical storage system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116149550A (en) * | 2022-12-29 | 2023-05-23 | 北京趋动智能科技有限公司 | Data management method, device, storage medium and equipment |
| CN118092812A (en) * | 2024-04-18 | 2024-05-28 | 华侨大学 | Key value storage and read-write method based on memory table index and iterator reduction mechanism |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110825748B (en) | High-performance and easily-expandable key value storage method by utilizing differentiated indexing mechanism | |
| CN111399777B (en) | Differential key value data storage method based on data value classification | |
| CN110347852B (en) | File system embedded with transverse expansion key value storage system and file management method | |
| CN103838853B (en) | Mixed file system based on different storage media | |
| US10740251B2 (en) | Hybrid drive translation layer | |
| CN109582593B (en) | FTL address mapping reading and writing method based on calculation | |
| CN111026329B (en) | Key value storage system based on host management tile record disk and data processing method | |
| CN113821171B (en) | Key value storage method based on hash table and LSM tree | |
| CN114546296B (en) | ZNS solid state disk-based full flash memory system and address mapping method | |
| CN111475507B (en) | Key value data indexing method for workload adaptive single-layer LSMT | |
| CN110968269A (en) | SCM and SSD-based key value storage system and read-write request processing method | |
| CN116257523A (en) | Column type storage indexing method and device based on nonvolatile memory | |
| CN114416646A (en) | Data processing method and device of hierarchical storage system | |
| CN114996275A (en) | Key value storage method based on multi-tree conversion mechanism | |
| CN112732725B (en) | NVM (non volatile memory) hybrid memory-based adaptive prefix tree construction method, system and medium | |
| CN115964350A (en) | File system management model and system | |
| CN116382588A (en) | LSM-Tree storage engine read amplification problem optimization method based on learning index | |
| CN114415966B (en) | Method for constructing KV SSD storage engine | |
| JPS6154547A (en) | Data processing system with 3-level hierarchical memory | |
| CN115509437A (en) | Storage system, network card, processor, data access method, device and system | |
| CN114969069B (en) | Heat perception local updating method applied to key value storage system | |
| CN114968103B (en) | Fingerprint storage method based on persistent memory | |
| CN114398007B (en) | LSM-tree-based caching optimization method for KV storage system read performance | |
| CN113821177B (en) | Storage structure of LSM tree based on NVM and data storage method thereof | |
| US20150278117A1 (en) | Method and apparatus method and apparatus for controlling access to a hash-based disk |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |