WO2022063059A1 - Data management method for key-value storage system and device thereof - Google Patents

Data management method for key-value storage system and device thereof Download PDF

Info

Publication number
WO2022063059A1
WO2022063059A1 PCT/CN2021/119141 CN2021119141W WO2022063059A1 WO 2022063059 A1 WO2022063059 A1 WO 2022063059A1 CN 2021119141 W CN2021119141 W CN 2021119141W WO 2022063059 A1 WO2022063059 A1 WO 2022063059A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
key
data layer
ordered
layer
Prior art date
Application number
PCT/CN2021/119141
Other languages
French (fr)
Chinese (zh)
Inventor
姚婷
王道辉
宋驰
丁萌
Original Assignee
华为云计算技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为云计算技术有限公司 filed Critical 华为云计算技术有限公司
Publication of WO2022063059A1 publication Critical patent/WO2022063059A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/13—File access structures, e.g. distributed indices
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/11—File system administration, e.g. details of archiving or snapshots
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/11—File system administration, e.g. details of archiving or snapshots
    • G06F16/128—Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/13—File access structures, e.g. distributed indices
    • G06F16/137—Hash-based
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815—Journaling file systems

Definitions

  • the I/O operation request is a read operation request and the data to be operated is data to be read
  • performing I/O operations on the key-value pair in the target data structure including: searching the first ordered data layer of the target data structure Whether to store the key of the data to be read, the first ordered data layer is located in the memory, and the first ordered data layer supports the update write operation; when the key of the data to be read is not stored in the first ordered data layer, search Whether the key of the data to be read is stored in the second ordered data layer of the target data structure, the second ordered data layer is located in the memory, and the second ordered data layer does not support modification operations; When the key of the data to be read is not stored, find out whether the key of the data to be read is stored in the third data layer of the target data structure, the third data layer is located in the hard disk, and the third data layer supports additional write operations; or, when When the key of the data to be read is not stored in the first ordered data layer, and the second ordered data layer does not exist in the target data structure, check whether
  • the root node also stores the node identifier of the intermediate node used to store the key covered by the key stored by the root node
  • the intermediate node also stores the node identifier of the leaf node used to store the key covered by the key stored by the intermediate node.
  • FIG. 4 is a schematic diagram of a key-value storage system provided by an embodiment of the present application.
  • FIG. 5 is a schematic diagram of reading data according to a data management method of a key-value storage system provided by an embodiment of the present application
  • FIG. 7 is a schematic structural diagram of a data management device of a key-value storage system provided by an embodiment of the present application.
  • FIG. 1 is a schematic structural diagram of an application scenario involved in a data management method of a key-value storage system provided by an embodiment of the present application.
  • the application scenario includes a computer system 10 , a client 20 and a storage pool 30 .
  • the storage pool 30 is used to store the data indicated by the client 20 to be stored.
  • the client 20 and the computer system 10 are connected through a network, and the client 20 can send an input/output (I/O) operation request to the computer system 10 through the network.
  • the computer system 10 and the storage pool 30 are connected through a network, and the computer system 10 can perform data interaction with the storage pool 30 through the network, so as to realize the I/O operation requested by the I/O operation request sent by the client 20 .
  • FIG. 2 is a flowchart of a method for writing data to be written into a key-value storage system according to a write operation request provided by an embodiment of the present application. As shown in Figure 2, the process of writing data includes:
  • the key in the key-value pair indicates the keyword used to find the physical storage address of the data written by the user, the value in the key-value pair indicates the physical storage address of the data, or the value in the key-value pair is the data itself . And, when the value in the key-value pair is the physical storage address of the data to be written, the key-value storage system is used to manage the metadata of the data to be written. When the value in the key-value pair is the data to be written itself, the key-value storage system is used to manage the data to be written.
  • the process of determining the key-value pair of the data to be written includes the process of respectively determining the key and the value in the key-value pair.
  • the index information itself may be determined as the key.
  • the quadruple includes the identifier of the cloud disk, the logical address of the data to be operated, the linked clone identifier and the snapshot identifier of the cloud disk, the quadruple can be directly determined as the key.
  • Step 204 After the computer system writes the write request into the data log, it sends a write operation response to the write operation request.
  • Step S1 Read the key-value pair in the third data layer into the memory.
  • the log file a also includes the mapping table log a2.
  • the hash mapping table Any modification of the recorded correspondence can be persisted to the mapping table log a2, so that when the key-value storage system fails, the key-value storage system can restore the hash mapping table by replaying the mapping table log a2.
  • the first ordered data layer supports update write operations.
  • the first ordered data layer please refer to the relevant content in the foregoing process of writing data, which will not be repeated here.
  • the search when searching for the key of the data to be read in the target data structure, the search can be performed in the target data structure of the latest version to ensure that the key-value storage system can The latest state is searched to ensure that accurate data is read according to the search results.
  • the computer system can count the total number of references to the target data structure, so as to determine the usage of the target data structure according to the total number. For example, when performing an I/O operation on the target data structure, the total number of references to the target data structure can be incremented by one to indicate that the target data structure is in use, and when the I/O operation on the target data structure is completed, the The total number of references to the target data structure is decremented by one.
  • the reference counts of the data structures of different versions are all 0 when they are created, that is, the initial values of the reference counts of the data structures are all 0.
  • the garbage collection process of the data structure does not affect the service provided by the key-value storage system, when the data structure needs to be garbage collected, it is necessary to ensure that the reference count of the data structure is 0 and the version of the data structure is not the latest version. .
  • the data management method of the key-value storage system further includes a process of deleting snapshots through the key-value storage system.
  • FIG. 6 is a flowchart of a method for deleting snapshots through the key-value storage system provided by an embodiment of the present application. As shown in Figure 6, the snapshot deletion process includes:
  • the determining module 702 is configured to determine the key of the data to be operated based on the index information.
  • the index information includes a keyword for indexing the key
  • the determining module 702 is specifically configured to: perform an index based on the keyword to obtain a key, and the key includes the identifier of the initiator of the I/O operation request and the logic of the data to be operated. address.
  • a receiving module 704 configured to receive a snapshot deletion request
  • the processing module 703 is further configured to perform the snapshot deletion operation indicated by the snapshot deletion request based on the snapshot deletion request recorded in the log file.
  • the memory 902 stores computer instructions and data, and the memory 902 may store computer instructions and data required to implement the data management method of the key-value storage system provided by the present application.
  • the memory 902 can be any one or any combination of the following storage media: non-volatile memory (such as read-only memory (Read-Only Memory, ROM), solid state disk (Solid State Disk, SSD), hard disk (Hard Disk) Drive, HDD), optical disc, etc., volatile memory.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The present application relates to the technical field of data storage, and disclosed thereby is a data management method for a key-value storage system. A logical address space of the key-value storage system comprises: a plurality of sub-logical address spaces, the plurality of sub-logical address spaces not overlapping one another, and data stored in the plurality of sub-logical address spaces being respectively managed in one-to-one correspondence by using a plurality of data structures. The method comprises: acquiring an input/output (I/O) operation request, the I/O operation request bearing index information of data to be operated; on the basis of the index information, determining a key of the data to be operated; on the basis of the key, determining, among the plurality of data structures, a target data structure for storing a key-value pair of the data to be operated; and performing an I/O operation on the key-value pair in the target data structure. The present application improves the efficiency of the key-value storage system in responding to an I/O operation request according to a data structure and the performance of the key-value storage system.

Description

键值存储***的数据管理方法及其装置Data management method and device for key-value storage system
本申请要求于2020年09月23日提交的申请号为202011011617.5、发明名称为“一种键值存储***”的中国专利申请的优先权，及于2021年02月26日提交的申请号为202110220635.2、发明名称为“键值存储***的数据管理方法及其装置”的中国专利申请的优先权，其全部内容通过引用结合在本公开中。This application claims the priority of the Chinese patent application with the application number 202011011617.5 and the invention name "a key-value storage system" filed on September 23, 2020, and the application number 202110220635.2 filed on February 26, 2021 , the priority of the Chinese patent application entitled "Data Management Method and Device for Key-Value Storage System", the entire contents of which are incorporated in this disclosure by reference.
技术领域technical field
本申请涉及数据存储技术领域，特别涉及一种键值存储***的数据管理方法及其装置。The present application relates to the technical field of data storage, and in particular, to a data management method and device of a key-value storage system.
背景技术Background technique
为了便于对计算机设备上的存储介质中存储的数据进行管理，目前基于存储介质提供的存储空间构建有存储***，用户可以通过该存储***在存储介质中读写数据。并且，为便于存储***对存储介质中存储的数据进行管理，在通过存储***向存储介质中写数据时，不仅要写入用户的数据，还需要写入用户的数据的元数据。元数据为用于描述存储***中存储的数据的描述信息，根据元数据可以确定该元数据所描述的数据的存储位置等信息。In order to facilitate the management of data stored in a storage medium on a computer device, a storage system is currently constructed based on the storage space provided by the storage medium, and users can read and write data in the storage medium through the storage system. In addition, in order to facilitate the storage system to manage the data stored in the storage medium, when writing data to the storage medium through the storage system, not only the user's data but also the metadata of the user's data needs to be written. The metadata is description information used to describe the data stored in the storage system, and information such as the storage location of the data described by the metadata can be determined according to the metadata.
相关技术中，通常使用数据结构对存储***中的所有元数据进行管理。在写入待写入数据的过程中，需要在该数据结构中写入该待写入数据的元数据。在读取待读取数据的过程中，可以先在该数据结构中获取该待读取数据的元数据，然后根据该元数据的指示在存储***中获取待读取数据。In the related art, a data structure is generally used to manage all metadata in a storage system. In the process of writing the data to be written, metadata of the data to be written needs to be written in the data structure. In the process of reading the data to be read, the metadata of the data to be read can be obtained from the data structure first, and then the data to be read can be obtained from the storage system according to the instruction of the metadata.
但是，当数据结构中存储的数据量较大时，会导致该存储***的读性能较差。However, when the amount of data stored in the data structure is large, the read performance of the storage system is poor.
发明内容SUMMARY OF THE INVENTION
本申请提供了一种键值存储***的数据管理方法及其装置。本申请提供的技术方案如下：The present application provides a data management method and device of a key-value storage system. The technical solutions provided by this application are as follows:
第一方面，本申请提供了一种键值存储***的数据管理方法。键值存储***的逻辑地址空间包括：多个子逻辑地址空间，多个子逻辑地址空间互不重叠，多个子逻辑地址空间中存储的数据分别使用多个数据结构一一对应管理，该方法包括：获取输入/输出I/O操作请求，I/O操作请求中携带有待操作数据的索引信息；基于索引信息，确定待操作数据的键；基于键，在多个数据结构中确定用于存储待操作数据的键值对的目标数据结构；在目标数据结构中对键值对进行I/O操作。In a first aspect, the present application provides a data management method of a key-value storage system. The logical address space of the key-value storage system includes: multiple sub-logical address spaces, the multiple sub-logical address spaces do not overlap each other, and the data stored in the multiple sub-logical address spaces are managed in a one-to-one correspondence with multiple data structures, and the method includes: obtaining Input/output I/O operation request, the I/O operation request carries the index information of the data to be operated; based on the index information, the key of the data to be operated is determined; based on the key, it is determined in multiple data structures for storing the data to be operated The target data structure of the key-value pair; perform I/O operations on the key-value pair in the target data structure.
通过在用于管理键值存储***的逻辑地址空间的多个数据结构中，确定用于存储所述待操作数据的键值对的目标数据结构，并在目标数据结构中对键值对进行I/O操作，由于该键值存储***中的多个子逻辑地址空间分别由多个数据结构进行管理，相较于采用一个数据结构管理键值存储***的逻辑地址空间，减小了该多个数据结构中每个数据结构需要存储的数据量，提高了该键值存储***根据该数据结构对I/O操作请求进行响应的效率和键值存储***的性能，能够通过该数据结构向用户提供稳定且可预测的读性能。By determining a target data structure for storing the key-value pair of the data to be operated from among the multiple data structures used to manage the logical address space of the key-value storage system, and performing an operation on the key-value pair in the target data structure /O operation, since multiple sub-logical address spaces in the key-value storage system are managed by multiple data structures, compared with using one data structure to manage the logical address space of the key-value storage system, the multiple data structures are reduced. The amount of data that needs to be stored in each data structure in the structure improves the efficiency of the key-value storage system in responding to I/O operation requests according to the data structure and the performance of the key-value storage system, and can provide users with stability through the data structure. and predictable read performance.
并且，由于键值存储***的多个子逻辑地址空间分别由多个数据结构管理，键值存储***能够通过该多个数据结构提供存储服务，提高该键值存储***的并行处理能力，提高了键值存储***的服务性能。Moreover, since the multiple sub-logical address spaces of the key-value storage system are managed by multiple data structures, the key-value storage system can provide storage services through the multiple data structures, improve the parallel processing capability of the key-value storage system, and improve the key-value storage system. Service performance of the value store system.
在一种可实现方式中，索引信息可以是键本身。例如，索引信息和键均包括I/O操作请求的发起者的标识和所述待操作数据的逻辑地址。此时，基于索引信息，确定待操作数据的键，包括：将索引信息确定为键。可选地，索引信息和键还可以包括I/O操作请求的发起者的链接克隆标识和快照标识，即索引信息和键均可以包括四元组，该四元组包括：I/O操作请求的发起者的逻辑单元号标识、链接克隆标识、逻辑块地址和快照标识。In one implementation, the index information may be the keys themselves. For example, both the index information and the key include the identifier of the initiator of the I/O operation request and the logical address of the data to be operated. At this time, determining the key of the data to be operated based on the index information includes: determining the index information as the key. Optionally, the index information and the key may also include the linked clone identifier and the snapshot identifier of the initiator of the I/O operation request, that is, the index information and the key may both include a quadruple, and the quadruple includes: the I/O operation request. The originator's LUN ID, Linked Clone ID, Logical Block Address, and Snapshot ID.
在一种实现场景中，当用户使用云盘时，该I/O操作请求的发起者为云盘，则索引信息和键均包括云盘的标识、所述待操作数据的逻辑地址、云盘的链接克隆标识和快照标识。In an implementation scenario, when a user uses a cloud disk, the initiator of the I/O operation request is the cloud disk, and the index information and keys both include the identifier of the cloud disk, the logical address of the data to be operated, and the cloud disk. linked clone ID and snapshot ID.
在另一种可实现方式中，索引信息包括用于索引键的关键字，键可以根据该关键字索引得到。例如，键包括所述I/O操作请求的发起者的标识和所述待操作数据的逻辑地址，索引信息为用于索引到该I/O操作请求的发起者的标识和所述待操作数据的逻辑地址的关键字。此时，索引信息包括用于索引键的关键字，基于索引信息，确定待操作数据的键，包括：基于关键字进行索引，得到键，键包括I/O操作请求的发起者的标识和待操作数据的逻辑地址。In another implementation manner, the index information includes a keyword for indexing the key, and the key can be obtained by indexing the keyword. For example, the key includes the identifier of the initiator of the I/O operation request and the logical address of the data to be operated, and the index information is used for indexing to the identifier of the initiator of the I/O operation request and the data to be operated. The keyword of the logical address. At this time, the index information includes a keyword for indexing the key, and based on the index information, determining the key of the data to be operated includes: indexing based on the keyword, and obtaining a key, where the key includes the identifier of the initiator of the I/O operation request and the data to be operated. The logical address of the operation data.
当I/O操作请求为写操作请求，待操作数据为待写入数据时，在目标数据结构中对键值对进行I/O操作，包括：将键值对存储在目标数据结构的第一有序数据层中，第一有序数据层位于内存中，且第一有序数据层支持更新写操作；当达到参考条件时，将第一有序数据层转换为第二有序数据层，第二有序数据层位于内存中，且第二有序数据层不支持修改操作；将第二有序数据层中的键值对写入目标数据结构的第三数据层中，第三数据层位于硬盘中，且第三数据层支持追加写操作；当第三数据层的数据量达到第二数据量阈值时，使用第三数据层中的键值对和目标数据结构的第四有序数据层中的键值对，更新第四有序数据层中的键值对，第四有序数据层位于硬盘中，且第四有序数据层支持追加写操作。When the I/O operation request is a write operation request and the data to be operated is data to be written, performing I/O operations on the key-value pair in the target data structure, including: storing the key-value pair in the first part of the target data structure In the ordered data layer, the first ordered data layer is located in the memory, and the first ordered data layer supports the update write operation; when the reference condition is reached, the first ordered data layer is converted into the second ordered data layer, The second ordered data layer is located in the memory, and the second ordered data layer does not support modification operations; the key-value pairs in the second ordered data layer are written into the third data layer of the target data structure, and the third data layer It is located in the hard disk, and the third data layer supports additional write operations; when the data volume of the third data layer reaches the second data volume threshold, the key-value pair in the third data layer and the fourth ordered data of the target data structure are used The key-value pairs in the layer, update the key-value pairs in the fourth ordered data layer, which is located in the hard disk, and the fourth ordered data layer supports additional write operations.
当I/O操作请求为读操作请求，待操作数据为待读取数据时，在目标数据结构中对键值对进行I/O操作，包括：查找目标数据结构的第一有序数据层中是否存储有待读取数据的键，第一有序数据层位于内存中，且第一有序数据层支持更新写操作；当第一有序数据层中未存储待读取数据的键时，查找目标数据结构的第二有序数据层中是否存储有待读取数据的键，第二有序数据层位于内存中，且第二有序数据层不支持修改操作；当第二有序数据层中未存储待读取数据的键时，查找目标数据结构的第三数据层中是否存储有待读取数据的键，第三数据层位于硬盘中，且第三数据层支持追加写操作；或者，当第一有序数据层中未存储待读取数据的键，且目标数据结构中不存在第二有序数据层时，查找第三有序数据层中是否存储有待读取数据的键；当第三数据层中未存储待读取数据的键时，查找目标数据结构的第四有序数据层中是否存储有待读取数据的键，第四有序数据层位于硬盘中，且第四有序数据层支持追加写操作。When the I/O operation request is a read operation request and the data to be operated is data to be read, performing I/O operations on the key-value pair in the target data structure, including: searching the first ordered data layer of the target data structure Whether to store the key of the data to be read, the first ordered data layer is located in the memory, and the first ordered data layer supports the update write operation; when the key of the data to be read is not stored in the first ordered data layer, search Whether the key of the data to be read is stored in the second ordered data layer of the target data structure, the second ordered data layer is located in the memory, and the second ordered data layer does not support modification operations; When the key of the data to be read is not stored, find out whether the key of the data to be read is stored in the third data layer of the target data structure, the third data layer is located in the hard disk, and the third data layer supports additional write operations; or, when When the key of the data to be read is not stored in the first ordered data layer, and the second ordered data layer does not exist in the target data structure, check whether the key of the data to be read is stored in the third ordered data layer; When the key of the data to be read is not stored in the third data layer, find out whether the key of the data to be read is stored in the fourth ordered data layer of the target data structure, the fourth ordered data layer is located in the hard disk, and the fourth ordered data layer is located in the hard disk. The data layer supports append write operations.
在一种可实现方式中，第四有序数据层包括根节点、中间节点和叶子节点，根节点和中间节点均用于存储待操作数据的键，根节点能够存储的键的范围覆盖中间节点能够存储的键的范围，叶子节点存储有中间节点存储的键对应的键值对。In an achievable manner, the fourth ordered data layer includes a root node, an intermediate node, and a leaf node. Both the root node and the intermediate node are used to store keys of data to be operated, and the range of keys that the root node can store covers the intermediate nodes. The range of keys that can be stored. Leaf nodes store key-value pairs corresponding to keys stored by intermediate nodes.
可选地，根节点还存储有用于存储根节点存储的键所覆盖的键的中间节点的节点标识，中间节点中还存储有用于存储中间节点存储的键所覆盖的键的叶子节点的节点标识。Optionally, the root node also stores the node identifier of the intermediate node used to store the key covered by the key stored by the root node, and the intermediate node also stores the node identifier of the leaf node used to store the key covered by the key stored by the intermediate node. .
并且，为了进一步保证键值存储***的读速度，键值存储***中还存储有哈希映射表，哈希映射表用于存储根节点、中间节点和叶子节点中至少一个节点存储的节点标识与节点标识指示的节点的存储位置信息的对应关系。In addition, in order to further ensure the reading speed of the key-value storage system, the key-value storage system also stores a hash mapping table, and the hash mapping table is used to store the node identifier stored by at least one node in the root node, intermediate node and leaf node. The corresponding relationship of the storage location information of the node indicated by the node identifier.
采用通过节点和该节点的存储位置信息的对应关系实现指向节点的指向信息时，对于合并更新后位置发生变化的一个节点的更新，仅需更新该对应关系中记载的存储位置信息，因此能够避免节点更新造成的全路径更新。When using the corresponding relationship between the node and the storage location information of the node to realize the pointing information to the node, for the update of a node whose location changes after the merge and update, only the storage location information recorded in the corresponding relationship needs to be updated, so it is possible to avoid Full path update caused by node update.
在一种可实现方式中，第三数据层为无序数据层。当该第三数据层为无序数据层时，在将第二有序数据层中的数据写入第三数据层的过程中，无需对写入的数据进行排序，缩短了数据写入时间，能够加快将第二有序数据层中的数据写入第三数据层中的速度。且由于第三数据层位于硬盘中，第二有序数据层位于内存中，当将第二有序数据层中的数据写入第三数据层中的速度较快时，能够降低第二有序数据层中的数据因掉电出现丢失的概率，提高了写数据成功的概率。In one possible implementation, the third data layer is an unordered data layer. When the third data layer is an unordered data layer, in the process of writing the data in the second ordered data layer into the third data layer, there is no need to sort the written data, which shortens the data writing time. The speed of writing data in the second ordered data layer into the third data layer can be accelerated. And since the third data layer is located in the hard disk and the second ordered data layer is located in the memory, when the data in the second ordered data layer is written to the third data layer at a faster speed, the second ordered data layer can be reduced. The probability of data loss in the data layer due to power failure increases the probability of successful data writing.
可选地，第四有序数据层为树状数据结构，第四有序数据层的深度小于3。此时，能够保证读操作的整体开销可控，提供稳定的读性能。并且，能够减小数据结构的复杂度，减少写入数据过程中需要执行的操作，能够降低计算机***的写放大。Optionally, the fourth ordered data layer is a tree-like data structure, and the depth of the fourth ordered data layer is less than 3. In this case, the overall overhead of the read operation can be guaranteed to be controllable, and stable read performance can be provided. In addition, the complexity of the data structure can be reduced, the operations that need to be performed in the process of writing data can be reduced, and the write amplification of the computer system can be reduced.
在一种可实现方式中，更新后的第四有序数据层的存储位置与更新前的第四有序数据层的存储位置不同，且更新前的第四有序数据层和更新后的第四有序数据层使用不同版本号进行标识。In an implementation manner, the storage location of the fourth ordered data layer after the update is different from the storage location of the fourth ordered data layer before the update, and the fourth ordered data layer before the update and the fourth ordered data layer after the update are different. Four ordered data layers are identified using different version numbers.
并且，为了对合并排序前后的第四有序数据层进行区分，还可以为第四有序数据层设置版本号。同时，还可以为合并排序前的第四有序数据层设置延迟删除机制。也即是，在对第三数据层和第四有序数据层中的键值对进行合并排序的过程中，仍在硬盘中保存第四有序数据层，并在得到经过合并排序后的第四有序数据层后，不立即删除合并排序前的第四有序数据层。Moreover, in order to distinguish the fourth ordered data layer before and after the merge sort, a version number may also be set for the fourth ordered data layer. At the same time, a delayed deletion mechanism can also be set for the fourth ordered data layer before merge sort. That is, in the process of merging and sorting the key-value pairs in the third data layer and the fourth ordered data layer, the fourth ordered data layer is still stored in the hard disk, and after the merge-sorted No. After four ordered data layers, the fourth ordered data layer before merge sort is not immediately deleted.
这样一来，一方面能够在合并排序过程中使用合并排序前的第四有序数据层，向用户提供服务，使得读写操作能够并行执行，以不至于因合并排序导致读请求阻塞，避免产生读波动。另一方面能够在合并排序出现故障和/或根据应用需求，将第四有序数据层回退至较早的版本，保证数据的一致性。并且，键值存储***还可以分别使用第四有序数据层的多个版本提供服务，实现多线程的并发访问服务。In this way, on the one hand, the fourth ordered data layer before merge sort can be used in the merge sort process to provide services to users, so that read and write operations can be executed in parallel, so as not to block read requests due to merge sort and avoid generating Read volatility. On the other hand, the fourth ordered data layer can be rolled back to an earlier version when the merge sort fails and/or according to application requirements, so as to ensure data consistency. In addition, the key-value storage system can also use multiple versions of the fourth ordered data layer to provide services, so as to realize multi-threaded concurrent access services.
在一种可实现方式中，键包括待操作数据的逻辑地址，逻辑地址通过数值表示，多个数据结构中同一数据结构中存储的键值对中逻辑地址对指定数值的余数相等，不同数据结构中存储的键值对中逻辑地址对指定数值的余数不等。In an implementation manner, the key includes the logical address of the data to be operated, the logical address is represented by a numerical value, and the remainder of the logical address in the key-value pair stored in the same data structure in multiple data structures is equal to the specified value, and different data structures The remainder of the logical address pair stored in the key-value pair is not equal to the specified value.
可选地，键值存储***包括硬盘，硬盘中存储有日志文件，该方法还包括：接收快照删除请求；将快照删除请求写在日志文件中，并发送针对快照删除请求的操作完成响应；基于日志文件中记录的快照删除请求，执行快照删除请求指示的快照删除操作。Optionally, the key-value storage system includes a hard disk, and a log file is stored in the hard disk, and the method further includes: receiving a snapshot deletion request; writing the snapshot deletion request in the log file, and sending an operation completion response to the snapshot deletion request; For the snapshot deletion request recorded in the log file, execute the snapshot deletion operation indicated by the snapshot deletion request.
第二方面，本申请提供了一种键值存储***的数据管理装置。键值存储***的逻辑地址空间包括：多个子逻辑地址空间，多个子逻辑地址空间互不重叠，多个子逻辑地址空间中存储的数据分别使用多个数据结构一一对应管理。该装置包括：获取模块，用于获取输入/输出I/O操作请求，I/O操作请求中携带有待操作数据的索引信息；确定模块，用于基于索引信息，确定待操作数据的键；确定模块，还用于基于键，在多个数据结构中确定用于存储待操作数据的键值对的目标数据结构；处理模块，用于在目标数据结构中对键值对进行I/O操作。In a second aspect, the present application provides a data management device of a key-value storage system. The logical address space of the key-value storage system includes: multiple sub-logical address spaces, the multiple sub-logical address spaces do not overlap each other, and the data stored in the multiple sub-logical address spaces are managed in a one-to-one correspondence with multiple data structures. The device includes: an acquisition module for acquiring an input/output I/O operation request, the I/O operation request carrying index information of the data to be operated; a determination module for determining the key of the data to be operated based on the index information; determining The module is also used for determining a target data structure for storing key-value pairs of data to be operated in multiple data structures based on the key; the processing module is used for performing I/O operations on the key-value pairs in the target data structure.
可选地，索引信息包括I/O操作请求的发起者的标识和待操作数据的逻辑地址，确定模块，具体用于：将索引信息确定为键。Optionally, the index information includes the identifier of the initiator of the I/O operation request and the logical address of the data to be operated, and the determining module is specifically configured to: determine the index information as a key.
可选地，I/O操作请求的发起者为云盘。Optionally, the initiator of the I/O operation request is a cloud disk.
可选地，索引信息包括用于索引键的关键字，确定模块，具体用于：基于关键字进行索引，得到键，键包括I/O操作请求的发起者的标识和待操作数据的逻辑地址。Optionally, the index information includes a keyword for indexing the key, and the determining module is specifically used for: indexing based on the keyword to obtain a key, and the key includes the identifier of the initiator of the I/O operation request and the logical address of the data to be operated. .
可选地，I/O操作请求为写操作请求，处理模块，具体用于：将键值对存储在目标数据结构的第一有序数据层中，第一有序数据层位于内存中，且第一有序数据层支持更新写操作；当达到参考条件时，将第一有序数据层转换为第二有序数据层，第二有序数据层位于内存中，且第二有序数据层不支持修改操作；将第二有序数据层中的键值对写入目标数据结构的第三数据层中，第三数据层位于硬盘中，且第三数据层支持追加写操作；当第三数据层的数据量达到第二数据量阈值时，使用第三数据层中的键值对和目标数据结构的第四有序数据层中的键值对，更新第四有序数据层中的键值对，第四有序数据层位于硬盘中，且第四有序数据层支持追加写操作。Optionally, the I/O operation request is a write operation request, and the processing module is specifically configured to: store the key-value pair in the first ordered data layer of the target data structure, and the first ordered data layer is located in the memory, and The first ordered data layer supports update write operations; when the reference condition is reached, the first ordered data layer is converted to the second ordered data layer, the second ordered data layer is located in the memory, and the second ordered data layer is Modification operations are not supported; the key-value pairs in the second ordered data layer are written into the third data layer of the target data structure, the third data layer is located in the hard disk, and the third data layer supports additional write operations; When the data volume of the data layer reaches the second data volume threshold, use the key-value pair in the third data layer and the key-value pair in the fourth ordered data layer of the target data structure to update the key in the fourth ordered data layer Value pairs, the fourth ordered data layer is located on the hard disk, and the fourth ordered data layer supports append writes.
可选地，I/O操作请求为读操作请求，待操作数据为待读取数据，处理模块，具体用于：查找目标数据结构的第一有序数据层中是否存储有待读取数据的键，第一有序数据层位于内存中，且第一有序数据层支持更新写操作；当第一有序数据层中未存储待读取数据的键时，查找目标数据结构的第二有序数据层中是否存储有待读取数据的键，第二有序数据层位于内存中，且第二有序数据层不支持修改操作；当第二有序数据层中未存储待读取数据的键时，查找目标数据结构的第三数据层中是否存储有待读取数据的键，第三数据层位于硬盘中，且第三数据层支持追加写操作；或者，当第一有序数据层中未存储待读取数据的键，且目标数据结构中不存在第二有序数据层时，查找第三有序数据层中是否存储有待读取数据的键；当第三数据层中未存储待读取数据的键时，查找目标数据结构的第四有序数据层中是否存储有待读取数据的键，第四有序数据层位于硬盘中，且第四有序数据层支持追加写操作。Optionally, the I/O operation request is a read operation request, the data to be operated is data to be read, and the processing module is specifically used for: finding whether the key of the data to be read is stored in the first ordered data layer of the target data structure , the first ordered data layer is located in the memory, and the first ordered data layer supports the update write operation; when the key of the data to be read is not stored in the first ordered data layer, the second ordered data structure of the target data structure is searched. Whether the key of the data to be read is stored in the data layer, the second ordered data layer is located in the memory, and the second ordered data layer does not support modification operations; when the key of the data to be read is not stored in the second ordered data layer When the key of the data to be read is stored in the third data layer of the target data structure, the third data layer is located in the hard disk, and the third data layer supports additional write operations; or, when there is no key in the first ordered data layer When the key of the data to be read is stored, and the second ordered data layer does not exist in the target data structure, check whether the key of the data to be read is stored in the third ordered data layer; when the third data layer does not store the key of the data to be read When the key of the data is retrieved, it is checked whether the key of the data to be read is stored in the fourth ordered data layer of the target data structure, the fourth ordered data layer is located in the hard disk, and the fourth ordered data layer supports additional write operations.
可选地，第四有序数据层包括根节点、中间节点和叶子节点，根节点和中间节点均用于存储待操作数据的键，根节点能够存储的键的范围覆盖中间节点能够存储的键的范围，叶子节点存储有中间节点存储的键对应的键值对。Optionally, the fourth ordered data layer includes a root node, an intermediate node, and a leaf node. Both the root node and the intermediate node are used to store the keys of the data to be operated, and the range of the keys that the root node can store covers the keys that the intermediate nodes can store. The leaf node stores the key-value pair corresponding to the key stored by the intermediate node.
并且，根节点还存储有用于存储根节点存储的键所覆盖的键的中间节点的节点标识，中间节点中还存储有用于存储中间节点存储的键所覆盖的键的叶子节点的节点标识。In addition, the root node also stores the node identifier of the intermediate node used to store the key covered by the key stored by the root node, and the intermediate node also stores the node identifier of the leaf node used to store the key covered by the key stored by the intermediate node.
进一步地，键值存储***中还存储有哈希映射表，哈希映射表用于存储根节点、中间节点和叶子节点中至少一个节点存储的节点标识与节点标识指示的节点的存储位置信息的对应关系。Further, a hash mapping table is also stored in the key-value storage system, and the hash mapping table is used to store the node identifier stored by at least one node in the root node, the intermediate node and the leaf node and the storage location information of the node indicated by the node identifier. Correspondence.
可选地，第三数据层为无序数据层。Optionally, the third data layer is an unordered data layer.
可选地，第四有序数据层为树状数据结构，第四有序数据层的深度小于3。Optionally, the fourth ordered data layer is a tree-like data structure, and the depth of the fourth ordered data layer is less than 3.
可选地，更新后的第四有序数据层的存储位置与更新前的第四有序数据层的存储位置不同，且更新前的第四有序数据层和更新后的第四有序数据层使用不同版本号进行标识。Optionally, the storage location of the fourth ordered data layer after the update is different from the storage location of the fourth ordered data layer before the update, and the fourth ordered data layer before the update and the fourth ordered data layer after the update are different. Layers are identified using different version numbers.
可选地，键包括待操作数据的逻辑地址，逻辑地址通过数值表示，多个数据结构中同一数据结构中存储的键值对中逻辑地址对指定数值的余数相等，不同数据结构中存储的键值对中逻辑地址对指定数值的余数不等。Optionally, the key includes a logical address of the data to be operated, and the logical address is represented by a numerical value. The remainder of the logical address in the key-value pair stored in the same data structure in multiple data structures is equal to the specified value, and the keys stored in different data structures are equal. The remainder of the logical address pair in the value pair is unequal to the specified value.
可选地，键值存储***包括硬盘，硬盘中存储有日志文件，装置还包括：接收模块，用于接收快照删除请求；处理模块，还用于将快照删除请求写在日志文件中，并发送针对快照删除请求的操作完成响应；处理模块，还用于基于日志文件中记录的快照删除请求，执行快 照删除请求指示的快照删除操作。Optionally, the key-value storage system includes a hard disk in which log files are stored, and the device further includes: a receiving module for receiving a snapshot deletion request; a processing module for writing the snapshot deletion request in the log file and sending the request. A response to the operation of the snapshot deletion request is completed; the processing module is further configured to execute the snapshot deletion operation indicated by the snapshot deletion request based on the snapshot deletion request recorded in the log file.
第三方面，本申请提供了一种计算机设备，包括存储器和处理器，存储器存储有程序指令，处理器运行程序指令以执行本申请第一方面提供的方法。In a third aspect, the present application provides a computer device, including a memory and a processor, the memory stores program instructions, and the processor executes the program instructions to execute the method provided in the first aspect of the present application.
第四方面，本申请提供了一种计算机可读存储介质，该存储介质为非易失性计算机可读存储介质，该存储介质包括程序指令，当程序指令在计算机设备上运行时，使得计算机设备执行如本申请第一方面提供的方法。In a fourth aspect, the present application provides a computer-readable storage medium, the storage medium is a non-volatile computer-readable storage medium, the storage medium includes program instructions, and when the program instructions are executed on a computer device, the computer device A method as provided in the first aspect of the present application is performed.
第五方面，本申请提供了一种包含指令的计算机程序产品，当计算机程序产品在计算机上运行时，使得计算机执行本申请第一方面提供的方法。In a fifth aspect, the present application provides a computer program product comprising instructions, which, when the computer program product is run on a computer, causes the computer to execute the method provided in the first aspect of the present application.
附图说明Description of drawings
图1是本申请实施例提供的一种键值存储***的数据管理方法涉及的实施场景的示意图；1 is a schematic diagram of an implementation scenario involved in a data management method of a key-value storage system provided by an embodiment of the present application;
图2是本申请实施例提供的一种根据键值存储***的数据管理方法写数据的流程图；2 is a flow chart of writing data according to a data management method of a key-value storage system provided by an embodiment of the present application;
图3是本申请实施例提供的一种计算机***的结构示意图；3 is a schematic structural diagram of a computer system provided by an embodiment of the present application;
图4是本申请实施例提供的一种键值存储***的示意图；4 is a schematic diagram of a key-value storage system provided by an embodiment of the present application;
图5是本申请实施例提供的一种根据键值存储***的数据管理方法读数据的示意图；5 is a schematic diagram of reading data according to a data management method of a key-value storage system provided by an embodiment of the present application;
图6是本申请实施例提供的一种删除快照的方法流程图；6 is a flowchart of a method for deleting a snapshot provided by an embodiment of the present application;
图7是本申请实施例提供的一种键值存储***的数据管理装置的结构示意图；7 is a schematic structural diagram of a data management device of a key-value storage system provided by an embodiment of the present application;
图8是本申请实施例提供的另一种键值存储***的数据管理装置的结构示意图；8 is a schematic structural diagram of a data management device of another key-value storage system provided by an embodiment of the present application;
图9是本申请实施例提供的一种计算机设备的结构示意图。FIG. 9 is a schematic structural diagram of a computer device provided by an embodiment of the present application.
具体实施方式detailed description
为使本申请的目的、技术方案和优点更加清楚，下面将结合附图对本申请实施方式作进一步地详细描述。In order to make the objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.
为便于理解，下面先对本申请实施例中涉及的相关背景知识进行介绍。For ease of understanding, the relevant background knowledge involved in the embodiments of the present application is first introduced below.
键值(key value，KV)对是数据库最简单的组织形式。键值对中的键(key)指示用于查找用户写入的数据的物理存储地址的关键字，键值对中的值(value)指示该数据的物理存储地址，或者，键值对中的值是数据本身。在用户读数据的过程中，需要先查找到待读取数据的键，然后根据待读取数据的键找到该键指示的值，根据该值得到待读取数据，从而完成数据的读取操作。并且，键中的关键字可以根据应用需求进行设置。在一种可实现方式中，本申请实施例中的键包含数据的四元组，该四元组包括：逻辑单元号(logical unit number，LUN)标识、链接克隆(brunch)标识、逻辑块地址(logical block address，LBA)和快照(snap)标识。其中，快照标识用于指示根据用户写入的数据创建的快照。链接克隆标识用于指示实现键值存储***的虚拟机。A key-value (KV) pair is the simplest form of organization of a database. The key (key) in the key-value pair indicates the keyword used to find the physical storage address of the data written by the user, and the value (value) in the key-value pair indicates the physical storage address of the data, or, the value in the key-value pair The value is the data itself. In the process of reading data, the user needs to find the key of the data to be read first, then find the value indicated by the key according to the key of the data to be read, and obtain the data to be read according to the value, thereby completing the data reading operation . Moreover, the keywords in the keys can be set according to application requirements. In an implementation manner, the key in this embodiment of the present application includes a quadruple of data, where the quadruple includes: a logical unit number (logical unit number, LUN) identifier, a linked clone (brunch) identifier, and a logical block address (logical block address, LBA) and snapshot (snap) identification. The snapshot identifier is used to indicate a snapshot created according to the data written by the user. The linked clone ID is used to indicate the virtual machine that implements the key-value store system.
有序数据层是指该数据层中存储的数据是按照某种规律排序后存储在该数据层中的。在本申请实施例的一种可实现方式中，该排序规律是按照键的大小进行排序。The ordered data layer means that the data stored in the data layer is stored in the data layer after being sorted according to a certain rule. In an implementation manner of the embodiment of the present application, the sorting rule is sorting according to the size of the keys.
无序数据层是指该数据层中存储的数据不是按照规律进行排序后存储在该数据层中的。例如，无序数据层中存储的数据未按照键的大小进行排序，仅是按照写入该无序数据层的先后顺序存储在该无序数据层中的。An unordered data layer means that the data stored in the data layer is not stored in the data layer after being sorted according to the rules. For example, the data stored in the unordered data layer is not sorted according to the size of the keys, but is only stored in the unordered data layer according to the order of writing to the unordered data layer.
数据层支持更新写操作，是指在需要采用新的数据修改数据层中已存储的数据时，采用 新的数据覆盖数据层中已存储的数据。例如，当数据层中已存储有键为A的数据时，当写操作请求请求写入键为A的数据时，更新写是指在该数据层中已存储有键为A的数据的位置处，使用写操作请求请求写入的键为A的数据覆盖已存储的键为A的数据，从而实现对该键为A的数据的修改。The data layer supports the update write operation, which means that when the data stored in the data layer needs to be modified with new data, the new data is used to overwrite the data stored in the data layer. For example, when the data with the key A has been stored in the data layer, when the write operation request requests to write the data with the key A, the update write refers to the location where the data with the key A has been stored in the data layer , use the write operation request to request the written data with the key A to overwrite the stored data with the key A, so as to realize the modification of the data with the key A.
数据层支持追加写操作，是指在向该数据层中写数据时，可以将数据写在该数据层中所有已写入数据的尾部。例如，当数据层中已存储有键为A的数据时，当写操作请求请求写入键为A的数据时，更新写是指在该数据层中已存储有键为A的数据的位置处覆盖写入该写操作请求请求写入的数据。The data layer supports additional write operations, which means that when writing data to the data layer, the data can be written at the end of all written data in the data layer. For example, when the data with the key A has been stored in the data layer, when the write operation request requests to write the data with the key A, the update write refers to the location where the data with the key A has been stored in the data layer Overwrites the data requested to be written by this write operation.
第一存储单位和第二存储单位之间允许键值覆盖，是指第一存储单位能够存储的键的范围和第二存储范围能够存储的键的范围存在重叠的范围。例如，当数据层包括多个有序字符串表(sorted string table，SSTable)时，该数据层包括的多个SSTable存在键值覆盖，是指该多个SSTable能够存储的键的范围存在重叠的范围。The allowable key value coverage between the first storage unit and the second storage unit refers to the overlapping range between the range of keys that the first storage unit can store and the range of keys that the second storage range can store. For example, when the data layer includes multiple sorted string tables (SSTables), the multiple SSTables included in the data layer have key value coverage, which means that the ranges of keys that the multiple SSTables can store overlap. Scope.
第二存储单位被第一存储单位键值覆盖，是指第二存储单位能够存储的键的范围位于第一存储单位能够存储的键的范围内。例如，若第一存储单位能够存储的键的范围为[1,3]，第二存储单位能够存储的键的范围为[1.5,2.1]，范围[1.5,2.1]位于范围[1,3]内，则可知第二存储单位被第一存储单位键值覆盖。The second storage unit is covered by the key value of the first storage unit, which means that the range of keys that the second storage unit can store is within the range of keys that the first storage unit can store. For example, if the range of keys that the first storage unit can store is [1,3], the range of keys that the second storage unit can store is [1.5,2.1], and the range [1.5,2.1] is in the range [1,3] inside, it can be known that the second storage unit is covered by the key value of the first storage unit.
图1是本申请实施例提供的键值存储***的数据管理方法所涉及的应用场景的结构示意图。如图1所示，应用场景包括计算机***10、客户端20以及存储池30。存储池30用于存储客户端20指示需要存储的数据。其中，客户端20与计算机***10之间通过网络连接，客户端20可以通过网络向计算机***10发送输入/输出(input/output，I/O)操作请求。计算机***10与存储池30之间通过网络连接，计算机***10可以通过网络与该存储池30进行数据交互，以便于实现客户端20发送的I/O操作请求所请求的I/O操作。FIG. 1 is a schematic structural diagram of an application scenario involved in a data management method of a key-value storage system provided by an embodiment of the present application. As shown in FIG. 1 , the application scenario includes a computer system 10 , a client 20 and a storage pool 30 . The storage pool 30 is used to store the data indicated by the client 20 to be stored. The client 20 and the computer system 10 are connected through a network, and the client 20 can send an input/output (I/O) operation request to the computer system 10 through the network. The computer system 10 and the storage pool 30 are connected through a network, and the computer system 10 can perform data interaction with the storage pool 30 through the network, so as to realize the I/O operation requested by the I/O operation request sent by the client 20 .
在一种可实现方式中，客户端20可以通过网络向计算机***10发送写操作请求，计算机***10可以根据该写操作请求，将写操作请求携带的待写入数据写入存储池30中。或者，客户端20可以通过网络向计算机***10发送读操作请求，计算机***10可以根据读操作请求，从存储池30中读出读操作请求请求读取的数据，并向客户端20发送从存储池30中读取的数据。In an implementation manner, the client 20 may send a write operation request to the computer system 10 through the network, and the computer system 10 may write the to-be-written data carried by the write operation request into the storage pool 30 according to the write operation request. Alternatively, the client 20 can send a read operation request to the computer system 10 through the network, and the computer system 10 can read the data read by the read operation request from the storage pool 30 according to the read operation request, and send the data from the storage pool 30 to the client 20 Data read in pool 30.
可选地，图1中的计算机***10可以是包括若干台主机和/或计算设备的集群。存储池30可以是包括若干个存储设备的存储设备集群。客户端20例如为手机、平板电脑、虚拟机、容器、个人电脑等，在本申请实施例中客户端20可以是任何需从计算机***10获取key对应value的设备。客户端20的数量可以是多个，计算机***10可同时处理多个客户端20发送的I/O操作请求，实现并发处理。Alternatively, the computer system 10 in Figure 1 may be a cluster comprising several hosts and/or computing devices. The storage pool 30 may be a storage device cluster including several storage devices. The client 20 is, for example, a mobile phone, a tablet computer, a virtual machine, a container, a personal computer, etc. In this embodiment of the present application, the client 20 may be any device that needs to obtain a value corresponding to a key from the computer system 10 . The number of clients 20 can be multiple, and the computer system 10 can process I/O operation requests sent by multiple clients 20 at the same time, so as to realize concurrent processing.
在本申请实施例中，计算机***10中部署有键值存储***。该键值存储***用于采用键值对的方式，对存储池中存储的数据或数据的元数据进行管理。可选地，键值存储***可以通过部署在该计算机***10中的物理机、虚拟机或容器实现。In the embodiment of the present application, a key-value storage system is deployed in the computer system 10 . The key-value storage system is used to manage the data or metadata of the data stored in the storage pool by means of key-value pairs. Optionally, the key-value storage system can be implemented by a physical machine, a virtual machine or a container deployed in the computer system 10 .
需要说明的是，图1所示的应用场景是本申请实施例提供的键值存储***的数据管理方法所涉及的应用场景的一种示例，并不用于限定本申请实施例提供的键值存储***的数据管理方法所涉及的应用场景，该应用场景还可以具有其它类型的架构或变形。It should be noted that the application scenario shown in FIG. 1 is an example of an application scenario involved in the data management method of the key-value storage system provided by the embodiment of the present application, and is not intended to limit the key-value storage provided by the embodiment of the present application The application scenarios involved in the data management method of the system may also have other types of structures or variants.
本申请实施例提供了一种键值存储***的数据管理方法。该键值存储***的逻辑地址空间可以包括多个子逻辑地址空间，该多个子逻辑地址空间互不重叠，且多个子逻辑地址空间中存储的数据可以分别使用多个数据结构一一对应管理。在该方法中，通过获取I/O操作请求，基于I/O操作请求中携带的待操作数据的索引信息，在用于管理键值存储***的逻辑地址空间的多个数据结构中，确定用于存储待操作数据的键值对的目标数据结构，并在目标数据结构中对键值对进行I/O操作。The embodiment of the present application provides a data management method of a key-value storage system. The logical address space of the key-value storage system may include multiple sub-logical address spaces, the multiple sub-logical address spaces do not overlap with each other, and the data stored in the multiple sub-logical address spaces may be managed in a one-to-one correspondence with multiple data structures. In this method, by obtaining the I/O operation request, based on the index information of the data to be operated carried in the I/O operation request, among the multiple data structures used to manage the logical address space of the key-value storage system, determine the A target data structure for storing key-value pairs of data to be operated, and performing I/O operations on the key-value pairs in the target data structure.
由于该键值存储***中的多个子逻辑地址空间分别由多个数据结构进行管理，相较于采用一个数据结构管理键值存储***的逻辑地址空间，减小了该多个数据结构中每个数据结构需要存储的数据量，提高了该键值存储***根据该数据结构对I/O操作请求进行响应的效率和键值存储***的性能，能够通过该数据结构向用户提供稳定且可预测的读性能。Since the multiple sub-logical address spaces in the key-value storage system are managed by multiple data structures, compared with using one data structure to manage the logical address space of the key-value storage system, each of the multiple data structures is reduced in size. The amount of data that the data structure needs to store improves the efficiency of the key-value storage system in responding to I/O operation requests according to the data structure and the performance of the key-value storage system, and can provide users with stable and predictable data through the data structure. read performance.
并且，由于键值存储***的多个子逻辑地址空间分别由多个数据结构管理，键值存储***能够通过该多个数据结构提供存储服务，提高该键值存储***的并行处理能力，提高了键值存储***的服务性能。Moreover, since the multiple sub-logical address spaces of the key-value storage system are managed by multiple data structures, the key-value storage system can provide storage services through the multiple data structures, improve the parallel processing capability of the key-value storage system, and improve the key-value storage system. Service performance of the value store system.
可选地，将键值存储***的逻辑地址空间划分为多个子逻辑地址空间时，其划分方式和划分得到的子逻辑地址空间的总数，可以根据应用需求设置。例如，可以根据应用需求，将该键值存储***的逻辑地址空间平均地划分为多个子逻辑地址空间。或者，根据键值存储***的最大存储容量和应用需求，确定每个子逻辑地址空间的最大存储容量和该键值存储***中子逻辑地址空间的总数，然后按照该总数和多个子逻辑地址空间各自的最大存储容量，划分得到多个子逻辑地址空间。Optionally, when the logical address space of the key-value storage system is divided into multiple sub-logical address spaces, the division method and the total number of sub-logical address spaces obtained by division can be set according to application requirements. For example, the logical address space of the key-value storage system can be evenly divided into multiple sub-logical address spaces according to application requirements. Or, according to the maximum storage capacity and application requirements of the key-value storage system, determine the maximum storage capacity of each sub-logical address space and the total number of sub-logical address spaces in the key-value storage system, and then according to the total and multiple sub-logical address spaces respectively The maximum storage capacity is divided into multiple sub-logical address spaces.
在一种可实现方式中，当采用键值存储***管理存储池中存储的数据的元数据时，可以根据应用需求确定元数据与通过该元数据管理的数据的比例，并根据该比例和存储池的最大存储容量，确定该键值存储***的最大存储容量，并根据该键值存储***的最大存储容量和每个子逻辑地址空间的最大存储容量，确定键值存储***包括的子逻辑地址空间的总数，并根据该总数和每个子逻辑地址空间的最大存储容量，对键值存储***的逻辑地址空间进行划分。In an implementation manner, when the key-value storage system is used to manage the metadata of the data stored in the storage pool, the ratio of the metadata to the data managed by the metadata can be determined according to the application requirements, and the ratio and the storage The maximum storage capacity of the pool, determine the maximum storage capacity of the key-value storage system, and determine the sub-logical address space included in the key-value storage system according to the maximum storage capacity of the key-value storage system and the maximum storage capacity of each sub-logical address space and divide the logical address space of the key-value storage system according to the total number and the maximum storage capacity of each sub-logical address space.
并且，为了保证键值存储空间对I/O操作请求的处理效率，可以通过合理划分子逻辑地址空间，控制用于管理子逻辑地址空间的数据结构的存储容量。例如，当用于管理子逻辑地址空间的数据结构为树状数据结构时，可以根据该树状数据结构中的每个数据层的最大存储容量确定该树状数据结构的最大存储容量，然后根据该键值存储***的最大存储容量和树状数据结构的最大存储容量，对键值存储***的逻辑地址空间进行划分，以控制用于管理子逻辑地址空间的数据结构的深度小于指定深度，使得数据结构保持为较低的层次及数据结构中的数据层保持为较小的总层数。从而以保证I/O操作的整体开销可控，提供稳定的I/O性能。In addition, in order to ensure the processing efficiency of the I/O operation request by the key-value storage space, the storage capacity of the data structure for managing the sub-logical address space can be controlled by reasonably dividing the sub-logical address space. For example, when the data structure for managing the sub-logical address space is a tree-like data structure, the maximum storage capacity of the tree-like data structure can be determined according to the maximum storage capacity of each data layer in the tree-like data structure, and then according to the maximum storage capacity of each data layer in the tree-like data structure The maximum storage capacity of the key-value storage system and the maximum storage capacity of the tree-like data structure, the logical address space of the key-value storage system is divided to control the depth of the data structure used to manage the sub-logical address space to be less than the specified depth, so that The data structure is kept at a lower level and the data layers within the data structure are kept at a small total number of levels. Thereby, the overall cost of the I/O operation is controllable and stable I/O performance is provided.
I/O操作请求包括写操作请求和读操作请求。下面分别对根据写操作请求向键值存储***中写数据的过程，及根据读操作请求从键值存储***中读数据的过程进行说明。The I/O operation request includes a write operation request and a read operation request. The process of writing data to the key-value storage system according to the write operation request and the process of reading data from the key-value storage system according to the read operation request will be described below.
图2是本申请实施例提供的一种根据写操作请求向键值存储***中写入待写入数据的方法流程图。如图2所示，该写数据的过程包括：FIG. 2 is a flowchart of a method for writing data to be written into a key-value storage system according to a write operation request provided by an embodiment of the present application. As shown in Figure 2, the process of writing data includes:
步骤201、计算机***获取写操作请求，写操作请求中携带有待写入数据和待写入数据 的索引信息。 Step 201, the computer system obtains a write operation request, and the write operation request carries the data to be written and the index information of the data to be written.
用户需要向存储池中写入数据时，可以通过客户端向计算机***发送写操作请求，以便于计算机***根据该写操作请求执行数据写入操作。例如，当用户使用云盘时，用户可以通过云盘的客户端向计算机***发送写操作请求，该写操作请求可以携带有待写入数据和待写入数据的索引信息。When the user needs to write data into the storage pool, the client can send a write operation request to the computer system, so that the computer system can perform the data write operation according to the write operation request. For example, when a user uses a cloud disk, the user may send a write operation request to the computer system through the client of the cloud disk, and the write operation request may carry the data to be written and the index information of the data to be written.
其中，索引信息为用于索引待写入数据的键的信息。在一种可实现方式中，索引信息可以是键本身。例如，索引信息和键均包括I/O操作请求的发起者的标识和待操作数据的逻辑地址。可选地，索引信息和键还可以包括I/O操作请求的发起者的链接克隆标识和快照标识，即索引信息和键均可以包括四元组，该四元组包括：I/O操作请求的发起者的逻辑单元号标识、链接克隆标识、逻辑块地址和快照标识。其中，I/O操作请求的发起者的链接克隆标识用于指示针对该发起者生成的连接克隆。I/O操作请求的发起者的快照标识用于指示根据该发起者生成的快照。在一种实现场景中，当用户使用云盘时，该I/O操作请求的发起者为云盘，则索引信息和键均包括云盘的标识、待操作数据的逻辑地址、云盘的链接克隆标识和快照标识。The index information is information of a key used to index the data to be written. In one implementation, the index information may be the keys themselves. For example, both the index information and the key include the identifier of the initiator of the I/O operation request and the logical address of the data to be operated. Optionally, the index information and the key may also include the linked clone identifier and the snapshot identifier of the initiator of the I/O operation request, that is, the index information and the key may both include a quadruple, and the quadruple includes: the I/O operation request. The originator's LUN ID, Linked Clone ID, Logical Block Address, and Snapshot ID. The linked clone identifier of the initiator of the I/O operation request is used to indicate the linked clone generated for the initiator. The snapshot identifier of the initiator of the I/O operation request is used to indicate the snapshot generated according to the initiator. In an implementation scenario, when a user uses a cloud disk, the initiator of the I/O operation request is the cloud disk, and the index information and key include the identifier of the cloud disk, the logical address of the data to be operated, and the link of the cloud disk Clone ID and Snapshot ID.
在另一种可实现方式中，索引信息包括用于索引键的关键字，键可以根据该关键字索引得到。例如，键包括I/O操作请求的发起者的标识和待操作数据的逻辑地址，索引信息为用于索引到该I/O操作请求的发起者的标识和待操作数据的逻辑地址的关键字。在该实现方式中，关键字的实现方式可以根据应用需求确定，且键包括的内容也可以根据应用需求进行调整。In another implementation manner, the index information includes a keyword for indexing the key, and the key can be obtained by indexing the keyword. For example, the key includes the identifier of the initiator of the I/O operation request and the logical address of the data to be operated, and the index information is a keyword for indexing to the identifier of the initiator of the I/O operation request and the logical address of the data to be operated. . In this implementation manner, the implementation manner of the keyword can be determined according to the application requirements, and the content included in the key can also be adjusted according to the application requirements.
步骤202、计算机***根据写操作请求，确定待写入数据的键值对，得到携带有待写入数据和待写入数据的键值对的写请求。Step 202: The computer system determines the key-value pair of the data to be written according to the write operation request, and obtains a write request carrying the data to be written and the key-value pair of the data to be written.
在本申请实施例中，计算机***中部署有键值存储***。该键值存储***用于采用键值对的方式，对存储池中存储的数据或存储池中存储的数据的元数据进行管理。因此，计算机***获取写操作请求后，可以根据该写操作请求，确定该待写入数据的键值对，以便于键值存储***根据该键值对对待写入数据或待写入数据的元数据进行管理。In the embodiment of the present application, a key-value storage system is deployed in the computer system. The key-value storage system is used to manage the data stored in the storage pool or the metadata of the data stored in the storage pool in the form of key-value pairs. Therefore, after obtaining the write operation request, the computer system can determine the key-value pair of the data to be written according to the write operation request, so that the key-value storage system can determine the data to be written or the element of the data to be written according to the key-value pair. data is managed.
其中，键值对中的键指示用于查找用户写入的数据的物理存储地址的关键字，键值对中的值指示该数据的物理存储地址，或者，键值对中的值是数据本身。并且，当键值对中的值是待写入数据的物理存储地址时，键值存储***用于对待写入数据的元数据进行管理。当键值对中的值是待写入数据本身时，键值存储***用于对待写入数据进行管理。相应的，确定待写入数据的键值对的过程包括分别确定键值对中的键和值的过程。The key in the key-value pair indicates the keyword used to find the physical storage address of the data written by the user, the value in the key-value pair indicates the physical storage address of the data, or the value in the key-value pair is the data itself . And, when the value in the key-value pair is the physical storage address of the data to be written, the key-value storage system is used to manage the metadata of the data to be written. When the value in the key-value pair is the data to be written itself, the key-value storage system is used to manage the data to be written. Correspondingly, the process of determining the key-value pair of the data to be written includes the process of respectively determining the key and the value in the key-value pair.
根据步骤201中的内容可知，键可以根据索引信息确定。并且，根据步骤201中描述的索引信息和键之间的关系，根据索引信息确定键的实现方式至少包括以下两种：According to the content in step 201, the key can be determined according to the index information. And, according to the relationship between the index information and the key described in step 201, the implementation manner of determining the key according to the index information includes at least the following two:
在一种可实现方式中，可以将索引信息本身确定为键。例如，当索引信息包括四元组，且四元组包括云盘的标识、待操作数据的逻辑地址、云盘的链接克隆标识和快照标识时，可以直接将该四元组确定为键。In one implementation, the index information itself may be determined as the key. For example, when the index information includes a quadruple, and the quadruple includes the identifier of the cloud disk, the logical address of the data to be operated, the linked clone identifier and the snapshot identifier of the cloud disk, the quadruple can be directly determined as the key.
在另一种可实现方式中，当索引信息包括用于索引键的关键字时，在基于索引信息，确定待操作数据的键时，可以基于关键字进行索引，以得到键。例如，索引信息包括用于索引键的关键字时，可以基于该关键字索引得到键包括云盘的标识、待操作数据的逻辑地址、云盘的链接克隆标识和快照标识。In another implementation manner, when the index information includes a keyword for indexing the key, when determining the key of the data to be operated based on the index information, the index may be performed based on the keyword to obtain the key. For example, when the index information includes a keyword for indexing a key, the key can be obtained based on the keyword index, including the identifier of the cloud disk, the logical address of the data to be operated, the linked clone identifier and the snapshot identifier of the cloud disk.
另外，在本申请实施例中，键值对中的值可以指示待写入数据的物理存储地址，或者， 值为待写入数据本身。则针对键值对中值的不同实现方式，根据写操作请求，确定待写入数据的键值对，还包括：当值指示待写入数据的物理存储地址时，可以根据键值存储***的实现逻辑确定待写入数据的物理存储地址，并将该物理存储地址确定为键值对中的值，当值为待写入数据本身时，可以将该待写入请求中携带的待写入数据确定为键值对中的值。In addition, in this embodiment of the present application, the value in the key-value pair may indicate the physical storage address of the data to be written, or the value may be the data to be written itself. Then, for different implementations of the value in the key-value pair, according to the write operation request, determine the key-value pair of the data to be written, and further include: when the value indicates the physical storage address of the data to be written, the key-value storage system The implementation logic determines the physical storage address of the data to be written, and determines the physical storage address as the value in the key-value pair. When the value is the data to be written itself, the to-be-written carried in the to-be-written request can be used. Data is determined as values in key-value pairs.
在一种可实现方式中，如图3所示，计算机***10包括：I/O操作请求处理模块101和键值存储***102。该I/O操作请求处理模块101用于根据获取到的写操作请求，确定待写入数据的键值对，得到携带有待写入数据和键值对的写请求，并将该写请求发送至键值存储***。键值存储***102用于根据该写请求，对待写入数据的元数据或待写入数据本身进行管理。In an implementation manner, as shown in FIG. 3 , the computer system 10 includes: an I/O operation request processing module 101 and a key-value storage system 102 . The I/O operation request processing module 101 is configured to determine the key-value pair of the data to be written according to the obtained write operation request, obtain a write request carrying the data to be written and the key-value pair, and send the write request to Key-value storage system. The key-value storage system 102 is configured to manage the metadata of the data to be written or the data to be written itself according to the write request.
步骤203、计算机***将写请求写入数据日志。 Step 203, the computer system writes the write request into the data log.
在本申请实施例中，计算机***配置有日志文件，该日志文件可以包括数据日志。该日志文件可以部署在非易失存储介质上，以保证日志文件中存储的内容在出现掉电等故障后不会丢失。例如，如图4所示，硬盘上部署有日志文件a，该日志文件a包括数据日志a1。计算机***得到写请求后，可以先将该写请求写入数据日志中，再根据写请求将键值对存储在数据结构中，以保证写数据操作的故障可恢复以及***的数据一致性和可靠性。在一种可实现方式中，用于管理键值存储***的多个子逻辑地址空间的多个数据结构共用一个数据日志，这样可以降低键值存储***的实现成本。In this embodiment of the present application, the computer system is configured with a log file, and the log file may include a data log. The log file can be deployed on a non-volatile storage medium to ensure that the content stored in the log file will not be lost in the event of a power failure or other failure. For example, as shown in FIG. 4, a log file a is deployed on the hard disk, and the log file a includes the data log a1. After the computer system gets the write request, it can first write the write request into the data log, and then store the key-value pair in the data structure according to the write request to ensure that the failure of the write data operation can be recovered and the data of the system is consistent and reliable. sex. In an implementation manner, multiple data structures used to manage multiple sub-logical address spaces of the key-value storage system share a data log, which can reduce the implementation cost of the key-value storage system.
在一种可实现方式中，如图3所示，键值存储***102包括：日志模块1021。将写请求写入数据日志的操作可以由该日志模块1021执行。并且，如图3所示，该键值存储***102还包括：故障恢复模块1022，该故障恢复模块1022用于在计算机***出现故障后，通过重放日志文件，恢复异常关闭的***和数据，以实现故障恢复。In an implementation manner, as shown in FIG. 3 , the key-value storage system 102 includes: a log module 1021 . The operation of writing the write request to the data log can be performed by the log module 1021 . And, as shown in FIG. 3, the key-value storage system 102 further includes: a fault recovery module 1022, the fault recovery module 1022 is used to restore the abnormally shut down system and data by replaying the log file after the computer system fails, for failure recovery.
并且，为进一步保证数据的安全性，该键值存储***中还设置有缓存机制。缓存可以部署在非易失性存储介质上。在键值存储***接收到写操作请求或写请求后，可以先将该写操作请求或写请求写入缓存中，然后从缓存中将写操作请求或写请求写入日志文件中。由于写缓存的速度快于写日志文件的速度，通过该实现方式，能够避免在写日志文件的过程中键值存储***出现故障导致的数据丢失。Moreover, in order to further ensure the security of data, a cache mechanism is also set in the key-value storage system. Caches can be deployed on non-volatile storage media. After the key-value storage system receives the write operation request or write request, it can first write the write operation request or write request into the cache, and then write the write operation request or write request from the cache into the log file. Since the speed of writing the cache is faster than the speed of writing the log file, this implementation can avoid data loss caused by the failure of the key-value storage system during the process of writing the log file.
相应的，如图3所示，键值存储***102还包括：日志缓存模块1023。该日志缓存模块1023用于缓存写操作请求或写请求，以便于从缓存中将写操作请求或写请求写入日志文件中。Correspondingly, as shown in FIG. 3 , the key-value storage system 102 further includes: a log cache module 1023 . The log cache module 1023 is configured to cache the write operation request or the write request, so as to write the write operation request or the write request into the log file from the cache.
步骤204、计算机***将写请求写入数据日志后，发送针对写操作请求的写操作响应。Step 204: After the computer system writes the write request into the data log, it sends a write operation response to the write operation request.
在将写请求写入数据日志后，写请求就不会因掉电等故障后丢失，就能够保证完成该写请求所请求的写操作。因此，可以在将写请求写入数据日志后，发送针对写操作请求的写操作响应，以指示完成该写操作请求指示的写操作。通过在将写请求写入数据日志后立即发送写操作响应，能够实现对客户端的快速响应，提高用户体验。After the write request is written to the data log, the write request will not be lost due to failures such as power failure, and the write operation requested by the write request can be guaranteed to be completed. Therefore, after the write request is written into the data log, a write operation response to the write operation request may be sent to indicate that the write operation indicated by the write operation request is completed. By sending the write operation response immediately after the write request is written into the data log, a quick response to the client can be achieved and user experience can be improved.
步骤205、计算机***基于键在多个数据结构中确定用于存储待写入数据的键值对的目标数据结构。 Step 205 , the computer system determines a target data structure for storing the key-value pair of the data to be written in the plurality of data structures based on the key.
在一种可实现方式中，当键包括待操作数据的逻辑地址，逻辑地址通过数值表示时，在基于键确定用于存储待写入数据的键值对的目标数据结构时，可以通过对逻辑地址取余数实现。例如，基于键确定用于存储待写入数据的键值对的目标数据结构的规则可以为：多个数据结构中同一数据结构中存储的键值对中逻辑地址对指定数值的余数相等，不同数据结构中 存储的键值对中逻辑地址对指定数值的余数不等。In an implementation manner, when the key includes the logical address of the data to be operated, and the logical address is represented by a numerical value, when the target data structure for storing the key-value pair of the data to be written is determined based on the key, the logical address can be Address taking the remainder to achieve. For example, the rule for determining the target data structure for storing the key-value pair of the data to be written based on the key may be: the remainder of the logical address pair specified value in the key-value pair stored in the same data structure of multiple data structures is equal, and the The remainder of the logical address pair specified in the key-value pair stored in the data structure is unequal.
例如，假设键值存储***的逻辑地址空间包括5个子逻辑地址空间，该5个子逻辑地址空间中存储的数据分别使用5个数据结构进行管理，且该5个数据结构存储的键值对中逻辑地址对指定数值的余数分别为1、2、3、4和5，则当键值对中逻辑地址对指定数值的余数为1时，使用第一个数据结构对该写操作请求中待写入数据或其元数据进行管理，当键值对中逻辑地址对指定数值的余数为2时，使用第二个数据结构对该写操作请求中待写入数据或其元数据进行管理，当键值对中逻辑地址对指定数值的余数为3时，使用第三个数据结构对该写操作请求中待写入数据或其元数据进行管理，当键值对中逻辑地址对指定数值的余数为4时，使用第四个数据结构对该写操作请求中待写入数据或其元数据进行管理，当键值对中逻辑地址对指定数值的余数为5时，使用第五个数据结构对该写操作请求中待写入数据或其元数据进行管理。For example, suppose that the logical address space of the key-value storage system includes five sub-logical address spaces, the data stored in the five sub-logical address spaces are managed by five data structures, and the key-value pairs stored in the five data structures are logically The remainder of the specified value of the address pair is 1, 2, 3, 4, and 5, respectively, then when the remainder of the specified value of the logical address pair in the key-value pair is 1, the first data structure is used to write in the write operation request. The data or its metadata is managed. When the remainder of the specified value of the logical address pair in the key-value pair is 2, the second data structure is used to manage the data to be written or its metadata in the write operation request. When the remainder of the specified value of the logical address in the pair is 3, use the third data structure to manage the data to be written in the write operation request or its metadata, when the remainder of the logical address in the key-value pair to the specified value is 4 , use the fourth data structure to manage the data to be written or its metadata in the write operation request, and use the fifth data structure when the remainder of the specified value of the logical address pair in the key-value pair is 5 Manage the data to be written or its metadata in the operation request.
可选的，该规则可以为键值存储***中预置的规则，也可以是根据需求配置的规则。例如，键值存储***中可以预留应用程序接口，以通过该应用程序接口根据应用需求配置该规则。Optionally, the rule may be a preset rule in the key-value storage system, or a rule configured according to requirements. For example, an application program interface may be reserved in the key-value storage system, through which the rule can be configured according to application requirements.
通过在多个数据结构中确定用于存储待写入数据的键值对的目标数据结构，能够将不同的写操作请求较均匀地分配到不同的数据结构上，能够使不同数据结构中存储的数据的总容量处于较均衡的状态，能够保证键值存储***的整体性能。并且，在该确定目标数据结构的实现方式中，通过执行对键值对中逻辑地址取余的一次操作就能够确定目标数据结构，其实现过程简单，能够简化写操作的实现过程。By determining a target data structure for storing key-value pairs of data to be written in multiple data structures, different write operation requests can be allocated to different data structures more evenly, and the data stored in different data structures can be The total capacity of data is in a relatively balanced state, which can ensure the overall performance of the key-value storage system. In addition, in the implementation manner of determining the target data structure, the target data structure can be determined by performing one operation of taking the remainder of the logical address in the key-value pair, the implementation process is simple, and the implementation process of the write operation can be simplified.
其中，该规则可以看成是将写操作请求打散到多个数据结构中的目标数据结构进行处理，因此该规则可称为“静态打散规则”。且该规则可以由功能模块实现，相应的，如图4所示，该功能模块可以称为“静态索引层”。在一种可实现方式中，该静态索引层的功能可以通过图3中I/O操作请求处理模块101实现。并且，如图3所示，键值存储***102还包括：空间管理模块1024，该空间管理模块1024用于确定用于存储该写操作请求的键值对的物理存储空间。并且，该空间管理模块1024还用于管理存储空间的使用状态。例如，空间管理模块1024可以根据存储空间的使用状态，将存储空间标记为已使用或未使用，以便于为未使用的存储空间进行垃圾回收。Among them, the rule can be regarded as a target data structure that smashes the write operation request into multiple data structures for processing, so the rule can be called a "static smashing rule". And the rule can be implemented by a functional module, correspondingly, as shown in FIG. 4 , the functional module can be called a “static index layer”. In an implementation manner, the function of the static index layer can be implemented by the I/O operation request processing module 101 in FIG. 3 . And, as shown in FIG. 3 , the key-value storage system 102 further includes: a space management module 1024, where the space management module 1024 is configured to determine a physical storage space for storing the key-value pair requested by the write operation. In addition, the space management module 1024 is also used to manage the usage status of the storage space. For example, the space management module 1024 may mark the storage space as used or unused according to the usage status of the storage space, so as to facilitate garbage collection for the unused storage space.
步骤206、将待写入数据的键值对存储在目标数据结构的第一有序数据层中。Step 206: Store the key-value pair of the data to be written in the first ordered data layer of the target data structure.
如图4所示，本申请实施例提供的数据结构包括第一有序数据层、第二有序数据层、第三数据层和第四有序数据层。其中，第一有序数据层和第二有序数据层位于内存中，第三数据层和第四有序数据层位于硬盘中。该硬盘可以是固态硬盘(solid state drive，SSD)。待写入数据的键值对写入数据结构的过程可以看成是在第一有序数据层、第二有序数据层、第三数据层和第四有序数据层中传递，并存储在第四有序数据层中的过程。因此，计算机***确定用于存储待写入数据的键值对的目标数据结构后，可以先将该待写入数据的键值对写入该目标数据结构的第一有序数据层中。As shown in FIG. 4 , the data structure provided by the embodiment of the present application includes a first ordered data layer, a second ordered data layer, a third ordered data layer, and a fourth ordered data layer. The first ordered data layer and the second ordered data layer are located in the memory, and the third ordered data layer and the fourth ordered data layer are located in the hard disk. The hard disk may be a solid state drive (solid state drive, SSD). The process of writing the key-value pair of the data to be written into the data structure can be regarded as being transmitted in the first ordered data layer, the second ordered data layer, the third data layer and the fourth ordered data layer, and stored in the Processes in the fourth ordered data layer. Therefore, after the computer system determines the target data structure for storing the key-value pair of the data to be written, it can first write the key-value pair of the data to be written into the first ordered data layer of the target data structure.
该第一有序数据层支持更新写操作。并且，在向第一有序数据层中写键值对的过程中，可以使用链表结构(如跳表结构)对向该第一有序数据层中写入的键值对进行排序，以便于将键值对有序地写入该第一有序数据层。其中，在对键值对进行排序时，可以按照键的大小顺序进行。例如，当键包括上述四元组时，该四元组可以表示为64位的数字串，可以根据键 的数字串的大小对键进行排序。The first ordered data layer supports update write operations. In addition, in the process of writing key-value pairs to the first ordered data layer, a linked list structure (such as a skip list structure) can be used to sort the key-value pairs written to the first ordered data layer, so as to facilitate The key-value pairs are written to the first ordered data layer in order. Among them, when sorting the key-value pair, it can be carried out according to the order of the size of the keys. For example, when the key includes the above-mentioned quadruple, the quadruple can be represented as a 64-bit numeric string, and the keys can be sorted according to the size of the numeric string of the key.
在一种可实现方式中，第一有序数据层包括多个SSTable。该第一有序数据层使用该多个有序字符串表有序地存储键值对。可选地，该第一有序数据层可以为内存表(memTable)。In one implementation, the first ordered data layer includes a plurality of SSTables. The first ordered data layer uses the plurality of ordered string tables to store key-value pairs in an orderly manner. Optionally, the first ordered data layer may be a memTable.
步骤207、当达到参考条件时，计算机***将第一有序数据层转换为第二有序数据层。Step 207: When the reference condition is reached, the computer system converts the first ordered data layer into the second ordered data layer.
在一种可实现方式中，参考条件可以为第一有序数据层的数据量达到第一数据量阈值，此时，计算机***可以将该第一有序数据层转化为第二有序数据层，以待将该第二有序数据层中的键值对写入硬盘中，并使用新的第一有序数据层存储键值对，以便于能够持续地为用户提供写服务。其中，新的第一有序数据层是在达到第一参考条件时新生成的第一有序数据层，且该新的有序数据层是空的数据层。In an implementation manner, the reference condition may be that the data volume of the first ordered data layer reaches a first data volume threshold, and at this time, the computer system may convert the first ordered data layer into the second ordered data layer , to write the key-value pairs in the second ordered data layer into the hard disk, and use the new first ordered data layer to store the key-value pairs, so as to continuously provide write services for users. The new first ordered data layer is a first ordered data layer newly generated when the first reference condition is reached, and the new ordered data layer is an empty data layer.
对应于该实现方式，在向第一有序数据层中写数据的过程中，需要先判断该第一有序数据层中的数据是否达到第一数据量阈值，当该第一有序数据层中的数据未达到第一数据量阈值时，向该第一有序数据层中写入数据，当该第一有序数据层中的数据达到第一数据量阈值时，将该第一有序数据层转换为第二有序数据层，并生成新的第一有序数据层，然后将数据写入新的第一有序数据层。Corresponding to this implementation, in the process of writing data to the first ordered data layer, it is necessary to first determine whether the data in the first ordered data layer reaches the first data volume threshold. When the data in the first ordered data layer does not reach the first data volume threshold, write data into the first ordered data layer, and when the data in the first ordered data layer reaches the first data volume threshold, write the first ordered data layer The data layer is converted to a second ordered data layer, and a new first ordered data layer is generated, and then data is written to the new first ordered data layer.
在另一种可实现方式中，参考条件可以为计算机***中的数据日志的容量达到参考容量阈值。数据日志的容量可以根据为键值存储***分配的内存容量匹配设置，当数据日志的容量达到参考容量阈值时，说明为该键值存储***分配的内存容量即将耗尽，则需要将内存中存储的键值对写入硬盘中。当用于管理键值存储***的多个子逻辑地址空间的多个数据结构共用一个数据日志时，若该数据日志的容量达到参考容量阈值，说明该多个数据结构使用的内存容量即将耗尽，则可将该多个数据结构中的第一有序数据层均转换为第二有序数据层。当用于管理每个子逻辑地址空间的数据结构单独使用一个数据日志时，当任一数据结构使用的数据日志的容量达到参考容量阈值，说明该数据结构使用的内存容量即将耗尽，则可将该数据结构中的第一有序数据层均转换为第二有序数据层。In another implementation manner, the reference condition may be that the capacity of the data log in the computer system reaches a reference capacity threshold. The capacity of the data log can be set according to the memory capacity allocated for the key-value storage system. When the capacity of the data log reaches the reference capacity threshold, it means that the memory capacity allocated for the key-value storage system is about to be exhausted, and the memory needs to be stored in the memory. The key-value pair is written to the hard disk. When multiple data structures used to manage multiple sub-logical address spaces of the key-value storage system share a data log, if the capacity of the data log reaches the reference capacity threshold, it means that the memory capacity used by the multiple data structures is about to be exhausted. Then, the first ordered data layers in the multiple data structures can all be converted into second ordered data layers. When the data structure used to manage each sub-logical address space uses a separate data log, when the capacity of the data log used by any data structure reaches the reference capacity threshold, indicating that the memory capacity used by the data structure is about to be exhausted, the The first ordered data layer in the data structure is converted into the second ordered data layer.
并且，在数据日志的容量达到参考容量阈值后，为避免键值存储***提供的写服务出现阻塞，计算机***可以新建一个新的数据日志，以使用该新的数据日志实现该达到参考容量阈值的数据日志的功能。In addition, after the capacity of the data log reaches the reference capacity threshold, in order to avoid blocking of the write service provided by the key-value storage system, the computer system can create a new data log to use the new data log to achieve the reference capacity threshold. The function of the data log.
其中，该采用第二有序数据层保存键值对的过程，可以看成是从内存向硬盘写入键值对的过渡过程，此时，为了避免因第二有序数据层中数据的修改导致向硬盘中写操作失败，该第二有序数据层可以为不支持修改操作的数据层。Among them, the process of using the second ordered data layer to save key-value pairs can be regarded as a transition process of writing key-value pairs from memory to hard disk. At this time, in order to avoid the modification of data in the second ordered data layer As a result, the write operation to the hard disk fails, and the second ordered data layer may be a data layer that does not support modification operations.
由上可以看出，第一有序数据层和第二有序数据层均用于在内存中缓存键值对。通过向第一有序数据层中写键值对，并在达到参考条件时，将该第一有序数据层转换成第二有序数据层，然后将该第二有序数据层写入硬盘中，能够将写操作请求的键值对在内存中聚集，将多个数据量较小的写入过程转换为向硬盘中一次性写入具有较大数据块的写入过程，能够提高向硬盘上写数据的带宽利用率。As can be seen from the above, both the first ordered data layer and the second ordered data layer are used to cache key-value pairs in memory. By writing key-value pairs into the first ordered data layer, and when the reference conditions are met, the first ordered data layer is converted into a second ordered data layer, and then the second ordered data layer is written to the hard disk The key-value pairs requested by the write operation can be aggregated in memory, and multiple write processes with a small amount of data can be converted into write processes with large data blocks written to the hard disk at one time. Bandwidth utilization for write data.
在一种可实现方式中，第二有序数据层包括多个SSTable。该第二有序数据层使用该多个SSTable存储键值对。可选地，该第二有序数据层可以为不可修改内存表(immutable memTable)。In one implementation, the second ordered data layer includes a plurality of SSTables. The second ordered data layer uses the plurality of SSTables to store key-value pairs. Optionally, the second ordered data layer may be an immutable memTable.
步骤208、将第二有序数据层中的数据写入目标数据结构的第三数据层中，第三数据层位于硬盘中。Step 208: Write the data in the second ordered data layer into the third data layer of the target data structure, where the third data layer is located in the hard disk.
其中，第三数据层支持追加写操作。将第二有序数据层中的数据写入第三数据层中的过程，也称为将内存中的数据持久化到硬盘中的刷回操作。将第二有序数据层中的数据写入第三数据层中后，可以删除该第二有序数据层。并且，将第二有序数据层中的数据写入第三数据层中之后，由于数据已存储在非易失性存储介质中了，不会因掉电丢失，因此，可以删除数据日志中用于请求写入该数据的写请求。Among them, the third data layer supports additional write operations. The process of writing the data in the second ordered data layer into the third data layer is also called the flush operation of persisting the data in the memory to the hard disk. After the data in the second ordered data layer is written into the third data layer, the second ordered data layer can be deleted. Moreover, after the data in the second ordered data layer is written into the third data layer, since the data has been stored in the non-volatile storage medium, it will not be lost due to power failure. Therefore, the data used in the data log can be deleted. for a write request that requested to write this data.
可选的，第三数据层可以为无序数据层。当该第三数据层为无序数据层时，在将第二有序数据层中的数据写入第三数据层的过程中，无需对写入的数据进行排序，缩短了数据写入时间，能够加快将第二有序数据层中的数据写入第三数据层中的速度。且由于第三数据层位于硬盘中，第二有序数据层位于内存中，当将第二有序数据层中的数据写入第三数据层中的速度较快时，能够降低第二有序数据层中的数据因掉电出现丢失的概率，提高了写数据成功的概率。Optionally, the third data layer may be an unordered data layer. When the third data layer is an unordered data layer, in the process of writing the data in the second ordered data layer into the third data layer, there is no need to sort the written data, which shortens the data writing time. The speed of writing data in the second ordered data layer into the third data layer can be accelerated. And since the third data layer is located in the hard disk and the second ordered data layer is located in the memory, when the data in the second ordered data layer is written to the third data layer at a faster speed, the second ordered data layer can be reduced. The probability of data loss in the data layer due to power failure increases the probability of successful data writing.
并且，该第三数据层允许键值覆盖。当第三数据层支持键值覆盖时，在将第二有序数据层中的数据写入第三数据层的过程中，无需对同一键对应的键值对进行处理，缩短了数据写入时间，能够加快将第二有序数据层中的数据写入第三数据层中的速度，并提高写数据成功的概率。Also, this third data layer allows key-value overrides. When the third data layer supports key-value overwriting, in the process of writing the data in the second ordered data layer to the third data layer, it is not necessary to process the key-value pairs corresponding to the same key, which shortens the data writing time , which can speed up the speed of writing data in the second ordered data layer into the third data layer, and improve the probability of successful data writing.
在一种可实现方式中，第三有序数据层包括多个SSTable。该第三有序数据层使用该多个SSTable存储键值对。并且，该多个SSTable中存储的数据允许键值覆盖。In one implementation, the third ordered data layer includes a plurality of SSTables. The third ordered data layer uses the plurality of SSTables to store key-value pairs. Also, the data stored in the multiple SSTables allows key-value overwriting.
并且，该第三数据层不仅可以部署在硬盘中，还可以同时缓存在内存中。此时，当计算机***需要读取该第三数据层中的数据时，由于内存具有较快的读速度，能够加快从该第三数据层中读取数据的速度。此时，如图3所示，键值存储***还包括：数据缓存模块1025，该数据缓存模块1025用于在内存中缓存该第三数据层。Moreover, the third data layer can not only be deployed in the hard disk, but also be cached in the memory at the same time. At this time, when the computer system needs to read the data in the third data layer, since the memory has a faster reading speed, the speed of reading data from the third data layer can be accelerated. At this time, as shown in FIG. 3 , the key-value storage system further includes: a data cache module 1025, and the data cache module 1025 is configured to cache the third data layer in the memory.
进一步的，用于管理键值存储***的多个数据结构的第三数据层使用的硬盘资源或内存资源可以共享。例如，假设多个数据结构的第三数据层都被配置为包括5个SSTable，但是在键值存储***提供服务的过程中，可以根据应用需求调整每个数据结构包括的SSTable的总数。例如，部分数据结构的第三数据层能够被调整为包括6个SSTable，部分数据结构的第三数据层能够被调整为包括4个SSTable。Further, hard disk resources or memory resources used by the third data layer for managing multiple data structures of the key-value storage system can be shared. For example, suppose that the third data layer of multiple data structures is configured to include 5 SSTables, but in the process of providing services by the key-value storage system, the total number of SSTables included in each data structure can be adjusted according to application requirements. For example, the third data layer of the partial data structure can be adjusted to include 6 SSTables, and the third data layer of the partial data structure can be adjusted to include 4 SSTables.
步骤209、当第三数据层的数据量达到第二数据量阈值时，使用第三数据层中的键值对和目标数据结构的第四有序数据层中的键值对，更新第四有序数据层中的键值对。Step 209: When the data volume of the third data layer reaches the second data volume threshold, use the key-value pair in the third data layer and the key-value pair in the fourth ordered data layer of the target data structure to update the fourth data volume. key-value pairs in the ordinal data layer.
其中，第四有序数据层位于硬盘中，第四有序数据层支持追加写操作。也即是，在向第四有序数据层中写数据时，可以将数据写在该第四有序数据层中已写入数据的尾部。The fourth ordered data layer is located in the hard disk, and the fourth ordered data layer supports additional write operations. That is, when data is written into the fourth ordered data layer, the data can be written at the tail of the written data in the fourth ordered data layer.
当第三数据层的数据量达到第二数据量阈值时，需要将该第三数据层中的键值对写入到第四有序数据层中。且将该第三数据层中的键值对写入到第四有序数据层中时，可以将该第三数据层中的键值对和第四有序数据层中原有的键值对进行合并排序，然后使用第四有序数据层存储经过合并排序后的键值对，实现对该第四有序数据层中的键值对的更新。When the data volume of the third data layer reaches the second data volume threshold, the key-value pair in the third data layer needs to be written into the fourth ordered data layer. And when the key-value pair in the third data layer is written into the fourth ordered data layer, the key-value pair in the third data layer and the original key-value pair in the fourth ordered data layer can be processed. Merge sorting, and then use the fourth ordered data layer to store the merged key-value pairs, so as to update the key-value pairs in the fourth ordered data layer.
在一种可实现方式中，第三数据层中的键值对和第四有序数据层中原有的键值对的合并排序过程包括：In an implementation manner, the process of merging and sorting the key-value pairs in the third data layer and the original key-value pairs in the fourth ordered data layer includes:
步骤S1：将第三数据层中的键值对读到内存中。Step S1: Read the key-value pair in the third data layer into the memory.
可选的，当第三数据层为无序的数据层时，该第三数据层中包括同一个键对应的多个键值对。此时，该步骤还包括：删除该第三数据层中的无效键值对。其中，对于同一个键对应 的多个键值对，写入时间最晚的键值对为有效键值对，写入时间不是最晚的键值对是无效键值对。Optionally, when the third data layer is an unordered data layer, the third data layer includes multiple key-value pairs corresponding to the same key. At this time, the step further includes: deleting invalid key-value pairs in the third data layer. Among them, for multiple key-value pairs corresponding to the same key, the key-value pair with the latest writing time is a valid key-value pair, and the key-value pair whose writing time is not the latest is an invalid key-value pair.
并且，在删除第三数据层中的无效键值对后，还可以对第三数据层中的有效键值对进行排序。其中，对键值对进行排序是指按照键的大小对键值对进行排序。或者，也可以不在该步骤中对第三数据层中的有效键值对进行排序，该对第三数据层中的有效键值对进行排序的过程，可以在后续步骤S3中执行。Moreover, after the invalid key-value pairs in the third data layer are deleted, the valid key-value pairs in the third data layer can also be sorted. The sorting of the key-value pairs refers to sorting the key-value pairs according to the size of the keys. Alternatively, the valid key-value pairs in the third data layer may not be sorted in this step, and the process of sorting the valid key-value pairs in the third data layer may be performed in the subsequent step S3.
步骤S2：将第四有序数据层中与第三数据层中键相同的键值对读到内存中。Step S2: Read the key-value pair in the fourth ordered data layer with the same key as the key in the third data layer into the memory.
对第三数据层和第四有序数据层中的键值对进行合并排序，是指对第三数据层和第四有序数据层中具有相同键的键值对进行合并排序，因此该第四有序数据层中需要读取到内存中的键值对是与第三数据层中键相同的键值对。Merge-sorting the key-value pairs in the third data layer and the fourth ordered data layer refers to merging and sorting the key-value pairs with the same key in the third data layer and the fourth ordered data layer. The key-value pair that needs to be read into memory in the fourth ordered data layer is the same key-value pair as the key in the third data layer.
步骤S3：删除从第四有序数据层中读取到内存中的无效键值对，对从第三数据层和第四有序数据层读取到内存中的有效键值对进行排序，将排序后的有效键值对存储到第四有序数据层的数据节点中。Step S3: delete the invalid key-value pairs read from the fourth ordered data layer into the memory, sort the valid key-value pairs read from the third data layer and the fourth ordered data layer into the memory, The sorted valid key-value pairs are stored in the data nodes of the fourth sorted data layer.
其中，步骤S2中读入到内存中的键值对的写入时间早于步骤S1中读入的键值对的写入时间，该写入时间早的键值对即为从第四有序数据层中读取到内存中的无效键值对。Wherein, the writing time of the key-value pair read into the memory in step S2 is earlier than the writing time of the key-value pair read in step S1, and the key-value pair with the earlier writing time is the order from the fourth Invalid key-value pair read into memory in the data layer.
可选的，为保证键值存储***的服务性能，在该步骤S3中，将排序后的有效键值对存储到第四有序数据层的数据节点中时，还可以获取该第四有序数据层中未读取到内存中的数据，并在硬盘中的其他存储位置中存储排序后的有效键值对和该未读取到内存中的数据，即在其他位置存储合并排序后的第四有序数据层，实现对数据结构的异地更新，使得硬盘中保存有不同版本的第四有序数据层。该其他位置为未存储有合并排序前的第四有序数据层的存储位置。Optionally, in order to ensure the service performance of the key-value storage system, in step S3, when the sorted valid key-value pairs are stored in the data nodes of the fourth ordered data layer, the fourth ordered data layer can also be obtained. The data in the memory is not read in the data layer, and the sorted valid key-value pairs and the data that are not read in the memory are stored in other storage locations on the hard disk, that is, the merge-sorted No. 1 is stored in other locations. The fourth ordered data layer realizes the remote update of the data structure, so that different versions of the fourth ordered data layer are stored in the hard disk. The other location is a storage location where the fourth ordered data layer before merge sort is not stored.
并且，为了对合并排序前后的第四有序数据层进行区分，还可以为第四有序数据层设置版本号。同时，还可以为合并排序前的第四有序数据层设置延迟删除机制。也即是，在对第三数据层和第四有序数据层中的键值对进行合并排序的过程中，仍在硬盘中保存第四有序数据层，并在得到经过合并排序后的第四有序数据层后，不立即删除合并排序前的第四有序数据层。这样一方面能够在合并排序过程中使用合并排序前的第四有序数据层，向用户提供服务，使得读写操作能够并行执行，以不至于因合并排序导致读请求阻塞，避免产生读波动。另一方面能够在合并排序出现故障和/或根据应用需求，将第四有序数据层回退至较早的版本，保证数据的一致性。其中，延迟删除旧版本的第四有序数据层中的数据的时长可以根据应用需求或存储空间的大小等确定，本申请实施例对其不做具体限定。并且，键值存储***还可以分别使用第四有序数据层的多个版本提供服务，实现多线程的并发访问服务。Moreover, in order to distinguish the fourth ordered data layer before and after the merge sort, a version number may also be set for the fourth ordered data layer. At the same time, a delayed deletion mechanism can also be set for the fourth ordered data layer before merge sort. That is, in the process of merging and sorting the key-value pairs in the third data layer and the fourth ordered data layer, the fourth ordered data layer is still stored in the hard disk, and after the merge-sorted No. After four ordered data layers, the fourth ordered data layer before merge sort is not immediately deleted. In this way, the fourth ordered data layer before merge sort can be used in the merge sort process to provide services to users, so that read and write operations can be executed in parallel, so as not to block read requests due to merge sort and avoid read fluctuations. On the other hand, the fourth ordered data layer can be rolled back to an earlier version when the merge sort fails and/or according to application requirements, so as to ensure data consistency. The duration of delaying deletion of the data in the fourth ordered data layer of the old version may be determined according to application requirements or the size of the storage space, etc., which is not specifically limited in this embodiment of the present application. In addition, the key-value storage system can also use multiple versions of the fourth ordered data layer to provide services, so as to realize multi-threaded concurrent access services.
需要说明的是，对第三数据层和第四有序数据层的合并排序操作可以串行执行。也即是，若在当前合并排序的过程中，若第三数据层的数据量又达到第二数据量阈值，需要在当前合并排序过程完成后，再对第三数据层和第四有序数据层执行合并排序过程。这样一来，可以保证合并排序操作的有序实现。It should be noted that the merge sort operation on the third data layer and the fourth ordered data layer can be performed serially. That is, if in the current merge sort process, if the data volume of the third data layer reaches the second data volume threshold again, after the current merge sort process is completed, the third data layer and the fourth ordered data need to be sorted. The layer performs the merge sort process. This way, an ordered implementation of the merge sort operation is guaranteed.
在一种可实现方式中，第四有序数据层可以为有序树结构。该第四有序数据层包括根节点、中间节点和叶子节点。根节点和中间节点均用于存储待操作数据的键，叶子节点用于存储中间节点存储的键对应的键值对。其中，根节点中存储的键的范围与中间节点中存储的键的范围存在键值覆盖。根节点用于存储待操作数据的键和指向中间节点的指向信息。中间节 点用于存储待操作数据的键和指向叶子节点的指向信息。叶子节点用于存储待操作数据的键值对。In an implementation manner, the fourth ordered data layer may be an ordered tree structure. The fourth ordered data layer includes root nodes, intermediate nodes and leaf nodes. Both the root node and the intermediate node are used to store the key of the data to be operated, and the leaf node is used to store the key-value pair corresponding to the key stored by the intermediate node. The range of keys stored in the root node and the range of keys stored in the intermediate nodes have key value coverage. The root node is used to store the keys of the data to be operated and the pointing information to the intermediate nodes. The intermediate node is used to store the key of the data to be operated and the pointing information to the leaf node. Leaf nodes are used to store key-value pairs of data to be operated.
在一种实现方式中，根节点和中间节点中用于指向节点的指向信息可以通过指针实现。例如，根节点还用于存储指向与根节点存储的键的范围存在范围覆盖的中间节点的指针，中间节点还用于存储指向与中间节点存储的键的范围存在范围覆盖的叶子节点的指针。In an implementation manner, the pointing information used to point to the node in the root node and the intermediate node may be implemented by a pointer. For example, the root node is also used to store pointers to intermediate nodes that have range coverage with the range of keys stored by the root node, and the intermediate nodes are also used to store pointers to leaf nodes that have range coverage with the range of keys stored by the intermediate node.
可选的，当使用指针指示节点的存储位置时，叶子节点还用于存储指向相邻叶子节点的指针。其中，相邻叶子节点存储的键的范围与叶子节点存储的键的范围相邻。这样一来，在该第四有序数据层中进行范围查找时，能够通过该执行相邻叶子结点的指针快速确定当前叶子结点的相邻叶子结点的存储位置，以加快范围查找时的查找速度。Optionally, when a pointer is used to indicate the storage location of the node, the leaf node is also used to store a pointer to an adjacent leaf node. The range of keys stored by adjacent leaf nodes is adjacent to the range of keys stored by leaf nodes. In this way, when a range search is performed in the fourth ordered data layer, the storage positions of the adjacent leaf nodes of the current leaf node can be quickly determined through the pointer of the executing adjacent leaf node, so as to speed up the range search process. search speed.
在另一种实现方式中，根节点和中间节点中用于指向节点的指向信息可以通过节点和该节点的存储位置信息的对应关系表示。例如，该指向信息可以通过存储有根节点、中间节点和叶子节点中至少一个节点存储的节点标识与节点标识指示的节点的存储位置信息的对应关系表示。In another implementation manner, the pointing information used to point to the node in the root node and the intermediate node may be represented by the corresponding relationship between the node and the storage location information of the node. For example, the pointing information may be represented by storing the corresponding relationship between the node identifier stored by at least one of the root node, the intermediate node and the leaf node and the storage location information of the node indicated by the node identifier.
其中，根节点还存储有用于存储根节点存储的键所覆盖的键的中间节点的节点标识，中间节点中还存储有用于存储中间节点存储的键所覆盖的键的叶子节点的节点标识。此时，根据节点标识与节点标识指示的节点的存储位置信息的对应关系，能够直接确定节点中存储的节点标识指示的节点的存储位置信息，而无需根据节点中存储的指针在第四有序数据层中进行层层查找，加快了在第四有序数据层中查找数据的速度。The root node also stores the node identifier of the intermediate node used to store the key covered by the key stored by the root node, and the intermediate node also stores the node identifier of the leaf node used to store the key covered by the key stored by the intermediate node. At this time, according to the corresponding relationship between the node identifier and the storage location information of the node indicated by the node identifier, the storage location information of the node indicated by the node identifier stored in the node can be directly determined, without needing to store the pointer stored in the node in the fourth order. Layer-by-layer search is performed in the data layer, which speeds up the search for data in the fourth ordered data layer.
在一种可实现方式中，如图4所示，键值存储***的内存中还存储有哈希映射表，哈希映射表用于记录根节点、中间节点和叶子节点中至少一个节点存储的节点标识与节点标识指示的节点的存储位置信息的对应关系。该哈希映射表又称为节点转换层(node translation layer，NTL)表。并且，为了进一步保证键值存储***的读速度，一个数据结构中节点标识与节点标识指示的节点的存储位置信息的对应关系可以采用NTL表记录。In an implementation manner, as shown in FIG. 4 , the memory of the key-value storage system further stores a hash mapping table, and the hash mapping table is used to record the data stored by at least one of the root node, intermediate node and leaf node. The corresponding relationship between the node identifier and the storage location information of the node indicated by the node identifier. The hash map is also called a node translation layer (NTL) table. In addition, in order to further ensure the reading speed of the key-value storage system, the corresponding relationship between the node identifier and the storage location information of the node indicated by the node identifier in a data structure can be recorded in an NTL table.
另外，如图4所示，日志文件a还包括映射表日志a2，为了避免内存中的节点标识与节点标识指示的节点的存储位置信息的对应关系因掉电等故障丢失，对哈希映射表记载的对应关系的任意修改均可以先持久化到映射表日志a2中，以便于在键值存储***出现故障时，键值存储***能够通过重放映射表日志a2，恢复哈希映射表。In addition, as shown in FIG. 4, the log file a also includes the mapping table log a2. In order to avoid the corresponding relationship between the node identifier in the memory and the storage location information of the node indicated by the node identifier is lost due to failures such as power failure, the hash mapping table Any modification of the recorded correspondence can be persisted to the mapping table log a2, so that when the key-value storage system fails, the key-value storage system can restore the hash mapping table by replaying the mapping table log a2.
当通过用于指向节点的指向信息通过节点和该节点的存储位置信息的对应关系表示时，在对第三数据层和第四有序数据层中的数据进行合并排序后，节点和节点的存储位置信息的实际对应关系可能会根据合并排序结果发生变化，因此，在合并排序完成后，还需要根据数据的实际存储情况更新该对应关系。其中，更新该对应关系的实现方式包括：对于被更新的节点，修改对应关系中记载的被更新的节点的存储位置。对于新增的节点，在对应关系中增加新增的节点与其存储位置信息的对应关系。When the pointing information for pointing to the node is represented by the corresponding relationship between the node and the storage location information of the node, after the data in the third data layer and the fourth ordered data layer are merge-sorted, the storage location of the node and the node The actual corresponding relationship of the location information may change according to the merge sort result. Therefore, after the merge sort is completed, the corresponding relationship needs to be updated according to the actual storage situation of the data. The implementation manner of updating the corresponding relationship includes: for the updated node, modifying the storage location of the updated node recorded in the corresponding relationship. For the newly added node, the corresponding relationship between the newly added node and its storage location information is added in the corresponding relationship.
并且，当采用指针实现用于指向节点的指向信息时，对于合并更新后的部分节点(如图4中节点a)，其存储位置发生变化，该存储位置发生变化的节点的父节点(如图4中节点b)的指针需要被修改，而该父节点(如图4中节点b)的修改又会造成该父节点(如图4中节点b)的父节点(如图4中节点c)需要被修改。因此，合并更新后位置发生变化的一个节点的更新，会导致该节点上溯到根节点的全路径更新。而采用通过节点和该节点的存储位置信息的对应关系实现指向节点的指向信息时，对于合并更新后位置发生变化的一个节点的更新， 仅需更新该对应关系中记载的存储位置信息，因此能够避免节点更新造成的全路径更新。And, when the pointer is used to realize the pointing information for pointing to the node, for the merged and updated part of the node (as shown in Figure 4, the node a), its storage location changes, and the parent node of the node whose storage location changes (as shown in Figure 4). The pointer of node b) in 4 needs to be modified, and the modification of the parent node (node b in Figure 4) will cause the parent node of the parent node (node b in Figure 4) to be the parent node (node c in Figure 4) needs to be modified. Therefore, the update of a node whose position has changed after the merged update will result in a full-path update of the node going back to the root node. When using the corresponding relationship between the node and the storage location information of the node to realize the pointing information to the node, for the update of a node whose location changes after the merge and update, only the storage location information recorded in the corresponding relationship needs to be updated, so it can be Avoid full path updates caused by node updates.
可选地，第四有序数据层可以为B+树(B+Tree)。Optionally, the fourth ordered data layer may be a B+Tree (B+Tree).
进一步的，为了提高读性能，还可以在内存中缓存该第四有序数据层的根节点和中间节点，以利用内存较快的读速度快速确定存储有读请求请求读取的数据的叶子节点的存储位置，从而加快从该第四有序数据层中读取数据的速度。并且，这样还能够减小读操作过程中访问硬盘的次数，以减小读操作的硬盘访问开销。其中，当在内存中缓存第三数据层和第四有序数据层中的根节点和中间节点时，可以通过读取内存直接找到用于存储待读取数据的叶子节点的位置信息，然后可以直接根据该位置信息在硬盘中找到该叶子节点，并读取该待读取数据，因此能够通过一次访盘读取到待读取数据，能够将读操作的访问开销维持在一次访盘，减小了读操作的访问开销。此时，图3中的数据缓存模块1025还用于在内存中缓存第四有序数据层的根节点和中间节点。Further, in order to improve the read performance, the root node and the intermediate node of the fourth ordered data layer can also be cached in the memory, so as to use the faster read speed of the memory to quickly determine the leaf nodes that store the data read by the read request. storage location, thereby speeding up the reading of data from this fourth ordered data layer. In addition, this can also reduce the number of times the hard disk is accessed during the read operation, so as to reduce the hard disk access overhead of the read operation. Wherein, when the root nodes and intermediate nodes in the third data layer and the fourth ordered data layer are cached in the memory, the location information of the leaf nodes used to store the data to be read can be directly found by reading the memory, and then Find the leaf node in the hard disk directly according to the location information, and read the data to be read, so the data to be read can be read through one disk access, and the access cost of the read operation can be maintained at one disk access, reducing The access overhead of read operations is reduced. At this time, the data caching module 1025 in FIG. 3 is further configured to cache the root node and intermediate nodes of the fourth ordered data layer in the memory.
并且，可以通过设置用于管理键值存储***的数据结构的总数，以及每个数据结构中第四有序数据层中各层节点的总数，控制数据结构的层次和数据结构中第四有序数据层的深度，例如，将第四有序数据层的有序树的深度控制为小于3，进而保证读操作的整体开销可控，提供稳定的读性能。并且，能够减小数据结构的复杂度，减少写入数据过程中需要执行的操作，能够降低计算机***的写放大(write amplification)。其中，写放大是计算机***的实际写数据量与用户写入的数据量的比值，当写放大越大时，计算机***的额外开销越大。And, by setting the total number of data structures used to manage the key-value storage system, and the total number of nodes of each layer in the fourth ordered data layer in each data structure, the level of the data structure and the fourth order in the data structure can be controlled. The depth of the data layer, for example, the depth of the ordered tree of the fourth ordered data layer is controlled to be less than 3, thereby ensuring that the overall cost of the read operation is controllable and providing stable read performance. Moreover, the complexity of the data structure can be reduced, the operations that need to be performed in the process of writing data can be reduced, and the write amplification of the computer system can be reduced. Among them, the write amplification is the ratio of the actual amount of data written by the computer system to the amount of data written by the user. When the write amplification is larger, the extra overhead of the computer system is larger.
其中，如图3所示，键值存储***还包括：存储模块1026，该存储模块1026用于将键值对写入目标数据结构中，即实现上述步骤206至步骤209。Wherein, as shown in FIG. 3, the key-value storage system further includes: a storage module 1026, which is used for writing the key-value pair into the target data structure, that is, implementing the above steps 206 to 209.
图5是本申请实施例提供的一种根据读操作请求从键值存储***中读数据的方法流程图。其中，用于管理该键值存储***的子逻辑空间的数据结构的实现方式，请相应参考写数据过程中的相关描述，此处不再赘述。如图5所示，该读数据的过程包括：FIG. 5 is a flowchart of a method for reading data from a key-value storage system according to a read operation request provided by an embodiment of the present application. For the implementation of the data structure used to manage the sub-logical space of the key-value storage system, please refer to the relevant description in the process of writing data, which will not be repeated here. As shown in Figure 5, the process of reading data includes:
步骤501、计算机***获取读操作请求，读操作请求中携带有待读取数据的索引信息。Step 501: The computer system obtains a read operation request, where the read operation request carries index information of the data to be read.
用户需要从存储池中读取数据时，可以通过客户端向计算机***发送读操作请求，以便于计算机***根据该读操作请求执行数据读取操作。例如，当用户使用云盘时，用户可以通过云盘的客户端向计算机***发送读操作请求。该读操作请求携带有待读取数据的索引信息。When the user needs to read data from the storage pool, the client can send a read operation request to the computer system, so that the computer system can perform the data read operation according to the read operation request. For example, when the user uses the cloud disk, the user can send a read operation request to the computer system through the client of the cloud disk. The read operation request carries index information of the data to be read.
步骤502、计算机***根据读操作请求，确定待读取数据的键，得到携带有待读取数据的键的读请求。Step 502: The computer system determines the key of the data to be read according to the read operation request, and obtains a read request carrying the key of the data to be read.
计算机***可以根据读操作请求中携带的待读取数据的索引信息，确定待读取数据的键，以得到携带有待读取数据的键的读请求，以便于根据该待读取数据的键在键值存储***中获取待读取数据的键值对。该根据索引信息确定待读取数据的键的实现方式请相应参考步骤202中的相关内容，此处不再赘述。The computer system can determine the key of the data to be read according to the index information of the data to be read carried in the read operation request, so as to obtain the read request carrying the key of the data to be read, so that the key of the data to be read is Obtain the key-value pair of the data to be read in the key-value storage system. For the implementation manner of determining the key of the data to be read according to the index information, please refer to the relevant content in step 202, which will not be repeated here.
在一种可实现方式中，如图3所示，计算机***10包括：I/O操作请求处理模块101和键值存储***102。该I/O操作请求处理模块101用于根据获取到的读操作请求，确定待写入数据的键，得到携带有待读取数据的键的读请求，并将该读请求发送至键值存储***。键值存储***102用于根据该读请求，查找读请求中键对应的值。In an implementation manner, as shown in FIG. 3 , the computer system 10 includes: an I/O operation request processing module 101 and a key-value storage system 102 . The I/O operation request processing module 101 is configured to determine the key of the data to be written according to the obtained read operation request, obtain a read request carrying the key of the data to be read, and send the read request to the key-value storage system . The key-value storage system 102 is configured to search for the value corresponding to the key in the read request according to the read request.
步骤503、计算机***基于键在多个数据结构中确定用于存储待读取数据的键值对的目标数据结构。Step 503: The computer system determines a target data structure for storing the key-value pair of the data to be read among the multiple data structures based on the key.
在写数据的过程中，是根据键确定用于存储待写入数据的键值对的目标数据结构的，因此当需要读取写入的数据时，可以按照确定用于存储待写入数据的目标数据结构所使用的逻辑，确定用于存储待读取数据的键值对的目标数据结构。也即是，可以根据待读取数据的键定用于存储待读取数据的键值对的目标数据结构。并且，该根据键确定目标数据结构的实现方式，请相应参考写操作流程中根据键确定目标数据结构的实现方式，此处不再赘述。In the process of writing data, the target data structure of the key-value pair used to store the data to be written is determined according to the key. Therefore, when the written data needs to be read, the target data structure used to store the data to be written can be determined according to the key. The logic used by the target data structure to determine the target data structure for storing key-value pairs of data to be read. That is, the target data structure for storing the key-value pair of the data to be read can be determined according to the key of the data to be read. In addition, for the implementation manner of determining the target data structure according to the key, please refer to the implementation manner of determining the target data structure according to the key in the write operation flow, which will not be repeated here.
步骤504、计算机***基于读请求，查找目标数据结构的第一有序数据层中是否存储有待读取数据的键。 Step 504 , the computer system searches, based on the read request, whether the key of the data to be read is stored in the first ordered data layer of the target data structure.
第一有序数据层位于内存中，由于内存具有较快的读速度，且由于向目标数据结构写数据时，先写入第一有序数据层中，再将第一有序数据层转换为第二有序数据层，以待写入硬盘中。因此，当需要从目标数据结构中读取待读取数据时，计算机***可以先从第一有序数据层进行查找。并且，查找待读取数据时，可以根据待读取数据的键进行查找。在查找结果指示第一有序数据层中存储有待读取数据的键的值时，返回该键的值，以便于根据该键的值获取待读取数据。在查找结果指示第一有序数据层中未存储待读取数据的键时，继续执行步骤505或执行步骤206。The first ordered data layer is located in the memory, because the memory has a fast reading speed, and because when writing data to the target data structure, it is first written into the first ordered data layer, and then the first ordered data layer is converted into A second ordered data layer to be written to the hard disk. Therefore, when the data to be read needs to be read from the target data structure, the computer system can first search from the first ordered data layer. Moreover, when searching for the data to be read, the search can be performed according to the key of the data to be read. When the search result indicates that the value of the key of the data to be read is stored in the first ordered data layer, the value of the key is returned, so as to obtain the data to be read according to the value of the key. When the search result indicates that the key of the data to be read is not stored in the first ordered data layer, continue to execute step 505 or execute step 206 .
在一种可实现方式中，第一有序数据层支持更新写操作。并且，该第一有序数据层的实现方式请相应参考前述写数据过程中的相关内容，此处不再赘述。In one implementation, the first ordered data layer supports update write operations. In addition, for the implementation of the first ordered data layer, please refer to the relevant content in the foregoing process of writing data, which will not be repeated here.
需要说明的是，当目标数据结构有多个版本时，在目标数据结构中查找待读取数据的键时，可以在最新版本的目标数据结构中进行查找，以保证能够根据键值存储***的最新状态进行查找，保证根据查找结果读取准确的数据。并且，计算机***可以统计目标数据结构被引用的总数，以便于根据该总数确定目标数据结构的使用情况。例如，在对目标数据结构执行I/O操作时，可以将目标数据结构的引用总数加一，以指示目标数据结构正处于使用状态，并在完成对目标数据结构的I/O操作时，将目标数据结构的引用总数减一。其中，不同版本的数据结构在创建时的引用计数均为0，即数据结构的引用计数的初始值均为0。并且，为了保证对数据结构的垃圾回收过程不影响键值存储***提供服务，当需要对数据结构进行垃圾回收时，需要保证该数据结构的引用计数为0且该数据结构的版本不是当前最新版本。It should be noted that when the target data structure has multiple versions, when searching for the key of the data to be read in the target data structure, the search can be performed in the target data structure of the latest version to ensure that the key-value storage system can The latest state is searched to ensure that accurate data is read according to the search results. Also, the computer system can count the total number of references to the target data structure, so as to determine the usage of the target data structure according to the total number. For example, when performing an I/O operation on the target data structure, the total number of references to the target data structure can be incremented by one to indicate that the target data structure is in use, and when the I/O operation on the target data structure is completed, the The total number of references to the target data structure is decremented by one. The reference counts of the data structures of different versions are all 0 when they are created, that is, the initial values of the reference counts of the data structures are all 0. In addition, in order to ensure that the garbage collection process of the data structure does not affect the service provided by the key-value storage system, when the data structure needs to be garbage collected, it is necessary to ensure that the reference count of the data structure is 0 and the version of the data structure is not the latest version. .
步骤505、当第一有序数据层中未存储待读取数据的键时，计算机***基于读请求，查找目标数据结构的第二有序数据层中是否存储有待读取数据的键。Step 505: When the key of the data to be read is not stored in the first ordered data layer, the computer system searches, based on the read request, whether the key of the data to be read is stored in the second ordered data layer of the target data structure.
当第一有序数据层中未存储待读取数据的键时，则可以继续在目标数据结构位于内存中的数据层中进行查找，即查找目标数据结构的第二有序数据层中是否存储有待读取数据的键。并且，在查找结果指示第二有序数据层中存储有待读取数据的键的值时，返回该键的值，以便于根据该键的值获取待读取数据。在查找结果指示第二有序数据层中未存储待读取数据的键时，继续执行步骤506。其中，第二有序数据层位于内存中，且第二有序数据层不支持修改操作。When the key of the data to be read is not stored in the first ordered data layer, you can continue to search in the data layer where the target data structure is located in the memory, that is, to find out whether the data stored in the second ordered data layer of the target data structure is stored The key for which data is to be read. And, when the search result indicates that the value of the key of the data to be read is stored in the second ordered data layer, the value of the key is returned, so as to obtain the data to be read according to the value of the key. When the search result indicates that the key of the data to be read is not stored in the second ordered data layer, step 506 is continued. The second ordered data layer is located in the memory, and the second ordered data layer does not support modification operations.
步骤506、当第二有序数据层中未存储待读取数据的键时，计算机***基于读请求，查找目标数据结构的第三数据层中是否存储有待读取数据的键，或者，当第一有序数据层中未存储待读取数据的键，且目标数据结构中不存在第二有序数据层时，查找第三有序数据层中是否存储有待读取数据的键。Step 506: When the key of the data to be read is not stored in the second ordered data layer, the computer system searches, based on the read request, whether the key of the data to be read is stored in the third data layer of the target data structure, or, when the first When the key of the data to be read is not stored in an ordered data layer and the second ordered data layer does not exist in the target data structure, it is searched whether the key of the data to be read is stored in the third ordered data layer.
当第二有序数据层中未存储待读取数据的键时，则可以继续在目标数据结构位于硬盘中的数据层中进行查找。并且，由于将内存中的键值对写入硬盘是先写入第三数据层，再从第 三数据层写入第四有序数据层，因此，当第二有序数据层中未存储待读取数据的键时，先在目标数据结构的第三数据层中查找是否存储有待读取数据的键。并且，如前，目标数据结构中在一些时机不存在第二有序数据层，则当第一有序数据层中未存储待读取数据的键时，可以直接在第三有序数据层中查找是否存储有待读取数据的键。When the key of the data to be read is not stored in the second ordered data layer, the search can be continued in the data layer where the target data structure is located in the hard disk. Moreover, since the key-value pair in the memory is written to the hard disk, the third data layer is first written, and then the fourth ordered data layer is written from the third data layer. When the key of the data is read, it is firstly searched in the third data layer of the target data structure whether the key of the data to be read is stored. And, as before, there is no second ordered data layer in the target data structure at some times, then when the key of the data to be read is not stored in the first ordered data layer, it can be directly stored in the third ordered data layer Finds if the key that stores the data to be read.
在查找结果指示第三数据层中存储有待读取数据的键的值时，返回该键的值，以便于根据该键的值获取待读取数据。在查找结果指示第三数据层中未存储待读取数据的键时，继续执行步骤507。其中，第三数据层支持追加写操作。When the search result indicates that the value of the key of the data to be read is stored in the third data layer, the value of the key is returned, so as to obtain the data to be read according to the value of the key. When the search result indicates that the key of the data to be read is not stored in the third data layer, step 507 is continued. Among them, the third data layer supports additional write operations.
并且，当在内存中缓存第三数据层时，当查找结果指示第二有序数据层中未存储有待读取数据的键的值时，或者，当第一有序数据层中未存储待读取数据的键，且目标数据结构中不存在第二有序数据层时，可以直接在内存中缓存的第三数据层中查找待读取数据的键的值，以减少访问硬盘的次数，从而加快读速度。And, when the third data layer is cached in memory, when the search result indicates that the value of the key of the data to be read is not stored in the second ordered data layer, or, when the first ordered data layer is not stored in the data to be read. When the key of the data is taken, and the second ordered data layer does not exist in the target data structure, the value of the key of the data to be read can be directly searched in the third data layer cached in the memory to reduce the number of accesses to the hard disk, thereby Speed up reading.
步骤507、当第三数据层中未存储待读取数据的键时，计算机***基于读请求，查找目标数据结构的第四有序数据层中是否存储有待读取数据的键。Step 507: When the key of the data to be read is not stored in the third data layer, the computer system searches, based on the read request, whether the key of the data to be read is stored in the fourth ordered data layer of the target data structure.
当第三数据层中未存储待读取数据的键时，则可以继续在第四有序数据层中进行查找。并且，在查找结果指示第四有序数据层中存储有待读取数据的键的值时，返回该键的值，以便于根据该键的值获取待读取数据。在查找结果指示第四有序数据层中未存储待读取数据的键时，确定目标数据结构中未存储有待读取数据的键值对，此时可向客户端返回指示读取数据失败的信息。其中，第四有序数据层位于硬盘中，且第四有序数据层支持追加写操作。When the key of the data to be read is not stored in the third data layer, the search can be continued in the fourth ordered data layer. And, when the search result indicates that the value of the key of the data to be read is stored in the fourth ordered data layer, the value of the key is returned, so as to obtain the data to be read according to the value of the key. When the search result indicates that the key of the data to be read is not stored in the fourth ordered data layer, it is determined that the key-value pair of the data to be read is not stored in the target data structure, and at this time, a message indicating the failure to read the data can be returned to the client information. The fourth ordered data layer is located in the hard disk, and the fourth ordered data layer supports additional write operations.
并且，当在内存中缓存第四有序数据层中的根节点和中间节点时，当查找结果指示第三数据层中未存储有待读取数据的键的值时，可以直接在内存中，根据缓存的第四有序数据层中的根节点和中间节点，定位到该第四有序数据层中的叶子节点，然后在硬盘中对定位到的叶子节点进行查找，以查找叶子节点是否存储有待读取数据的键。通过根据内存中缓存的根节点和中间节点定位叶子节点，能够减少访问硬盘的次数，从而加快读速度。And, when the root node and the intermediate node in the fourth ordered data layer are cached in the memory, when the search result indicates that the value of the key of the data to be read is not stored in the third data layer, it can be directly stored in the memory, according to the The root node and intermediate node in the fourth ordered data layer of the cache are located to the leaf nodes in the fourth ordered data layer, and then the located leaf nodes are searched in the hard disk to find out whether the leaf nodes have storage pending Key to read data. By locating the leaf nodes according to the root node and intermediate nodes cached in the memory, the number of accesses to the hard disk can be reduced, thereby speeding up the reading speed.
键值存储***的数据管理方法还包括通过该键值存储***删除快照的过程。图6是本申请实施例提供的一种通过该键值存储***删除快照的方法流程图。如图6所示，该快照删除的过程包括：The data management method of the key-value storage system further includes a process of deleting snapshots through the key-value storage system. FIG. 6 is a flowchart of a method for deleting snapshots through the key-value storage system provided by an embodiment of the present application. As shown in Figure 6, the snapshot deletion process includes:
步骤601、计算机***接收快照删除请求。Step 601: The computer system receives a snapshot deletion request.
键值存储***包括硬盘。该硬盘中存储有日志文件，通过回放日志文件能够在计算机***出现故障时对计算机***进行故障恢复。本申请实施例提供的键值存储***支持快照服务。例如，该键值存储***支持删除快照的服务。用户需要删除日志文件中的快照时，可以通过客户端向计算机***发送快照删除请求，以请求计算机***删除对应的快照。Key-value storage systems include hard disks. The hard disk stores log files, and by playing back the log files, the computer system can be recovered from failure when the computer system fails. The key-value storage system provided by the embodiment of the present application supports the snapshot service. For example, this key-value store system supports services that delete snapshots. When the user needs to delete the snapshot in the log file, the user can send a snapshot deletion request to the computer system through the client to request the computer system to delete the corresponding snapshot.
其中，为区分不同的快照，计算机***可以通过快照号对不同的快照进行管理，相应的，该键值存储***存储的键值对的键包括快照号。因此，快照删除请求中可以携带有快照号，以通过该快照号指示需要删除的快照。Wherein, in order to distinguish different snapshots, the computer system can manage different snapshots by using the snapshot numbers. Correspondingly, the key of the key-value pair stored in the key-value storage system includes the snapshot number. Therefore, the snapshot deletion request may carry a snapshot number, so as to indicate the snapshot to be deleted through the snapshot number.
可选地，该键值存储***还支持创建快照的服务，计算机***可以根据用户的请求为用户创建快照。并且，计算机***在创建快照后，还可以为创建的快照分配快照号，以区分不同的快照。在一种可实现方式中，计算机***在创建快照后，可以对当前已有快照号加一，以得到新创建得到的快照的快照号。相应的，由于后创建的快照比先创建的快照更能反映键 值存储***的当前状态，因此，写入键值存储***中的键值对中的快照号通常为当前最新创建的快照的快照号，键值存储***可通过判断快照号的大小确定最新创建的快照，其中快照号最大的快照为最新创建的快照。Optionally, the key-value storage system also supports the service of creating snapshots, and the computer system can create snapshots for users according to the user's request. Moreover, after the snapshot is created, the computer system can also assign a snapshot number to the created snapshot, so as to distinguish different snapshots. In an implementation manner, after creating the snapshot, the computer system may add one to the current snapshot number to obtain the snapshot number of the newly created snapshot. Correspondingly, since the snapshot created later can better reflect the current state of the key-value storage system than the snapshot created first, the snapshot number in the key-value pair written in the key-value storage system is usually the snapshot of the latest created snapshot. number, the key-value storage system can determine the newly created snapshot by judging the size of the snapshot number, and the snapshot with the largest snapshot number is the newly created snapshot.
步骤602、计算机***将快照删除请求写在日志文件中，并发送针对快照删除请求的操作完成响应。Step 602: The computer system writes the snapshot deletion request in a log file, and sends an operation completion response to the snapshot deletion request.
计算机***接收快照删除请求后，可以先将该快照删除请求写在日志文件中，并在完成将快照删除请求写在日志文件中的操作后，向发送操作完成响应，以指示已完成快照删除请求指示的快照删除操作。其中，如图4所示，日志文件a包括快照日志a3，快照删除请求具体可以在写在快照日志a3中。After receiving the snapshot deletion request, the computer system can first write the snapshot deletion request in the log file, and after completing the operation of writing the snapshot deletion request in the log file, send an operation completion response to indicate that the snapshot deletion request has been completed. The indicated snapshot delete operation. Wherein, as shown in FIG. 4 , the log file a includes a snapshot log a3, and the snapshot deletion request may be specifically written in the snapshot log a3.
由于将快照删除请求写在日志文件中的耗时比执行快照删除请求指示的快照删除操作的耗时短，通过在将快照删除请求写在日志文件后发送操作完成响应，能够对快照删除请求进行快速响应，能够为用户提供低时延的快照服务，提高用户体验。Since the time-consuming of writing the snapshot deletion request in the log file is shorter than the time-consuming of executing the snapshot deletion operation indicated by the snapshot deletion request, by sending the operation completion response after writing the snapshot deletion request in the log file, the snapshot deletion request can be processed. Quick response can provide users with low-latency snapshot services and improve user experience.
步骤603、计算机***基于日志文件中记录的快照删除请求，执行快照删除请求指示的快照删除操作。Step 603: The computer system executes the snapshot deletion operation indicated by the snapshot deletion request based on the snapshot deletion request recorded in the log file.
在步骤602中，通过将快照删除请求写在日志文件中，能够实现对快照删除请求的记录，使得计算机***能够根据该记录执行快照删除操作。在一种可实现方式中，计算机***可以在第三数据层中的键值对和第四有序数据层中原有的键值对的合并排序过程中，执行快照删除操作。In step 602, the snapshot deletion request can be recorded by writing the snapshot deletion request in the log file, so that the computer system can perform the snapshot deletion operation according to the record. In an implementation manner, the computer system may perform a snapshot deletion operation in the process of merging and sorting the key-value pairs in the third data layer and the original key-value pairs in the fourth ordered data layer.
由于合并排序过程和执行快照删除操作的过程涉及一些相同的操作(如修改数据结构中节点的操作)，通过在合并排序的过程中执行快照删除操作，能够减小部分该相同的操作，减小计算机***因执行该部分操作的***开销，能够降低计算机***的写放大。并且，还能够将快照日志中记录的多个快照删除请求批量处理，批量处理快照删除请求的过程也能够减少相同操作的执行，能够进一步降低计算机***的写放大。Since the merge sort process and the process of performing the snapshot deletion operation involve some same operations (such as the operation of modifying the nodes in the data structure), by performing the snapshot deletion operation during the merge sort process, it is possible to reduce some of the same operations and reduce the The computer system can reduce the write amplification of the computer system due to the system overhead of performing this part of the operation. In addition, multiple snapshot deletion requests recorded in the snapshot log can be processed in batches, and the process of batch processing the snapshot deletion requests can also reduce the execution of the same operation, which can further reduce the write amplification of the computer system.
并且，在完成快照删除操作后，可以清空快照日志。其中，若快照日志中记录有快照删除请求，说明计算机***需要执行该快照删除请求所指示的快照删除操作。Also, after the snapshot deletion operation is completed, the snapshot log can be emptied. Wherein, if a snapshot deletion request is recorded in the snapshot log, it indicates that the computer system needs to perform the snapshot deletion operation indicated by the snapshot deletion request.
在一种可实现方式中，如图3所示，键值存储***102还包括：快照模块1027，该快照模块1027用于提供快照服务，如提供创建快照的服务和删除快照的服务。并且，如图3所示，该键值存储***102还包括：垃圾回收模块1028，该垃圾回收模块1028用于根据空间管理模块对存储空间的标记状态，对未使用的存储进行垃圾回收。In an implementation manner, as shown in FIG. 3 , the key-value storage system 102 further includes: a snapshot module 1027, the snapshot module 1027 is configured to provide snapshot services, such as providing a snapshot creation service and a snapshot deletion service. Moreover, as shown in FIG. 3 , the key-value storage system 102 further includes: a garbage collection module 1028, which is configured to perform garbage collection on unused storage according to the marked state of the storage space by the space management module.
综上所述，在本申请实施例提供的键值存储***的数据管理方法中，通过在用于管理键值存储***的逻辑地址空间的多个数据结构中，确定用于存储待操作数据的键值对的目标数据结构，并在目标数据结构中对键值对进行I/O操作，由于该键值存储***中的多个子逻辑地址空间分别由多个数据结构进行管理，相较于采用一个数据结构管理键值存储***的逻辑地址空间，减小了该多个数据结构中每个数据结构需要存储的数据量，提高了该键值存储***根据该数据结构对I/O操作请求进行响应的效率和键值存储***的性能，能够通过该数据结构向用户提供稳定且可预测的读性能。To sum up, in the data management method of the key-value storage system provided by the embodiment of the present application, among the multiple data structures used to manage the logical address space of the key-value storage system, the data structure for storing the data to be operated is determined. The target data structure of the key-value pair, and I/O operations are performed on the key-value pair in the target data structure. Since the multiple sub-logical address spaces in the key-value storage system are managed by multiple data structures, compared with using A data structure manages the logical address space of the key-value storage system, which reduces the amount of data that needs to be stored in each of the multiple data structures, and improves the performance of the key-value storage system on I/O operation requests according to the data structure. The efficiency of the response and the performance of the key-value storage system can provide users with stable and predictable read performance through this data structure.
并且，由于键值存储***的多个子逻辑地址空间分别由多个数据结构管理，键值存储***能够通过该多个数据结构提供存储服务，提高该键值存储***的并行处理能力，提高了键值存储***的服务性能。Moreover, since the multiple sub-logical address spaces of the key-value storage system are managed by multiple data structures, the key-value storage system can provide storage services through the multiple data structures, improve the parallel processing capability of the key-value storage system, and improve the key-value storage system. Service performance of the value store system.
需要说明的是，本申请实施例提供的键值存储***的数据管理方法的步骤先后顺序可以进行适当调整，步骤也可以根据情况进行相应增减。任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化的方法，都应涵盖在本申请的保护范围之内，因此不再赘述。It should be noted that, the sequence of steps of the data management method of the key-value storage system provided by the embodiment of the present application may be adjusted appropriately, and the steps may be correspondingly increased or decreased according to the situation. Any person skilled in the art who is familiar with the technical scope disclosed in the present application can easily think of any variation of the method, which should be covered by the protection scope of the present application, and thus will not be repeated here.
本申请实施例还提供了一种键值存储***的数据管理装置。键值存储***的逻辑地址空间包括：多个子逻辑地址空间，多个子逻辑地址空间互不重叠，多个子逻辑地址空间中存储的数据分别使用多个数据结构一一对应管理。如图7所示，该键值存储***的数据管理装置70包括：The embodiments of the present application also provide a data management device of a key-value storage system. The logical address space of the key-value storage system includes: multiple sub-logical address spaces, the multiple sub-logical address spaces do not overlap each other, and the data stored in the multiple sub-logical address spaces are managed in a one-to-one correspondence with multiple data structures. As shown in Figure 7, the data management device 70 of the key-value storage system includes:
获取模块701，用于获取输入/输出I/O操作请求，I/O操作请求中携带有待操作数据的索引信息。The obtaining module 701 is configured to obtain an input/output I/O operation request, where the I/O operation request carries index information of the data to be operated.
确定模块702，用于基于索引信息，确定待操作数据的键。The determining module 702 is configured to determine the key of the data to be operated based on the index information.
确定模块702，还用于基于键，在多个数据结构中确定用于存储待操作数据的键值对的目标数据结构。The determining module 702 is further configured to determine, based on the key, a target data structure for storing the key-value pair of the data to be operated among the multiple data structures.
处理模块703，用于在目标数据结构中对键值对进行I/O操作。The processing module 703 is configured to perform I/O operations on the key-value pair in the target data structure.
可选地，索引信息包括I/O操作请求的发起者的标识和待操作数据的逻辑地址，确定模块702，具体用于：将索引信息确定为键。Optionally, the index information includes the identifier of the initiator of the I/O operation request and the logical address of the data to be operated. The determining module 702 is specifically configured to: determine the index information as a key.
可选地，I/O操作请求的发起者为云盘。Optionally, the initiator of the I/O operation request is a cloud disk.
可选地，索引信息包括用于索引键的关键字，确定模块702，具体用于：基于关键字进行索引，得到键，键包括I/O操作请求的发起者的标识和待操作数据的逻辑地址。Optionally, the index information includes a keyword for indexing the key, and the determining module 702 is specifically configured to: perform an index based on the keyword to obtain a key, and the key includes the identifier of the initiator of the I/O operation request and the logic of the data to be operated. address.
可选地，I/O操作请求为写操作请求，处理模块703，具体用于：将键值对存储在目标数据结构的第一有序数据层中，第一有序数据层位于内存中，且第一有序数据层支持更新写操作；当达到参考条件时，将第一有序数据层转换为第二有序数据层，第二有序数据层位于内存中，且第二有序数据层不支持修改操作；将第二有序数据层中的键值对写入目标数据结构的第三数据层中，第三数据层位于硬盘中，且第三数据层支持追加写操作；当第三数据层的数据量达到第二数据量阈值时，使用第三数据层中的键值对和目标数据结构的第四有序数据层中的键值对，更新第四有序数据层中的键值对，第四有序数据层位于硬盘中，且第四有序数据层支持追加写操作。Optionally, the I/O operation request is a write operation request, and the processing module 703 is specifically configured to: store the key-value pair in the first ordered data layer of the target data structure, and the first ordered data layer is located in the memory, And the first ordered data layer supports update write operations; when the reference condition is reached, the first ordered data layer is converted into a second ordered data layer, the second ordered data layer is located in the memory, and the second ordered data layer is The layer does not support modification operations; the key-value pairs in the second ordered data layer are written into the third data layer of the target data structure, the third data layer is located in the hard disk, and the third data layer supports additional write operations; When the data volume of the third data layer reaches the second data volume threshold, use the key-value pair in the third data layer and the key-value pair in the fourth ordered data layer of the target data structure to update the data in the fourth ordered data layer For key-value pairs, the fourth ordered data layer is located on the hard disk, and the fourth ordered data layer supports additional write operations.
可选地，I/O操作请求为读操作请求，待操作数据为待读取数据，处理模块703，具体用于：查找目标数据结构的第一有序数据层中是否存储有待读取数据的键，第一有序数据层位于内存中，且第一有序数据层支持更新写操作；当第一有序数据层中未存储待读取数据的键时，查找目标数据结构的第二有序数据层中是否存储有待读取数据的键，第二有序数据层位于内存中，且第二有序数据层不支持修改操作；当第二有序数据层中未存储待读取数据的键时，查找目标数据结构的第三数据层中是否存储有待读取数据的键，第三数据层位于硬盘中，且第三数据层支持追加写操作；或者，当第一有序数据层中未存储待读取数据的键，且目标数据结构中不存在第二有序数据层时，查找第三有序数据层中是否存储有待读取数据的键；当第三数据层中未存储待读取数据的键时，查找目标数据结构的第四有序数据层中是否存储有待读取数据的键，第四有序数据层位于硬盘中，且第四有序数据层支持追加写操作。Optionally, the I/O operation request is a read operation request, and the data to be operated is data to be read. The processing module 703 is specifically configured to: find out whether the first ordered data layer of the target data structure stores the data to be read. key, the first ordered data layer is located in the memory, and the first ordered data layer supports the update write operation; when the key of the data to be read is not stored in the first ordered data layer, the second ordered data structure of the target data structure is searched. Whether the key of the data to be read is stored in the ordered data layer, the second ordered data layer is located in the memory, and the second ordered data layer does not support modification operations; when the second ordered data layer does not store the key of the data to be read When the key is used, check whether the key to be read is stored in the third data layer of the target data structure, the third data layer is located in the hard disk, and the third data layer supports additional write operations; or, when the first ordered data layer When the key of the data to be read is not stored and the second ordered data layer does not exist in the target data structure, check whether the key of the data to be read is stored in the third ordered data layer; when the key of the data to be read is not stored in the third data layer When reading the key of the data, check whether the key of the data to be read is stored in the fourth ordered data layer of the target data structure, the fourth ordered data layer is located in the hard disk, and the fourth ordered data layer supports additional write operations.
可选地，第四有序数据层包括根节点、中间节点和叶子节点，根节点和中间节点均用于 存储待操作数据的键，根节点能够存储的键的范围覆盖中间节点能够存储的键的范围，叶子节点存储有中间节点存储的键对应的键值对。Optionally, the fourth ordered data layer includes a root node, an intermediate node, and a leaf node. Both the root node and the intermediate node are used to store the keys of the data to be operated, and the range of the keys that the root node can store covers the keys that the intermediate nodes can store. The leaf node stores the key-value pair corresponding to the key stored by the intermediate node.
并且，根节点还存储有用于存储根节点存储的键所覆盖的键的中间节点的节点标识，中间节点中还存储有用于存储中间节点存储的键所覆盖的键的叶子节点的节点标识。In addition, the root node also stores the node identifier of the intermediate node used to store the key covered by the key stored by the root node, and the intermediate node also stores the node identifier of the leaf node used to store the key covered by the key stored by the intermediate node.
进一步地，键值存储***中还存储有哈希映射表，哈希映射表用于存储根节点、中间节点和叶子节点中至少一个节点存储的节点标识与节点标识指示的节点的存储位置信息的对应关系。Further, a hash mapping table is also stored in the key-value storage system, and the hash mapping table is used to store the node identifier stored by at least one node in the root node, the intermediate node and the leaf node and the storage location information of the node indicated by the node identifier. Correspondence.
可选地，第三数据层为无序数据层。Optionally, the third data layer is an unordered data layer.
可选地，第四有序数据层为树状数据结构，第四有序数据层的深度小于3。Optionally, the fourth ordered data layer is a tree-like data structure, and the depth of the fourth ordered data layer is less than 3.
可选地，更新后的第四有序数据层的存储位置与更新前的第四有序数据层的存储位置不同，且更新前的第四有序数据层和更新后的第四有序数据层使用不同版本号进行标识。Optionally, the storage location of the fourth ordered data layer after the update is different from the storage location of the fourth ordered data layer before the update, and the fourth ordered data layer before the update and the fourth ordered data layer after the update are different. Layers are identified using different version numbers.
可选地，键包括待操作数据的逻辑地址，逻辑地址通过数值表示，多个数据结构中同一数据结构中存储的键值对中逻辑地址对指定数值的余数相等，不同数据结构中存储的键值对中逻辑地址对指定数值的余数不等。Optionally, the key includes a logical address of the data to be operated, and the logical address is represented by a numerical value. The remainder of the logical address in the key-value pair stored in the same data structure in multiple data structures is equal to the specified value, and the keys stored in different data structures are equal. The remainder of the logical address pair in the value pair is unequal to the specified value.
可选地，键值存储***包括硬盘，硬盘中存储有日志文件，如图8所示，键值存储***的数据管理装置70还包括：Optionally, the key-value storage system includes a hard disk, and log files are stored in the hard disk. As shown in FIG. 8 , the data management device 70 of the key-value storage system further includes:
接收模块704，用于接收快照删除请求；a receiving module 704, configured to receive a snapshot deletion request;
处理模块703，还用于将快照删除请求写在日志文件中，并发送针对快照删除请求的操作完成响应；The processing module 703 is further configured to write the snapshot deletion request in the log file, and send an operation completion response for the snapshot deletion request;
处理模块703，还用于基于日志文件中记录的快照删除请求，执行快照删除请求指示的快照删除操作。The processing module 703 is further configured to perform the snapshot deletion operation indicated by the snapshot deletion request based on the snapshot deletion request recorded in the log file.
综上所述，在本申请实施例提供的键值存储***的数据管理装置中，确定模块在用于管理键值存储***的逻辑地址空间的多个数据结构中，确定用于存储待操作数据的键值对的目标数据结构，处理模块在目标数据结构中对键值对进行I/O操作，由于该键值存储***中的多个子逻辑地址空间分别由多个数据结构进行管理，相较于采用一个数据结构管理键值存储***的逻辑地址空间，减小了该多个数据结构中每个数据结构需要存储的数据量，提高了该键值存储***根据该数据结构对I/O操作请求进行响应的效率和键值存储***的性能，能够通过该数据结构向用户提供稳定且可预测的读性能。To sum up, in the data management device of the key-value storage system provided by the embodiment of the present application, the determination module determines, among the multiple data structures used to manage the logical address space of the key-value storage system, to store the data to be operated The target data structure of the key-value pair, the processing module performs I/O operations on the key-value pair in the target data structure. Since multiple sub-logical address spaces in the key-value storage system are managed by multiple data structures, compared to Because one data structure is used to manage the logical address space of the key-value storage system, the amount of data that needs to be stored in each data structure in the multiple data structures is reduced, and the key-value storage system can perform I/O operations according to the data structure. The efficiency of responding to requests and the performance of key-value storage systems can provide users with stable and predictable read performance through this data structure.
并且，由于键值存储***的多个子逻辑地址空间分别由多个数据结构管理，键值存储***能够通过该多个数据结构提供存储服务，提高该键值存储***的并行处理能力，提高了键值存储***的服务性能。Moreover, since the multiple sub-logical address spaces of the key-value storage system are managed by multiple data structures, the key-value storage system can provide storage services through the multiple data structures, improve the parallel processing capability of the key-value storage system, and improve the key-value storage system. Service performance of the value store system.
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的装置和模块的具体工作过程，可以参考前述方法实施例中的对应内容，在此不再赘述。Those skilled in the art can clearly understand that, for the convenience and brevity of description, for the specific working process of the above-described devices and modules, reference may be made to the corresponding content in the foregoing method embodiments, which will not be repeated here.
本申请实施例提供了一种计算机设备。图9示例性的提供了计算机设备的一种可能的架构图。如图9所示，该计算机设备90可以包括处理器901、存储器902、通信接口903和总线904。在计算机设备中，处理器901的数量可以是一个或多个，图9仅示意了其中一个处理器901。可选的，处理器901可以是中央处理器(Central Processing Unit，CPU)。若计算机设备具有多个处理器901，多个处理器901的类型可以不同，或者可以相同。可选的，计 算机设备的多个处理器还可以集成为多核处理器。The embodiments of the present application provide a computer device. FIG. 9 exemplarily provides a possible architectural diagram of a computer device. As shown in FIG. 9 , the computer device 90 may include a processor 901 , a memory 902 , a communication interface 903 and a bus 904 . In a computer device, the number of processors 901 may be one or more, and FIG. 9 only illustrates one of the processors 901 . Optionally, the processor 901 may be a central processing unit (Central Processing Unit, CPU). If the computer device has multiple processors 901, the multiple processors 901 may be of different types, or may be the same. Optionally, multiple processors of the computer device can also be integrated into a multi-core processor.
存储器902存储计算机指令和数据，存储器902可以存储实现本申请提供的键值存储***的数据管理方法所需的计算机指令和数据。存储器902可以是以下存储介质的任一种或任一种组合：非易失性存储器(如只读存储器(Read-Only Memory，ROM)、固态硬盘(Solid State Disk，SSD)、硬盘(Hard Disk Drive，HDD)、光盘等、易失性存储器。The memory 902 stores computer instructions and data, and the memory 902 may store computer instructions and data required to implement the data management method of the key-value storage system provided by the present application. The memory 902 can be any one or any combination of the following storage media: non-volatile memory (such as read-only memory (Read-Only Memory, ROM), solid state disk (Solid State Disk, SSD), hard disk (Hard Disk) Drive, HDD), optical disc, etc., volatile memory.
通信接口903可以是以下器件的任一种或任一种组合：网络接口(如以太网接口)、无线网卡等具有网络接入功能的器件。The communication interface 903 may be any one or any combination of the following devices: a network interface (eg, an Ethernet interface), a wireless network card, and other devices with a network access function.
通信接口903用于计算机设备与其他节点或者其他计算机设备进行数据通信。The communication interface 903 is used for data communication between the computer device and other nodes or other computer devices.
图9还示例性地绘制出总线904。总线904可以将处理器901与存储器902、通信接口903连接。这样，通过总线904，处理器901可以访问存储器902，还可以利用通信接口903与其他节点或者其他计算机设备进行数据交互。FIG. 9 also exemplarily depicts bus 904 . The bus 904 can connect the processor 901 with the memory 902 and the communication interface 903 . In this way, through the bus 904, the processor 901 can access the memory 902, and can also use the communication interface 903 to perform data interaction with other nodes or other computer devices.
在本申请中，计算机设备执行存储器902中的计算机指令，可以实现本申请提供的键值存储***的数据管理方法。例如，获取输入/输出I/O操作请求，I/O操作请求中携带有待操作数据的索引信息；基于索引信息，确定待操作数据的键；基于键，在多个数据结构中确定用于存储待操作数据的键值对的目标数据结构；在目标数据结构中对键值对进行I/O操作。并且，计算机设备通过执行存储器902中的计算机指令，执行计算机设备的步骤的实现过程可以相应参考上述方法实施例中对应的描述。In the present application, the computer device executes the computer instructions in the memory 902 to implement the data management method of the key-value storage system provided by the present application. For example, obtain an input/output I/O operation request, and the I/O operation request carries the index information of the data to be operated; based on the index information, determine the key of the data to be operated; The target data structure of the key-value pairs of the data to be operated; perform I/O operations on the key-value pairs in the target data structure. Moreover, by executing the computer instructions in the memory 902 by the computer device, the implementation process of executing the steps of the computer device may refer to the corresponding descriptions in the foregoing method embodiments.
本申请实施例还提供了一种存储介质，该存储介质为非易失性计算机可读存储介质，当存储介质中的指令被处理器执行时，实现如本申请实施例提供的键值存储***的数据管理方法。Embodiments of the present application further provide a storage medium, which is a non-volatile computer-readable storage medium, and when instructions in the storage medium are executed by a processor, the key-value storage system provided by the embodiments of the present application is implemented. method of data management.
本申请实施例还提供了一种包含指令的计算机程序产品，当计算机程序产品在计算机上运行时，使得计算机执行本申请实施例提供的键值存储***的数据管理方法。Embodiments of the present application also provide a computer program product including instructions, which, when the computer program product runs on a computer, enables the computer to execute the data management method of the key-value storage system provided by the embodiments of the present application.
本领域普通技术人员可以理解实现上述实施例的全部或部分步骤可以通过硬件来完成，也可以通过程序来指令相关的硬件完成，所述的程序可以存储于一种计算机可读存储介质中，上述提到的存储介质可以是只读存储器，磁盘或光盘等。Those of ordinary skill in the art can understand that all or part of the steps of implementing the above embodiments can be completed by hardware, or can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. The storage medium mentioned may be a read-only memory, a magnetic disk or an optical disk, etc.
在本申请实施例中，术语“第一”、“第二”和“第三”仅用于描述目的，而不能理解为指示或暗示相对重要性。术语“至少一个”是指一个或多个，术语“多个”指两个或两个以上，除非另有明确的限定。In the embodiments of the present application, the terms "first", "second" and "third" are only used for description purposes, and cannot be understood as indicating or implying relative importance. The term "at least one" refers to one or more, and the term "plurality" refers to two or more, unless expressly limited otherwise.
本申请中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。The term "and/or" in this application is only an association relationship to describe associated objects, which means that there can be three kinds of relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, independently There are three cases of B. In addition, the character "/" in this document generally indicates that the related objects are an "or" relationship.
以上所述仅为本申请的可选实施例，并不用以限制本申请，凡在本申请的构思和原则之内，所作的任何修改、等同替换、改进等，均应包含在本申请的保护范围之内。The above are only optional embodiments of the present application, and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the concept and principles of the present application shall be included in the protection of the present application. within the range.

Claims (26)

  1. 一种键值存储***的数据管理方法，其特征在于，所述键值存储***的逻辑地址空间包括：多个子逻辑地址空间，所述多个子逻辑地址空间互不重叠，所述多个子逻辑地址空间中存储的数据分别使用多个数据结构一一对应管理，所述方法包括：A data management method for a key-value storage system, wherein the logical address space of the key-value storage system comprises: multiple sub-logical address spaces, the multiple sub-logical address spaces do not overlap each other, and the multiple sub-logical address spaces The data stored in the space is managed in a one-to-one correspondence with multiple data structures, and the method includes:
    获取输入/输出I/O操作请求，所述I/O操作请求中携带有待操作数据的索引信息；Obtain an input/output I/O operation request, where the I/O operation request carries index information of the data to be operated;
    基于所述索引信息，确定所述待操作数据的键；Based on the index information, determine the key of the data to be operated;
    基于所述键，在所述多个数据结构中确定用于存储所述待操作数据的键值对的目标数据结构；Based on the key, determining a target data structure for storing the key-value pair of the data to be operated in the plurality of data structures;
    在所述目标数据结构中对所述键值对进行I/O操作。I/O operations are performed on the key-value pair in the target data structure.
  2. 根据权利要求1所述的方法，其特征在于，所述索引信息包括所述I/O操作请求的发起者的标识和所述待操作数据的逻辑地址，所述基于所述索引信息，确定所述待操作数据的键，包括：The method according to claim 1, wherein the index information includes an identifier of an initiator of the I/O operation request and a logical address of the data to be operated, and the index information is determined based on the index information. Keys that describe the data to be operated, including:
    将所述索引信息确定为所述键。The index information is determined as the key.
  3. 根据权利要求2所述的方法，其特征在于，I/O操作请求的发起者为云盘。The method according to claim 2, wherein the initiator of the I/O operation request is a cloud disk.
  4. 根据权利要求1所述的方法，其特征在于，所述索引信息包括用于索引所述键的关键字，所述基于所述索引信息，确定所述待操作数据的键，包括：The method according to claim 1, wherein the index information includes a keyword for indexing the key, and the determining the key of the data to be operated based on the index information includes:
    基于所述关键字进行索引，得到所述键，所述键包括所述I/O操作请求的发起者的标识和所述待操作数据的逻辑地址。Indexing is performed based on the keyword to obtain the key, where the key includes the identifier of the initiator of the I/O operation request and the logical address of the data to be operated.
  5. 根据权利要求1至4任一所述的方法，其特征在于，所述I/O操作请求为写操作请求，所述待操作数据为待写入数据，所述在所述目标数据结构中对所述键值对进行I/O操作，包括：The method according to any one of claims 1 to 4, wherein the I/O operation request is a write operation request, the data to be operated is data to be written, and the data to be written in the target data structure The key-value pair performs I/O operations, including:
    将所述键值对存储在所述目标数据结构的第一有序数据层中，所述第一有序数据层位于内存中，且所述第一有序数据层支持更新写操作；storing the key-value pair in a first ordered data layer of the target data structure, the first ordered data layer is located in memory, and the first ordered data layer supports update write operations;
    当达到参考条件时，将所述第一有序数据层转换为第二有序数据层，所述第二有序数据层位于内存中，且所述第二有序数据层不支持修改操作；When the reference condition is reached, the first ordered data layer is converted into a second ordered data layer, the second ordered data layer is located in the memory, and the second ordered data layer does not support modification operations;
    将所述第二有序数据层中的键值对写入所述目标数据结构的第三数据层中，所述第三数据层位于硬盘中，且所述第三数据层支持追加写操作；writing the key-value pairs in the second ordered data layer into the third data layer of the target data structure, where the third data layer is located in the hard disk, and the third data layer supports additional write operations;
    当所述第三数据层的数据量达到第二数据量阈值时，使用所述第三数据层中的键值对和所述目标数据结构的第四有序数据层中的键值对，更新所述第四有序数据层中的键值对，所述第四有序数据层位于硬盘中，且所述第四有序数据层支持追加写操作。When the data volume of the third data layer reaches the second data volume threshold, use the key-value pair in the third data layer and the key-value pair in the fourth ordered data layer of the target data structure to update The key-value pair in the fourth ordered data layer, the fourth ordered data layer is located in the hard disk, and the fourth ordered data layer supports additional write operations.
  6. 根据权利要求1至5任一所述的方法，其特征在于，所述I/O操作请求为读操作请求，所述待操作数据为待读取数据，所述在所述目标数据结构中对所述键值对进行I/O操作，包括：The method according to any one of claims 1 to 5, wherein the I/O operation request is a read operation request, the data to be operated is data to be read, and the data to be read in the target data structure The key-value pair performs I/O operations, including:
    查找所述目标数据结构的第一有序数据层中是否存储有所述待读取数据的键，所述第一 有序数据层位于内存中，且所述第一有序数据层支持更新写操作；Find out whether the key of the data to be read is stored in the first ordered data layer of the target data structure, the first ordered data layer is located in the memory, and the first ordered data layer supports update writing operate;
    当所述第一有序数据层中未存储所述待读取数据的键时，查找所述目标数据结构的第二有序数据层中是否存储有所述待读取数据的键，所述第二有序数据层位于内存中，且所述第二有序数据层不支持修改操作；When the key of the data to be read is not stored in the first ordered data layer, find out whether the key of the data to be read is stored in the second ordered data layer of the target data structure, and the The second ordered data layer is located in memory, and the second ordered data layer does not support modification operations;
    当所述第二有序数据层中未存储所述待读取数据的键时，查找所述目标数据结构的第三数据层中是否存储有所述待读取数据的键，所述第三数据层位于硬盘中，且所述第三数据层支持追加写操作；或者，当所述第一有序数据层中未存储所述待读取数据的键，且所述目标数据结构中不存在所述第二有序数据层时，查找所述第三有序数据层中是否存储有所述待读取数据的键；When the key of the data to be read is not stored in the second ordered data layer, find out whether the key of the data to be read is stored in the third data layer of the target data structure, and the third data layer The data layer is located in the hard disk, and the third data layer supports additional write operations; or, when the key of the data to be read is not stored in the first ordered data layer, and the target data structure does not exist In the second ordered data layer, searching whether the key of the data to be read is stored in the third ordered data layer;
    当所述第三数据层中未存储所述待读取数据的键时，查找所述目标数据结构的第四有序数据层中是否存储有所述待读取数据的键，所述第四有序数据层位于硬盘中，且所述第四有序数据层支持追加写操作。When the key of the data to be read is not stored in the third data layer, find out whether the key of the data to be read is stored in the fourth ordered data layer of the target data structure, and the fourth order data layer The in-order data layer is located in the hard disk, and the fourth in-order data layer supports additional write operations.
  7. 根据权利要求5或6所述的方法，其特征在于，所述第四有序数据层包括根节点、中间节点和叶子节点，所述根节点和所述中间节点均用于存储所述待操作数据的键，所述根节点能够存储的键的范围覆盖所述中间节点能够存储的键的范围，所述叶子节点存储有所述中间节点存储的键对应的键值对；The method according to claim 5 or 6, wherein the fourth ordered data layer includes a root node, an intermediate node and a leaf node, and the root node and the intermediate node are both used for storing the to-be-operated the key of the data, the range of the key that the root node can store covers the range of the key that the intermediate node can store, and the leaf node stores the key-value pair corresponding to the key stored by the intermediate node;
    所述根节点还存储有用于存储所述根节点存储的键所覆盖的键的中间节点的节点标识，所述中间节点中还存储有用于存储所述中间节点存储的键所覆盖的键的叶子节点的节点标识；The root node also stores the node identifier of the intermediate node used to store the key covered by the key stored by the root node, and the intermediate node also stores the leaf used to store the key covered by the key stored by the intermediate node. Node ID of the node;
    所述键值存储***中还存储有哈希映射表，所述哈希映射表用于存储所述根节点、所述中间节点和所述叶子节点中至少一个节点存储的节点标识与节点标识指示的节点的存储位置信息的对应关系。A hash mapping table is also stored in the key-value storage system, and the hash mapping table is used to store the node identification and node identification indication stored by at least one node in the root node, the intermediate node and the leaf node. The corresponding relationship of the storage location information of the nodes.
  8. 根据权利要求5至7任一所述的方法，其特征在于，所述第三数据层为无序数据层。The method according to any one of claims 5 to 7, wherein the third data layer is an unordered data layer.
  9. 根据权利要求5至8任一所述的方法，其特征在于，所述第四有序数据层为树状数据结构，所述第四有序数据层的深度小于3。The method according to any one of claims 5 to 8, wherein the fourth ordered data layer is a tree-like data structure, and the depth of the fourth ordered data layer is less than 3.
  10. 根据权利要求5至9任一所述的方法，其特征在于，更新后的第四有序数据层的存储位置与更新前的第四有序数据层的存储位置不同，且更新前的第四有序数据层和更新后的第四有序数据层使用不同版本号进行标识。The method according to any one of claims 5 to 9, wherein the storage location of the fourth ordered data layer after the update is different from the storage location of the fourth ordered data layer before the update, and the storage location of the fourth ordered data layer before the update is different. The ordered data layer and the updated fourth ordered data layer are identified with different version numbers.
  11. 根据权利要求1至10任一所述的方法，其特征在于，所述键包括所述待操作数据的逻辑地址，所述逻辑地址通过数值表示，所述多个数据结构中同一数据结构中存储的键值对中逻辑地址对指定数值的余数相等，不同数据结构中存储的键值对中逻辑地址对指定数值的余数不等。The method according to any one of claims 1 to 10, wherein the key includes a logical address of the data to be operated, the logical address is represented by a numerical value, and the same data structure of the plurality of data structures stores The remainder of the logical address pair in the key-value pair is equal to the specified value, and the remainder of the logical address pair in the key-value pair stored in different data structures is not equal to the specified value.
  12. 根据权利要求1至11任一所述的方法，其特征在于，所述键值存储***包括硬盘，所述硬盘中存储有日志文件，所述方法还包括：The method according to any one of claims 1 to 11, wherein the key-value storage system comprises a hard disk in which log files are stored, and the method further comprises:
    接收快照删除请求；Receive a snapshot deletion request;
    将所述快照删除请求写在所述日志文件中，并发送针对所述快照删除请求的操作完成响应；Write the snapshot deletion request in the log file, and send an operation completion response for the snapshot deletion request;
    基于所述日志文件中记录的快照删除请求，执行所述快照删除请求指示的快照删除操作。Based on the snapshot deletion request recorded in the log file, the snapshot deletion operation indicated by the snapshot deletion request is performed.
  13. 一种键值存储***的数据管理装置，其特征在于，所述键值存储***的逻辑地址空间包括：多个子逻辑地址空间，所述多个子逻辑地址空间互不重叠，所述多个子逻辑地址空间中存储的数据分别使用多个数据结构一一对应管理，所述装置包括：A data management device for a key-value storage system, characterized in that the logical address space of the key-value storage system includes: multiple sub-logical address spaces, the multiple sub-logical address spaces do not overlap each other, and the multiple sub-logical address spaces The data stored in the space is managed in a one-to-one correspondence with multiple data structures, and the device includes:
    获取模块，用于获取输入/输出I/O操作请求，所述I/O操作请求中携带有待操作数据的索引信息；an acquisition module, configured to acquire an input/output I/O operation request, where the I/O operation request carries index information of the data to be operated;
    确定模块，用于基于所述索引信息，确定所述待操作数据的键；a determining module, configured to determine the key of the data to be operated based on the index information;
    所述确定模块，还用于基于所述键，在所述多个数据结构中确定用于存储所述待操作数据的键值对的目标数据结构；The determining module is further configured to determine a target data structure for storing the key-value pair of the data to be operated in the plurality of data structures based on the key;
    处理模块，用于在所述目标数据结构中对所述键值对进行I/O操作。A processing module, configured to perform I/O operations on the key-value pair in the target data structure.
  14. 根据权利要求13所述的装置，其特征在于，所述索引信息包括所述I/O操作请求的发起者的标识和所述待操作数据的逻辑地址，所述确定模块，具体用于：The device according to claim 13, wherein the index information includes an identifier of an initiator of the I/O operation request and a logical address of the data to be operated, and the determining module is specifically configured to:
    将所述索引信息确定为所述键。The index information is determined as the key.
  15. 根据权利要求14所述的装置，其特征在于，I/O操作请求的发起者为云盘。The apparatus according to claim 14, wherein the initiator of the I/O operation request is a cloud disk.
  16. 根据权利要求13所述的装置，其特征在于，所述索引信息包括用于索引所述键的关键字，所述确定模块，具体用于：The apparatus according to claim 13, wherein the index information includes a keyword for indexing the key, and the determining module is specifically configured to:
    基于所述关键字进行索引，得到所述键，所述键包括所述I/O操作请求的发起者的标识和所述待操作数据的逻辑地址。Indexing is performed based on the keyword to obtain the key, where the key includes the identifier of the initiator of the I/O operation request and the logical address of the data to be operated.
  17. 根据权利要求13至16任一所述的装置，其特征在于，所述I/O操作请求为写操作请求，所述处理模块，具体用于：The device according to any one of claims 13 to 16, wherein the I/O operation request is a write operation request, and the processing module is specifically configured to:
    将所述键值对存储在所述目标数据结构的第一有序数据层中，所述第一有序数据层位于内存中，且所述第一有序数据层支持更新写操作；storing the key-value pair in a first ordered data layer of the target data structure, the first ordered data layer is located in memory, and the first ordered data layer supports update write operations;
    当达到参考条件时，将所述第一有序数据层转换为第二有序数据层，所述第二有序数据层位于内存中，且所述第二有序数据层不支持修改操作；When the reference condition is reached, the first ordered data layer is converted into a second ordered data layer, the second ordered data layer is located in the memory, and the second ordered data layer does not support modification operations;
    将所述第二有序数据层中的键值对写入所述目标数据结构的第三数据层中，所述第三数据层位于硬盘中，且所述第三数据层支持追加写操作；writing the key-value pairs in the second ordered data layer into the third data layer of the target data structure, where the third data layer is located in the hard disk, and the third data layer supports additional write operations;
    当所述第三数据层的数据量达到第二数据量阈值时，使用所述第三数据层中的键值对和所述目标数据结构的第四有序数据层中的键值对，更新所述第四有序数据层中的键值对，所述第四有序数据层位于硬盘中，且所述第四有序数据层支持追加写操作。When the data volume of the third data layer reaches the second data volume threshold, use the key-value pair in the third data layer and the key-value pair in the fourth ordered data layer of the target data structure to update The key-value pair in the fourth ordered data layer, the fourth ordered data layer is located in the hard disk, and the fourth ordered data layer supports additional write operations.
  18. 根据权利要求13至17任一所述的装置，其特征在于，所述I/O操作请求为读操作请 求，所述待操作数据为待读取数据，所述处理模块，具体用于：The device according to any one of claims 13 to 17, wherein the I/O operation request is a read operation request, the data to be operated is data to be read, and the processing module is specifically used for:
    查找所述目标数据结构的第一有序数据层中是否存储有所述待读取数据的键，所述第一有序数据层位于内存中，且所述第一有序数据层支持更新写操作；Find out whether the key of the data to be read is stored in the first ordered data layer of the target data structure, the first ordered data layer is located in the memory, and the first ordered data layer supports update writing operate;
    当所述第一有序数据层中未存储所述待读取数据的键时，查找所述目标数据结构的第二有序数据层中是否存储有所述待读取数据的键，所述第二有序数据层位于内存中，且所述第二有序数据层不支持修改操作；When the key of the data to be read is not stored in the first ordered data layer, find out whether the key of the data to be read is stored in the second ordered data layer of the target data structure, and the The second ordered data layer is located in memory, and the second ordered data layer does not support modification operations;
    当所述第二有序数据层中未存储所述待读取数据的键时，查找所述目标数据结构的第三数据层中是否存储有所述待读取数据的键，所述第三数据层位于硬盘中，且所述第三数据层支持追加写操作；或者，当所述第一有序数据层中未存储所述待读取数据的键，且所述目标数据结构中不存在所述第二有序数据层时，查找所述第三有序数据层中是否存储有所述待读取数据的键；When the key of the data to be read is not stored in the second ordered data layer, find out whether the key of the data to be read is stored in the third data layer of the target data structure, and the third data layer The data layer is located in the hard disk, and the third data layer supports additional write operations; or, when the key of the data to be read is not stored in the first ordered data layer, and the target data structure does not exist In the second ordered data layer, searching whether the key of the data to be read is stored in the third ordered data layer;
    当所述第三数据层中未存储所述待读取数据的键时，查找所述目标数据结构的第四有序数据层中是否存储有所述待读取数据的键，所述第四有序数据层位于硬盘中，且所述第四有序数据层支持追加写操作。When the key of the data to be read is not stored in the third data layer, find out whether the key of the data to be read is stored in the fourth ordered data layer of the target data structure, and the fourth order data layer The in-order data layer is located in the hard disk, and the fourth in-order data layer supports additional write operations.
  19. 根据权利要求17或18所述的装置，其特征在于，所述第四有序数据层包括根节点、中间节点和叶子节点，所述根节点和所述中间节点均用于存储所述待操作数据的键，所述根节点能够存储的键的范围覆盖所述中间节点能够存储的键的范围，所述叶子节点存储有所述中间节点存储的键对应的键值对；The apparatus according to claim 17 or 18, wherein the fourth ordered data layer comprises a root node, an intermediate node and a leaf node, and the root node and the intermediate node are both used to store the to-be-operated the key of the data, the range of the key that the root node can store covers the range of the key that the intermediate node can store, and the leaf node stores the key-value pair corresponding to the key stored by the intermediate node;
    所述根节点还存储有用于存储所述根节点存储的键所覆盖的键的中间节点的节点标识，所述中间节点中还存储有用于存储所述中间节点存储的键所覆盖的键的叶子节点的节点标识；The root node also stores the node identifier of the intermediate node used to store the key covered by the key stored by the root node, and the intermediate node also stores the leaf used to store the key covered by the key stored by the intermediate node. Node ID of the node;
    所述键值存储***中还存储有哈希映射表，所述哈希映射表用于存储所述根节点、所述中间节点和所述叶子节点中至少一个节点存储的节点标识与节点标识指示的节点的存储位置信息的对应关系。A hash mapping table is also stored in the key-value storage system, and the hash mapping table is used to store the node identification and node identification indication stored by at least one node in the root node, the intermediate node and the leaf node. The corresponding relationship of the storage location information of the nodes.
  20. 根据权利要求17至19任一所述的装置，其特征在于，所述第三数据层为无序数据层。The apparatus according to any one of claims 17 to 19, wherein the third data layer is an unordered data layer.
  21. 根据权利要求17至20任一所述的装置，其特征在于，所述第四有序数据层为树状数据结构，所述第四有序数据层的深度小于3。The apparatus according to any one of claims 17 to 20, wherein the fourth ordered data layer is a tree-like data structure, and the depth of the fourth ordered data layer is less than 3.
  22. 根据权利要求17至21任一所述的装置，其特征在于，更新后的第四有序数据层的存储位置与更新前的第四有序数据层的存储位置不同，且更新前的第四有序数据层和更新后的第四有序数据层使用不同版本号进行标识。The device according to any one of claims 17 to 21, wherein the storage location of the fourth ordered data layer after the update is different from the storage location of the fourth ordered data layer before the update, and the storage location of the fourth ordered data layer before the update is different. The ordered data layer and the updated fourth ordered data layer are identified with different version numbers.
  23. 根据权利要求13至22任一所述的装置，其特征在于，所述键包括所述待操作数据的逻辑地址，所述逻辑地址通过数值表示，所述多个数据结构中同一数据结构中存储的键值对中逻辑地址对指定数值的余数相等，不同数据结构中存储的键值对中逻辑地址对指定数值的余数不等。The device according to any one of claims 13 to 22, wherein the key includes a logical address of the data to be operated, the logical address is represented by a numerical value, and a same data structure among the multiple data structures stores The remainder of the logical address pair in the key-value pair is equal to the specified value, and the remainder of the logical address pair in the key-value pair stored in different data structures is not equal to the specified value.
  24. 根据权利要求13至23任一所述的装置，其特征在于，所述键值存储***包括硬盘，所述硬盘中存储有日志文件，所述装置还包括：The apparatus according to any one of claims 13 to 23, wherein the key-value storage system comprises a hard disk, and the hard disk stores log files, and the apparatus further comprises:
    接收模块，用于接收快照删除请求；The receiving module is used to receive the snapshot deletion request;
    所述处理模块，还用于将所述快照删除请求写在所述日志文件中，并发送针对所述快照删除请求的操作完成响应；The processing module is further configured to write the snapshot deletion request in the log file, and send an operation completion response for the snapshot deletion request;
    所述处理模块，还用于基于所述日志文件中记录的快照删除请求，执行所述快照删除请求指示的快照删除操作。The processing module is further configured to perform the snapshot deletion operation indicated by the snapshot deletion request based on the snapshot deletion request recorded in the log file.
  25. 一种计算机设备，其特征在于，包括存储器和处理器，所述存储器存储有程序指令，所述处理器运行所述程序指令以执行权利要求1至12任一所述的方法。A computer device, comprising a memory and a processor, wherein the memory stores program instructions, and the processor executes the program instructions to execute the method according to any one of claims 1 to 12.
  26. 一种计算机可读存储介质，其特征在于，包括程序指令，当所述程序指令在计算机设备上运行时，使得所述计算机设备执行如权利要求1至12任一所述的方法。A computer-readable storage medium, characterized by comprising program instructions, which, when the program instructions are executed on a computer device, cause the computer device to execute the method according to any one of claims 1 to 12.
PCT/CN2021/119141 2020-09-23 2021-09-17 Data management method for key-value storage system and device thereof WO2022063059A1 (en)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN202011011617 2020-09-23
CN202011011617.5 2020-09-23
CN202110220635.2 2021-02-26
CN202110220635.2A CN114253908A (en) 2020-09-23 2021-02-26 Data management method and device of key value storage system

Publications (1)

Publication Number Publication Date
WO2022063059A1 true WO2022063059A1 (en) 2022-03-31

Family

ID=80790906

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/119141 WO2022063059A1 (en) 2020-09-23 2021-09-17 Data management method for key-value storage system and device thereof

Country Status (2)

Country Link
CN (1) CN114253908A (en)
WO (1) WO2022063059A1 (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114780500A (en) * 2022-06-21 2022-07-22 平安科技（深圳）有限公司 Data storage method, device, equipment and storage medium based on log merging tree
CN115586871A (en) * 2022-10-28 2023-01-10 北京百度网讯科技有限公司 Data appending and writing method, device, equipment and medium for cloud computing scene
CN116069267A (en) * 2023-04-06 2023-05-05 苏州浪潮智能科技有限公司 Writing cache method, system, equipment and storage medium of RAID card
WO2024022329A1 (en) * 2022-07-25 2024-02-01 华为云计算技术有限公司 Data management method based on key value storage system and related device thereof

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117131012B (en) * 2023-08-28 2024-04-16 中国科学院软件研究所 Sustainable and extensible lightweight multi-version ordered key value storage system

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107315745A (en) * 2016-04-26 2017-11-03 北京京东尚科信息技术有限公司 A kind of personal letter storage method and system
CN109085997A (en) * 2017-06-13 2018-12-25 西部数据技术公司 Memory-efficient for nonvolatile memory continues key assignments storage
CN109947709A (en) * 2019-04-02 2019-06-28 北京百度网讯科技有限公司 Date storage method and device
CN109977111A (en) * 2017-12-28 2019-07-05 英特尔公司 Using the data management system based on hash and the key-value data structure based on tree
US20200134100A1 (en) * 2018-10-31 2020-04-30 EMC IP Holding Company LLC Storing metadata in a cuckoo tree

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107315745A (en) * 2016-04-26 2017-11-03 北京京东尚科信息技术有限公司 A kind of personal letter storage method and system
CN109085997A (en) * 2017-06-13 2018-12-25 西部数据技术公司 Memory-efficient for nonvolatile memory continues key assignments storage
CN109977111A (en) * 2017-12-28 2019-07-05 英特尔公司 Using the data management system based on hash and the key-value data structure based on tree
US20200134100A1 (en) * 2018-10-31 2020-04-30 EMC IP Holding Company LLC Storing metadata in a cuckoo tree
CN109947709A (en) * 2019-04-02 2019-06-28 北京百度网讯科技有限公司 Date storage method and device

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114780500A (en) * 2022-06-21 2022-07-22 平安科技（深圳）有限公司 Data storage method, device, equipment and storage medium based on log merging tree
WO2024022329A1 (en) * 2022-07-25 2024-02-01 华为云计算技术有限公司 Data management method based on key value storage system and related device thereof
CN115586871A (en) * 2022-10-28 2023-01-10 北京百度网讯科技有限公司 Data appending and writing method, device, equipment and medium for cloud computing scene
CN115586871B (en) * 2022-10-28 2023-10-27 北京百度网讯科技有限公司 Cloud computing scene-oriented data additional writing method, device, equipment and medium
CN116069267A (en) * 2023-04-06 2023-05-05 苏州浪潮智能科技有限公司 Writing cache method, system, equipment and storage medium of RAID card

Also Published As

Publication number Publication date
CN114253908A (en) 2022-03-29

Similar Documents

Publication Publication Date Title
WO2022063059A1 (en) Data management method for key-value storage system and device thereof
EP3746906B1 (en) Cache for efficient record lookups in an lsm data structure
US10564850B1 (en) Managing known data patterns for deduplication
US20230359644A1 (en) Cloud-based replication to cloud-external systems
US10102150B1 (en) Adaptive smart data cache eviction
US9317519B2 (en) Storage system for eliminating duplicated data
US7711916B2 (en) Storing information on storage devices having different performance capabilities with a storage system
JP2021508879A (en) Systems and methods for database management using additional dedicated storage devices
US11580162B2 (en) Key value append
US11093472B2 (en) Using an LSM tree file structure for the on-disk format of an object storage platform
US20200097205A1 (en) System and method for early removal of tombstone records in database
US11416145B2 (en) Efficient space management for high performance writable snapshots
US11210006B2 (en) Distributed scalable storage
WO2023165196A1 (en) Journal storage acceleration method and apparatus, and electronic device and non-volatile readable storage medium
US11169968B2 (en) Region-integrated data deduplication implementing a multi-lifetime duplicate finder
US11620261B2 (en) Writing data to an LSM tree file structure using consistent cache staging
US10824610B2 (en) Balancing write amplification and space amplification in buffer trees
US11016676B2 (en) Spot coalescing of distributed data concurrent with storage I/O operations
US10416901B1 (en) Storage element cloning in presence of data storage pre-mapper with multiple simultaneous instances of volume address using virtual copies
US11256434B2 (en) Data de-duplication
US20210133158A1 (en) Methods and systems for logging data transactions and managing hash tables
US10423533B1 (en) Filtered data cache eviction
US11620062B1 (en) Resource allocation techniques using a metadata log
CN115840662A (en) Data backup system and device
WO2021257994A1 (en) Sparse file system implemented with multiple cloud services

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21871429

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21871429

Country of ref document: EP

Kind code of ref document: A1