WO2017020576A1 - 一种键值存储***中文件压实的方法和装置 - Google Patents

一种键值存储***中文件压实的方法和装置 Download PDF

Info

Publication number
WO2017020576A1
WO2017020576A1 PCT/CN2016/074043 CN2016074043W WO2017020576A1 WO 2017020576 A1 WO2017020576 A1 WO 2017020576A1 CN 2016074043 W CN2016074043 W CN 2016074043W WO 2017020576 A1 WO2017020576 A1 WO 2017020576A1
Authority
WO
WIPO (PCT)
Prior art keywords
key
sstable
log
compacted
delete log
Prior art date
Application number
PCT/CN2016/074043
Other languages
English (en)
French (fr)
Inventor
陈世敏
朱良昌
张军
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to EP16832065.3A priority Critical patent/EP3316150B1/en
Publication of WO2017020576A1 publication Critical patent/WO2017020576A1/zh
Priority to US15/878,589 priority patent/US11232073B2/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/174—Redundancy elimination performed by the file system
    • G06F16/1748—De-duplication implemented within the file system, e.g. based on file segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/174—Redundancy elimination performed by the file system
    • G06F16/1744—Redundancy elimination performed by the file system using compression, e.g. sparse files
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/13—File access structures, e.g. distributed indices
    • G06F16/137—Hash-based
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
    • G06F16/162—Delete operations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G06F16/2282—Tablespace storage structures; Management thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/282—Hierarchical databases, e.g. IMS, LDAP data stores or Lotus Notes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0614—Improving the reliability of storage systems
    • G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
    • G06F3/065—Replication mechanisms
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
    • G06F3/0652—Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Definitions

  • the present invention relates to the field of data processing technologies, and in particular, to a method and apparatus for file compaction in a KV-Store (Key-Value Store) system.
  • KV-Store Key-Value Store
  • the KV-Store system is widely used in big data storage and processing. Its data model is based on Key-Value pairs.
  • the basic operations provided include GET and PUT operations. .
  • the process of the PUT operation of the server usually includes: first writing the Key-Value pair into the MemTable (Memory Table) in the memory; when the MemTable is full, creating in the external storage (referred to as external storage, such as a disk, etc.) An SSTable (Sorted String Table), and then the Key-Value pair in the MemTable is sorted into the SSTable.
  • the PUT operation replaces the old value with the new value of the Key-Value pair, it is still not Save the old value in an external SSTable to delete; and so on, as the PUT operation increases, the external memory can contain one or more SSTables, which contain a large number of old values that are accumulated and replaced. Space occupancy and performance issues. Subsequently, all or part of the SSTable is merged by the Compaction mechanism to remove the Key-Value pairs corresponding to the non-latest Value corresponding to the same Key, thereby achieving the purpose of saving storage space.
  • Compaction is generally multi-file compaction, that is, the integration between multiple SSTables; specifically, it can be divided into Major Compaction (full compaction) and Minor Compaction (partial compaction).
  • Major Compaction refers to all SSTables merged at one time
  • Minor Compaction refers to a partial merged SSTable.
  • Embodiments of the present invention provide a method and apparatus for file compaction in a key value storage system to reduce I/O bandwidth and memory resources required to perform a compaction operation, thereby enabling a compaction operation process Does not affect the rate at which other operations are performed to enhance the user experience.
  • a method for file compaction in a key value storage KV-Store system including:
  • the to-be-compacted SSTable is compacted according to the log to be deleted corresponding to the to-be-decompressed ordered string table SSTable, and a new one is generated.
  • the method further includes:
  • the target key is recorded in the Delete Log.
  • the recording the target key in the Delete Log includes:
  • the target key is recorded in the Delete Log when it is determined that the target key is not included in the Delete Log.
  • the to-be-compacted sequence string table corresponding to the SSTable to be deleted The log is deleted, and the SSTable to be compacted is compacted to generate a new SSTable, including:
  • the to-be-compacted SSTable is compacted according to the log to be deleted corresponding to the to-be-decompressed ordered string table SSTable, and a new one is generated.
  • the method further includes:
  • the SSTable in which the new value corresponding to the to-be-searched key is located is determined, and the to-be-searched is recorded in the Delete Log corresponding to the determined SSTable. Key.
  • the KV-Store system is applied to an incremental storage scenario, where the log to be deleted corresponding to the to-be-compacted string table SSTable is deleted, Before the compaction of the SSTable is performed to generate a new SSTable, the method further includes:
  • the SSTable in which the latest value corresponding to the to-be-searched key is located is determined, and the to-be-searched key is recorded in the Delete Log corresponding to the determined SSTable. ;
  • the to-be-searched key is recorded in the Delete Log corresponding to the determined SSTable ,include:
  • the key to be searched is not included in the Delete Log corresponding to the determined SSTable, the key to be searched is recorded in the Delete Log corresponding to the determined SSTable.
  • the to-be-decompacted SSTable is compacted according to the log to be deleted corresponding to the to-be-compacted string table SSTable, and a new SSTable is generated.
  • the SSTable to be compacted is compacted according to the Delete Log to generate a new one. SSTable.
  • the to-be-compacted SSTable corresponds to a Bloom filter
  • the Bloom Filter records the Delete Log. Key
  • the method further includes:
  • the initial value of the Bloom Filter is set to be empty
  • the KV-Store system is a distributed storage system
  • the Delete Log is a local Delete Log
  • the key is determined according to the key recorded in the global Delete Log.
  • An initial value of the Bloom Filter wherein the Key in the local Delete Log is recorded in the global Delete Log.
  • a server including a key value storage KV-Store system, the server further comprising:
  • a compacting unit configured to compact the SSTable to be compacted according to the log to be deleted corresponding to the SSTable to be compacted, to generate a new SSTable; wherein the record in the Delete Log is a key key corresponding to the non-latest value in the KV-Store system saved in the compacted SSTable, where the new SSTable does not include a Key-Value pair corresponding to the Key in the Delete Log;
  • the deleting unit is configured to delete the to-be-compacted SSTable.
  • the server further includes:
  • a first determining unit configured to determine, in the to-be-compacted SSTable, a key corresponding to a non-latest Value in the KV-Store system as a target key;
  • a recording unit for recording the target key in the Delete Log.
  • the recording unit is specifically configured to: when determining that the target key is not included in the Delete Log, in the Delete The target key is recorded in the Log.
  • the compacting unit is specifically configured to: A new SSTable is generated by referring to the Key-Value corresponding to the Key in the Deleted SSTable that does not belong to the Delete Log.
  • the server further includes:
  • a receiving unit configured to receive a read GET operation carrying a key to be searched
  • a second determining unit configured to obtain, according to the GET operation, the corresponding Key to be searched After the latest value is determined, the SSTable in which the second new value corresponding to the to-be-searched key is located is determined, and the to-be-searched key is recorded in the Delete Log corresponding to the determined SSTable.
  • the KV-Store system is applied to an incremental storage scenario, and the server further includes:
  • a receiving unit configured to receive a read GET operation carrying a key to be searched
  • a second determining unit configured to determine, according to the GET operation, the latest value corresponding to the to-be-find key, determine an SSTable in which the latest value corresponding to the to-be-searched key is located, and a Delete Log corresponding to the determined SSTable Recording the to-be-searched key;
  • the receiving unit is further configured to receive a PUT operation carrying the key to be searched.
  • the second determining unit is configured to perform the Delete Log corresponding to the determined SSTable.
  • the key to be searched is recorded, it is specifically used to:
  • the key to be searched is not included in the Delete Log corresponding to the determined SSTable, the key to be searched is recorded in the Delete Log corresponding to the determined SSTable.
  • the compacting unit is specifically configured to: when the number of keys in the Delete Log corresponding to the to-be-deleted log corresponding to the SSTable to be compacted is greater than or equal to When the threshold is set, the SSTable to be compacted is compacted according to the Delete Log to generate a new SSTable.
  • the to-be-compacted SSTable corresponds to a Bloom filter
  • the key in the Delete Log is recorded in the Bloom Filter
  • the server further includes : a third determining unit for:
  • the initial value of the Bloom Filter is set to be empty
  • the KV-Store system is a distributed storage system
  • the Delete Log is a local Delete Log
  • the key is determined according to the key recorded in the global Delete Log.
  • An initial value of the Bloom Filter wherein the Key in the local Delete Log is recorded in the global Delete Log.
  • the compacted SSTable is compacted to generate a new SSTable; wherein the Delete Log records the to be compacted
  • the Key corresponding to the non-latest Value in the KV-Store system saved in the SSTable the new SSTable does not include the Key-Value pair corresponding to the Key in the Delete Log; then, the SSTable to be compacted is deleted, thereby realizing Single file compaction.
  • FIG. 1 is a schematic diagram of a process of performing a PUT operation provided in the prior art
  • FIG. 2 is a schematic diagram of a process for performing a GET operation provided in the prior art
  • FIG. 3 is a schematic flowchart of a method for file compaction in a KV-Store system according to Embodiment 1 of the present invention
  • FIG. 4 is a schematic diagram of a data structure involved in a server according to Embodiment 2 of the present invention.
  • FIG. 5 is a schematic flowchart of a method for recording a Key in a Delete Log according to Embodiment 2 of the present invention
  • FIG. 6 is a schematic flowchart diagram of another method for recording a Key in a Delete Log according to Embodiment 2 of the present invention.
  • FIG. 7 is a schematic flowchart diagram of a method for file compaction based on the KV-Store system of FIG. 5 or FIG. 6 according to Embodiment 3 of the present invention.
  • FIG. 8 is a schematic structural diagram of a Delete Log according to an embodiment of the present disclosure.
  • FIG. 9 is a schematic structural diagram of a server according to Embodiment 4 of the present invention.
  • FIG. 10 is a schematic structural diagram of another server according to Embodiment 4 of the present invention.
  • FIG. 11 is a schematic structural diagram of a server according to Embodiment 5 of the present invention.
  • the "KV-Store system” in the embodiment of the present invention may be a distributed storage system or a local storage system.
  • the KV-Store system when the "KV-Store system" is a distributed storage system, the KV-Store system can be distributed on multiple servers, and each part of the KV-Store system runs on each server; all data of the KV-Store system is Divided into multiple subsets, each subset is managed by a server's KV-Store system process.
  • the "KV-Store System” is a local file storage system, the KV-Store system is included on one server, and all data of the KV-Store system is managed by the KV-Store system process on the server.
  • the PUT operation carries a Key-Value pair, which can be represented as a PUT (Key, Value) operation.
  • the process of the PUT (Key, Value) operation of the server may include: first writing the Key-Value pair carried in the PUT operation into the MemTable in the memory; when the MemTable is full, creating an SSTable in the external memory, and the MemTable is The Key-Value pairs are written into the SSTable.
  • the "writing the Key-Value pair carried in the PUT operation to the MemTable” may include: if the MemTable already contains the Key carried by the PUT operation, the value corresponding to the Key included in the MemTable is updated to be carried by the PUT operation. If the MemTable does not contain the Key carried by the PUT operation, the Key-Value pair carried in the PUT operation is written in the MemTable.
  • the SSTable is only written once at the time of creation, and then becomes a read-only file. Can't be modified anymore, but can be deleted.
  • the PUT operation of the KV-Store system replaces the old Value with the new Value of the Key-Value pair, the old Value is still stored in an external SSTable.
  • the actual value of the actual external storage consumes more space than the value in the latest Key-Value pair.
  • the number of SSTables in the external memory will gradually increase.
  • the Keys of any two Key-Value pairs in an SSTable are different.
  • Different SSTables may contain Key-Value pairs with the same Key.
  • Different SSTables are arranged in the order of creation time. The Key-Value pairs in the same SSTable are sorted according to the size of the Key.
  • KV-Store system is a distributed storage system
  • all Key-Value pairs corresponding to one Key are recorded in the same server; Key-Value pairs corresponding to different Keys may be Recorded on a different server.
  • FIG. 1 there is shown a schematic diagram of a process for performing a PUT operation provided in the prior art.
  • the GET operation carries a Key, which can be represented as a GET (Key) operation, and is used to find the latest Value corresponding to the Key. It should be noted that the Key carried by the GET operation is hereinafter referred to as “Key to be searched”.
  • the process of performing a GET (Key) operation by the server may include: first searching for a Key carried by the GET operation in the MemTable, and outputting a Value corresponding to the Key if found; if not, searching from the latest SSTable according to the creation time. Find the Key carried by the GET operation in the oldest SSTable, and if found, output the Value corresponding to the Key. The value that is output is the latest Value corresponding to the key.
  • the server's memory caches a Bloom Filter and a Block Index corresponding to each SSTable.
  • the vector is the Bloom Filter representing A.
  • the Bloom Filter representing A.
  • the k hash functions of b we can calculate the k hash functions of b, and test the corresponding positions in the bit vector, that is, the positions corresponding to all Hashi(b), only when all the k bits in the hash function calculation result of b When both are 1, b may appear in A. If any of the k bits in the hash function calculation result of b is 0, then b must not be in A. Therefore, the Bloom Filter can quickly filter out the case where b does not belong to A.
  • the Block Index is used to record the Key and the location of the byte at a certain number of Key-Value pairs in the SSTable.
  • Querying the Key carried by the GET operation from the SSTable may include: after the test of the Bloom Filter corresponding to the SSTable, searching for the Key carried by the GET operation from the Block Index corresponding to the SSTable; if found in a Block Index The key searches for the key from the block corresponding to the Block Index, and outputs the value corresponding to the Key.
  • an SSTable can include multiple blocks.
  • FIG. 2 it is a schematic diagram of a process for performing a GET operation provided in the prior art.
  • BF in FIG. 2 indicates a Bloom Filter, that is, BFs described below;
  • Ind indicates a Block Index.
  • the Delete log is a log file created for the SSTable in the embodiment of the present invention, and is used to record the key corresponding to the non-latest Value in the corresponding SSTable.
  • the Log file is a special file that records information of an operation in chronological order, and only performs an append operation at the end of the file.
  • the Delete Log is located in the external memory.
  • a Delete Log is created for the SSTable; or, when the server generates a preset number of SSTables in the external storage, a Delete Log is created for the preset number of SSTables.
  • the value of the “predetermined quantity” and the value selection manner are not limited in the embodiment of the present invention.
  • a Bloom Filter is set in memory for the SSTable.
  • the "Bloom Filter corresponding to the SSTable” in the prior art is referred to as BFs
  • the newly added "Bloom Filter corresponding to the SSTable” in the embodiment of the present invention is referred to as BFd.
  • the BFd corresponding to the SSTable is used to filter the Key in the Delete Log corresponding to the SSTable.
  • FIG. 3 is a schematic flowchart diagram of a method for file compaction in a KV-Store system according to an embodiment of the present invention.
  • the method shown in Figure 3 is applied to a server containing a KV-Store system.
  • the method includes the following steps S301-S302:
  • S301 compact the SSTable according to the Delete Log corresponding to the SSTable to be compacted to generate a new SSTable, where the Delete Log records the non-latest Value in the KV-Store system to be compacted in the SSTable to be compacted. Key, the new SSTable does not contain the Key-Value pair corresponding to the Key in the Delete Log.
  • the execution body of the method provided by this embodiment may be a server that includes a KV-Store system.
  • the "server including the KV-Store system” may be any one of the servers distributed by the KV-Store system, that is, any server including the KV-Store system process.
  • the "to-compact SSTable” can be any SSTable in the KV-Store system.
  • the “to-compact SSTable” can be independently mapped to a Delete Log, or it can share a Delete Log with other SSTables in the KV-Store system.
  • the "Value in the KV-Store system” includes the value recorded in the memory and the external memory of all the servers where the KV-Store system is located, including the values recorded in the MemTable and the SSTable in all the servers where the KV-Store system is located. "The latest Value in the KV-Store system” can be recorded in the MemTable or SSTable. The “non-latest Value in the KV-Store system” is generally recorded in the SSTable.
  • the non-latest value refers to a value in a Key-Value pair that has multiple Key-Value pairs with the same Key and is not recorded at the latest time; specifically, may include: multiple with the same Key Key-Value pair, and in the Value of any one or more of 1, 2, 3, ..., (i-1) times; wherein, the value recorded by the i-1th time is the "secondary Value" corresponding to the key; i is an integer greater than or equal to 1. It should be noted that a plurality of Key-Value pairs having the same Key may be recorded in multiple SSTables, or may be recorded in MemTable and one/multiple SSTables.
  • the Key recorded in the "Delete Log” is recorded in at least the MemTable or the creation time of an SSTable after the SSTable to be compacted (ie, an SSTable updated than the SSTable to be compacted).
  • the specific implementation manner of recording a Key in the Delete Log is not limited in the embodiment of the present invention.
  • the step S301 is specifically implemented as: copying a Key-Value pair in the SSTable to be compacted and not belonging to the Key in the Delete Log, and generating a new SSTable. For example, by merging the SSTable and the Delete Log, the Key-Value pair corresponding to the Key in the Deleted SSTable that is not in the Delete Log is copied, and a new SSTable is generated.
  • the implementation method of “merging and compacting the SSTable and the Delete Log” can refer to the merge operation in the prior art.
  • the method may further include: creating a new Delete Log for the new SSTable to prepare for performing the compact operation on the new SSTable.
  • Step S302 can be understood as: replacing the to-be-compacted SSTable with the new SSTable generated in step S301.
  • the method may further include: deleting the information corresponding to the SSTable to be compacted, where the information corresponding to the SSTable to be compacted may include but not limited to one or more of the following information: The Delete Log corresponding to the SSTable, the Delete Log Buffer (the log cache to be deleted), and the BFd corresponding to the SSTable to be compacted.
  • the method for file compaction in the KV-Store system is to compact the SSTable according to the Delete Log corresponding to the SSTable to be compacted, and generate a new SSTable; wherein the Delete Log records the SSTable to be compacted.
  • the key corresponding to the non-latest Value in the saved KV-Store system the new SSTable does not include the Key-Value pair corresponding to the Key in the Delete Log; then, the SSTable to be compacted is deleted, thereby realizing the single The file is compacted.
  • Intraoperative multi-file compaction method requires less I/O bandwidth and less memory resources. In this way, while performing compaction operations using this method, more resources can be reserved for other operations without affecting The rate at which other operations are performed, thereby improving the user experience.
  • the single file compacting method provided by the embodiment of the present invention requires less data to be processed in each process of performing the compacting operation, and thus is required. The time is shorter; this way, more processing time can be reserved for other operations, thereby improving the user experience.
  • the multi-file compaction method provided in the prior art needs to read data in multiple SSTables each time the compaction operation is performed. Therefore, it is necessary to set a large read and write buffer in the memory, which requires a large memory resource.
  • the single file compaction method provided by the embodiment of the present invention only needs to read data in one SSTable, so compared with the prior art, the read/write buffer required to be set in the memory takes up less space, that is, Said, the use of less memory resources; this way, you can reserve more memory resources for other operations, thus improving the performance of the KV-Store system.
  • the step S301 may include: when the number of Keys in the Delete Log corresponding to the SSTable to be compacted is greater than or equal to a preset threshold, the compacted SSTable is compacted according to the Delete Log.
  • the compacted SSTable may be compacted under other triggering conditions, for example, compacting the SSTable at a predetermined time or periodically; wherein the predetermined time may be a specific time, or For the idle time of the server, etc.
  • the value of the preset threshold value and the method for obtaining the value are not limited.
  • the number of keys in the SSTable to be compacted may be half, and of course, other values may be used.
  • the method may further include: determining, in the to-be-compacted SSTable, a Key corresponding to the non-latest Value in the KV-Store system as the target key; in the Delete The target key is recorded in the Log.
  • the server can compare the SSTable and the MemTable to be compacted, so that the SSTable is to be compacted, and Recorded in MemTable Key is the target key. If the SSTable to be compacted is not the latest SSTable created in the KV-Store system, the server can compare the SSTable and the MemTable to be compacted, and compare the SSTable to be compacted and the SSTable after the SSTable to be compacted. Therefore, the key in the SSTable to be compacted in the SSTable and recorded in the MemTable/creation time after the SSTable to be compacted is used as the target key.
  • recording the target key in the Delete Log may include: recording the target key in the Delete Log when it is determined that the target key is not included in the Delete Log.
  • a Bloom Filter may be established in the memory of the server in advance, and the Key in the Delete Log is recorded in the Bloom Filter; in this case, “When it is determined that the target key is not included in the Delete Log, Recording the target key in the Delete Log may include: recording the target key in the Delete Log when it is determined that the target key is not included in the Bloom Filter.
  • the server can set a Bloom Filter (ie, BFd) for each SSTable in advance.
  • a Bloom Filter ie, BFd
  • the target key if the target key does not pass the test of the BFd corresponding to the SSTable to be compacted, the target key must not be recorded in the Delete Log.
  • the above "The target key is not included in the Bloom Filter" is considered that the target key has not passed the test of the Bloom Filter.
  • the Delete Log corresponds to a Delete Log Buffer in the memory; in this case, recording the target key in the Delete Log may include: when the number of target keys in the Delete Log Buffer is greater than or equal to a second preset threshold. When the target key is recorded in the Delete Log.
  • the server may set a Delete Log Buffer for each Delete Log in the memory, and first record the target Key in the Delete Log Buffer corresponding to the Delete Log; when the Delete Log Buffer is in the When the number of target keys is greater than or equal to the second preset threshold, the Key in the Delete Log Buffer is sequentially written to the Delete Log. Since the server reads and writes data in memory much faster than the rate of reading and writing data in the external memory, this optional implementation enables the server to speed up the rate of reading and writing data, thereby improving KV-Store. The overall performance of the system.
  • the embodiment of the present invention does not limit the specific value of the second preset threshold and the method for obtaining the value, for example, the maximum value of the number of target keys that can be accommodated in the Delete Log Buffer.
  • the embodiment of the present invention further provides a technical solution for recording a Key in the Delete according to the GET operation, and specifically includes the following two optional implementation manners:
  • An optional implementation manner is as follows: receiving a GET operation carrying a key to be searched; determining, after obtaining the latest value corresponding to the key to be searched according to the GET operation, determining an SSTable in which the second new value corresponding to the key to be searched is located, and corresponding to the determined SSTable The key to be found is recorded in the Delete Log.
  • the optional implementation is determined under the trigger condition that the server performs the GET operation in the prior art, that is, the trigger condition of the latest value corresponding to the to-be-searched key carried in the GET operation is determined by the server.
  • the SSTable in which the second new value corresponding to the key is located is searched, and the to-be-searched key is recorded in the Delete Log corresponding to the SSTable.
  • the "non-latest Value" in step S301 is embodied as "secondary new value" in the optional implementation manner.
  • the latest value corresponding to the key to be searched may be the value in the MemTable or the SSTable.
  • the server obtains the second new value corresponding to the key to be searched according to the method of obtaining the latest value corresponding to the key to be searched.
  • Optional implementation manner 2 When the KV-Store system is applied to the incremental storage scenario, the GET operation carrying the key to be searched for the key is received; after the latest value corresponding to the key to be searched is obtained according to the GET operation, the to-be-find is determined.
  • the SSTable in which the latest value corresponding to the Key is located records the key to be searched in the Delete Log corresponding to the determined SSTable.
  • each Key-Value pair is only GET once, and after the server performs the GET operation, a new Value for the Key is generated, and the implementation carries the Key and the new The PUT operation of the Value.
  • the method may further include: receiving a PUT operation carrying the key to be searched.
  • the “receiving the PUT operation carrying the key to be searched” may be performed after performing the “receiving the read GET operation carrying the key to be searched Key”, and performing the next GET operation or any one carrying the key other than the to-be-searched key. of Execute in any step before the PUT operation of the other Key.
  • the server will perform a PUT (New Key) operation after performing the GET operation, so that the PUT is executed after the PUT is executed.
  • the new Value After the Key, the new Value) operation, the "latest Value" obtained by the GET (Key to Find) will become the second new Value. Therefore, the optional implementation manner 2 is the same as the basic idea of the foregoing optional implementation manner 1.
  • the foregoing optional implementation manner 1 can be applied to the incremental storage scenario and the non-incremental storage scenario; and is applicable to the scenario in which the “latest Value” is in the MemTable or the SSTable.
  • the optional implementation 2 is applied to the incremental storage scenario; and is applicable to the "latest Value” in the scenario in the SSTable.
  • the “key to be searched” in the optional implementation manners one and two is The above "target key”.
  • the alternative implementations one and two can be used in conjunction with any one or more of the other alternatives provided herein.
  • the “recording the to-be-searched key in the Deleted Log corresponding to the determined SSTable” may include: corresponding to the determined SSTable when the key to be searched is not included in the Deleted Log corresponding to the determined SSTable. The key to be found is recorded in the Delete Log.
  • the embodiment of the present invention further provides how to create a Bloom Filter (ie, BFd) corresponding to the SSTable to be compacted after the server fails and is restored. Specifically, the following manners may be included:
  • the initial value of the Bloom Filter corresponding to the SSTable to be compacted is set to null; and then, according to the optional implementation provided above, the Bloom Filter corresponding to the SSTable to be compacted is written. Key.
  • the method 1 may cause the key recorded in the Delete Log corresponding to the SSTable to be compacted to be repeated to some extent, it can ensure that after the server recovers from the fault, the single solution according to the technical solution provided above is performed.
  • the correct rate in the process of file compaction; in addition, this method has the advantage of being simple to implement.
  • each server that includes the process of the KV-Store system can create a Delete Log for each local SSTable in its own external storage, the Delete Log. It can be called "local Delete Log"; in addition, you can create a global Delete Log in one or more of these servers to record the key in the local Delete Log in each server that contains the process of the KV-Store system. To achieve a backup of the local Delete Log. In this way, after the server of one or more processes including the KV-Store system fails, the Key in the local Delete Log in the failed server can be obtained from the global Delete Log.
  • each server that includes the KV-Store system process can periodically write the local Delete Log to the global Delete Log periodically or periodically.
  • other non-failed servers can transfer KV-Store system data from these failed servers to their own storage units.
  • the embodiment of the present invention does not limit the transfer mode. For example, the transfer may be performed by information recorded in the global Delete Log.
  • This embodiment provides two methods for recording a Key in a Delete Log.
  • the two methods are applied to a server containing a KV-Store system; the data structure involved in the server is shown in FIG.
  • the server in Figure 4 includes memory and external memory; there is a MemTable, several BFs, several BFds and several Delete Log Buffers in the memory; several SSTables and several Delete Logs are set in the external memory.
  • Each SSTable corresponds to one BFs; each SSTable corresponds to one Delete Log, each SSTable corresponds to one BFd, and each Delete Log corresponds to one Delete Log Buffer.
  • each SSTable in the server shown in FIG. 4 may also correspond to one or more Block Indexes, and the Block Index is not shown in FIG. 4.
  • FIG. 5 it is a schematic flowchart of a method for recording a Key in a Delete Log according to an embodiment of the present disclosure.
  • the method shown in Figure 5 can be applied to both incremental storage and non-incremental storage scenarios.
  • the method can include the following steps S501-S504:
  • step S501 When receiving the GET operation carrying the key to be searched, obtain the latest Value corresponding to the key to be searched according to the GET operation, and find the secondary new value corresponding to the key to be searched. If it is found, step S502 is performed; if not, it is ended.
  • S502 Determine an SSTable that records a second new value corresponding to the Key to be searched.
  • S503 Determine whether the to-be-searched key is recorded in the BFd corresponding to the determined SSTable.
  • step S504 is performed.
  • S504 Record the to-be-find key in the determined BFd corresponding to the SSTable, and record the to-be-find key in the determined Deleted corresponding SSTable.
  • the step S504 may include: recording the to-be-find key in the BFd corresponding to the SSTable to be compacted, and recording the to-be-searched key in the Delete Log Buffer corresponding to the SSTable to be compacted; if the Delete Log Buffer is full, Write the key recorded in the Delete Log Buffer to the Delete Log corresponding to the SSTable to be compacted.
  • the server can perform the operation on the GET (Key) according to the following algorithm:
  • the code in the fifth line is that the server finds the key to be searched for the first time, and the value corresponding to the key to be found that is found this time is the latest value corresponding to the key to be searched; and the SSTable in which the latest value is located is SSTable. [i].
  • the code in the 10th line above indicates that the server searches for the key to be searched for the second time. The value corresponding to the key to be searched for this time is the new value corresponding to the key to be searched; the SSTable of the new value is SSTable[j] .
  • FIG. 6 another flow chart of a method for recording a Key in a Delete Log according to the embodiment is provided.
  • the method shown in FIG. 6 is applied to an incremental storage scenario, and the method may include the following steps S601-S604:
  • step S601 When receiving the GET operation carrying the key to be searched, obtain the latest Value corresponding to the key to be searched according to the GET operation. If it is found, step S602 is performed; if not, it is ended.
  • S602 Determine an SSTable in which the latest Value is recorded.
  • S603-S604 Same as S503-S504 described above.
  • the server may perform an incremental storage scenario according to the following algorithm:
  • FIG. 7 is a schematic flowchart diagram of a method for file compaction in a KV-Store system based on FIG. 5 or FIG. 6 according to the embodiment.
  • the method shown in Figure 7 includes the following steps S701-S704:
  • S701 Determine whether the number of Keys in the Delete Log corresponding to the SSTable to be compacted is greater than or equal to a preset threshold.
  • step S702 If yes, go to step S702; if no, end.
  • the “to-compact SSTable” may be any SSTable in the server where the KV-Store system is located.
  • the server may perform step S701 for any SSTable at a time or periodically or in a triggering manner.
  • step S701 determines whether the server has performed the Delete Log as shown in Figure 6. If the result of the determination in step S701 is "NO", the server does not perform the operation of compacting the table to be compacted after performing the determining operation of step S701; in this case, the server may continue to execute the above FIG. 5 or Record the Key in the Delete Log as shown in Figure 6.
  • the "sorting" performed in this step is the basis for performing the "merging" operation in the subsequent step S703.
  • S703 merge the SSTable and the sorted Delete Log to obtain a new SSTable; wherein the Key-Value pair recorded in the new SSTable is the SSTable to be compacted The Key-Value pair corresponding to the Key in the Delete Log.
  • step S703 can be implemented by the following algorithm:
  • MemTable/SSTable Key MemTable Key1, Key2, Key3 SSTable1 Key1, Key2, Key3, Key4, Key5 SSTable2 Key1, Key4, Key5, Key7, Key8 SSTable3 Key2, Key3, Key6, Key8, Key9
  • the server can merge SSTable1 and Delete Log1, and merge to obtain a new SSTable1.
  • the Keys recorded in the MemTable and SSTable in the KV-Store system are as shown in Table 3:
  • MemTable/SSTable Key MemTable Key1, Key2, Key3 New SSTable1 Key4, Key5 SSTable2 Key1, Key4, Key5, Key7, Key8 SSTable3 Key2, Key3, Key6, Key8, Key9
  • the server successively executes GET (Key1), GET (Key2), GET (Key3), GET (Key4), GET (Key5), and the Delete Log corresponding to each SSTable in the KV-Store system.
  • the Key is shown in Table 4:
  • MemTable/SSTable Key MemTable Key1, Key2, Key3 New SSTable1 Key4, Key5 New SSTable2 Key7, Key8 SSTable3 Key2, Key3, Key6, Key8, Key9
  • the server can perform compaction operation on any SSTable whose number of keys in the corresponding Delete Log meets “greater than or equal to a preset threshold” according to the above method, thereby achieving the purpose of saving storage space.
  • the Delete Log may be divided into multiple Delete Blocks, and each Delete Block is used to record an SSTable.
  • the Key corresponding to the non-latest Value, and the Key corresponding to the non-latest Value in an SSTable can be recorded in multiple Delete Blocks respectively.
  • the Keys corresponding to the non-latest Values in an SSTable can be recorded in consecutive or non-contiguous Delete Blocks in the Delete Log, respectively.
  • the server can save each in memory.
  • the position of the last Delete Block corresponding to the SSTable, and when the next Delete Block is generated, the position of the previous Delete Block is recorded into the new Delete Block, thus forming a single chain from the back to the front.
  • the key recorded in the plurality of Delete Blocks corresponding to the SSTable may be read from the back and forward according to the single chain.
  • FIG. 8 is a schematic structural diagram of a Delete Log according to an embodiment of the present invention.
  • an SSTable corresponds to multiple non-contiguous Delete Blocks.
  • SSTable[i] represents the i-th SSTable
  • SSTable[j] represents the j-th SSTable
  • i, j are integers greater than or equal to 0, and i is not equal to j.
  • FIG. 9 is a schematic structural diagram of a server according to an embodiment of the present invention.
  • the server shown in FIG. 9 is for performing the method of file compaction in any of the KV-Store systems provided above.
  • the related content in this embodiment reference may be made to the foregoing method embodiments, and details are not described herein again.
  • the server 9 shown in FIG. 9 includes a KV-Store system, which may include a compacting unit 91 and a deleting unit 92. specific:
  • the compacting unit 91 is configured to compact the SSTable to be compacted according to the log to be deleted corresponding to the table string table SSTable to be compacted, to generate a new SSTable, wherein the Delete Log records The key Key corresponding to the non-latest value Value in the KV-Store system to be compacted in the SSTable, and the new SSTable does not include the Key-Value pair corresponding to the Key in the Delete Log.
  • the deleting unit 92 is configured to delete the to-be-compacted SSTable.
  • the server 9 may further include:
  • a first determining unit 93 configured to determine, in the to-be-compacted SSTable, a Key corresponding to the non-latest Value in the KV-Store system as a target key;
  • the recording unit 94 is configured to record the target key in the Delete Log.
  • the recording unit 96 may be specifically configured to record the target key in the Delete Log when it is determined that the target key is not included in the Delete Log.
  • the compacting unit 91 is specifically configured to: copy a Key-Value corresponding to the Key in the Deleted SSTable that does not belong to the Delete Log, and generate a new SSTable.
  • the server 1 may further include: a receiving unit 95, configured to receive a read GET operation carrying a key to be searched; and a second determining unit 96, configured to After the GET operation obtains the latest value corresponding to the to-be-find key, And determining the to-be-find key in the Delete Log corresponding to the determined SSTable.
  • a receiving unit 95 configured to receive a read GET operation carrying a key to be searched
  • a second determining unit 96 configured to After the GET operation obtains the latest value corresponding to the to-be-find key, And determining the to-be-find key in the Delete Log corresponding to the determined SSTable.
  • the KV-Store system is applied to an incremental storage scenario.
  • the server 1 may further include: a receiving unit 95, configured to receive and carry The read GET operation of the key to be searched; the second determining unit 96 is configured to determine, after obtaining the latest value corresponding to the key to be searched according to the GET operation, the SSTable in which the latest value corresponding to the to-be-searched key is located, The key to be searched is recorded in the Delete Log corresponding to the determined SSTable.
  • the receiving unit 95 is further configured to receive a PUT operation carrying the key to be searched.
  • the second determining unit 96 records the to-be-searched key in the Delete Log corresponding to the determined SSTable, specifically, the When the key to be searched is not included in the Delete Log corresponding to the determined SSTable, the key to be searched is recorded in the Delete Log corresponding to the determined SSTable.
  • the compacting unit 91 is specifically configured to: when the number of keys in the Delete Log corresponding to the to-be-deleted ordered string table SSTable is greater than or equal to a preset threshold, according to the Delete Log And compacting the to-be-compacted SSTable to generate a new SSTable.
  • the to-be-compacted SSTable corresponds to a Bloom filter; the key in the Delete Log is recorded in the Bloom Filter, as shown in FIG. 10, the server 9 may further include: a determining unit 97, configured to: when the server where the KV-Store system is located is faulty, set an initial value of the Bloom Filter to be empty; or, in the KV-Store system, a distributed storage system, and When the Delete Log is the local Delete Log, after the server where the KV-Store system is located is recovered, the initial value of the Bloom Filter is determined according to the Key recorded in the global Delete Log; wherein the global Delete Log records The Key in the local Delete Log.
  • a determining unit 97 configured to: when the server where the KV-Store system is located is faulty, set an initial value of the Bloom Filter to be empty; or, in the KV-Store system, a distributed storage system, and When the Delete Log is the local Delete Log, after the server where the KV-Store system is located is recovered,
  • the server provided in this embodiment compresses the SSTable according to the Delete Log corresponding to the SSTable to be compacted, and generates a new SSTable.
  • the Delete Log records the KV-Store system to be compacted in the SSTable.
  • the key corresponding to the non-latest Value, the new SSTable does not include the Key-Value pair corresponding to the Key in the Delete Log; then, the SSTable to be compacted is deleted, thereby achieving single file compaction.
  • each compaction In operation only one data in the SSTable needs to be read; compared with the multi-file compaction method in the prior art, the I/O bandwidth and memory resources to be occupied are small, so that the compaction operation is performed by using the method. At the same time, more resources can be reserved for other operations so as not to affect the rate at which other operations are performed, thereby improving the user experience.
  • processors embedded in or independent of the server 9 in hardware form may also be stored in the memory of the server 9 in software, so that the processor calls to perform operations corresponding to the above modules, and the processor may be central.
  • Processing unit CPU, Central Processing Unit
  • microprocessor microcontroller, etc.
  • FIG. 11 is a schematic structural diagram of a server according to an embodiment of the present invention.
  • the server shown in FIG. 11 is for performing the method of file compaction in any of the KV-Store systems provided above.
  • the related content in this embodiment reference may be made to the foregoing method embodiments, and details are not described herein again.
  • the server 11 shown in FIG. 11 includes a KV-Store system, which may include a memory 11A, a processor 11B, a wireless interface 11C, and a bus system 11D. Among them, the memory 11A, the processor 11B and the wireless interface 11C are coupled together by the bus system 11D.
  • the memory 11A may include a high speed RAM (Random Access Memory), and may also include a non-volatile memory such as at least one disk memory.
  • a high speed RAM Random Access Memory
  • non-volatile memory such as at least one disk memory.
  • the wireless interface 11C is used by the server 11 to communicate with other devices.
  • the bus system 11D may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component) bus, or an EISA (Extended Industry Standard Architecture) bus; In addition to the data bus, it can also include a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are labeled as the bus system 11D in the figure.
  • ISA Industry Standard Architecture
  • PCI Peripheral Component
  • EISA Extended Industry Standard Architecture
  • the memory 11A is for storing a program.
  • the program may include program code, the process The sequence code includes computer operating instructions.
  • the processor 11B executes the program stored in the memory 11A to implement the file compaction method in the KV-Store system provided by the embodiment of the present invention.
  • the method may include: compacting the to-be-compacted SSTable according to the log to be deleted corresponding to the to-be-decoded string table SSTable, to generate a new SSTable; wherein the Delete Log records the The key key corresponding to the non-latest value in the KV-Store system saved in the SSTable is compacted, and the new SSTable does not include the Key-Value pair corresponding to the Key in the Delete Log; Compact the SSTable.
  • the processor 11B is further configured to: determine, in the to-be-compacted SSTable, a Key corresponding to the non-latest Value in the KV-Store system as a target key; and record the location in the Delete Log The target key.
  • the processor 11B may be specifically configured to: when determining that the target key is not included in the Delete Log, record the in the Delete Log Target Key.
  • the processor 11B performs the compaction of the to-be-deprecated SSTable according to the to-be-deprecated log Delete Log corresponding to the to-be-compressed ordered string table SSTable, and generates a new SSTable, which may be specifically used for: copying The Key-Value corresponding to the Key in the Deleted SSTable that is not in the Delete Log generates a new SSTable.
  • the processor 11B is further configured to: receive a read GET operation carrying a key to be searched; and after obtaining the latest value corresponding to the to-be-find key according to the GET operation, Determining the SSTable in which the sub-new value corresponding to the to-be-searched key is located, and recording the to-be-find key in the Delete Log corresponding to the determined SSTable.
  • the KV-Store system is applied to an incremental storage scenario, and the processor 11B is further configured to: receive a read GET operation carrying a key to be searched; and according to the GET After the operation obtains the latest value corresponding to the to-be-find key, the SSTable in which the latest value corresponding to the to-be-searched key is located is determined, and the to-be-searched key is recorded in the Delete Log corresponding to the determined SSTable; The PUT operation of the Key to be searched.
  • the processor 11B when the processor 11B records the to-be-searched key in the Delete Log corresponding to the determined SSTable, the processor 11B is specifically configured to: When the key to be searched is not included in the Delete Log corresponding to the determined SSTable, the key to be searched is recorded in the Delete Log corresponding to the determined SSTable.
  • the processor 11B is configured to perform compaction on the to-be-deprecated SSTable corresponding to the to-be-deprecated log table corresponding to the table to be compacted, and to generate a new SSTable. If the number of keys in the Delete Log corresponding to the table to be deleted is greater than or equal to the preset threshold, the SSTable to be compacted is compacted according to the Delete Log to generate a new SSTable. .
  • the to-be-compacted SSTable corresponds to a Bloom filter; the Bloom Filter records the Key in the Delete Log, and the processor 11B is further configured to: when the KV-Store system is located After the server fails, the initial value of the Bloom Filter is set to be empty; or, when the KV-Store system is a distributed storage system, and the Delete Log is a local Delete Log, when the KV-Store After the fault is rectified, the initial value of the Bloom Filter is determined according to the key recorded in the global Delete Log. The key in the local Delete Log is recorded in the global Delete Log.
  • the server provided in this embodiment compresses the SSTable according to the Delete Log corresponding to the SSTable to be compacted, and generates a new SSTable.
  • the Delete Log records the KV-Store system to be compacted in the SSTable.
  • the key corresponding to the non-latest Value, the new SSTable does not include the Key-Value pair corresponding to the Key in the Delete Log; then, the SSTable to be compacted is deleted, thereby achieving single file compaction.
  • the disclosed system, apparatus, and method may be implemented in other manners.
  • the device embodiments described above are merely illustrative.
  • the division of the unit is only a logical function division, and may be implemented in actual implementation.
  • multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.
  • each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit.
  • the above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units.
  • the above-described integrated unit implemented in the form of a software functional unit can be stored in a computer readable storage medium.
  • the software functional units described above are stored in a storage medium and include instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to perform portions of the steps of the methods described in various embodiments of the present invention.
  • the foregoing storage medium includes: a U disk, a mobile hard disk, a ROM (Read-Only Memory), a RAM, a magnetic disk, or an optical disk, and the like, which can store program codes.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Human Computer Interaction (AREA)
  • Computer Security & Cryptography (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Debugging And Monitoring (AREA)

Abstract

一种键值存储***中文件压实的方法和装置，涉及数据处理技术领域，用以减少执行压实操作时所需要占用的I/O带宽和内存资源，从而使得在执行压实操作的过程中，不影响执行其他操作的速率，以提升用户体验。提供的方法包括：根据待压实SSTable对应的Delete Log，对待压实SSTable进行压实，生成新的SSTable；其中，该Delete Log中记录有待压实SSTable中保存的键值存储KV-Store***中的非最新Value所对应的Key，该新的SSTable中不包含该Delete Log中的Key对应的Key-Value对（S301）；删除待压实SSTable（S302）。

Description

一种键值存储***中文件压实的方法和装置 技术领域
本发明涉及数据处理技术领域，尤其涉及一种KV-Store(Key-Value Store，键值存储)***中文件压实的方法和装置。
背景技术
KV-Store***被广泛应用于大数据存储与处理中，其数据模型以Key-Value(键-值)对为基本单位；其提供的基本的操作包括GET(读)操作和PUT(写)操作。
服务器执行PUT操作的过程通常包括：先将Key-Value对写入内存中的MemTable(Memory Table，内存表)中；当MemTable已满时，在外部存储(简称外存，例如磁盘等)中创建一个SSTable(Sorted String Table，有序字符串表)，然后将MemTable中的Key-Value对排序写入该SSTable中，PUT操作在用Key-Value对的新值代替旧值的时候，并不对仍然保存在外存的某个SSTable中的旧值进行删除；依次类推，随着PUT操作的增多，外存中可以包含一个或多个SSTable，其中包含的不断积累的大量被替代的旧值也会带来空间占用和影响性能的问题。后续通过Compaction(压实)机制归并全部或部分SSTable，以去除同一Key对应的非最新Value所对应的Key-Value对，从而实现节省存储空间的目的。
目前，Compaction一般为多文件压实，即实现多个SSTable之间的归并；具体可以分为Major Compaction(全部压实)和Minor Compaction(部分压实)。其中，Major Compaction是指一次归并所有SSTable，Minor Compaction是指一次归并部分SSTable。
利用上述提供的方法执行压实操作时，需要同时读取多个待压实SSTable中的数据，这需要占用很大的I/O带宽和内存资源；这样，在利用上述提供的方法执行压实操作的同时，能够预留给其他操作的资源较少，使得执行其他操作的速率会很慢，从而影响用户的使用。
发明内容
本发明的实施例提供一种键值存储***中文件压实的方法和装置，用以减少执行压实操作时所需要占用的I/O带宽和内存资源，从而使得在执行压实操作的过程中，不影响执行其他操作的速率，以提升用户体验。
为达到上述目的，本发明的实施例采用如下技术方案：
第一方面，提供一种键值存储KV-Store***中文件压实的方法，包括：
根据待压实有序字符串表SSTable对应的待删除日志Delete Log，对所述待压实SSTable进行压实，生成新的SSTable；其中，所述Delete Log中记录有所述待压实SSTable中保存的所述KV-Store***中的非最新值Value所对应的键Key，所述新的SSTable中不包含所述Delete Log中的Key对应的Key-Value对；
删除所述待压实SSTable。
结合第一方面，在第一种可能的实现方式中，在所述根据待压实有序字符串表SSTable对应的待删除日志Delete Log，对所述待压实SSTable进行压实，生成新的SSTable之前，所述方法还包括：
在所述待压实SSTable中，确定所述KV-Store***中的非最新Value所对应的Key，作为目标Key；
在所述Delete Log中记录所述目标Key。
结合第一方面的第一种可能的实现方式，在第二种可能的实现方式中，所述在所述Delete Log中记录所述目标Key，包括：
在确定所述Delete Log中不包含所述目标Key时，在所述Delete Log中记录所述目标Key。
结合第一方面、第一方面的一种可能的实现方式或第二种可能的实现方式，在第三种可能的实现方式中，所述根据待压实有序字符串表SSTable对应的待删除日志Delete Log，对所述待压实SSTable进行压实，生成新的SSTable，包括：
拷贝所述待压实SSTable中的、且不属于所述Delete Log中的Key对应的Key-Value，生成新的SSTable。
结合第一方面，在第四种可能的实现方式中，在所述根据待压实有序字符串表SSTable对应的待删除日志Delete Log，对所述待压实SSTable进行压实，生成新的SSTable之前，所述方法还包括：
接收携带有待查找Key的读GET操作；
在根据所述GET操作获取到所述待查找Key对应的最新Value后，确定所述待查找Key对应的次新Value所在的SSTable，在所述确定的SSTable对应的Delete Log中记录所述待查找Key。
结合第一方面，在第五种可能的实现方式中，所述KV-Store***应用于增量存储场景，在所述根据待压实有序字符串表SSTable对应的待删除日志Delete Log，对所述待压实SSTable进行压实，生成新的SSTable之前，所述方法还包括：
接收携带有待查找键Key的读GET操作；
在根据所述GET操作获取到所述待查找Key对应的最新Value后，确定所述待查找Key对应的最新Value所在的SSTable，在所述确定的SSTable对应的Delete Log中记录所述待查找Key；
接收携带有所述待查找Key的PUT操作。
结合第一方面的第四种可能的实现方式或第五种可能的实现方式，在第六种可能的实现方式中，所述在所述确定的SSTable对应的Delete Log中记录所述待查找Key，包括：
在所述确定的SSTable对应的Delete Log中不包含所述待查找Key时，在所述确定的SSTable对应的Delete Log中记录所述待查找Key。
结合第一方面，在第七种可能的实现方式中，所述根据待压实有序字符串表SSTable对应的待删除日志Delete Log，对所述待压实SSTable进行压实，生成新的SSTable，包括：
当待压实有序字符串表SSTable对应的待删除日志Delete Log中的Key的数量大于或等于预设阈值时，根据所述Delete Log，对所述待压实SSTable进行压实，生成新的SSTable。
结合第一方面，在第八种可能的实现方式中，所述待压实SSTable对应一个布隆过滤器Bloom Filter，所述Bloom Filter中记录有所述Delete Log中 的Key，所述方法还包括：
当所述KV-Store***所在的服务器故障恢复后，将所述Bloom Filter的初始值设置为空；
或，在所述KV-Store***为分布式存储***、且所述Delete Log为本地Delete Log时，当所述KV-Store***所在的服务器故障恢复后，根据全局Delete Log中记录的Key确定所述Bloom Filter的初始值；其中，所述全局Delete Log中记录有所述本地Delete Log中的Key。
第二方面，提供一种服务器，包括键值存储KV-Store***，所述服务器还包括：
压实单元，用于根据待压实有序字符串表SSTable对应的待删除日志Delete Log，对所述待压实SSTable进行压实，生成新的SSTable；其中，所述Delete Log中记录有所述待压实SSTable中保存的所述KV-Store***中的非最新值Value所对应的键Key，所述新的SSTable中不包含所述Delete Log中的Key对应的Key-Value对；
删除单元，用于删除所述待压实SSTable。
结合第二方面，在第一种可能的实现方式中，所述服务器还包括：
第一确定单元，用于在所述待压实SSTable中，确定所述KV-Store***中的非最新Value所对应的Key，作为目标Key；
记录单元，用于在所述Delete Log中记录所述目标Key。
结合第二方面的第一种可能的实现方式，在第二种可能的实现方式中，所述记录单元具体用于：在确定所述Delete Log中不包含所述目标Key时，在所述Delete Log中记录所述目标Key。
结合第二方面、第二方面的第一种可能的实现方式或第二方面的第二种可能的实现方式，在第三种可能的实现方式中，所述压实单元具体用于：拷贝所述待压实SSTable中的、且不属于所述Delete Log中的Key对应的Key-Value，生成新的SSTable。
结合第二方面，在第四种可能的实现方式中，所述服务器还包括：
接收单元，用于接收携带有待查找Key的读GET操作；
第二确定单元，用于在根据所述GET操作获取到所述待查找Key对应 的最新Value后，确定所述待查找Key对应的次新Value所在的SSTable，在所述确定的SSTable对应的Delete Log中记录所述待查找Key。
结合第二方面，在第五种可能的实现方式中，所述KV-Store***应用于增量存储场景，所述服务器还包括：
接收单元，用于接收携带有待查找Key的读GET操作；
第二确定单元，用于在根据所述GET操作获取到所述待查找Key对应的最新Value后，确定所述待查找Key对应的最新Value所在的SSTable，在所述确定的SSTable对应的Delete Log中记录所述待查找Key；
所述接收单元还用于，接收携带有所述待查找Key的PUT操作。
结合第二方面的第四种可能的实现方式或第五种可能的实现方式，在第六种可能的实现方式中，所述第二确定单元在执行在所述确定的SSTable对应的Delete Log中记录所述待查找Key时，具体用于：
在所述确定的SSTable对应的Delete Log中不包含所述待查找Key时，在所述确定的SSTable对应的Delete Log中记录所述待查找Key。
结合第二方面，在第七种可能的实现方式中，所述压实单元具体用于：当待压实有序字符串表SSTable对应的待删除日志Delete Log中的Key的数量大于或等于预设阈值时，根据所述Delete Log，对所述待压实SSTable进行压实，生成新的SSTable。
结合第二方面，在第八种可能的实现方式中，所述待压实SSTable对应一个布隆过滤器Bloom Filter，所述Bloom Filter中记录有所述Delete Log中的Key，所述服务器还包括：第三确定单元，用于：
当所述KV-Store***所在的服务器故障恢复后，将所述Bloom Filter的初始值设置为空；
或，在所述KV-Store***为分布式存储***、且所述Delete Log为本地Delete Log时，当所述KV-Store***所在的服务器故障恢复后，根据全局Delete Log中记录的Key确定所述Bloom Filter的初始值；其中，所述全局Delete Log中记录有所述本地Delete Log中的Key。
上述技术方案中，根据待压实SSTable对应的Delete Log，对待压实SSTable进行压实，生成新的SSTable；其中，该Delete Log中记录有待压实 SSTable中保存的KV-Store***中的非最新Value所对应的Key，该新的SSTable中不包含该Delete Log中的Key对应的Key-Value对；然后，删除该待压实SSTable，从而实现了单文件压实。该方法中，每次压实操作时只需要读取一个SSTable中的数据；相比现有技术中的多文件压实的方法，需要占用的I/O带宽和内存资源较小，这样，在利用该方法执行压实操作的同时，可以为其他操作预留更多的资源，以不影响执行其他操作的速率，从而提升了用户体验。
附图说明
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1为现有技术中提供的一种执行PUT操作的过程的示意图；
图2为现有技术中提供的一种执行GET操作的过程的示意图；
图3为本发明实施例一提供的一种KV-Store***中文件压实的方法的流程示意图；
图4为本发明实施例二提供的一种服务器中所涉及的数据结构的示意图；
图5为本发明实施例二提供的一种在Delete Log中记录Key的方法的流程示意图；
图6为本发明实施例二提供的另一种在Delete Log中记录Key的方法的流程示意图；
图7为本发明实施例三提供的一种基于图5或图6的KV-Store***中文件压实的方法的流程示意图；
图8为本发明实施例提供的一种Delete Log的结构示意图；
图9为本发明实施例四提供的一种服务器的结构示意图；
图10为本发明实施例四提供的另一种服务器的结构示意图；
图11为本发明实施例五提供的一种服务器的结构示意图。
具体实施方式
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
下面首先对本发明实施例中所涉及的部分术语进行解释说明，以方便本领域技术人员理解。
1)、KV-Store***
本发明实施例中的“KV-Store***”可以为分布式存储***，也可以为本地存储***。其中，当“KV-Store***”为分布式存储***时，KV-Store***可以分布在多个服务器上，每个服务器上运行该KV-Store***的部分进程；KV-Store***的全部数据被划分为多个子集，每个子集由一个服务器的KV-Store***进程进行管理。当“KV-Store***”为本地文件存储***时，KV-Store***包含在一个服务器上，KV-Store***的全部数据由该服务器上的KV-Store***进程进行管理。
2)、PUT操作
PUT操作携带有一个Key-Value对，可以表示为PUT(Key，Value)操作。
服务器执行PUT(Key，Value)操作的过程可以包括：先将PUT操作所携带的Key-Value对写入内存中的MemTable中；当MemTable已满时，在外存中创建一个SSTable，并将MemTable中的Key-Value对排序写入SSTable中。
其中，“将PUT操作所携带的Key-Value对写入MemTable中”可以包括：若MemTable中已包含PUT操作所携带的Key，则将MemTable中包含的该Key对应的Value更新为PUT操作所携带的Value；若MemTable中未包含PUT操作所携带的Key，则在MemTable中写入PUT操作所携带的Key-Value对。
需要说明的是，SSTable只在创建时被写入一次，之后就成为只读文件， 不能再被修改，但是可以被删除。从上面的描述可以看出，KV-Store***的PUT操作虽然使Key-Value对的新Value代替了旧Value，但是旧Value仍然保存在外存的某个SSTable中。尤其对于频繁修改更新的Key，实际的外存存储的旧Value比最新Key-Value对中的Value消耗多倍的空间。
另外需要说明的是，随着PUT操作的增多，外存中的SSTable的数量会逐渐增多。一般地，一个SSTable中的任意两个Key-Value对的Key不同，不同SSTable中可能包含具有相同Key的Key-Value对。不同的SSTable按照创建时间的先后顺序进行排列，同一SSTable中的Key-Value对按照Key的大小进行排序。
另外需要说明的是，当KV-Store***为分布式存储***时，一般地，一个Key对应的所有Key-Value对均会被记录在同一个服务器中；不同Key对应的Key-Value对可能被记录在不同的服务器中。
参见图1，为现有技术中提供的一种执行PUT操作的过程的示意图。
3)、GET操作
GET操作携带有一个Key，可以表示为GET(Key)操作，用于查找该Key对应的最新Value。需要说明的是，下文中将GET操作携带的Key称为“待查找Key”。
服务器执行GET(Key)操作的过程可以包括：先在MemTable中查找GET操作所携带的Key，若查找到，则输出该Key对应的Value；若没有查找到，则按照创建时间依次从最新SSTable到最旧SSTable中查找GET操作所携带的Key，若查找到，则输出该Key对应的Value。其中，所输出的Value即为该Key对应的最新Value。
需要说明的是，具体实现时，服务器的内存中缓存有每个SSTable对应的Bloom Filter(布隆过滤器)和Block Index(块索引)。其中，Bloom Filter一般用于提供针对查询的过滤功能，例如，对于集合A＝{a0,a1,…,an}，若想知道是否b∈A，则可以首先建立一个m位的位向量(bit vector)，其中，Bit vector的每一位初始为0；其次，确定k个互不相同的哈希函数(例如，Hash0，Hash1，…，Hashk)。其中，m和k是可调参数。然后，对于A中的每个元素aj，计算哈希函数，并把相应的位置设为1，也就是Hashi(aj)位置都设为1；依次类推，对A中的所有元素都完成此操作，最终获得的bit vector就是代表A的Bloom Filter。对于需要查询的元素b，可以计算b的k个哈希函数，测试bit vector中相应的位置，即所有Hashi(b)所对应的位置，只有当b的哈希函数计算结果中所有k个bit都为1时，b才可能出现在A中。如果b的哈希函数计算结果中k个bit中任意一个为0，那么b一定不在A中。于是，通过Bloom Filter就可以快速地过滤掉b不属于A的情况。Block Index用于在SSTable中每隔一定个数的Key-Value对记录Key和所在的字节位置。
“从SSTable中查找GET操作所携带的Key”可以包括：在通过SSTable对应的Bloom Filter的测试之后，从SSTable对应的Block Index中查找GET操作所携带的Key；若在某个Block Index中查找到该Key，则从该Block Index所对应的Block中查找该Key，并输出该Key对应的Value。其中，一个SSTable可以包括多个Block。
参见图2，为现有技术中提供的一种执行GET操作的过程的示意图。其中，图2中的“BF”表示Bloom Filter，即下述的BFs；“Ind”表示Block Index。
4)、Delete Log(待删除日志)
Delete Log是本发明实施例中为SSTable创建的Log文件，用于记录其所对应的SSTable中的非最新Value所对应的Key。其中，Log文件是一种按照时间先后顺序记录某种操作的信息的一种特殊的文件，只在文件末尾进行追加操作。
Delete Log位于外存中。具体实现时，可以为每个SSTable创建一个Delete Log，也可以为多个SSTable共同创建一个Delete Log。例如，当服务器在外存中生成一个SSTable时，为该SSTable创建一个Delete Log；或者，当服务器在外存中生成预设数量个SSTable时，为该预设数量个SSTable创建一个Delete Log。其中，本发明实施例对“预设数量”的取值及取值方式不进行限定。
5)、BFs和BFd
在本发明的一些实施例/实现方式中，在内存中为SSTable设置了Bloom Filter。为了区分现有技术中的GET操作所涉及的“SSTable对应的Bloom Filter”和本发明实施例新增加的“SSTable对应的Bloom Filter”，本文中 将现有技术中的“SSTable对应的Bloom Filter”称为BFs，将本发明实施例中新增加的“SSTable对应的Bloom Filter”称为BFd。其中，“SSTable对应的BFd”用于过滤该SSTable对应的Delete Log中的Key。
6)、本文中的“和/或”仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。“/”表示“或”。“多个”是指两个或两个以上。
实施例一
参见图3，为本发明实施例提供的一种KV-Store***中文件压实的方法的流程示意图。图3所示的方法应用于包含KV-Store***的服务器中。该方法包括以下步骤S301-S302：
S301：根据待压实SSTable对应的Delete Log，对待压实SSTable进行压实，生成新的SSTable；其中，该Delete Log中记录有待压实SSTable中保存的KV-Store***中的非最新Value所对应的Key，该新的SSTable中不包含该Delete Log中的Key对应的Key-Value对。
其中，本实施例提供的方法的执行主体可以为包含KV-Store***的服务器。其中，当“KV-Store***”为分布式存储***时，“包含KV-Store***的服务器”可以为KV-Store***所分布的任一个服务器，即包含KV-Store***进程的任一个服务器。
“待压实SSTable”可以为KV-Store***中的任一个SSTable。“待压实SSTable”可以独立对应一个Delete Log，也可以与KV-Store***中的其他SSTable共享一个Delete Log。
“KV-Store***中的Value”包括KV-Store***所在的所有服务器的内存和外存中记录的Value，具体包括KV-Store***所在的所有服务器中的MemTable和SSTable中记录的Value。“KV-Store***中的最新Value”可以记录在MemTable或SSTable中。“KV-Store***中的非最新Value”一般记录在SSTable中。其中，“非最新Value”是指具有相同Key的多个Key-Value对中的、且被记录的时间不是最晚的一个Key-Value对中的Value；具体可以包括：具有相同Key的多个Key-Value对中的、且在“第 1、2、3、……、(i-1)次”中的任意一次或多次被记录的Value；其中，第i-1次被记录的Value为该Key对应的“次新Value”；i为大于或等于1的整数。需要说明的是，多个具有相同Key的Key-Value对可以被记录在多个SSTable中，也可以被记录在MemTable和一个/多个SSTable中。
由上述描述可知，“Delete Log”中所记录的Key至少在MemTable中或者创建时间在待压实SSTable之后的一个SSTable(即比待压实SSTable更新的一个SSTable)中被记录过。本发明实施例对在Delete Log中记录Key的具体实现方式不进行限定。
可选的，步骤S301具体可以实现为：拷贝待压实SSTable中的、且不属于该Delete Log中的Key对应的Key-Value对，生成新的SSTable。举例而言，通过归并待压实SSTable和该Delete Log，拷贝待压实SSTable中的、且不属于该Delete Log中的Key对应的Key-Value对，生成新的SSTable。其中，“归并待压实SSTable和该Delete Log”的实现方法可以参考现有技术中的归并操作。
需要说明的是，在步骤S301之后，该方法还可以包括：为新的SSTable创建新的Delete Log，以对新的SSTable执行压实操作作准备。
S302：删除待压实SSTable。
步骤S302可以理解为：用步骤S301中生成的新的SSTable替换待压实SSTable。
需要说明的是，在执行S302之后，该方法还可以包括：删除待压实SSTable对应的信息，其中，待压实SSTable对应的信息可以包括但不限于以下信息中的一种或几种：待压实SSTable对应的Delete Log、后续可选的实现方式中所涉及的Delete Log Buffer(待删除日志缓存)、待压实SSTable对应的BFd等。
本实施例提供的KV-Store***中文件压实的方法，根据待压实SSTable对应的Delete Log，对待压实SSTable进行压实，生成新的SSTable；其中，该Delete Log中记录有待压实SSTable中保存的KV-Store***中的非最新Value所对应的Key，该新的SSTable中不包含该Delete Log中的Key对应的Key-Value对；然后，删除该待压实SSTable，从而实现了单文件压实。该方法中，每次压实操作时只需要读取一个SSTable中的数据；相比现有技 术中的多文件压实的方法，需要占用的I/O带宽和内存资源较小，这样，在利用该方法执行压实操作的同时，可以为其他操作预留更多的资源，以不影响执行其他操作的速率，从而提升了用户体验。
另外，相比现有技术提供的多文件压实的方法，利用本发明实施例提供的单文件压实的方法，在每次执行压实操作的过程中需要处理的数据较少，因此所需要的时间较短；这样，能够为其他操作预留更多的处理时间，从而提高了用户体验。
进一步地，为了保证读写效率，一般需要在内存中设置读写缓冲，现有技术中提供的多文件压实的方法在每次执行压实操作时，需要读取多个SSTable中的数据，因此需要在内存中设置很大的读写缓冲，这需要占用很大的内存资源。而利用本发明实施例提供的单文件压实的方法，只需要读取一个SSTable中的数据，因此相比现有技术，需要在内存中设置的读写缓冲所占用的空间较少，也就是说，占用的内存资源较少；这样，可以为其他操作预留更多的内存资源，从而提高了KV-Store***的性能。
在一种可选的实现方式中，步骤S301可以包括：当待压实SSTable对应的Delete Log中的Key的数量大于或等于预设阈值时，根据该Delete Log，对待压实SSTable进行压实。
举例而言，在该可选的实现方式中，将“待压实SSTable对应的Delete Log中的Key的数量大于或等于预设阈值”作为对待压实SSTable进行压实的触发条件。具体实现时，也可以在其他触发条件下对待压实SSTable进行压实，例如，在预定的时刻或周期性地对待压实SSTable进行压实；其中，预定的时刻可以为一具体时刻，也可以为服务器的空闲时刻等。本发明实施例中对“预设阈值”的取值及该取值的获取方式不进行限定，例如，具体可以为待压实SSTable中的Key的数量的一半，当然还可以为其他值。
在一种可选的实现方式中，在步骤S301之前，该方法还可以包括：在待压实SSTable中，确定KV-Store***中的非最新Value所对应的Key，作为目标Key；在该Delete Log中记录该目标Key。
举例而言，若待压实SSTable为KV-Store***中创建时间最晚的SSTable(即最新SSTable)，那么，服务器可以通过对比待压实SSTable与MemTable，从而将待压实SSTable中的、且被记录在MemTable中的 Key作为目标Key。若待压实SSTable不为KV-Store***中创建时间最晚的SSTable，那么，服务器可以对比待压实SSTable与MemTable，以及对比待压实SSTable与创建时间在该待压实SSTable之后的SSTable，从而将待压实SSTable中的、且被记录在MemTable/创建时间在该待压实SSTable之后的SSTable中的Key作为目标Key。
需要说明的是，在该可选的实现方式中，若待压实SSTable中的每个Key均没有被记录在MemTable和创建时间在待压实SSTable之后的所有SSTable中，那么，该待压实SSTable中不包含目标Key。
举例而言，在该Delete Log中记录所述目标Key，可以包括：在确定该Delete Log中不包含该目标Key时，在该Delete Log中记录该目标Key。该方式，能够避免在该Delete Log中重复记录目标Key。具体的：可以预先在服务器的内存中为待压实SSTable建立一个Bloom Filter，该Bloom Filter中记录有该Delete Log中的Key；该情况下，“在确定该Delete Log中不包含该目标Key时，在该Delete Log中记录该目标Key”可以包括：在确定该Bloom Filter中不包含目标Key时，在该Delete Log中记录目标Key。
具体实现时，服务器可以预先为每个SSTable设置一个Bloom Filter(即BFd)。根据Bloom Filter的基本原理可知，若目标Key没有通过待压实SSTable对应的BFd的测试，则目标Key一定没有被记录在该Delete Log中。上述“该Bloom Filter中不包含目标Key”被认为目标Key没有通过该Bloom Filter的测试。
进一步可选的，该Delete Log对应内存中的一个Delete Log Buffer；该情况下，在Delete Log中记录目标Key，可以包括：当Delete Log Buffer中的目标Key的数量大于或等于第二预设阈值时，在该Delete Log中记录目标Key。
举例而言，该可选的实现方式中，服务器可以在内存中为每个Delete Log设置一个Delete Log Buffer，并首先在该Delete Log对应的Delete Log Buffer中记录目标Key；当该Delete Log Buffer中的目标Key的数量大于或等于第二预设阈值时，再将该Delete Log Buffer中的Key依次写入该Delete Log。由于服务器在内存中读写数据的速率远大于在外存中读写数据的速率，因此该可选的实现方式能够服务器加快读写数据的速率，从而提高KV-Store ***的整体性能。其中，本发明实施例对“第二预设阈值”的具体取值以及取值的获取方法不进行限定，例如，可以为该Delete Log Buffer中能够容纳的目标Key的数量的最大值等。
本发明实施例还提供了根据GET操作在Delete中记录Key的技术方案，具体可以包括以下两种可选的实现方式中：
可选的实现方式一：接收携带有待查找Key的GET操作；在根据GET操作获取到待查找Key对应的最新Value后，确定待查找Key对应的次新Value所在的SSTable，在所确定的SSTable对应的Delete Log中记录该待查找Key。
举例而言，该可选的实现方式在服务器执行完现有技术中的GET操作的触发条件下，即服务器获取到GET操作中携带的待查找Key对应的最新Value的触发条件下，确定该待查找Key对应的次新Value所在的SSTable，从而在该SSTable对应的Delete Log中记录该待查找Key。
其中，步骤S301中的“非最新Value”在该可选的实现方式中具体体现为“次新Value”。“待查找Key对应的最新Value”可以为MemTable或SSTable中的Value。服务器根据GET操作获取待查找Key对应的最新Value的实现方法可以参考上文，此处不再赘述。服务器可以按照获取待查找Key对应的最新Value的方法，获取待查找Key对应的次新Value。
可选的实现方式二：当KV-Store***应用于增量存储场景时，接收携带有待查找键Key的GET操作；在根据该GET操作获取到待查找Key对应的最新Value后，确定该待查找Key对应的最新Value所在的SSTable，在所确定的SSTable对应的Delete Log中记录该待查找Key。
其中，“增量存储场景“的特点为：每个Key-Value对仅被GET一次，并且服务器在执行GET操作之后，会产生针对该Key的新的Value，并执行携带有该Key和该新的Value的PUT操作。
在该可选的实现方式二中，在确定待压实SSTable之后，该方法还可以包括：接收携带有该待查找Key的PUT操作。其中，“接收携带有该待查找Key的PUT操作”可以在执行“接收携带有待查找键Key的读GET操作”之后、且在执行下一个GET操作或任意一个携带有除该待查找Key之外的 其他Key的PUT操作之前的任一步骤中执行。
举例而言，根据增量存储场景的特点可知，服务器在执行完GET(待查找Key)操作后会接着执行PUT(待查找Key，新的Value)操作，这样，在执行完该PUT(待查找Key，新的Value)操作后，由该GET(待查找Key)操作得到的“最新Value”会成为次新Value。由此可知，该可选的实现方式二与上述可选的实现方式一的基本思想相同。
需要说明的是，上述可选的实现方式一可以适用于增量存储场景和非增量存储场景中；并且，适用于“最新Value”在MemTable或SSTable中的场景中。该可选的实现方式二应用于增量存储场景中；并且，适用于“最新Value”在SSTable中的场景中。
另外需要说明的是，在该可选的实现方式一、二与上述涉及“目标Key”的实现方式结合使用的场景中，该可选的实现方式一、二中的“待查找Key”即为上述“目标Key”。另外，该可选的实现方式一、二可以与本文中提供的任一种或多种其他可选的方式结合使用。具体的：上述“在所确定的SSTable对应的Delete Log中记录该待查找Key”，可以包括：在所确定的SSTable对应的Delete Log中不包含该待查找Key时，在所确定的SSTable对应的Delete Log中记录该待查找Key。
具体实现时，当服务器出现故障时，其内存中的数据会丢失，其外存中的数据不会丢失。本发明实施例还提供了当服务器出现故障并恢复后，如何创建待压实SSTable对应的Bloom Filter(即BFd)。具体可以包括以下方式一和方式二：
方式一：当服务器故障恢复后，将待压实SSTable对应的Bloom Filter的初始值设置为空。
举例而言，当服务器故障恢复后，将待压实SSTable对应的Bloom Filter的初始值设置为空；后续再按照上文提供的可选的实现方式向待压实SSTable对应的Bloom Filter中写入Key。
需要说明的是，该方式一虽然在一定程度上会使得待压实SSTable对应的Delete Log中记录的Key出现重复，但是，能够保证在服务器在故障恢复后，按照上文提供的技术方案进行单文件压实的过程中的正确率；另外，该方式一具有实现简单的优点。
方式二：在KV-Store***为分布式存储***、且待压实SSTable对应的Delete Log为本地Delete Log时，当服务器故障恢复后，根据全局Delete Log中记录的Key确定待压实SSTable对应的Bloom Filter的初始值；其中，全局Delete Log中记录有该本地Delete Log中的Key。
举例而言，当KV-Store***为分布式存储***时，每个包含该KV-Store***的进程的服务器可以在自身的外存中，为本地的每个SSTable创建一个Delete Log，该Delete Log可以称为“本地Delete Log”；另外，还可以在其中的一个或多个服务器中创建全局Delete Log，用于记录每个包含该KV-Store***的进程的服务器中的本地Delete Log中的Key，以实现对本地Delete Log的备份。这样，当其中一个或多个包含该KV-Store***的进程的服务器故障恢复后，可以从全局Delete Log中获取该故障恢复的服务器中的本地Delete Log中的Key。
需要说明的是，具体实现时，包含KV-Store***进程的每个服务器可以定期地或周期性地或触发性地将本地Delete Log写入全局Delete Log。另外，当其中的一个或多个服务器发生故障后，其他未发生故障的服务器可以将这些发生故障的服务器中的KV-Store***数据转移到自身的存储单元中。本发明实施例对该转移方式不进行限定，例如，可以通过全局Delete Log中记录的信息等进行转移。
实施例二
本实施例提供两种在Delete Log中记录Key的方法。该两种方法应用于包含KV-Store***的服务器中；该服务器中所涉及的数据结构如图4所示。图4中的服务器包括内存和外存；内存中设置有一个MemTable、若干个BFs、若干个BFd和若干个Delete Log Buffer；外存中设置有若干个SSTable和若干个Delete Log。其中，每个SSTable对应一个BFs；每个SSTable对应一个Delete Log，每个SSTable对应一个BFd，每个Delete Log对应一个Delete Log Buffer。
需要说明的是，具体实现时，图4所示的服务器中的每个SSTable还可以对应一个或多个Block Index，图4中未示出Block Index。
在Delete Log中记录Key的方法一
参见图5，为本实施例提供的一种在Delete Log中记录Key的方法的流程示意图。图5所示的方法可以应用于增量存储场景和非增量存储场景中。该方法可以包括以下步骤S501-S504：
S501：在接收到携带有待查找Key的GET操作时，根据GET操作获取待查找Key对应的最新Value，并查找待查找Key对应的次新Value。若查找到，则执行步骤S502；若没有查找到，则结束。
S502：确定记录有待查找Key对应的次新Value的SSTable。
S503：判断所确定的SSTable对应的BFd中是否记录有该待查找Key。
若是，说明待压实SSTable对应的Delete Log中已经记录有该待查找Key，则结束；若否，则执行步骤S504。
S504：在所确定的SSTable对应的BFd中记录该待查找Key，并在所确定的SSTable对应的Delete Log中记录该待查找Key。
具体的，步骤S504可以包括：在待压实SSTable对应的BFd中记录该待查找Key，并在待压实SSTable对应的Delete Log Buffer中记录该待查找Key；若该Delete Log Buffer已满，则将Delete Log Buffer中所记录的Key写入待压实SSTable对应的Delete Log中。
举例而言，在该方法一中，服务器可以根据以下算法执行GET(Key)上操作：
Figure PCTCN2016074043-appb-000001
Figure PCTCN2016074043-appb-000002
需要说明的是，上述第5行代码为服务器第一次查找到待查找Key，本次查找到的待查找Key对应的Value为待查找Key对应的最新Value；另外该最新Value所在的SSTable为SSTable[i]。上述第10行代码说明服务器第二次查找到待查找Key，本次查找到的待查找Key对应的Value为待查找Key对应的次新Value；另外该次新Value所在的SSTable为SSTable[j]。
在Delete Log中记录Key的方法二
参见图6，为本实施例提供的另一种在Delete Log中记录Key的方法的流程示意图。图6所示的方法应用于增量存储场景中，该方法可以包括以下步骤S601-S604：
S601：在接收到携带有待查找Key的GET操作时，根据GET操作获取待查找Key对应的最新Value。若查找到，则执行步骤S602；若没有查找到，则结束。
S602：确定记录有该最新Value的SSTable。
S603-S604：与上述S503-S504相同。
举例而言，在该方法二中，服务器可以根据以下算法执行增量存储场景 下的GET(Key)操作：
增量存储场景下的GET(Key)
Figure PCTCN2016074043-appb-000003
实施例三
参见图7，为本实施例提供的基于图5或图6的一种KV-Store***中文件压实的方法的流程示意图。图7所示的方法包括以下步骤S701-S704：
S701：判断待压实SSTable对应的Delete Log中的Key的数量是否大于或等于预设阈值。
若是，则执行步骤S702；若否，则结束。
其中，“待压实SSTable”可以为KV-Store***所在的服务器中的任一个SSTable。具体实现时，服务器可以时刻或周期性地或触发性地针对任一个SSTable执行步骤S701。
需要说明的是，若步骤S701的判断结果为“否”，则服务器在执行步骤S701的判断操作之后不执行对待压实SSTable进行压实的操作；该情况下，服务器可以继续执行上述图5或图6所示的在Delete Log中记录Key。
S702：对待压实SSTable对应的Delete Log中的Key进行排序。
其中，该步骤中所执行的“排序”是后续步骤S703中执行“归并”操作的基础。
S703：归并待压实SSTable和排序后的该Delete Log，得到新的SSTable；其中，新的SSTable中记录的Key-Value对为待压实SSTable 中的、且不属于该Delete Log中的Key对应的Key-Value对。
S704：删除待压实SSTable。
举例而言，步骤S703可以通过以下算法实现：
SingleCompact(SSTable)
1.Sort delete_log；
2.key_d＝delete_log.getNext()；
3.Init SSTable_new；
4.while(((key_s,value)＝SSTable.getNext())！＝null){
5.   while(key_s>key_d)key_d＝delete_log.getNext()；
6.   if(key_s＝＝key_d){//删除这个key
7.   }
8.   else if(key_s<key_d){//保留这个key
9.      SSTable_new.add(key_s,value)；
10.      }
11.  }
下面通过一个具体的示例对图5提供的在Delete Log中记录Key的方法和基于图5的KV-Store***中文件压实的方法进行示例性说明：
假设某一时刻，KV-Store***中的每个SSTable对应的Delete Log均为空，也就是说，每个Delete Log中均未记录任何一个Key；并且，此时KV-Store***中的MemTable和SSTable中记录的Key如表1所示：
表1
MemTable/SSTable Key
MemTable Key1、Key2、Key3
SSTable1 Key1、Key2、Key3、Key4、Key5
SSTable2 Key1、Key4、Key5、Key7、Key8
SSTable3 Key2、Key3、Key6、Key8、Key9
基于表1，服务器在连续执行GET(Key1)、GET(Key2)、GET(Key3)、GET(Key4)、GET(Key5)之后，KV-Store***中的每个SSTable对应的Delete Log中的Key如表2所示：
表2
SSTable Delete Log Key
SSTable1 Delete Log1 Key1、Key2、Key3
SSTable2 Delete Log2 Key4、Key5
SSTable3 Delete Log3 空
假设“预设阈值”为3，那么，在表2中，Delete Log1中Key的数量满足“大于或等于预设阈值”；此时，服务器可以归并SSTable1和Delete Log1，归并后得到新的SSTable1。该情况下，KV-Store***中的MemTable和SSTable中记录的Key如表3所示：
表3
MemTable/SSTable Key
MemTable Key1、Key2、Key3
新的SSTable1 Key4、Key5
SSTable2 Key1、Key4、Key5、Key7、Key8
SSTable3 Key2、Key3、Key6、Key8、Key9
接着，基于表3，服务器在连续执行GET(Key1)、GET(Key2)、GET(Key3)、GET(Key4)、GET(Key5)之后，KV-Store***中的每个SSTable对应的Delete Log中的Key如表4所示：
表4
SSTable Delete Log Key
新的SSTable1 新的Delete Log1 空
SSTable2 Delete Log2 Key1、Key4、Key5
SSTable3 Delete Log3 Key2、Key3
假设“预设阈值”为3，那么，在表4中，Delete Log2中Key的数量满足“大于或等于预设阈值”；此时，服务器可以归并SSTable2和Delete  Log2，归并后得到新的SSTable2。该情况下，KV-Store***中的MemTable和SSTable中记录的Key如表5所示：
表5
MemTable/SSTable Key
MemTable Key1、Key2、Key3
新的SSTable1 Key4、Key5
新的SSTable2 Key7、Key8
SSTable3 Key2、Key3、Key6、Key8、Key9
依此类推，服务器可以按照上述方法对所对应的Delete Log中的Key的数量满足“大于或等于预设阈值”的任一个SSTable执行压实操作，从而实现节省存储空间的目的。
需要说明的是，根据该示例，本领域技术人员应当能够在不付出创造性的劳动下，得出图6提供的在Delete Log中记录Key的方法和基于图6的在KV-Store***中文件压实的方法的具体示例。
需要说明的是，在上述任一实施例或可选的实现方式中，当多个SSTable共享一个Delete Log时，可以将Delete Log划分为多个Delete Block，每个Delete Block用于记录一个SSTable中的非最新Value所对应的Key，一个SSTable中的非最新Value所对应的Key可以分别被记录在多个Delete Block中。其中，一个SSTable中的非最新Value所对应的Key可以分别被记录在Delete Log中的连续的或非连续的多个Delete Block中。
当一个SSTable中的非最新Value所对应的Key被记录在Delete Log中的非连续的多个Delete Block中时，即一个SSTable对应多个非连续的Delete Block时，服务器可以在内存中保存每个SSTable对应的最后一个Delete Block的位置，并在产生下一个Delete Block时，将上一个Delete Block的位置记录进新的Delete Block，如此就形成了从后向前的单链。在需要读取一个SSTable中的非最新Value所对应的Key时，可以根据该单链，从后向前地读取记录该SSTable对应的多个Delete Block中所记录的Key。
参见图8，为本发明实施例提供的一种Delete Log的结构示意图。图8所示的Delete Log中，一个SSTable对应多个非连续的Delete Block。其中，SSTable[i]表示第i个SSTable，SSTable[j]表示第j个SSTable；i、j均为大于或等于0的整数，且i不等于j。
实施例四
参见图9，为本发明实施例提供的一种服务器的结构示意图。图9所示的服务器用于执行上述提供的任一种KV-Store***中文件压实的方法。本实施例中相关内容的解释可以参考上述方法实施例，此处不再赘述。
图9所示的服务器9包含KV-Store***，该服务器9可以包括：压实单元91和删除单元92。具体的：
压实单元91，用于根据待压实有序字符串表SSTable对应的待删除日志Delete Log，对所述待压实SSTable进行压实，生成新的SSTable；其中，所述Delete Log中记录有所述待压实SSTable中保存的所述KV-Store***中的非最新值Value所对应的键Key，所述新的SSTable中不包含所述Delete Log中的Key对应的Key-Value对。
删除单元92，用于删除所述待压实SSTable。
可选的，如图10所示，所述服务器9还可以包括：
第一确定单元93，用于在所述待压实SSTable中，确定所述KV-Store***中的非最新Value所对应的Key，作为目标Key；
记录单元94，用于在所述Delete Log中记录所述目标Key。
举例而言，所述记录单元96具体可以用于：在确定所述Delete Log中不包含所述目标Key时，在所述Delete Log中记录所述目标Key。
可选的，所述压实单元91具体可以用于：拷贝所述待压实SSTable中的、且不属于所述Delete Log中的Key对应的Key-Value，生成新的SSTable。
在一种可选的实现方式中，如图10所示，所述服务器1还可以包括：接收单元95，用于接收携带有待查找Key的读GET操作；第二确定单元96，用于在根据所述GET操作获取到所述待查找Key对应的最新Value后，确 定所述待查找Key对应的次新Value所在的SSTable，在所述确定的SSTable对应的Delete Log中记录所述待查找Key。
在另一种可选的实现方式中，所述KV-Store***应用于增量存储场景，该情况下，如图10所示，所述服务器1还可以包括：接收单元95，用于接收携带有待查找Key的读GET操作；第二确定单元96，用于在根据所述GET操作获取到所述待查找Key对应的最新Value后，确定所述待查找Key对应的最新Value所在的SSTable，在所述确定的SSTable对应的Delete Log中记录所述待查找Key；所述接收单元95还可以用于，接收携带有所述待查找Key的PUT操作。
举例而言，在上述两种可选的实现方式中，所述第二确定单元96在执行在所述确定的SSTable对应的Delete Log中记录所述待查找Key时，具体用于：在所述确定的SSTable对应的Delete Log中不包含所述待查找Key时，在所述确定的SSTable对应的Delete Log中记录所述待查找Key。
可选的，所述压实单元91具体可以用于：当待压实有序字符串表SSTable对应的待删除日志Delete Log中的Key的数量大于或等于预设阈值时，根据所述Delete Log，对所述待压实SSTable进行压实，生成新的SSTable。
可选的，所述待压实SSTable对应一个布隆过滤器Bloom Filter；所述Bloom Filter中记录有所述Delete Log中的Key，如图10所示，所述服务器9还可以包括：第三确定单元97，用于：当所述KV-Store***所在的服务器故障恢复后，将所述Bloom Filter的初始值设置为空；或，在所述KV-Store***为分布式存储***、且所述Delete Log为本地Delete Log时，当所述KV-Store***所在的服务器故障恢复后，根据全局Delete Log中记录的Key确定所述Bloom Filter的初始值；其中，所述全局Delete Log中记录有所述本地Delete Log中的Key。
本实施例提供的服务器，根据待压实SSTable对应的Delete Log，对待压实SSTable进行压实，生成新的SSTable；其中，该Delete Log中记录有待压实SSTable中保存的KV-Store***中的非最新Value所对应的Key，该新的SSTable中不包含该Delete Log中的Key对应的Key-Value对；然后，删除该待压实SSTable，从而实现了单文件压实。该方法中，每次压实 操作时只需要读取一个SSTable中的数据；相比现有技术中的多文件压实的方法，需要占用的I/O带宽和内存资源较小，这样，在利用该方法执行压实操作的同时，可以为其他操作预留更多的资源，以不影响执行其他操作的速率，从而提升了用户体验。
实施例五
在硬件实现上，上述实施例四中的压实单元91、删除单元92、第一确定单元93、记录单元94、接收单元95、第二确定单元96和第三确定单元97中的一种或几种以硬件形式内嵌于或独立于服务器9的处理器中，也可以以软件形式存储于服务器9的存储器中，以便于处理器调用执行以上各个模块对应的操作，该处理器可以为中央处理单元(CPU，Central Processing Unit)、微处理器、单片机等。
参见图11，为本发明实施例提供的一种服务器的结构示意图。图11所示的服务器用于执行上述提供的任一种KV-Store***中文件压实的方法。本实施例中相关内容的解释可以参考上述方法实施例，此处不再赘述。
图11所示的服务器11包含KV-Store***，该服务器11可以包括：存储器11A、处理器11B、无线接口11C和总线***11D。其中，存储器11A、处理器11B和无线接口11C之间是通过总线***11D耦合在一起的。
存储器11A可能包含高速RAM(Random Access Memory，随机存取存储器)，也可能包含非易失性存储器(non-volatile memory)，例如至少一个磁盘存储器。
无线接口11C用于所述服务器11与其他设备进行通信。
总线***11D可以是ISA(Industry Standard Architecture，工业标准体系结构)总线、PCI(Peripheral Component，外部设备互连)总线或EISA(Extended Industry Standard Architecture，扩展工业标准体系结构)总线等；总线***11D除包括数据总线之外，还可以包括电源总线、控制总线和状态信号总线等。但是为了清楚说明起见，在图中将各种总线都标为总线***11D。
存储器11A，用于存放程序。具体地，程序可以包括程序代码，所述程 序代码包括计算机操作指令。
处理器11B执行所述存储器11A中存放的程序，以实现本发明实施例提供的KV-Store***中文件压实的方法。具体可以包括：根据待压实有序字符串表SSTable对应的待删除日志Delete Log，对所述待压实SSTable进行压实，生成新的SSTable；其中，所述Delete Log中记录有所述待压实SSTable中保存的所述KV-Store***中的非最新值Value所对应的键Key，所述新的SSTable中不包含所述Delete Log中的Key对应的Key-Value对；删除所述待压实SSTable。
可选的，处理器11B还可以用于：在所述待压实SSTable中，确定所述KV-Store***中的非最新Value所对应的Key，作为目标Key；在所述Delete Log中记录所述目标Key。
举例而言，处理器11B在执行所述Delete Log中记录所述目标Key时，具体可以用于：在确定所述Delete Log中不包含所述目标Key时，在所述Delete Log中记录所述目标Key。
可选的，处理器11B在执行根据待压实有序字符串表SSTable对应的待删除日志Delete Log，对所述待压实SSTable进行压实，生成新的SSTable时，具体可以用于：拷贝所述待压实SSTable中的、且不属于所述Delete Log中的Key对应的Key-Value，生成新的SSTable。
在一种可选的实现方式中，所述处理器11B还可以用于：接收携带有待查找Key的读GET操作；并在根据所述GET操作获取到所述待查找Key对应的最新Value后，确定所述待查找Key对应的次新Value所在的SSTable，在所述确定的SSTable对应的Delete Log中记录所述待查找Key。
在另一种可选的实现方式中，所述KV-Store***应用于增量存储场景，所述处理器11B还可以用于：接收携带有待查找Key的读GET操作；并在根据所述GET操作获取到所述待查找Key对应的最新Value后，确定所述待查找Key对应的最新Value所在的SSTable，在所述确定的SSTable对应的Delete Log中记录所述待查找Key；以及接收携带有所述待查找Key的PUT操作。
举例而言，在上述两种可选的实现方式中，所述处理器11B在执行在所述确定的SSTable对应的Delete Log中记录所述待查找Key时，具体用于： 在所述确定的SSTable对应的Delete Log中不包含所述待查找Key时，在所述确定的SSTable对应的Delete Log中记录所述待查找Key。
可选的，处理器11B在执行根据待压实有序字符串表SSTable对应的待删除日志Delete Log，对所述待压实SSTable进行压实，生成新的SSTable时，具体可以用于：当待压实有序字符串表SSTable对应的待删除日志Delete Log中的Key的数量大于或等于预设阈值时，根据所述Delete Log，对所述待压实SSTable进行压实，生成新的SSTable。
可选的，所述待压实SSTable对应一个布隆过滤器Bloom Filter；所述Bloom Filter中记录有所述Delete Log中的Key，处理器11B还可以用于：当所述KV-Store***所在的服务器故障恢复后，将所述Bloom Filter的初始值设置为空；或，在所述KV-Store***为分布式存储***、且所述Delete Log为本地Delete Log时，当所述KV-Store***所在的服务器故障恢复后，根据全局Delete Log中记录的Key确定所述Bloom Filter的初始值；其中，所述全局Delete Log中记录有所述本地Delete Log中的Key。
本实施例提供的服务器，根据待压实SSTable对应的Delete Log，对待压实SSTable进行压实，生成新的SSTable；其中，该Delete Log中记录有待压实SSTable中保存的KV-Store***中的非最新Value所对应的Key，该新的SSTable中不包含该Delete Log中的Key对应的Key-Value对；然后，删除该待压实SSTable，从而实现了单文件压实。该方法中，每次压实操作时只需要读取一个SSTable中的数据；相比现有技术中的多文件压实的方法，需要占用的I/O带宽和内存资源较小，这样，在利用该方法执行压实操作的同时，可以为其他操作预留更多的资源，以不影响执行其他操作的速率，从而提升了用户体验。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的***，装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的***，装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可 以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个***，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本发明各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理包括，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用硬件加软件功能单元的形式实现。
上述以软件功能单元的形式实现的集成的单元，可以存储在一个计算机可读取存储介质中。上述软件功能单元存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本发明各个实施例所述方法的部分步骤。而前述的存储介质包括：U盘、移动硬盘、ROM(Read-Only Memory，只读存储器)、RAM、磁碟或者光盘等各种可以存储程序代码的介质。
最后应说明的是：以上实施例仅用以说明本发明的技术方案，而非对其限制；尽管参照前述实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本发明各实施例技术方案的精神和范围。

Claims (18)

  1. 一种键值存储KV-Store***中文件压实的方法，其特征在于，包括：
    根据待压实有序字符串表SSTable对应的待删除日志Delete Log，对所述待压实SSTable进行压实，生成新的SSTable；其中，所述Delete Log中记录有所述待压实SSTable中保存的所述KV-Store***中的非最新值Value所对应的键Key，所述新的SSTable中不包含所述Delete Log中的Key对应的Key-Value对；
    删除所述待压实SSTable。
  2. 根据权利要求1所述的方法，其特征在于，在所述根据待压实有序字符串表SSTable对应的待删除日志Delete Log，对所述待压实SSTable进行压实，生成新的SSTable之前，所述方法还包括：
    在所述待压实SSTable中，确定所述KV-Store***中的非最新Value所对应的Key，作为目标Key；
    在所述Delete Log中记录所述目标Key。
  3. 根据权利要求2所述的方法，其特征在于，所述在所述Delete Log中记录所述目标Key，包括：
    在确定所述Delete Log中不包含所述目标Key时，在所述Delete Log中记录所述目标Key。
  4. 根据权利要求1-3任一项所述的方法，其特征在于，所述根据待压实有序字符串表SSTable对应的待删除日志Delete Log，对所述待压实SSTable进行压实，生成新的SSTable，包括：
    拷贝所述待压实SSTable中的、且不属于所述Delete Log中的Key对应的Key-Value，生成新的SSTable。
  5. 根据权利要求1所述的方法，其特征在于，在所述根据待压实有序字符串表SSTable对应的待删除日志Delete Log，对所述待压实SSTable进行压实，生成新的SSTable之前，所述方法还包括：
    接收携带有待查找Key的读GET操作；
    在根据所述GET操作获取到所述待查找Key对应的最新Value后，确定所述待查找Key对应的次新Value所在的SSTable，在所述确定的SSTable对应的Delete Log中记录所述待查找Key。
  6. 根据权利要求1所述的方法，其特征在于，所述KV-Store***应用于增量存储场景，在所述根据待压实有序字符串表SSTable对应的待删除日志Delete Log，对所述待压实SSTable进行压实，生成新的SSTable之前，所述方法还包括：
    接收携带有待查找键Key的读GET操作；
    在根据所述GET操作获取到所述待查找Key对应的最新Value后，确定所述待查找Key对应的最新Value所在的SSTable，在所述确定的SSTable对应的Delete Log中记录所述待查找Key；
    接收携带有所述待查找Key的写PUT操作。
  7. 根据权利要求5或6所述的方法，其特征在于，所述在所述确定的SSTable对应的Delete Log中记录所述待查找Key，包括：
    在所述确定的SSTable对应的Delete Log中不包含所述待查找Key时，在所述确定的SSTable对应的Delete Log中记录所述待查找Key。
  8. 根据权利要求1所述的方法，其特征在于，所述根据待压实有序字符串表SSTable对应的待删除日志Delete Log，对所述待压实SSTable进行压实，生成新的SSTable，包括：
    当待压实有序字符串表SSTable对应的待删除日志Delete Log中的Key的数量大于或等于预设阈值时，根据所述Delete Log，对所述待压实SSTable进行压实，生成新的SSTable。
  9. 根据权利要求1所述的方法，其特征在于，所述待压实SSTable对应一个布隆过滤器Bloom Filter，所述Bloom Filter中记录有所述Delete Log中的Key，所述方法还包括：
    当所述KV-Store***所在的服务器故障恢复后，将所述Bloom Filter的初始值设置为空；
    或，在所述KV-Store***为分布式存储***、且所述Delete Log为本地Delete Log时，当所述KV-Store***所在的服务器故障恢复后，根据全局Delete Log中记录的Key确定所述Bloom Filter的初始值；其中，所述全局Delete Log中记录有所述本地Delete Log中的Key。
  10. 一种服务器，其特征在于，包括键值存储KV-Store***，所述服务器还包括：
    压实单元，用于根据待压实有序字符串表SSTable对应的待删除日志Delete Log，对所述待压实SSTable进行压实，生成新的SSTable；其中，所述Delete Log中记录有所述待压实SSTable中保存的所述KV-Store***中的非最新值Value所对应的键Key，所述新的SSTable中不包含所述Delete Log中的Key对应的Key-Value对；
    删除单元，用于删除所述待压实SSTable。
  11. 根据权利要求10所述的服务器，其特征在于，所述服务器还包括：
    第一确定单元，用于在所述待压实SSTable中，确定所述KV-Store***中的非最新Value所对应的Key，作为目标Key；
    记录单元，用于在所述Delete Log中记录所述目标Key。
  12. 根据权利要求11所述的服务器，其特征在于，所述记录单元具体用于：在确定所述Delete Log中不包含所述目标Key时，在所述Delete Log中记录所述目标Key。
  13. 根据权利要求10-12任一项所述的服务器，其特征在于，所述压实单元具体用于：拷贝所述待压实SSTable中的、且不属于所述Delete Log中的Key对应的Key-Value，生成新的SSTable。
  14. 根据权利要求10所述的服务器，其特征在于，所述服务器还包括：
    接收单元，用于接收携带有待查找Key的读GET操作；
    第二确定单元，用于在根据所述GET操作获取到所述待查找Key对应的最新Value后，确定所述待查找Key对应的次新Value所在的SSTable，在所述确定的SSTable对应的Delete Log中记录所述待查找Key。
  15. 根据权利要求10所述的服务器，其特征在于，所述KV-Store系 统应用于增量存储场景，所述服务器还包括：
    接收单元，用于接收携带有待查找Key的读GET操作；
    第二确定单元，用于在根据所述GET操作获取到所述待查找Key对应的最新Value后，确定所述待查找Key对应的最新Value所在的SSTable，在所述确定的SSTable对应的Delete Log中记录所述待查找Key；
    所述接收单元还用于，接收携带有所述待查找Key的写PUT操作。
  16. 根据权利要求14或15所述的服务器，其特征在于，所述第二确定单元在执行在所述确定的SSTable对应的Delete Log中记录所述待查找Key时，具体用于：
    在所述确定的SSTable对应的Delete Log中不包含所述待查找Key时，在所述确定的SSTable对应的Delete Log中记录所述待查找Key。
  17. 根据权利要求10所述的服务器，其特征在于，所述压实单元具体用于：当待压实有序字符串表SSTable对应的待删除日志Delete Log中的Key的数量大于或等于预设阈值时，根据所述Delete Log，对所述待压实SSTable进行压实，生成新的SSTable。
  18. 根据权利要求10所述的服务器，其特征在于，所述待压实SSTable对应一个布隆过滤器Bloom Filter，所述Bloom Filter中记录有所述Delete Log中的Key，所述服务器还包括：第三确定单元，用于：
    当所述KV-Store***所在的服务器故障恢复后，将所述Bloom Filter的初始值设置为空；
    或，在所述KV-Store***为分布式存储***、且所述Delete Log为本地Delete Log时，当所述KV-Store***所在的服务器故障恢复后，根据全局Delete Log中记录的Key确定所述Bloom Filter的初始值；其中，所述全局Delete Log中记录有所述本地Delete Log中的Key。
PCT/CN2016/074043 2015-07-31 2016-02-18 一种键值存储***中文件压实的方法和装置 WO2017020576A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP16832065.3A EP3316150B1 (en) 2015-07-31 2016-02-18 Method and apparatus for file compaction in key-value storage system
US15/878,589 US11232073B2 (en) 2015-07-31 2018-01-24 Method and apparatus for file compaction in key-value store system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510466697.6 2015-07-31
CN201510466697.6A CN106407224B (zh) 2015-07-31 2015-07-31 一种键值存储***中文件压实的方法和装置

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US15/878,589 Continuation US11232073B2 (en) 2015-07-31 2018-01-24 Method and apparatus for file compaction in key-value store system

Publications (1)

Publication Number Publication Date
WO2017020576A1 true WO2017020576A1 (zh) 2017-02-09

Family

ID=57942345

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/074043 WO2017020576A1 (zh) 2015-07-31 2016-02-18 一种键值存储***中文件压实的方法和装置

Country Status (4)

Country Link
US (1) US11232073B2 (zh)
EP (1) EP3316150B1 (zh)
CN (1) CN106407224B (zh)
WO (1) WO2017020576A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108628542A (zh) * 2017-03-22 2018-10-09 华为技术有限公司 一种文件合并方法及控制器

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10732853B2 (en) * 2017-04-12 2020-08-04 Oracle International Corporation Dynamic memory management techniques
US10691695B2 (en) 2017-04-12 2020-06-23 Oracle International Corporation Combined sort and aggregation
US10824558B2 (en) 2017-04-26 2020-11-03 Oracle International Corporation Optimized sorting of variable-length records
CN107291541B (zh) * 2017-06-23 2020-07-10 安徽大学 面向Key-Value***的compaction粗粒度进程级并行优化方法及***
CN109271343B (zh) * 2018-07-24 2020-12-15 华为技术有限公司 一种应用于键值存储***中的数据合并方法和装置
KR20210081888A (ko) 2019-12-24 2021-07-02 삼성전자주식회사 키-밸류 기반으로 데이터를 저장하는 스토리지 장치 및 이의 동작 방법
CN113094372A (zh) 2021-04-16 2021-07-09 三星（中国）半导体有限公司 数据存取方法、数据存取控制装置及数据存取***
CN113626431A (zh) * 2021-07-28 2021-11-09 浪潮云信息技术股份公司 一种基于lsm树的延迟垃圾回收的键值分离存储方法及***
CN114780500B (zh) * 2022-06-21 2022-09-20 平安科技（深圳）有限公司 基于日志合并树的数据存储方法、装置、设备及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103218365A (zh) * 2012-01-20 2013-07-24 阿里巴巴集团控股有限公司 一种SSTable文件数据处理方法及其***
CN103473239A (zh) * 2012-06-08 2013-12-25 腾讯科技（深圳）有限公司 一种非关系型数据库数据更新方法和装置
CN103744628A (zh) * 2014-01-27 2014-04-23 北京奇虎科技有限公司 SSTable文件存储方法及装置
CN103812877A (zh) * 2014-03-12 2014-05-21 西安电子科技大学 基于Bigtable分布式存储***的数据压缩方法
US20140258234A1 (en) * 2013-03-11 2014-09-11 AppGlu, Inc. Synchronization of cms data to mobile device storage

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7428524B2 (en) * 2005-08-05 2008-09-23 Google Inc. Large scale data storage in sparse tables
US7567973B1 (en) * 2005-08-05 2009-07-28 Google Inc. Storing a sparse table using locality groups
US7548928B1 (en) * 2005-08-05 2009-06-16 Google Inc. Data compression of large scale data stored in sparse tables
US9280570B2 (en) * 2013-03-28 2016-03-08 Avaya Inc. System and method for deletion compactor for large static data in NoSQL database
US9411539B2 (en) * 2014-09-24 2016-08-09 International Business Machines Corporation Providing access information to a storage controller to determine a storage tier for storing data

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103218365A (zh) * 2012-01-20 2013-07-24 阿里巴巴集团控股有限公司 一种SSTable文件数据处理方法及其***
CN103473239A (zh) * 2012-06-08 2013-12-25 腾讯科技（深圳）有限公司 一种非关系型数据库数据更新方法和装置
US20140258234A1 (en) * 2013-03-11 2014-09-11 AppGlu, Inc. Synchronization of cms data to mobile device storage
CN103744628A (zh) * 2014-01-27 2014-04-23 北京奇虎科技有限公司 SSTable文件存储方法及装置
CN103812877A (zh) * 2014-03-12 2014-05-21 西安电子科技大学 基于Bigtable分布式存储***的数据压缩方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3316150A4 *

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108628542A (zh) * 2017-03-22 2018-10-09 华为技术有限公司 一种文件合并方法及控制器
CN108628542B (zh) * 2017-03-22 2021-08-03 华为技术有限公司 一种文件合并方法及控制器
US11403021B2 (en) 2017-03-22 2022-08-02 Huawei Technologies Co., Ltd. File merging method and controller

Also Published As

Publication number Publication date
EP3316150A4 (en) 2018-05-02
EP3316150B1 (en) 2020-09-30
US20180150472A1 (en) 2018-05-31
EP3316150A1 (en) 2018-05-02
CN106407224A (zh) 2017-02-15
US11232073B2 (en) 2022-01-25
CN106407224B (zh) 2019-09-13

Similar Documents

Publication Publication Date Title
WO2017020576A1 (zh) 一种键值存储***中文件压实的方法和装置
US11775392B2 (en) Indirect replication of a dataset
KR102007070B1 (ko) 메모리 관리 시의 중복 제거를 위해서 기준 세트로 기준 블록을 취합하는 기법
US9792306B1 (en) Data transfer between dissimilar deduplication systems
US9201800B2 (en) Restoring temporal locality in global and local deduplication storage systems
JP5732536B2 (ja) 重複排除に基づくストレージシステムにおけるスケーラブル参照管理のためのシステム、方法及び非一時的なコンピュータ可読ストレージ媒体
US10949405B2 (en) Data deduplication device, data deduplication method, and data deduplication program
CN108255647B (zh) 一种samba服务器集群下的高速数据备份方法
US9977746B2 (en) Processing of incoming blocks in deduplicating storage system
US20160196320A1 (en) Replication to the cloud
WO2013086969A1 (zh) 重复数据查找方法、装置及***
WO2014094479A1 (zh) 重复数据删除方法和装置
JP2016513306A (ja) データ格納方法、データストレージ装置、及びストレージデバイス
CN110908589B (zh) 数据文件的处理方法、装置、***和存储介质
WO2014067063A1 (zh) 重复数据检索方法及设备
US20170123678A1 (en) Garbage Collection for Reference Sets in Flash Storage Systems
CN108415671B (zh) 一种面向绿色云计算的重复数据删除方法及***
US20170123689A1 (en) Pipelined Reference Set Construction and Use in Memory Management
US12045203B2 (en) Systems and methods for physical capacity estimation of logical space units
CN113867627A (zh) 一种存储***性能优化方法及***
CN107798063B (zh) 快照处理方法和快照处理装置
CN104484402B (zh) 一种删除重复数据的方法及装置
JP6113816B1 (ja) 情報処理システム、情報処理装置、及びプログラム
CN116868173A (zh) 降低在恢复操作期间网络延时的影响
US20210117096A1 (en) Method, device and computer program product for backuping data

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16832065

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2016832065

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE