WO2020248598A1 - 一种数据分块存储方法、装置及电子设备 - Google Patents

一种数据分块存储方法、装置及电子设备 Download PDF

Info

Publication number
WO2020248598A1
WO2020248598A1 PCT/CN2020/071130 CN2020071130W WO2020248598A1 WO 2020248598 A1 WO2020248598 A1 WO 2020248598A1 CN 2020071130 W CN2020071130 W CN 2020071130W WO 2020248598 A1 WO2020248598 A1 WO 2020248598A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
layer
macroblock
lsm tree
jth
Prior art date
Application number
PCT/CN2020/071130
Other languages
English (en)
French (fr)
Inventor
阳振坤
赵裕众
Original Assignee
创新先进技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 创新先进技术有限公司 filed Critical 创新先进技术有限公司
Priority to US16/884,497 priority Critical patent/US10983975B2/en
Publication of WO2020248598A1 publication Critical patent/WO2020248598A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0608—Saving storage space on storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/064—Management of blocks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/0644—Management of space entities, e.g. partitions, extents, pools
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G06F3/0674—Disk device
    • G06F3/0676—Magnetic disk device

Definitions

  • the embodiments of this specification relate to the field of electronic technology, and in particular, to a data block storage method, device, and electronic equipment.
  • LSM-Tree Log-Structured Merge Tree
  • LSM tree is a hard disk-based data structure that can significantly reduce the overhead of the hard disk arm and provide high-speed insertion and deletion of files in a long time.
  • LSM trees perform poorly in some cases, especially when queries require fast response.
  • the LSM tree is a multi-layer data structure, and the data in each layer is organized by files. When merging data between layers, the lower layer files need to be collectively rewritten.
  • the embodiments of this specification provide a data block storage method, device and electronic equipment.
  • the embodiment of this specification provides a data block storage method, which is applied to a database using an LSM tree.
  • the LSM tree uses an N-layer data structure, and the storage space of the i-th layer is smaller than the storage space of the i+1th layer.
  • I contains an integer from 0 to N-1, and the method includes: writing multiple data macroblocks in the 0th layer of the LSM tree through the following steps:
  • the data microblock is appended to the current data macroblock in the 0th layer until the size of the current data macroblock reaches a preset value, and the next data microblock is written into the next data macroblock.
  • the embodiment of this specification provides a data block storage device, which is applied to a database using an LSM tree.
  • the LSM tree uses an N-layer data structure, and the storage space of the i-th layer is smaller than the storage space of the i+1th layer , I contains an integer from 0 to N-1, and the device includes a compression unit and a storage unit: the compression unit and the storage unit execute the following steps respectively to write multiple data in the 0th layer of the LSM tree Macro block:
  • the compression unit obtains M pieces of ordered data in the memory according to the data order, and compresses the M pieces of ordered data to obtain data microblocks, where M is an integer greater than 0;
  • the storage unit appends the data microblock to the current data macroblock in layer 0 until the size of the current data macroblock reaches a preset value, and writes the next formed data macroblock into the next data macroblock Piece.
  • an embodiment of this specification provides an electronic device, including: a memory; one or more processors; and a data block storage device in the second aspect, which is stored in the memory and configured to be composed of one or more Processor execution.
  • the embodiments of this specification provide a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the data block storage method in the first aspect are implemented.
  • the data block storage method provided in the embodiment of this specification is applied to a database using an LSM tree.
  • the LSM tree uses an N-layer data structure.
  • the storage space of the i-th layer is smaller than the storage space of the i+1th layer.
  • the 0th layer is filled with multiple data macroblocks: Obtain M ordered data in the memory according to the data sequence, and compress the M ordered data to obtain data microblocks, and append the data microblocks to the 0th layer
  • For the current data macroblock until the size of the current data macroblock reaches the preset value, the next data microblock is written into the next data macroblock. In this way, the data in the 0th layer of the LSM tree is organized by data macroblocks.
  • each layer in the LSM tree is dumped in the form of data macroblocks to the lower layer, so that each layer in the LSM tree is organized by data macro Block organization, and further, when merging data between layers, the data can be merged in the form of data macro blocks, thereby avoiding collective rewriting of the entire layer data, and effectively reducing disk IO and temporary space consumption.
  • each data macroblock includes a plurality of variable-length data microblocks, data can be stored in the storage space in a more compact manner, and the waste of space is very small, and the storage space of each layer can be fully utilized.
  • FIG. 1 is a schematic flowchart of a data block storage method provided by the first aspect of the embodiments of this specification;
  • FIG. 2 is a schematic diagram of data macro block construction in the data block storage method provided by the first aspect of the embodiments of this specification;
  • FIG. 3 is a schematic diagram of merging data macroblocks in the data block storage method provided by the first aspect of the embodiments of this specification;
  • FIG. 4 is a schematic structural diagram of a data block storage device provided by the second aspect of the embodiments of this specification.
  • FIG. 5 is a schematic diagram of the structure of an electronic device provided by the third aspect of the embodiments of this specification.
  • MicroBlock data micro block: A storage block composed of several pieces of data. The length is not fixed. The general recommended value is 16KB. The actual storage length may be shorter or longer than 16KB. It is the basic unit of data reading.
  • MacroBlock (data macro block): A storage block composed of several micro blocks, with a fixed length. The generally recommended value is 2MB, which is the basic unit of data writing.
  • Log Structured Merged Tree A multi-layered data structure in which the data of each layer is ordered, but the data between different layers is crossed. When writing, only the latest layer is written. When querying Need to merge the data of all layers.
  • the embodiment of this specification provides a data block storage method, which is applied to a database using an LSM tree.
  • the LSM tree uses an N-layer data structure.
  • the storage space of the i-th layer is smaller than the storage space of the i+1th layer, i Contains an integer from 0 to N-1.
  • the method includes: Refer to Figure 1, and write multiple data macroblocks in the 0th layer of the LSM tree through the following steps:
  • S101 Obtain M pieces of ordered data in the memory according to the data sequence, and compress the M pieces of ordered data to obtain data micro-blocks, where M is an integer greater than 0;
  • S102 Append the data microblock to the current data macroblock in the 0th layer until the size of the current data macroblock reaches a preset value, and write the next data microblock to the next data macroblock.
  • the data block storage method in this embodiment is applied to a database that uses LSM.
  • the persistent storage space uses an LSM tree structure, and the data is placed in layers, and the data is initially written to the storage system Specifically, the data of the key-value structure is written into the memory space in a log manner. Specifically, it needs to be written in the order of the data's primary key (key) value from small to large, and then from top to top It is gradually written to other levels of hard disk drives (HDD, HardDiskDrive) or solid state drives (SSD, SolidStateDrive) and other persistent storage spaces.
  • HDD hard disk drives
  • SSD SolidStateDrive
  • an area can be allocated in the memory in advance, or a storage table can be established to store the data that needs to be placed in the LSM tree later.
  • a storage table can be established to store the data that needs to be placed in the LSM tree later.
  • the LSM tree can be shaped like a pyramid, including the 0th layer, the 1st layer, the 2nd layer,..., the N-1th layer, for a total of N layers.
  • the storage space of the i-th layer is smaller than the storage space of the i+1th layer, that is, the storage space of the lower layer is larger than the storage space of the upper layer.
  • the storage space of the second layer is larger than that of the first layer.
  • the storage space of layer 1 is larger than that of layer 0. It can be followed when storing, first store in the 0th layer, after the 0th layer is full, write the data of the 0th layer into the 1st layer together, and so on, realize the data is written into the LSM tree, and the data is continuously changed from The upper storage space moves to the lower storage space.
  • the read and write sizes of data blocks are all fixed-length, usually 8KB or 16KB in size.
  • Fixed-length data blocks are convenient for storage management, but not conducive to data compression. Assuming that a data block has a size of 16KB, the size becomes 9KB after compression by some compression algorithms, but 16KB of space is still required for storage, and about 50% of the space is wasted.
  • Oracle/MySQL fixed-length blocks need to be randomly written to the disk during the persistence process, which is very unfriendly to SSD disks and mechanical disks. The performance of random writes in small blocks will be much lower than that of continuous large blocks. Write performance.
  • Figure 2 shows the construction process of layer 0 data micro-blocks and data macro-blocks.
  • M can be set according to actual needs.
  • the embodiment of this specification does not limit it.
  • the length of the data macroblock is 2MB, and the length of the data microblock is about 16KB.
  • the size of the data macroblock reaches 2M-16KB, and there is still 16KB to reach 2M.
  • the microblocks to be added at this time The length is just a little bit larger than 16KB, so it cannot be put in. At this time, less than 0.8% of 16KB/2MB space will be wasted.
  • the preset compression algorithm may be differential encoding, RLE, Huffman encoding, LZW encoding, LZ compression method, etc., which can be set according to actual needs.
  • the embodiment of this specification does not limit it.
  • the data in the memory needs to be dumped to the persistent storage space, that is: the memory
  • the data is stored in the data macroblocks in the 0th layer in the form of data microblocks until the size of the formed data macroblock reaches the preset value.
  • the data macroblock size is a fixed value of 2M
  • the preset value is 2M
  • the next data macro block will be opened, waiting to store the data micro block dumped from the memory, and the next formed data micro block will be written to the next One data macro block forms a new data macro block.
  • the data macroblocks of the 0th layer are dumped to the first layer. After the first layer is full, the data can be stored to the second layer, and so on, to realize the establishment of the LSM tree.
  • Each layer of the LSM tree stores multiple data macroblocks, and the data in the storage space of each layer is organized by data macroblocks, and there are multiple ordered data microblocks in each data macroblock.
  • the data macroblocks in each layer are stored in the form of variable-length data microblocks, which can be stored in the form of data macroblocks in a more compact manner, reducing the waste of space.
  • the write unit of the persistent storage space is the size of the data macro block
  • the read unit is the size of the data macro block.
  • the jth layer to be merged is first determined, which can be specifically as follows Steps to achieve:
  • the first method Use the layer in the LSM tree where the remaining storage space cannot store the data to be stored as the jth layer.
  • the layer that cannot store the data to be stored is taken as the jth layer that requires data merging.
  • the data to be stored can be divided into the following two situations:
  • the data to be stored is the data microblocks to be written received by the 0th layer. If the remaining storage space of the 0th layer is less than the size of one data macroblock, it indicates that the 0th layer cannot store the data microblocks to be written. , The jth layer is the 0th layer.
  • layer 0 receives the data micro-blocks dumped by the memory, and the data micro-blocks are obtained by sequentially compressing multiple pieces of ordered data in the memory through a preset compression algorithm, that is, the data in the memory Read multiple pieces of sequence data in sequence, and then compress them according to a preset compression algorithm to form data microblocks, and then dump them to the current data macroblocks of layer 0 until the total number of data microblocks contained in the current data macroblock
  • the size reaches the preset value, such as: 2M, can not fit the next data microblock, open a new data macroblock, store the next data microblock, so, by analogy, multiple data in order are formed in the 0th layer
  • the data of the 0th layer needs to be merged into the 1st layer. In this way, the 0th layer is determined to be the jth layer to be merged.
  • the second case the data to be stored is the data macroblock in the upper layer of the current layer in the LSM tree.
  • the 0th layer stores 1G data macroblocks and the first layer has 10G storage space. If 5G data has been stored in the first layer, after the 0th layer is full of 1G, the data macroblocks of the 0th layer and the first The data macroblocks of layer 1 are merged and stored in the first layer, and the merged data macroblocks are up to 6G, and the first layer has enough space for storage. Therefore, the first layer is not the jth layer to be merged. If 9.5G of data has been stored in the first layer, after the 0th layer is full of 1G, the data macroblocks of the 0th layer and the data macroblocks of the 1st layer are merged and stored in the 1st layer. The merged data macroblocks If the size exceeds 10G, the first layer does not have enough space for storage. Therefore, the first layer is the jth layer to be merged, and it needs to be merged with the macroblock data in the second layer.
  • the second method Use the layer with the remaining storage space in the LSM tree less than the preset capacity as the jth layer.
  • the preset capacity can be set according to actual needs. For the kth layer, it can be set to 10% of the storage space of the kth layer, or set to the size of the storage space of the k-1th layer, or 90% of the storage space of the k-1th layer.
  • the kth layer can have enough storage space to store the data to be written to ensure the timeliness of data writing.
  • it can also be other numerical values, and the embodiments of this specification do not limit it here.
  • the data macroblocks of the jth layer are merged into the j+1th layer, and the jth layer is cleared to store the data macroblocks merged from the j-1th layer.
  • the first data macroblock is directly dumped to the j+1 layer in the data order;
  • a new data macro block is formed, and the new data macro block is stored in order to the first j+1 layer.
  • FIG. 3 shows the organization structure of data macroblocks and the merging process.
  • Each square in the figure represents a data macro block.
  • the data in the data macro block in each layer is in order, arranged according to the key value from small to large, forming an ordered tree in the data structure Find the tree.
  • the data macroblocks of the first layer and the second layer are expressed in different colors.
  • the data macroblocks in the first layer are gray, and the data macroblocks in the second layer are white.
  • the data in the 3 data macroblocks in the first layer is just larger than the data macroblocks in the second layer, that is: the smallest key among the 3 data macroblocks in the first layer All are larger than the largest key in the data macro block in the second layer, so these 3 macro blocks can be reused directly, and the other data macro blocks in the second layer have not changed, and can be reused directly, only some intermediate
  • the node can be rewritten, and the rewritten macro block is represented in black. It can be seen from the figure that more than 80% of data macroblocks are directly reused, which reduces disk IO and temporary space consumption by 80% compared with ordinary merge.
  • data macroblocks of layer 0 For example, if it is necessary to merge data macroblocks of layer 0 into layer 1. Assuming that the 0th layer includes data macroblock 1 and data macroblock 2, and the first layer includes data macroblock 3, data macroblock 4, and data macroblock 5.
  • Data macro block 1 includes data with key ⁇ 1, 2, 3, 4, 5 ⁇
  • data macro block 2 includes data with key ⁇ 11, 12, 13, 14, 15 ⁇
  • data macro block 3 includes The key is ⁇ 1,3,5,6,7 ⁇ data
  • data macroblock 4 includes data with key ⁇ 16,17,18,19,20 ⁇
  • data macroblock 5 includes data with key ⁇ 21,22 , 23, 24, 25 ⁇ data.
  • data macroblock 1 in layer 0 and data macroblock 3 in layer 1 have data intersections, which are data with keys 1, 3, and 5, respectively.
  • Data macroblock 1 and data macroblock 3 Read into the memory, and then re-sort after de-duplication, and the sorted data is reorganized into data macro blocks.
  • data macro block 1 including data with key ⁇ 1, 2, 3, 4, 5 ⁇
  • data macro block 3 including data with key ⁇ 6, 7 ⁇
  • data macro block 2 including ⁇ 11, 12 , 13, 14, 15 ⁇ data
  • data macro block 4 includes data with key ⁇ 16, 17, 18, 19, 20 ⁇
  • data macro block 5 includes key ⁇ 21, 22, 23, 24, 25 ⁇
  • the data is stored in the first layer in order.
  • each layer is organized by data macroblocks
  • many data macroblocks without data intersection can be directly reused, such as data macroblock 2, data macroblock 4, and data macroblock 5 in this example. Transfer directly to the j+1 layer. Only the data macroblocks with data intersection (such as data macroblock 1 and data macroblock 3 in this example) need to be re-arranged and organized into data macroblocks, and stored to the j+1th layer of the jth layer in data order Therefore, compared with the prior art merging method, the consumption of disk IO resources and temporary memory space is reduced.
  • the size of the data macro block can be set to a fixed length, and the fixed length data macro block is convenient for storage space management, does not cause internal space fragmentation, and facilitates the recovery and reuse of disk space.
  • the specific length of the data macroblock is not limited, and variable-length data macroblocks can also be used. Generally speaking, as long as the length of the data macroblock is greater than the length of the data macroblock.
  • the target data in the database when reading target data in the database, it is first determined whether the target data is stored in the memory. That is, if a data read request is received, considering that the data in the memory is the fastest to find, it can be searched in the memory first to determine whether there is the requested target data in the memory. If it is found in the memory, there is no need to search in the LSM tree. If the target data is not read in the memory, it will search layer by layer from the 0th layer in the LSM tree.
  • the data macro block to which the key belongs can be located according to the key of the target data, and the value corresponding to the key of the target data is read from the data macro block. Because of the search from top to bottom, the upper layer and the lower layer are even There is data crossover, and the data stored in the upper layer is also the latest data, ensuring that the latest target data can be quickly read.
  • the data macroblocks of each layer are arranged in order, and the data macroblock containing the key can be quickly located according to the key of the target data, and then the data macroblock can be quickly read To the value corresponding to the key, ensure the efficiency of reading data.
  • the embodiments of this specification provide a data block storage device, which is applied to a database using an LSM tree.
  • the LSM tree uses an N-layer data structure, and the storage space of the i-th layer is smaller than that of the i+th layer.
  • the storage space of layer 1, i contains an integer from 0 to N-1.
  • the device includes a compression unit 401 and a storage unit 402: the compression unit 401 and the storage unit 402 perform the following steps respectively, Write multiple data macroblocks in the 0th layer of the LSM tree:
  • the compression unit 401 obtains M pieces of ordered data in the memory according to the data sequence, and compresses the M pieces of ordered data to obtain data microblocks, where M is an integer greater than 0;
  • the storage unit 402 adds the data microblock to the current data macroblock in layer 0 until the size of the current data macroblock reaches a preset value, and writes the next data microblock to the next data Macro block.
  • the device further includes:
  • a determining unit configured to determine the jth layer to be merged in the LSM tree after the 0th layer of the LSM tree is filled with multiple data macroblocks, where j is an integer from 0 to N-1;
  • a merging unit configured to merge data macroblocks in the jth layer to the j+1th layer
  • the emptying unit is used to empty the jth layer, and the empty jth layer is used to store the dumped data macroblocks of the j-1th layer.
  • the determining unit is specifically configured to:
  • the layer where the remaining storage space in the LSM tree cannot store the data to be stored is taken as the jth layer.
  • the data to be stored is the data microblock to be written received by the 0th layer, and if the remaining storage space of the 0th layer is less than the size of one data macroblock, it indicates that the first layer Layer 0 cannot store the data micro-blocks to be written; or
  • the data to be stored is a data macroblock in a layer above the current layer in the LSM tree.
  • the determining unit is specifically configured to:
  • the combining unit is specifically configured to:
  • the first data macroblock is directly dumped to The j+1 layer;
  • the data content of the same primary key in the third data macroblock is updated to The data content of the same primary key in the second data macro block obtains the updated third data macro block;
  • a new data macro block is formed, and the new data macro block
  • the blocks are stored in order to the j+1th layer.
  • the present invention also provides an electronic device, as shown in FIG. 5, which includes a memory 504, a processor 502, and is stored on the memory 504.
  • bus 500 may include any number of interconnected buses and bridges
  • bus 500 will include one or more processors represented by processor 502 and memory 504 represented by The various circuits of the memory are linked together.
  • the bus 500 can also link various other circuits such as peripherals, voltage regulators, and power management circuits, etc., which are all known in the art. Therefore, the method in this embodiment will not be performed on them. Further description.
  • the bus interface 506 provides an interface between the bus 500 and the receiver 501 and transmitter 503.
  • the receiver 501 and the transmitter 503 may be the same element, namely a transceiver, which provides a unit for communicating with various other devices on the transmission medium.
  • the processor 502 is responsible for managing the bus 500 and general processing, and the memory 504 may be used to store data used by the processor 502 when performing operations.
  • the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the foregoing data block is realized Steps of any method of storage method.
  • These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing equipment to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction device.
  • the device implements the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.
  • These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operation steps are executed on the computer or other programmable equipment to produce computer-implemented processing, so as to execute on the computer or other programmable equipment.
  • the instructions provide steps for implementing functions specified in a flow or multiple flows in the flowchart and/or a block or multiple blocks in the block diagram.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本说明书实施例提供一种数据分块存储方法，应用于采用LSM树的数据库，LSM树中第0层中的数据按数据宏块组织，LSM树中数据由上层转移至下层转储时，以数据宏块形式转储至低层，使得LSM树中各层均按数据宏块形式组织，进而，在进行层与层之间的数据合并时，可以以数据宏块的方式进行合并，可以避免对整层数据进行集体重写，能有效减少磁盘IO和临时空间消耗。并且，由于每个数据宏块中包括多个变长的数据微块，数据可以以更紧凑的方式存储在存储空间中，对于空间的浪费非常小，能充分利用各层的存储空间。

Description

一种数据分块存储方法、装置及电子设备 技术领域
本说明书实施例涉及电子技术领域，尤其涉及一种数据分块存储方法、装置及电子设备。
背景技术
LSM-Tree(Log-Structured Merge Tree，LSM树)是一种基于硬盘的数据结构，能显著地减少硬盘磁盘臂的开销，并能在较长的时间提供对文件的高速***以及删除。然而LSM树在某些情况下，特别是在查询需要快速响应时性能不佳。在现有的基于LSM树的存储引擎中，LSM树是一个多层的数据结构，每层中的数据按文件组织。在进行层与层之间的数据合并时，需要对下层文件集体重写。
发明内容
本说明书实施例提供及一种数据分块存储方法、装置及电子设备。
第一方面，本说明书实施例提供一种数据分块存储方法，应用于采用LSM树的数据库，所述LSM树采用N层数据结构，第i层的存储空间小于第i+1层的存储空间，i包含0至N-1的整数，所述方法包括：通过如下步骤在所述LSM树的第0层写满多个数据宏块：
按数据顺序获得内存中M条有序数据，并对所述M条有序数据进行压缩，获得数据微块，M为大于0的整数；
将所述数据微块追加至第0层中的当前数据宏块，直至所述当前数据宏块的大小达到预设数值，将下一个形成的数据微块写入下一个数据宏块。
第二方面，本说明书实施例提供一种数据分块存储装置，应用于采用LSM树的数据库，所述LSM树采用N层数据结构，第i层的存储空间小于第i+1层的存储空间，i包含0至N-1的整数，所述装置包括压缩单元和存储单元：通过所述压缩单元和所述存储单元分别执行如下步骤，在所述LSM树的第0层写满多个数据宏块：
所述压缩单元按数据顺序获得内存中M条有序数据，并对所述M条有序数据进行压缩，获得数据微块，M为大于0的整数；
所述存储单元将所述数据微块追加至第0层中的当前数据宏块，直至所述当前数据宏块的大小达到预设数值，将下一个形成的数据微块写入下一个数据宏块。
第三方面，本说明书实施例提供一种电子设备，包括：存储器；一个或多个处理器；及第二方面中数据分块存储装置，存储于所述存储器中并被配置成由一个或多个处理器执行。
第四方面，本说明书实施例提供一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现上述第一方面数据分块存储方法的步骤。
本说明书实施例有益效果如下：
本说明书实施例提供的数据分块存储方法，应用于采用LSM树的数据库，LSM树采用N层数据结构，第i层的存储空间小于第i+1层的存储空间，通过如下步骤在LSM树的第0层写满多个数据宏块：按数据顺序获得内存中M条有序数据，并对M条有序数据进行压缩，获得数据微块，将数据微块追加至第0层中的当前数据宏块，直至当前数据宏块的大小达到预设数值，将下一个形成的数据微块写入下一个数据宏块。这样，LSM树中第0层中的数据按数据宏块组织，LSM树中数据由上层转移至下层转储时，以数据宏块形式转储至低层，使得LSM树中各层均按数据宏块形式组织，进而，在进行层与层之间的数据合并时，可以以数据宏块的方式进行合并，从而避免对整层数据进行集体重写，能有效减少磁盘IO和临时空间消耗。并且，由于每个数据宏块中包括多个变长的数据微块，数据可以以更紧凑的方式存储在存储空间中，对于空间的浪费非常小，能充分利用各层的存储空间。
附图说明
图1为本说明书实施例第一方面提供的数据分块存储方法流程示意图；
图2为本说明书实施例第一方面提供的数据分块存储方法中构建数据宏块示意图；
图3为本说明书实施例第一方面提供的数据分块存储方法中合并数据宏块示意图；
图4为本说明书实施例第二方面提供的数据分块存储装置结构示意图；
图5为本说明书实施例第三方面提供的电子设备结构示意图。
具体实施方式
为了更好的理解上述技术方案，下面通过附图以及具体实施例对本说明书实施例的技术方案做详细的说明，应当理解本说明书实施例以及实施例中的具体特征是对本说明书实施例技术方案的详细的说明，而不是对本说明书技术方案的限定，在不冲突的情况下，本说明书实施例以及实施例中的技术特征可以相互组合。
为了更好地说明本说明书实施例，下面先对几个术语进行解释如下：
MicroBlock(数据微块)：若干条数据组成的存储块，长度不固定，一般建议值为16KB，实际存储长度可能比16KB更短或者更长，为数据读取的基本单位。
MacroBlock(数据宏块)：若干个微块组成的存储块，长度固定，一般建议值为2MB，为数据写入的基本单位。
Log Structured Merged Tree(LSM树)：一种多层的数据结构，其中每层数据都是有序的，但不同层之间的数据存在交叉，写入时只写入最新的一层，查询时需要对所有层的数据做归并。
第一方面，本说明书实施例提供一种数据分块存储方法，应用于采用LSM树的数据库，LSM树采用N层数据结构，第i层的存储空间小于第i+1层的存储空间，i包含0至N-1的整数，方法包括：请参考图1，通过如下步骤在LSM树的第0层写满多个数据宏块：
S101：按数据顺序获得内存中M条有序数据，并对M条有序数据进行压缩，获得数据微块，M为大于0的整数；
S102：将数据微块追加至第0层中的当前数据宏块，直至当前数据宏块的大小达到预设数值，将下一个形成的数据微块写入下一个数据宏块。
本实施例中的数据分块存储方法应用于采用LSM的数据库，在采用LSM的数据库中，持久化存储空间采用的是LSM树结构，数据分层放置，数据在最初时写入的是存储***中的内存空间，具体的，key-value结构的数据以log的方式写入内存空间，具体的，需要按数据的数据主键(key)值由小至大的顺序写入，然后再从上到下逐渐写入其他层次的硬盘驱动器(HDD，HardDiskDrive)或固态硬盘(SSD，SolidStateDrive)等持久化存储空间中。
具体的，内存中可以预先分配一块区域，或者是建立一个存储表格，用于存储后续 需要放入LSM树中的数据。在有数据需要存入LSM树的时候，可以是先存储在内存中，等到内存中的数据满了之后，才会将这些数据一起存入LSM树的第0层。LSM树可以是类似金字塔的形状，包括第0层、第1层、第2层、…、第N-1层，共计N层。第i层的存储空间小于第i+1层的存储空间，即：越下面的层比越上面的层的存储空间更大一些，例如，第2层的存储空间是大于第1层的，第1层的存储空间是大于第0层的。在存储的时候可以遵循，先存储在第0层，第0层满了之后将第0层的数据一起写入第1层中，以此类推，实现将数据写入LSM树中，数据不断由高层存储空间向低层存储空间移动。
在现有LSMTree的存储***中，为了给新写入的数据腾出存储空间，会经常执行数据合并操作，在HBase、RocksDB等基于LSM树的存储引擎中，每层中的数据是通过文件的方式来组织，假设第1层的文件大小是1GB，第2层的文件大小是10GB，那么在HBase和RocksDB对第1层和第2层数据合并的过程中，将第1层中的全部数据(1G的数据)和第2层的数据(10G的数据)写入内存后，在内存中对key值相同的交叉数据的更新，然后对数据重新进行排序，这样，需要对第2层数据整体重写，那么就需要额外的11GB内存空间并消耗11GB的读写IO。
并且，在Oracle、MySQL等传统数据库中，数据块的读写大小均为定长的，通常为8KB或者16KB的大小。定长数据块便于存储管理，但不利于数据压缩。假设一个数据块大小为16KB，通过一些压缩算法压缩后大小变为9KB，但是在存储上仍然需要16KB的空间，有大约50％的空间都被浪费掉了。同时Oracle/MySQL的定长块在持久化的过程中，需要随机写磁盘，这对于SSD磁盘和机械磁盘来说都是很不友好的，小块随机写的性能会远远低于大块连续写的性能。
请参考图2，图2展示了第0层数据微块和数据宏块的构造过程，采用本实施例中的方案，首先把M条有序的数据组成一个原始数据块，并对其采用预设压缩算法进行压缩，压缩后的原始数据块就成为一个数据微块，由于每次获得M条数据，形成的原始数据块大小固定，不同数据类型压缩效果不同，所以，形成的数据微块的大小也不同，所以，数据微块长度是不固定的。然后把形成的数据微块按照key值由小至大的数据顺序追加到一个宏块中，直到数据宏块放不下更多微块为止。在具体实施过程中，M可根据实际需要进行设定，在此，本说明书实施例不做限制。在建议参数下，数据宏块长度为2MB，数据微块长度为16KB左右，在最坏情况下，数据宏块的大小达到2M-16KB，还差16KB才达到2M，此时要添加的微块长度刚好比16KB大一点点导致无法放入， 此时会浪费16KB/2MB大约不到0.8％的空间。在具体实施过程中，预设压缩算法可以是差分编码、RLE、Huffman编码、LZW编码、LZ压缩方法等，可根据实际需要进行设定，在此，本说明书实施例不做限制。
这样，和Oracle/MySQL等传统数据库的定长存储块相比，在本实施例中，由于微块数据长度为变长的，可以以更紧凑的方式存储在宏块中，对于空间的浪费非常小，和Oracle等数据库相比可以节省50％的存储空间。同时一次数据读的大小仍然是16KB左右，读性能也没有受到影响。
具体的，数据在写入时，首先写入内存，内存中的数据仍然按顺序写入，在内存空间不足情况下，需要将内存中的数据转储至持久化存空间，即：将内存中的数据按数据微块的形式存储至第0层中的数据宏块中，直到形成的数据宏块的大小达到预设数值，比如，数据宏块大小为固定值2M，则预设数值为2M，也可设置为比2M小一点，例如：2M-16K等，然后会开启下一数据宏块，等待存储从内存中转储来的数据微块，将下一个形成的数据微块写入下一个数据宏块形成新的数据宏块。依次类推，直至第0层的剩余存空间小于数据宏块的长度。进而，将第0层的数据宏块转储至第1层，第1层存储满了之后，就可以将数据存储至第2层，依次类推，实现LSM树的建立。LSM树的各层中存储了多个数据宏块，各层存储空间的数据按数据宏块组织，并且，每个数据宏块中存在多个有序的数据微块。每层中数据宏块中以变长的数据微块形式存储的，可以以更紧凑的方式以数据宏块的形式存储，减少了空间的浪费。持久化存储空间的写入单位为数据宏块的大小，读取单位为数据微块大小。
进而，在数据存储至LSM树中各层存储空间过程中，为了能存储新的数据，会不断进行各层间的数据合并，具体的，首先确定出待合并的第j层，具体可通过如下步骤实现：
第一种方式：将LSM树中剩余存储空间不能存储待存储数据的层作为第j层。
具体的，在本实施例中，将不能存储待存储数据的层作为需要进行数据合并的第j层。待存储数据可分为以下两种情况：
第一种情况：待存储数据为第0层接收到待写入的数据微块，如果第0层的剩余存储空间小于一个数据宏块大小，表明第0层不能存储待写入的数据微块，第j层即为第0层。
具体的，在本实施例中，第0层接收内存转储的数据微块，数据微块为通过预设压 缩算法依次对内存中多条有序数据进行压缩后得到，即对内存中的有序数据顺序读取多条，再按预设压缩算法对其进行压缩，形成数据微块后，转储至第0层的当前数据宏块，直至当前数据宏块所包含的数据微块的总大小达到预设数值，如：2M，放不下下一个数据微块，开启新的数据宏块，存储下一个数据微块，这样，以此类推，第0层中形成了有序的多个数据宏块，直至第0层剩余的存储空间不足一个数据宏块的大小，需要将第0层的数据合并至第1层，这样，确定第0层为待合并的第j层。
第二种情况：待存储数据为LSM树中当前层的上一层中的数据宏块。
具体的，在本实施例中，针对LSM的中间层第p层，第p-1层也存储有多个数据宏块，如果第p-1层的数据宏块合并至第p层，判断第p层是否能存储p-1层所包含的所有的数据宏块，如果不能，则确定j＝p，第p层为待合并的第j层。
假设第0层存储了1G的数据宏块，第1层的存储空间为10G，如果第1层中已经存储了5G数据，第0层存满1G后，将第0层的数据宏块与第1层的数据宏块合并后存储在第1层，合并后的数据宏块最多6G，第1层具备足够的空间存储，所以，第1层不是待合并的第j层。如果第1层中已经存储了9.5G数据，第0层存满1G后，将第0层的数据宏块与第1层的数据宏块合并后存储在第1层，合并后的数据宏块大小超过10G，第1层不具备足够的空间存储，所以，第1层是待合并的第j层，需要将其与第2层中的宏块数据合并。
第二种方式：将LSM树中剩余存储空间小于预设容量的层作为第j层。
具体的，在本实施例中，可按预设时间间隔去检测各层剩余存储空间的情况，如果存在第k层剩余存储空间小于预设容量，则j＝k，第k层为待合并的第j层。在具体实施过程中，预设容量可根据实际需要进行设定，针对第k层，可设定为第k层存储空间的10％，或者设置为第k-1层存储空间的大小，也可第k-1层存储空间的90％，这样，可确保有数据写入LSM树第k层时，第k层能有足够的存储空间存储待写入数据，确保数据写入的及时性。当然，还可以是其他数值，在此，本说明书实施例不做限制。
进而，在确定出待合并的第j层以后，将第j层的数据宏块合并至第j+1层，再清空第j层，以便存储第j-1层合并而来的数据宏块。
在将第j层数据宏块合并至第j+1时，可通过如下步骤实现：
如果第j层中的第一数据宏块与第j+1层中的数据宏块间不存在主键相同的交叉数据，将第一数据宏块按数据顺序直接转储至j+1层；
如果第j层中第二数据宏块与第j+1层的第三数据宏块间存在主键相同的交叉数据，将第三数据宏块中相同主键的数据内容更新为第二数据宏块中相同主键的数据内容，获得更新后的第三数据宏块；
将第二数据宏块中除交叉数据之外的其他数据与更新后的第三数据宏块中的数据按顺序排列后，形成新的数据宏块，将新的数据宏块按顺序存储至第j+1层。
具体的，在本实施例中，图3展示了数据宏块的组织结构以及合并的过程。图中的每一个方块都表示一个数据宏块，在每个层中的数据宏块中的数据都是有序的，按key值由小至大排列，在数据结构上构成一棵有序的查找树。
第1层和第2层的数据宏块使用不同颜色表示，第1层中的数据宏块颜色为灰色，第2层中数据宏块颜色为白色，当要将第1层和第2层的数据宏块进行合并时，发现第1层中3个数据宏块中的数据刚好比第2层中的数据宏块的数据都大，即：第1层中3个数据宏块中最小的key均比第2层中的数据宏块中最大的key大，因此可以直接重用这3个宏块，而第2层中其他数据宏块都没有变化，也可以直接重用掉，只需要对一些中间节点进行重写就可以了，被重写的宏块使用黑色进行表示。从图中可以看到，有超过80％的数据宏块都是被直接重用的，和普通的合并相比，减少了80％的磁盘IO和临时空间消耗。
举例而言，如果需要将第0层的数据宏块合并至第1层。假设第0层中包括数据宏块1和数据宏块2，第1层中包括数据宏块3、数据宏块4和数据宏块5。数据宏块1中包括key为{1，2，3，4，5}的数据，数据宏块2中包括key为{11，12，13，14，15}的数据，数据宏块3中包括key为{1，3，5，6，7}的数据，数据宏块4中包括key为{16，17，18，19，20}的数据，数据宏块5中包括key为{21，22，23，24，25}的数据。
在进行数据合并时，第0层中的数据宏块1与第1层中的数据宏块3存在数据交集，分别为key为1，3，5的数据，将数据宏块1和数据宏块3读取至内存，然后进行去重处理后重新排序，排序后的数据重新组织成数据宏块。假设在第0层中，key＝1对应的value＝a，key＝3对应的value＝b，key＝3对应的value＝c。在第1层中，key＝1对应的value＝e，key＝3对应的value＝f，key＝3对应的value＝g，则去重处理后，将key为1，3，5的值最终更新为第0层中的数值，即：key＝1对应的value＝a，key＝3对应的value＝b，key＝3对应的value＝c。然后重新组织形成数据宏块1，包括key为{1，2，3，4，5}的数据，数据宏块3包括key为{6，7}的数据，数据宏块2包括{11，12，13，14，15}的数据，数据宏块4中包括key为{16，17，18，19，20}的数据，数据宏块5中包括key为{21， 22，23，24，25}的数据，按顺序存入了第1层。
由上述示例可知，由于各层按数据宏块组织，在合并时，许多没有数据交集的数据宏块可以直接重用，如本示例中的数据宏块2、数据宏块4和数据宏块5，直接转存至第j+1层。仅需对存在数据交集的数据宏块(如本示例中的数据宏块1和数据宏块3)进行去重排列组织成数据宏块，按数据顺序存储至第j层的第j+1层，所以，和现有技术合并方式相比，减少了的磁盘IO资源以及临时内存空间的消耗。
进一步，在本实施例中，数据宏块的大小可以设定为定长，定长数据宏块便于进行存储空间的管理，不会造成内部空间碎片，有利于磁盘空间的回收和重用。但在具体实施过程中，对数据宏块的具体长度并不做限制，可也采用变长的数据宏块，一般来说，只要数据宏块长度大于数据微块长度即可。
进一步，在本实施例中，在读取数据库中的目标数据时，首先确定内存中是否存储有目标数据。即，如果接收到一个数据读取请求，考虑到内存中的数据是查找最快的，那么可以先到内存中查找，确定内存中是否有请求的目标数据。如果在内存中查找到了，就不需要到LSM树中进行查找了。如果内存中没有读到目标数据，则再到LSM树中从第0层开始逐层进行查找。
在对LSM树进行查找的时候，可以选择从第0层开始，采用逐层查找的方式确定LSM树中是否有所请求的数据。具体的，可根据目标数据的key，定位到该key所属的数据宏块，在从该数据宏块中读取目标数据的key对应的value，由于从上至下查找，上层与下层间即使存在数据交叉，存放在上层的数据也是最新的数据，确保能快速读取到最新的目标数据。并且，由于各层数据按数据宏块组织，每层的数据宏块间按顺序排列，可根据目标数据的key快速定位到包含该key的数据宏块，进而从该数据宏块中快速读取到与key对应的value，确保读数据的高效。
第二方面，基于同一发明构思，本说明书实施例提供一种数据分块存储装置，应用于采用LSM树的数据库，所述LSM树采用N层数据结构，第i层的存储空间小于第i+1层的存储空间，i包含0至N-1的整数，请参考图4，所述装置包括压缩单元401和存储单元402：通过所述压缩单元401和所述存储单元402分别执行如下步骤，在所述LSM树的第0层写满多个数据宏块：
所述压缩单元401按数据顺序获得内存中M条有序数据，并对所述M条有序数据进行压缩，获得数据微块，M为大于0的整数；
所述存储单元402将所述数据微块追加至第0层中的当前数据宏块，直至所述当前数据宏块的大小达到预设数值，将下一个形成的数据微块写入下一个数据宏块。
在一种可选方式中，所述装置还包括：
确定单元，用于在所述LSM树的第0层写满多个数据宏块之后，确定所述LSM树中待合并的第j层，j为0至N-1的整数；
合并单元，用于将所述第j层中的数据宏块合并至第j+1层；
清空单元，用于清空所述第j层，清空后的所述第j层用于存储第j-1层转储的数据宏块。
在一种可选方式中，所述确定单元具体用于：
将所述LSM树中剩余存储空间不能存储待存储数据的层作为所述第j层。
在一种可选方式中，所述待存储数据为所述第0层接收到待写入的数据微块，如果所述第0层的剩余存储空间小于一个数据宏块大小，表明所述第0层不能存储所述待写入的数据微块；或
所述待存储数据为所述LSM树中当前层的上一层中的数据宏块。
在一种可选方式中，所述确定单元具体用于：
将所述LSM树中剩余存储空间小于预设容量的层作为所述第j层。
在一种可选方式中，所述合并单元具体用于：
如果所述第j层中的第一数据宏块与所述第j+1层中的数据宏块间不存在主键相同的交叉数据，将所述第一数据宏块按数据顺序直接转储至所述j+1层；
如果所述第j层中第二数据宏块与所述第j+1层的第三数据宏块间存在主键相同的交叉数据，将所述第三数据宏块中相同主键的数据内容更新为所述第二数据宏块中所述相同主键的数据内容，获得更新后的第三数据宏块；
将所述第二数据宏块中除所述交叉数据之外的其他数据与更新后的第三数据宏块中的数据按顺序排列后，形成新的数据宏块，将所述新的数据宏块按顺序存储至所述第j+1层。
本实施例中的数据分块存储装置进行数据处理的详细过程已在前述第一方面实施例中详细阐述，具体可参照第一方面实施例中对应的内容，在此，本说明书实施例不做赘 述。
第三方面，基于与前述实施例中数据分块存储方法同样的发明构思，本发明还提供一种电子设备，如图5所示，包括存储器504、处理器502及存储在存储器504上并可在处理器502上运行的计算机程序，处理器502执行程序时实现前文数据分块存储方法的任一方法的步骤。
其中，在图5中，总线架构(用总线500来代表)，总线500可以包括任意数量的互联的总线和桥，总线500将包括由处理器502代表的一个或多个处理器和存储器504代表的存储器的各种电路链接在一起。总线500还可以将诸如***设备、稳压器和功率管理电路等之类的各种其他电路链接在一起，这些都是本领域所公知的，因此，本实施例中的方法不再对其进行进一步描述。总线接口506在总线500和接收器501和发送器503之间提供接口。接收器501和发送器503可以是同一个元件，即收发机，提供用于在传输介质上与各种其他装置通信的单元。处理器502负责管理总线500和通常的处理，而存储器504可以被用于存储处理器502在执行操作时所使用的数据。
第六方面，基于与前述实施例中数据分块存储方法的发明构思，本发明还提供一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现前文数据分块存储方法的任一方法的步骤。
本说明书是参照根据本说明书实施例的方法、设备(***)、和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的设备。
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令设备的制造品，该指令设备实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或 其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。
尽管已描述了本说明书的优选实施例，但本领域内的技术人员一旦得知了基本创造性概念，则可对这些实施例作出另外的变更和修改。所以，所附权利要求意欲解释为包括优选实施例以及落入本说明书范围的所有变更和修改。
显然，本领域的技术人员可以对本说明书进行各种改动和变型而不脱离本说明书的精神和范围。这样，倘若本说明书的这些修改和变型属于本说明书权利要求及其等同技术的范围之内，则本说明书也意图包含这些改动和变型在内。

Claims (15)

  1. 一种数据分块存储方法，应用于采用LSM树的数据库，所述LSM树采用N层数据结构，第i层的存储空间小于第i+1层的存储空间，i包含0至N-1的整数，所述方法包括：通过如下步骤在所述LSM树的第0层写满多个数据宏块：
    按数据顺序获得内存中M条有序数据，并对所述M条有序数据进行压缩，获得数据微块，M为大于0的整数；
    将所述数据微块追加至第0层中的当前数据宏块，直至所述当前数据宏块的大小达到预设数值，将下一个形成的数据微块写入下一个数据宏块。
  2. 如权利要求1所述的方法，在所述LSM树的第0层写满多个数据宏块之后，所述方法还包括：
    确定所述LSM树中待合并的第j层，j为0至N-1的整数；
    将所述第j层中的数据宏块合并至第j+1层；
    清空所述第j层，清空后的所述第j层用于存储第j-1层转储的数据宏块。
  3. 如权利要求1所述的方法，所述确定所述LSM树中待合并的第j层，包括：
    将所述LSM树中剩余存储空间不能存储待存储数据的层作为所述第j层。
  4. 如权利要求3所述的方法，所述待存储数据为所述第0层接收到待写入的数据微块，如果所述第0层的剩余存储空间小于一个数据宏块大小，表明所述第0层不能存储所述待写入的数据微块；或
    所述待存储数据为所述LSM树中当前层的上一层中的数据宏块。
  5. 如权利要求2所述的方法，所述确定所述LSM树中待合并的第j层，包括：
    将所述LSM树中剩余存储空间小于预设容量的层作为所述第j层。
  6. 如权利要求2所述的方法，所述将所述第j层中的数据宏块合并至第j+1层，包括：
    如果所述第j层中的第一数据宏块与所述第j+1层中的数据宏块间不存在主键相同的交叉数据，将所述第一数据宏块按数据顺序直接转储至所述j+1层；
    如果所述第j层中第二数据宏块与所述第j+1层的第三数据宏块间存在主键相同的交叉数据，将所述第三数据宏块中相同主键的数据内容更新为所述第二数据宏块中所述相同主键的数据内容，获得更新后的第三数据宏块；
    将所述第二数据宏块中除所述交叉数据之外的其他数据与更新后的第三数据宏块中的数据按顺序排列后，形成新的数据宏块，将所述新的数据宏块按顺序存储至所述第j+1层。
  7. 如权利要求1-6任一项所述的方法，所述LSM树中各层存储的数据宏块的大小为预设的固定数值。
  8. 一种数据分块存储装置，应用于采用LSM树的数据库，所述LSM树采用N层数据结构，第i层的存储空间小于第i+1层的存储空间，i包含0至N-1的整数，所述装置包括压缩单元和存储单元：通过所述压缩单元和所述存储单元分别执行如下步骤，在所述LSM树的第0层写满多个数据宏块：
    所述压缩单元按数据顺序获得内存中M条有序数据，并对所述M条有序数据进行压缩，获得数据微块，M为大于0的整数；
    所述存储单元将所述数据微块追加至第0层中的当前数据宏块，直至所述当前数据宏块的大小达到预设数值，将下一个形成的数据微块写入下一个数据宏块。
  9. 如权利要求8所述的装置，所述装置还包括：
    确定单元，用于在所述LSM树的第0层写满多个数据宏块之后，确定所述LSM树中待合并的第j层，j为0至N-1的整数；
    合并单元，用于将所述第j层中的数据宏块合并至第j+1层；
    清空单元，用于清空所述第j层，清空后的所述第j层用于存储第j-1层转储的数据宏块。
  10. 如权利要求9所述的装置，所述确定单元具体用于：
    将所述LSM树中剩余存储空间不能存储待存储数据的层作为所述第j层。
  11. 如权利要求10所述的装置，所述待存储数据为所述第0层接收到待写入的数据微块，如果所述第0层的剩余存储空间小于一个数据宏块大小，表明所述第0层不能存储所述待写入的数据微块；或
    所述待存储数据为所述LSM树中当前层的上一层中的数据宏块。
  12. 如权利要求9所述的装置，所述确定单元具体用于：
    将所述LSM树中剩余存储空间小于预设容量的层作为所述第j层。
  13. 如权利要求9所述的装置，所述合并单元具体用于：
    如果所述第j层中的第一数据宏块与所述第j+1层中的数据宏块间不存在主键相同的交叉数据，将所述第一数据宏块按数据顺序直接转储至所述j+1层；
    如果所述第j层中第二数据宏块与所述第j+1层的第三数据宏块间存在主键相同的交叉数据，将所述第三数据宏块中相同主键的数据内容更新为所述第二数据宏块中所述相同主键的数据内容，获得更新后的第三数据宏块；
    将所述第二数据宏块中除所述交叉数据之外的其他数据与更新后的第三数据宏块 中的数据按顺序排列后，形成新的数据宏块，将所述新的数据宏块按顺序存储至所述第j+1层。
  14. 一种电子设备，包括：
    存储器；
    一个或多个处理器；及
    权利要求8-13中任一项所述的数据分块存储装置，存储于所述存储器中并被配置成由一个或多个处理器执行。
  15. 一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现权利要求1-7中任一项所述方法所包含的步骤。
PCT/CN2020/071130 2019-06-13 2020-01-09 一种数据分块存储方法、装置及电子设备 WO2020248598A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US16/884,497 US10983975B2 (en) 2019-06-13 2020-05-27 Data block storage method and apparatus, and electronic device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910511095.6 2019-06-13
CN201910511095.6A CN110377227B (zh) 2019-06-13 2019-06-13 一种数据分块存储方法、装置及电子设备

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/884,497 Continuation US10983975B2 (en) 2019-06-13 2020-05-27 Data block storage method and apparatus, and electronic device

Publications (1)

Publication Number Publication Date
WO2020248598A1 true WO2020248598A1 (zh) 2020-12-17

Family

ID=68250265

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/071130 WO2020248598A1 (zh) 2019-06-13 2020-01-09 一种数据分块存储方法、装置及电子设备

Country Status (3)

Country Link
CN (1) CN110377227B (zh)
TW (1) TWI734364B (zh)
WO (1) WO2020248598A1 (zh)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110377227B (zh) * 2019-06-13 2020-07-07 阿里巴巴集团控股有限公司 一种数据分块存储方法、装置及电子设备
CN111104403B (zh) * 2019-11-30 2022-06-07 北京浪潮数据技术有限公司 一种lsm树数据处理方法、***、设备及计算机介质
CN113609090B (zh) * 2021-08-06 2024-06-18 杭州网易云音乐科技有限公司 数据存储方法及装置、计算机可读存储介质、电子设备

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104142958A (zh) * 2013-05-10 2014-11-12 华为技术有限公司 一种键值对***中数据的存储方法和相关装置
CN105447059A (zh) * 2014-09-29 2016-03-30 华为技术有限公司 一种数据处理方法及装置
CN107038206A (zh) * 2017-01-17 2017-08-11 阿里巴巴集团控股有限公司 Lsm树的建立方法、lsm树的数据读取方法和服务器
CN107436738A (zh) * 2017-08-17 2017-12-05 北京理工大学 一种数据存储方法及***
US20180107402A1 (en) * 2016-10-19 2018-04-19 Acronis International Gmbh System and method for data storage using log-structured merge trees
CN110377227A (zh) * 2019-06-13 2019-10-25 阿里巴巴集团控股有限公司 一种数据分块存储方法、装置及电子设备

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9116793B2 (en) * 2012-06-12 2015-08-25 International Business Machines Corporation Maintaining versions of data in solid state memory
US9037791B2 (en) * 2013-01-22 2015-05-19 International Business Machines Corporation Tiered caching and migration in differing granularities
US9311252B2 (en) * 2013-08-26 2016-04-12 Globalfoundries Inc. Hierarchical storage for LSM-based NoSQL stores
CN105447035B (zh) * 2014-08-29 2018-12-25 华为技术有限公司 数据扫描方法及装置
CN105159915B (zh) * 2015-07-16 2018-07-10 中国科学院计算技术研究所 可动态适应的lsm树合并方法及***
CN106708427B (zh) * 2016-11-17 2019-05-10 华中科技大学 一种适用于键值对数据的存储方法
CN106599247B (zh) * 2016-12-19 2020-04-17 北京奇虎科技有限公司 LSM-tree结构中数据文件的合并方法及装置
CN106682184B (zh) * 2016-12-29 2019-12-20 华中科技大学 一种基于日志合并树结构的轻量级合并方法
CN108733306B (zh) * 2017-04-14 2020-04-21 华为技术有限公司 一种文件合并方法及装置
CN108090372A (zh) * 2017-12-23 2018-05-29 王彬 一种数据存储方法及装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104142958A (zh) * 2013-05-10 2014-11-12 华为技术有限公司 一种键值对***中数据的存储方法和相关装置
CN105447059A (zh) * 2014-09-29 2016-03-30 华为技术有限公司 一种数据处理方法及装置
US20180107402A1 (en) * 2016-10-19 2018-04-19 Acronis International Gmbh System and method for data storage using log-structured merge trees
CN107038206A (zh) * 2017-01-17 2017-08-11 阿里巴巴集团控股有限公司 Lsm树的建立方法、lsm树的数据读取方法和服务器
CN107436738A (zh) * 2017-08-17 2017-12-05 北京理工大学 一种数据存储方法及***
CN110377227A (zh) * 2019-06-13 2019-10-25 阿里巴巴集团控股有限公司 一种数据分块存储方法、装置及电子设备

Also Published As

Publication number Publication date
TW202046118A (zh) 2020-12-16
CN110377227A (zh) 2019-10-25
CN110377227B (zh) 2020-07-07
TWI734364B (zh) 2021-07-21

Similar Documents

Publication Publication Date Title
WO2020248598A1 (zh) 一种数据分块存储方法、装置及电子设备
US10761758B2 (en) Data aware deduplication object storage (DADOS)
US10289315B2 (en) Managing I/O operations of large data objects in a cache memory device by dividing into chunks
CN102364474B (zh) 用于机群文件***的元数据存储***和管理方法
CN103488709B (zh) 一种索引建立方法及***、检索方法及***
CN103678160B (zh) 一种存储数据的方法和装置
CN108804510A (zh) 键值文件***
CN109582593B (zh) 一种基于计算的ftl地址映射读、写方法
CN110032549B (zh) 分区***方法、装置、电子设备及可读存储介质
CN105683898A (zh) 对储存***中数据高效存储与检索的组相关哈希表组织
CN101504670A (zh) 数据操作方法、***、客户端和数据服务器
CN103150122B (zh) 一种磁盘缓存空间管理方法和装置
CN106570113B (zh) 一种海量矢量切片数据云存储方法及***
CN106681668A (zh) 一种基于固态盘缓存的混合式存储***及存储方法
CN105824881A (zh) 一种基于负载均衡的重复数据删除数据放置方法器
US10983975B2 (en) Data block storage method and apparatus, and electronic device
CN101271429A (zh) 一种数据存储方法及装置
CN107817945A (zh) 一种混合内存结构的数据读取方法和***
WO2024061108A1 (en) Distributed storage systems and methods thereof, device and storage medium
CN104035925A (zh) 数据存储方法、装置和存储***
Zhan et al. RangeKV: An efficient key-value store based on hybrid DRAM-NVM-SSD storage structure
CN110597912A (zh) 一种区块存储方法及装置
Zhang et al. FlameDB: A key-value store with grouped level structure and heterogeneous Bloom filter
CN112711564B (zh) 合并处理方法以及相关设备
CN113868245A (zh) 使用节点缓冲区搜索键值索引

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20822178

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20822178

Country of ref document: EP

Kind code of ref document: A1