CN115809011A

CN115809011A - Data reconstruction method and device in storage system

Info

Publication number: CN115809011A
Application number: CN202111070821.9A
Authority: CN
Inventors: 李大权; 易鹏
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2021-09-13
Filing date: 2021-09-13
Publication date: 2023-03-17

Abstract

The embodiment of the application provides a data reconstruction method and device in a storage system, relates to the field of data storage, and can improve the security of data in the storage system. The method comprises the following steps: determining a faulty block group of a plurality of block groups of a storage system; wherein, the block group comprises a plurality of blocks which respectively correspond to different hard disks in the storage system, and an erasure code mechanism is arranged among the blocks in the block group; the fault block group comprises a first fault block group and a second fault block group; the number of the fault blocks C1 in the first fault block group is larger than the number of the fault blocks C2 in the second fault block group; wherein, C1 and C2 are both positive integers, and C1 is less than or equal to the number of blocks which can be erased by an erasure code mechanism; and preferentially reconstructing data in k fault blocks in the first fault block group, wherein k is the difference value between C1 and C2.

Description

Data reconstruction method and device in storage system

Technical Field

The embodiment of the application relates to the field of data storage, in particular to a method and a device for reconstructing data in a storage system.

Background

With the development of internet technology, the concern of each large enterprise on the security of data is higher and higher, and a redundant array of independent hard disks (also called a hard disk array) is widely applied to the field of data storage as a data storage system with fault tolerance.

At present, redundant Arrays of Independent Disks (RAID) have different versions of data storage systems, such as RAID 0, RAID 3, RAID 5, RAID6, and RAID 10, taking RAID6 as an example, RAID6 constitutes one hard disk array from at least 5 hard disks, when data to be written needs to be stored in the hard disk array, the data to be written is divided into X parts of sub data, and the X parts of sub data and 2 erasure codes corresponding to the X parts of sub data are respectively stored in blocks (chunks, CK) on different hard disks in the hard disk array (the blocks may also be referred to as fixed-granularity pieces of hard disks), where a block corresponding to each of the X parts of sub data and a block corresponding to 2 erasure codes of the X parts of sub data (i.e., X +2 blocks) form a block group (chunk group, CKG), and a block group may also be referred to as a combination of fixed-granularity pieces of hard disks. When the hard disk where 1 or 2 blocks in a block group are located fails, the processor reconstructs data on the failed block (i.e., the block belonging to the block group in the failed hard disk) according to data in other healthy blocks in the block group and based on an erasure code mechanism, and the data of other block groups on the failed hard disk corresponding to the failed block also needs to be reconstructed by a similar method, so as to reconstruct all data on the failed hard disk, thereby improving the fault tolerance and stability of the data.

When a plurality of hard disks in the hard disk array fail and a failed hard disk comprises blocks in a plurality of block groups, reconstructing data in the failed blocks according to the position sequence of the failed blocks in the failed hard disk, in this case, if the failed blocks in a certain block group change from the previous 2 failed blocks to 3 failed blocks, the number of the failed blocks in the block group exceeds a predetermined failed block threshold corresponding to an erasure code mechanism, so that data in the failed blocks in the block group cannot be reconstructed, and thus, data in a part of blocks in the block group is lost.

Disclosure of Invention

The embodiment of the application provides a data reconstruction method and device in a storage system, which can improve the security of data in the storage system.

In order to achieve the above purpose, the embodiment of the present application adopts the following technical solutions:

in a first aspect, an embodiment of the present application provides a method and an apparatus for reconstructing data in a storage system, where the method includes: determining a faulty block group of a plurality of block groups of a storage system; the block group comprises a plurality of blocks, the blocks respectively correspond to different hard disks in the storage system, and an erasure code mechanism is arranged among the blocks in the block group; the fault block group comprises a first fault block group and a second fault block group; the number C1 of the fault blocks in the first fault block group is greater than the number C2 of the fault blocks in the second fault block group; wherein, C1 and C2 are both positive integers, and C1 is less than or equal to the number of blocks that can be erased by the erasure code mechanism; and preferentially reconstructing data in k fault blocks in the first fault block group, wherein k is the difference value between C1 and C2.

In the data reconstruction method in the storage system provided in the embodiment of the present application, the apparatus for data reconstruction determines a faulty block group in a plurality of block groups of the storage system, where the faulty block group includes a first faulty block group and a second faulty block group, the number C1 of faulty blocks in the first faulty block group is greater than the number C2 of faulty blocks in the second faulty block group, and C1 is less than or equal to the number of blocks that can be erased by an erasure coding mechanism in the storage system; then, data in k faulty blocks in the first faulty block group is preferentially reconstructed, where k is a difference between C1 and C2. In this embodiment of the present application, when a hard disk failure occurs in a storage system, partial data of a failed block group (for example, the first failed block group) with the most failed blocks in the storage system is preferentially reconstructed, so that it is possible to avoid data loss of the first failed block group due to a failure of a hard disk in which another block in the first failed block group is located.

In a possible implementation manner, the determining a faulty block group in the plurality of block groups of the storage system includes: determining a failed hard disk in the storage system; and determining a fault block group in the plurality of block groups according to the fault hard disk.

In a possible implementation manner, the preferentially reconstructing data in k faulty blocks in the first faulty block group specifically includes: based on the erasure code mechanism, reconstructing data in k fault blocks in the first fault block group according to data in the healthy blocks in the first fault block group; storing the data in k fault blocks in the reconstructed first fault block group into k hot standby blocks; and replacing k fault blocks in the first fault block group with k hot standby blocks.

In a possible implementation manner, the hard disk where the hot standby block is located is a hot standby disk.

In a possible implementation manner, the failed block in the first failed block group corresponds to at least two failed hard disks in the storage system.

In a possible implementation manner, after preferentially reconstructing data in k failed blocks in the first failed block group, when k hard disk failures occur again in the storage system, the data in the failed blocks in the first failed block group is reconstructed based on a fir tree erasure code mechanism.

In the data reconstruction method in the storage system provided in the embodiment of the application, partial data in a faulty block group with the largest number of faulty blocks (for example, data in k faulty blocks in a first faulty block group are reconstructed preferentially), but data in all faulty blocks in the first faulty block group are not reconstructed preferentially, and in a short time when data in k faulty blocks in the first faulty block group are reconstructed preferentially, the probability of failure of other hard disks in the storage system is low, so that after data in k faulty blocks in the first faulty block group are reconstructed preferentially, whether a new faulty hard disk exists in the storage system is periodically detected, and in the case that a new faulty hard disk exists in the storage system, the faulty block group with the largest number of faulty blocks in the storage system is determined, and partial data in the faulty block group with the largest number of faulty blocks is reconstructed preferentially based on an erasure correction and deletion mechanism, so that the risk of data loss in the storage system can be reduced, and the reliability of the storage system is further improved.

In a possible implementation manner, after data in k failed blocks in the first failed block group is preferentially reconstructed, a failure occurs again in k hard disks in the storage system, where the total number of currently failed hard disks in the storage system is M, and data in the failed blocks in the first failed block group is reconstructed based on the erasure correcting code mechanism, where M is greater than the number of blocks that can be erased by the erasure correcting code mechanism.

In a possible implementation manner, the erasure code mechanism is a redundant array of independent hard disks mechanism.

In a possible implementation manner, the hard disk is a solid state hard disk.

In a possible implementation manner, the hard disk is a mechanical hard disk.

In a second aspect, an embodiment of the present application provides a data reconstruction apparatus in a storage system, including: a determining module and a reconstructing module; the determining module is used for determining a fault block group in a plurality of block groups of the storage system; wherein, the block group comprises a plurality of blocks which respectively correspond to different hard disks in the storage system, and an erasure code mechanism is arranged among the blocks in the block group; the fault block group comprises a first fault block group and a second fault block group; the number of the fault blocks C1 in the first fault block group is larger than the number of the fault blocks C2 in the second fault block group; wherein, C1 and C2 are both positive integers, and C1 is less than or equal to the number of blocks that can be erased by the erasure code mechanism; the reconstruction module is used for preferentially reconstructing data in k fault blocks in the first fault block group, wherein k is a difference value between C1 and C2.

In a possible implementation manner, the determining module is specifically configured to determine a failed hard disk in the storage system; and determining a fault block group in the plurality of block groups according to the fault hard disk.

In a possible implementation manner, the data reconstructing apparatus in the storage system further includes: the device comprises a storage module and an updating module; the reconstruction module is specifically configured to reconstruct data in k faulty blocks in the first faulty block group according to data in healthy blocks in the first faulty block group based on the erasure code mechanism; the storage module is used for storing the data in k fault blocks in the reconstructed first fault block group into k hot standby blocks; the updating module is used for replacing k fault blocks in the first fault block group with k hot standby blocks.

In a possible implementation manner, the reconstruction module is configured to reconstruct data in a failed block in the first failed block group based on the erasure correcting code mechanism when k hard disk failures occur again in the storage system.

In a possible implementation manner, after preferentially reconstructing data in k failed blocks in the first failed block group, k hard disk failures occur again in the storage system, where the total number of currently failed hard disks in the storage system is M, and data in the failed blocks in the first failed block group is reconstructed based on the erasure code mechanism, where M is greater than the number of blocks that can be erased by the erasure code mechanism.

In a possible implementation manner, the hard disk is a solid state hard disk.

In a possible implementation manner, the hard disk is a mechanical hard disk.

In a third aspect, an embodiment of the present application provides a storage system, including a controller and a hard disk, where the controller includes a processor, and the processor is configured to determine a faulty block group in a plurality of block groups of the storage system; wherein, the block group comprises a plurality of blocks which respectively correspond to different hard disks in the hard disks, and an erasure code mechanism is arranged among the blocks in the block group; the fault block group comprises a first fault block group and a second fault block group; the number of the fault blocks C1 in the first fault block group is larger than the number of the fault blocks C2 in the second fault block group; wherein, C1 and C2 are both positive integers, and C1 is less than or equal to the number of blocks which can be erased by an erasure code mechanism; and preferentially reconstructing data in k fault blocks in the first fault block group, wherein k is the difference value between C1 and C2.

In a possible implementation manner, the processor is specifically configured to determine a failed hard disk in the storage system; and determining a fault block group in the plurality of block groups according to the fault hard disk.

In a possible implementation manner, the processor is specifically configured to reconstruct, based on an erasure code mechanism and according to data in healthy blocks in the first faulty block group, data in k faulty blocks in the first faulty block group; storing data in k fault blocks in the reconstructed first fault block group into k hot standby blocks; and replacing k fault blocks in the first fault block group with k hot standby blocks.

In a possible implementation manner, the processor is further configured to reconstruct the data in the failed block in the first failed block group based on an erasure code mechanism when k hard disk failures occur again in the storage system after preferentially reconstructing the data in the k failed blocks in the first failed block group.

In a possible implementation manner, the processor is further configured to reconstruct, after preferentially reconstructing data in k failed blocks in the first failed block group, data in the failed blocks in the first failed block group again, where the total number of currently failed hard disks in the storage system is M, and reconstruct the data in the failed blocks in the first failed block group based on an erasure correcting code mechanism, where M is greater than the number of blocks that can be erased by the erasure correcting code mechanism.

In a possible implementation manner, the erasure code mechanism is a redundant array of independent disks mechanism.

In a possible implementation manner, the hard disk is a solid state hard disk.

In a possible implementation manner, the hard disk is a mechanical hard disk.

In a fourth aspect, an embodiment of the present application provides a computer storage medium including computer instructions, which, when run on a computing device, cause the computing device to perform the method described in any one of the first aspect and its possible implementation manners.

In a fifth aspect, embodiments of the present application provide a computer program product, which when run on a computer, causes the computer to execute the method described in any one of the first aspect and possible implementation manners.

Drawings

Fig. 1 is a schematic diagram of a data partitioning method according to an embodiment of the present application;

FIG. 2 is a schematic diagram of a data storage according to an embodiment of the present application;

FIG. 3 is a schematic diagram of a reconstructed data storage according to an embodiment of the present application;

FIG. 4 is a schematic diagram of another data storage provided by an embodiment of the present application;

FIG. 5 is a schematic structural diagram of a data storage system according to an embodiment of the present application;

fig. 6 is a schematic flowchart of a data reconstruction method in a storage system according to an embodiment of the present disclosure;

fig. 7 is a schematic flowchart of a data reconstruction method in another storage system according to an embodiment of the present disclosure;

fig. 8 is a schematic flowchart illustrating a data reconstruction method in a storage system according to an embodiment of the present disclosure;

fig. 9 is a schematic flowchart illustrating a data reconstruction method in a storage system according to an embodiment of the present disclosure;

FIG. 10 is a schematic diagram of data storage during a reconstruction process according to an embodiment of the present application;

fig. 11 is a schematic diagram illustrating a data reconstruction method in a storage system according to an embodiment of the present disclosure;

fig. 12 is a schematic diagram of a data reconstruction apparatus according to an embodiment of the present application;

fig. 13 is a schematic diagram of another data reconstruction apparatus according to an embodiment of the present application.

Detailed Description

The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone.

The terms "first" and "second," and the like, in the description and in the claims of the embodiments of the present application are used for distinguishing between different objects and not for describing a particular order of the objects. For example, the first faulty block group and the second faulty block group, etc. are for distinguishing different faulty block groups, rather than for describing a specific order of the faulty block groups.

In the embodiments of the present application, the words "exemplary" or "such as" are used herein to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "e.g.," is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Rather, use of the word "exemplary" or "such as" is intended to present concepts related in a concrete fashion.

In the description of the embodiments of the present application, the meaning of "a plurality" means two or more unless otherwise specified. For example, a plurality of block groups refers to two or more block groups.

First, some concepts related to a method and an apparatus for reconstructing data in a storage system provided in an embodiment of the present application are explained.

The Erasure Coding (EC) checking mechanism is to divide data into at least two data fragments, calculate the check fragments of the at least two data fragments according to a certain checking algorithm, and at this moment, when one of the two data fragments is lost, recover data (i.e., reconstruct data) by using the other data fragment and the check fragment.

For example, as shown in fig. 1, the target data is divided into data a and data B, which are stored in different areas, and then, the verification data C of the target data is calculated according to a verification algorithm, for example: data of

Data B = check data C, wherein,

is an XOR operator; if the data B is lost, the data B can be reconstructed according to the data A and the check data C.

With the advent of the big data era, the security of data storage is more and more concerned by various large enterprises, and RAID is applied to the field of data storage as an EC (redundant array of independent disks) verification mechanism.

Currently, RAID includes data storage systems with different fault tolerance modes, such as RAID 0, RAID 3, RAID 5, RAID6, and RAID 10. Taking RAID6 as an example, in the data storage system shown in fig. 2, when first data needs to be written into the data storage system corresponding to RAID6, a processor corresponding to the storage system divides the first data into 3 pieces of sub data (data fragments), and then performs xor operation on the 3 pieces of sub data to obtain 2 pieces of check data (i.e., 2 check fragments); further, the 3 pieces of sub data and the 2 pieces of check data corresponding to the 3 pieces of sub data are respectively stored in corresponding blocks, which are respectively blocks 1-3 (i.e., the block numbered 3 in the hard disk 1), 2-2, 3-5, 4-6 and 5-2, so that the blocks 1-3, 2-2, 3-5, 4-6 and 5-2 form a first block group. Based on the method for writing the first data into the data storage system corresponding to the RAID6, the second data and 2 check data corresponding to the subdata of the second data are written into the blocks 1-2, 3-3, 4-2, 5-1, and 6-4, respectively, so that the blocks 1-2, 3-3, 4-2, 5-1, and 6-4 constitute a second block group.

When the hard disk 1 and the hard disk 2 in the data storage system are in failure, the hard disk 1 and the hard disk 2 become failure hard disks, so that the blocks 1-3 and the blocks 2-2 in the first block group are failure blocks, and further the data stored in the blocks 1-3 and the blocks 2-2 in the first block group are lost; similarly, the blocks 1-2 in the second block group are failure blocks, that is, the data stored in the blocks 1-2 in the second block group is lost. In this case, an erasure code mechanism may be adopted to reconstruct the data in the failed block, specifically, the processor reconstructs the data in the failed block according to the position order of the blocks in the failed hard disk; for example, in the failed hard disk 1, the data on the failed block 1-2 in the failed hard disk 1 is reconstructed, then the data on the failed block 1-3 in the failed hard disk 1 is reconstructed, and in the failed hard disk 2, the data in the block 2-1 is reconstructed, and then the data in the block 2-2 is reconstructed. Taking reconstruction of the failed block 1-2 as an example, the failed block 1-2 belongs to the second block group, so that the data stored on the failed block 1-2 can be reconstructed from the data stored in the healthy blocks (i.e., block 3-3, block 4-2, block 5-1, and block 6-4) in the second block group.

Based on the above description, in the data storage system corresponding to RAID6, data reconstruction is performed in units of hard disks (that is, after all data in all blocks on one failed hard disk are reconstructed, data in all blocks on the next failed hard disk are reconstructed).

However, in the data storage system corresponding to RAID6 shown in fig. 2, for the failed hard disk 1, when the processor is after reconstructing the data in the blocks 1-2 and the processor has not reconstructed the data in the remaining failed blocks in the failed hard disk 1 (i.e. has not reconstructed the data on the blocks 1-3), if the hard disk 5 in the data storage system becomes the failed hard disk, as shown in fig. 4, there are 3 failed hard disks in the data storage system corresponding to RAID6, and the blocks 1-3, 2-2, and 5-2 in the first block group are all the failed blocks, and 6 supports the number of the failed blocks in one RAID block group at maximum as 2, so as to cause the first data corresponding to the first block group to be lost as the hard disk 5 becomes the failed hard disk, thereby reducing the security of the data in the storage system.

For the problem of low data security in the data storage system in the above scheme, in the data reconstruction method and apparatus in the storage system according to the embodiment of the present application, the apparatus for data reconstruction determines a faulty block group in a plurality of block groups of the storage system; wherein, a block group comprises a plurality of blocks which respectively correspond to different hard disks in the storage system, and erasure code mechanisms are arranged among the blocks in the block group; the fault block group comprises a first fault block group and a second fault block group; the number of the fault blocks C1 in the first fault block group is larger than the number of the fault blocks C2 in the second fault block group; wherein, C1 and C2 are both positive integers, and C1 is less than or equal to the number of blocks that can be erased by the erasure correcting code mechanism; and preferentially reconstructing data in k fault blocks in the first fault block group, wherein k is the difference value between C1 and C2. By the technical scheme provided by the embodiment of the application, the safety of the data in the data storage system can be improved.

The data reconstruction method and apparatus in the data storage system provided in the embodiment of the present application may be applied to the storage system shown in fig. 5, where the storage system may be a storage system composed of a solid state disk, or a storage system composed of other types of storage media. As shown in fig. 5, the data storage system includes a controller (abbreviated as "master") 501 and a plurality of hard disks 505, wherein the master 501 includes: the processor 502 and, optionally, the controller 501 further includes a host interface 504 and n (n > 0) channel controllers 503.

The master 501 is used to issue executable commands to a plurality of hard disks 505, so as to read or write data on the hard disks 505.

The host interface 504 is used for communicating with a host, receiving a command request from the host, and forwarding the command request to the processor 502, where the host is not limited to any device such as a server, a personal computer, or an array controller

Specifically, the processor 502 is configured to execute the data reconstruction method in the storage system provided in the embodiment of the present application, for example, the processor 502 is configured to determine a faulty block group in a plurality of block groups of the storage system, and preferentially reconstruct data in a partially faulty block in a first faulty block group based on an erasure code mechanism between the plurality of blocks in the block group, where the processor 502 includes one or more CPUs. The CPU may be a single-core CPU (Single-CPU) or a multi-core CPU (Multi-CPU).

The channel controller 503 is used to carry executable commands issued by the processor 502 to the hard disk 505.

Optionally, the data storage system further comprises a bus 506, and the processor 502, the channel controller 503, the host interface 504, and the hard disk 505 are typically connected to each other through the bus 506, or in other manners.

When the data storage system receives a data writing request of a host, a host interface 504 in the master control 501 forwards the data writing request to a processor 502 in the master control 501, the processor 502 divides the data to be written into M parts of sub-data, and obtains X parts of erasure codes according to the M parts of sub-data, wherein the sum of M and X is less than or equal to the number n of hard disks in the storage system; the processor 502 sends the write request of the (M + X) pieces of sub data to the (M + X) hard disks 505 of the n hard disks 505 through the channel controller 503 to write the data to be written to the hard disks 505.

Optionally, the apparatus for performing the data reconstruction method in the storage system according to the embodiment of the present application may be a processor in the controller in the storage system shown in fig. 5, as shown in fig. 6, and the method may include S610-S620.

S610, determining a fault block group in a plurality of block groups of the storage system.

In the embodiment of the present application, each block group in the plurality of block groups includes a plurality of blocks, the plurality of blocks are respectively located on different hard disks in the storage system, and an EC mechanism is provided between the plurality of blocks in each block group.

Illustratively, the storage system shown in fig. 3 includes a first block group and a second block group, where the first block group includes 5 blocks, which are, respectively, block 3-5, block 4-6, block 5-2, block 7-4, and block 8-1; the second block group also includes 5 blocks, block 3-3, block 4-2, block 5-1, block 6-4 and block 7-3.

Optionally, the EC mechanism may be an RAID mechanism or other erasure correcting code mechanisms, and the embodiment of the present application is not limited in particular.

Optionally, the hard disk may be a solid state hard disk, a mechanical hard disk, or other storage media, and a suitable hard disk is specifically selected according to actual requirements, which is not limited in this embodiment of the application.

With reference to fig. 6 and as shown in fig. 7, the method for determining a faulty block group in multiple block groups in a storage system specifically includes: S710-S720.

And S710, determining a fault hard disk in the storage system.

It should be understood that the blocks in the above-mentioned failed hard disk are failed blocks, that is, data in all blocks in the failed hard disk has been lost.

Optionally, the method for determining a failed hard disk in the storage system includes: determining a failed hard disk in the storage system according to the state information of each hard disk in the storage system, optionally, the state information of the hard disk includes: at least one of a temperature of the hard disk, a read performance of the hard disk, and a write performance of the hard disk. For example, when the temperature of the hard disk is higher than the temperature threshold of the hard disk, the hard disk is determined as a failed hard disk.

Optionally, other methods may also be used to determine the failed hard disk in the storage system, and the method for determining the failed hard disk in the storage system is not limited in the embodiment of the present application.

S720, determining a fault block group in the plurality of block groups in the storage system according to the fault hard disk.

It should be understood that the above-mentioned faulty block group refers to a block group containing a faulty block, that is, as long as a faulty block exists in a block group, the block group is a faulty block group.

The method for determining a faulty block group among a plurality of block groups in the storage system includes: determining a block group associated with a failed hard disk in a plurality of block groups in the storage system as a failed block group, namely: and determining the block group corresponding to the fault block in the hard disk containing the fault as the fault block group.

For example, in the storage system shown in fig. 2, if the hard disk 1 is a failed hard disk, the failed block of the hard disk 1 for storing data includes: fault blocks 1-2 and fault blocks 1-3; if the block group to which the fault block 1-2 belongs is the second block group, determining that the second block group is the fault block group; similarly, if the block group to which the fault block 1-3 belongs is the first block group, it is determined that the first block group is also the fault block group.

In this embodiment of the application, the number of failed hard disks in the storage system may be multiple, for example, the storage system includes a first failed block group and a second failed block group, where the number of failed blocks in the first failed block group is C1, the number of failed blocks in the second failed block group is C2, and C1> C2, where C1 and C2 are both positive integers, and C1 is less than or equal to the number of blocks that can be corrected by the EC mechanism corresponding to the failed block group. C1 is more than or equal to 2, namely: the failure block in the first failure block group corresponds to at least two failure hard disks in the storage system.

Illustratively, taking a storage system corresponding to RAID6 as an example, there are two failed blocks in the maximum allowable block group in the storage system, in this case, the number of the failed blocks in the first failed block group is 2, and the number of the failed blocks in the second failed block group is 1.

S620, data in k fault blocks in the first fault block group are reconstructed preferentially.

Where k is the difference between the number of failed blocks C1 in the first failed block group and the number of failed blocks C2 in the second failed block group, i.e., k = C1-C2.

In this embodiment of the present application, which faulty block group to reconstruct data preferentially may be determined according to the number of faulty blocks included in the faulty block group, and specifically, the data in the faulty block group with the largest number of faulty blocks in the faulty block group is reconstructed preferentially.

For example, it is assumed that the value of C1 is 2 and the value of C2 is 1; determining that the value of k is 1 according to the difference value of C1 and C2; the data reconstruction means in the storage system preferentially reconstructs data in 1 defective block in the first defective block group.

It should be noted that the k fault blocks are any k fault blocks in C1 fault blocks in the first fault block group, and in the process of reconstructing data in the k fault blocks, data reconstruction is performed on the k fault blocks one by one, and the order of reconstructing the k fault blocks is not limited in the embodiment of the present application.

With reference to fig. 6 or fig. 7, as shown in fig. 8, the method for preferentially reconstructing data in k faulty blocks in the first faulty block group specifically includes: S810-S830.

And S810, reconstructing data in k fault blocks in the first fault block group according to the data in the healthy blocks in the first fault block group based on an erasure code mechanism.

For the above method for reconstructing data in k faulty blocks in a first faulty block group according to data in healthy blocks in the first faulty block group based on an erasure code mechanism, reference may be made to the relevant description of the EC checking mechanism in the above embodiment, and details are not described here again.

When the first faulty block group includes a plurality of block groups (that is, the number of faulty blocks in the faulty block groups in the storage system is equal, and the number of faulty blocks in any faulty block group in the faulty block groups is greater than the number of faulty blocks in the second faulty block group), and k >1, the process of reconstructing data in k faulty blocks in the first faulty block group is as follows: and reconstructing data in one fault block in all the first fault block groups respectively, and then reconstructing data in another fault block in all the first fault block groups respectively, so as to complete reconstruction of data in k fault blocks in the first fault block groups.

For example, suppose that there may be at most 3 faulty blocks in a block group in the storage system a, suppose there are 3 faulty blocks in the faulty block group B and the faulty block group C in the storage system a, respectively, and there are 1 faulty block in the faulty block group E, that is: the fault block group B and the fault block group C are both a first fault block group, and the fault block group E is a second fault block group. The difference between the number of faulty blocks 3 in the first faulty block group and the number of faulty blocks 1 in the second faulty block group is 2, i.e. the value of k is 2. At this moment, when the storage system a preferentially reconstructs data in 2 fault blocks in the first fault block group, first, data in one fault block in the fault block group B and the fault block group C are reconstructed, optionally, data in any fault block in 3 fault blocks in the fault block group B may be reconstructed first, and then data in any fault block in 3 fault blocks in the fault block group C may be reconstructed, or data in any fault block in 3 fault blocks in the fault block group C may be reconstructed first, and then data in any fault block in 3 fault blocks in the fault block group B may be reconstructed; then, the data in another fault block in the remaining fault blocks in the fault block group B and the fault block group C are reconstructed respectively, optionally, the data in any fault block in 2 fault blocks in the fault block group B may be reconstructed first, then the data in any fault block in 2 fault blocks in the fault block group C may be reconstructed, or the data in any fault block in 2 fault blocks in the fault block group C may be reconstructed first, and then the data in any fault block in 2 fault blocks in the fault block group B may be reconstructed.

S820, storing the data in the k fault blocks in the reconstructed first fault block group into the k hot standby blocks.

It should be noted that the hard disk where the hot standby block is located is a hot standby disk, and the k hot standby blocks correspond to k different hot standby disks.

S830, replacing k fault blocks in the first fault block group with k hot standby blocks.

In the embodiment of the application, k fault blocks in the first fault block group are deleted, and k hot spare blocks corresponding to the k fault blocks are added into the first fault block group.

Illustratively, in the storage system shown in fig. 2, the first failure block group includes: blocks 1-3, blocks 2-2, blocks 3-5, blocks 4-6, and blocks 5-2, wherein blocks 1-3 and blocks 2-2 are fault blocks; the second failure block group includes: blocks 1-2, 3-3, 4-2, 5-1, and 6-4, where blocks 1-2 are faulty blocks. Calculating to obtain the value of k as 1 according to the number of the fault blocks in the first fault block group and the second fault block group; as shown in FIG. 7, the storage system preferentially reconstructs the data in 1 failed block in the first set of failed blocks (i.e., reconstructs the data in blocks 1-3); and then writing the reconstructed data into the hot standby block 7-3, and replacing the block 1-3 in the first failure block group with the hot standby block 7-3, so that the first failure block group comprises: block 2-2, block 3-5, block 4-6, block 5-2, and hot spare block 7-3.

It can be understood that, after the above reconstruction of the data in the k failed blocks in the first failed block group, the number of the failed blocks in the first failed block group is the same as that in the second failed block group, the number of the failed blocks in the first failed block group is C2, and the number of the failed blocks in the second failed block group is also C2.

In the data reconstruction method in the storage system provided in the embodiment of the present application, the apparatus for data reconstruction determines a faulty block group in a plurality of block groups of the storage system, where the faulty block group includes a first faulty block group and a second faulty block group, the number C1 of faulty blocks in the first faulty block group is greater than the number C2 of faulty blocks in the second faulty block group, and C1 is less than or equal to the number of blocks that can be erased by an erasure coding mechanism in the storage system; then, data in k faulty blocks in the first faulty block group is preferentially reconstructed, where k is the difference between C1 and C2. In this embodiment of the present application, when a hard disk failure occurs in a storage system, a portion of data of a failed block group (for example, the first failed block group) with the most failed blocks in the storage system is preferentially reconstructed, so that it is possible to avoid that a hard disk in which other blocks in the first failed block group are located fails to cause data loss of the first failed block group.

Optionally, with reference to fig. 8 and as shown in fig. 9, after the data in the k failed blocks in the first failed block group is successfully reconstructed, the method for reconstructing data in a storage system provided in the embodiment of the present application further includes: s910 to S930.

S910, judging whether a new fault hard disk exists in the storage system.

It should be understood that the new failed hard disk refers to a failed hard disk in the storage system excluding the hard disk corresponding to the failed block in the first failed block group and the hard disk corresponding to the failed block in the second failed block group.

The method for judging whether the storage system has a new fault hard disk comprises the following steps: and determining whether the other hard disks have faults according to the state information of the other hard disks in the storage system, wherein the other hard disks are hard disks in the storage system except for the fault hard disks associated with the first fault block group and the second fault block group. The failed hard disk associated with the first failed block group and the second failed block group refers to the failed hard disk in which all the failed blocks in the first failed block group and the second failed block group are located.

When there is a new failed hard disk in the storage system, the data reconstruction apparatus in the storage system performs S920.

S920, re-determining the fault block group in the plurality of block groups in the storage system, determining the fault block group with the most fault blocks in the fault block group, and preferentially reconstructing data in part of fault blocks in the fault block group with the most fault blocks based on an erasure code mechanism.

It should be understood that the partial faulty blocks in the faulty block group with the most faulty blocks are specifically determined according to the number of faulty blocks in each faulty block group.

Assuming that there are p new failed hard disks in the storage system, when p is less than or equal to k, if the p new failed hard disks include blocks in the first failed block group, at this time, since the number of failed blocks in the first failed block group is less than or equal to the number of blocks that can be erased by the erasure correction mechanism, the storage system can still reconstruct data in the failed blocks in the first failed block group based on the erasure correction code mechanism.

It should be noted that after the data in the k failed blocks in the first failed block group is preferentially reconstructed, the p hard disks in the storage system again fail, and the total number of failed hard disks in the current storage system is M, where M is greater than the number of blocks that can be erased by the erasure correcting code mechanism.

Illustratively, based on the above example, in the storage system shown in fig. 2, the hard disk 1 and the hard disk 2 are failure hard disks, and the first failure block group is composed of blocks 1-3, 2-2, 3-5, 4-6 and 5-2, where blocks 1-3 and 2-2 are failure blocks; the second faulty block group is composed of blocks 1-2, 3-3, 4-2, 5-1 and 6-4, where blocks 1-2 are faulty blocks. Determining that the value of k is 1 according to the number of the fault blocks in the first fault block group and the number of the fault blocks in the second fault block group, reconstructing data on the blocks 1-3 in the first fault block group according to data in healthy blocks in the first fault block group based on an erasure correcting code mechanism, and writing the reconstructed data into hot spare blocks 7-4, at this moment, the first fault block group is composed of blocks 2-2, 3-5, 4-6, 5-2 and 7-4, wherein the blocks 2-2 are fault blocks, as shown in fig. 10, at this moment, the hard disk 5 fails to become a fault hard disk, that is, there are 3 fault hard disks in the storage system, and the maximum number of the blocks capable of being erased by RAID6 is 2, that is, the number of the fault hard disks in the storage system is greater than the maximum number of the blocks capable of being erased by RAID 6. Currently, the block 2-2 and the block 5-2 in the first faulty block group are faulty blocks, and data in the faulty blocks in the first faulty block group are reconstructed based on an erasure code mechanism.

If p new failed hard disks exist in the storage system, when p is greater than k and the first block group comprises a failed block on at least one failed hard disk in the p failed hard disks, judging whether the number of the failed blocks in the first failed block group is greater than the number of blocks which can be erased by the erasure code mechanism; when the number of the failed blocks in the first failed block group is greater than the number of blocks that can be erased by the erasure correcting code mechanism, the storage system cannot reconstruct the failed blocks in the first block group based on the erasure correcting mechanism. When the number of the failed blocks in the first failed block group is less than or equal to the number of the blocks that can be erased by the erasure correcting code mechanism, the storage system reconstructs the failed blocks in the first block group based on the erasure correcting mechanism.

When there is no new failed hard disk in the above storage system, the data reconstruction apparatus in the storage system performs S930.

And S930, reconstructing data in the C2 fault blocks in the first fault block group and the second fault block group respectively.

It should be understood that after preferentially reconstructing data in k faulty blocks in the first faulty block group based on S820, the number of faulty blocks in the first faulty block group is equal to the number of faulty blocks in the second faulty block group.

It should be noted that, the method for reconstructing C2 faulty blocks in the first faulty block group and the second faulty block group respectively is similar to the method for preferentially reconstructing data in k faulty blocks in the first faulty block group in S620, and for specific reference, detailed description of S620 is given, and details are not repeated here.

In one implementation, a data reconstruction device in a storage system periodically detects a failed hard disk in the storage system. If it is detected that a new failed hard disk is generated in the storage system, the data reconstruction apparatus in the storage system stops executing S930 and executes 920 while reconstructing data in C2 failed blocks in the first failed block group and the second failed block group, respectively, after preferentially reconstructing k failed blocks in the first failed block group (i.e., while executing S930).

For example, as shown in fig. 11, at time T1, a hard disk 1 in a storage system fails to become a failed hard disk 1, and the data reconstruction device in the storage system starts to execute task 1, where task 1 reconstructs data on the failed hard disk 1; when the hard disk 2 in the storage system becomes the failed hard disk 2 at time T2, the data reconstruction device in the storage system stops (or suspends) execution of task 1 and starts execution of task 2, where task 2 is the step S920; after the data reconstruction device in the storage system finishes executing the task 2 at the time of T3, starting to execute the task 1; after the data reconstruction device in the storage system completes the task 1 at the time of T4 (that is, after all data in the failed hard disk 1 is reconstructed), the data reconstruction device in the storage system starts to execute the task 3, wherein the task 3 is to reconstruct the data on the failed hard disk 2 until the data reconstruction device in the storage system completes the task at the time of T5.

Accordingly, an embodiment of the present application provides a data reconstruction apparatus (hereinafter, simply referred to as a data reconstruction apparatus) in a storage system, where the data reconstruction apparatus is configured to execute steps of the data reconstruction method in the data storage, and the data reconstruction apparatus may be divided into functional modules according to the above method example, for example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The division of the modules in the embodiment of the present application is schematic, and is only a logic function division, and there may be another division manner in actual implementation.

Fig. 12 is a schematic diagram showing a possible structure of the data reconstruction device according to the above embodiment, in a case where each functional module is divided for each function. As shown in fig. 12, the data reconstruction apparatus includes: a determination module 1210 and a reconstruction module 1220.

The determining module 1210 is configured to determine a faulty block group in the plurality of block groups of the storage system, for example, to execute step S610 in the above method embodiment.

The reconstructing module 1220 is configured to preferentially reconstruct the data in the k failed blocks in the first failed block group, for example, execute step S620 in the foregoing method embodiment.

Optionally, the determining module 1210 is specifically configured to determine a failed hard disk in the storage system, and determine a failed block group in the plurality of block groups in the storage system according to the failed hard disk, for example, execute steps S710 to S720 in the foregoing method embodiment.

Optionally, an embodiment of the present application provides a data reconstruction apparatus, further including: a storage module 1230 and an update module 1240;

the reconstructing module 1220 is specifically configured to reconstruct data in k failed blocks in the first failed block group according to data in healthy blocks in the first failed block group based on an erasure code mechanism, for example, execute step S810 in the foregoing method embodiment.

The storing module 1230 is configured to store the data in the k failed blocks in the reconstructed first failed block group into the k hot spare blocks, for example, to execute step S820 in the foregoing method embodiment.

The updating module 1240 is configured to replace the k failed blocks in the first failed block group with the k hot spare blocks, for example, to execute step S830 in the foregoing method embodiment.

Optionally, the reconstructing module 1220 is further configured to reconstruct data in the failed block in the first failed block group based on an erasure code mechanism when k hard disk failures occur again in the storage system, for example, to execute step S920 in the foregoing method embodiment.

Each module of the data reconstruction apparatus may also be configured to execute other actions in the method embodiment, and all relevant contents of each step related to the method embodiment may be referred to the functional description of the corresponding functional module, which is not described herein again.

In the case of using an integrated unit, a schematic structural diagram of the data reconstruction apparatus provided in the embodiment of the present application is shown in fig. 13. In fig. 13, the data reconstruction apparatus includes: a processing module 1301 and a communication module 1302. The processing module 1301 is used to control and manage the actions of the data reconstruction apparatus, such as performing the steps performed by the determining module 1210, the reconstructing module 1220, and the updating module 1240, and/or other processes for performing the techniques described herein. The communication module 1302 is configured to support interaction between the data reconstruction apparatus and other devices, and as shown in fig. 13, the data reconstruction apparatus may further include a storage module 1303, where the storage module 1303 is configured to execute the steps executed by the storage module 1230.

The processing module 1301 may be a processor or a controller, such as the processor 502 in the controller 501 in fig. 5. The communication module 1302 may be a transceiver, an RF circuit or a communication interface, etc., such as the host interface 504 in the controller 501 and/or the channel controller 503 in the controller 501 in fig. 5. The storage module 1303 may be a memory, such as the hard disk 505 in fig. 5.

In the above embodiments, all or part of the implementation may be realized by software, hardware, firmware, or any combination thereof. When implemented using a software program, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The computer instructions, when loaded and executed on a computer, result in all, or in part, the processes or functions described in the embodiments of the application. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored on a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device including one or more available media integrated servers, data centers, and the like. The usable medium may be a magnetic medium (e.g., a floppy disk, a magnetic tape), an optical medium (e.g., a Digital Video Disk (DVD)), or a semiconductor medium (e.g., a Solid State Drive (SSD)), among others.

Through the description of the foregoing embodiments, it will be clear to those skilled in the art that, for convenience and simplicity of description, only the division of the functional modules is illustrated, and in practical applications, the above function distribution may be completed by different functional modules as needed, that is, the internal structure of the apparatus may be divided into different functional modules to complete all or part of the above described functions. For the specific working processes of the system, the apparatus and the unit described above, reference may be made to the corresponding processes in the foregoing method embodiments, and details are not described here again.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules or units is only one type of logical functional division, and other divisions may be realized in practice, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) or a processor to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media that can store program code, such as flash memory, removable hard drive, read-only memory, random-access memory, magnetic or optical disk, etc.

The above description is only an embodiment of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions within the technical scope of the present disclosure should be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for reconstructing data in a storage system, comprising:

determining a failed block group of a plurality of block groups of the storage system; the block group comprises a plurality of blocks, the blocks respectively correspond to different hard disks in the storage system, and erasure code mechanisms are arranged among the blocks in the block group; the fault block group comprises a first fault block group and a second fault block group; the number of the fault blocks C1 in the first fault block group is greater than the number of the fault blocks C2 in the second fault block group; wherein C1 and C2 are both positive integers, and C1 is less than or equal to the number of blocks that can be erased by the erasure code mechanism;

and preferentially reconstructing data in k fault blocks in the first fault block group, wherein k is the difference value between C1 and C2.

2. The method of claim 1, wherein determining a faulty block group of the plurality of block groups of the storage system comprises:

determining a failed hard disk in the storage system;

and determining a fault block group in the plurality of block groups according to the fault hard disk.

3. The method according to claim 1 or 2, wherein the preferentially reconstructing data in k faulty blocks in the first faulty block group specifically comprises:

reconstructing data in k fault blocks in the first fault block group according to data in healthy blocks in the first fault block group based on the erasure code mechanism;

storing the reconstructed data in the k fault blocks in the first fault block group into k hot standby blocks;

and replacing k fault blocks in the first fault block group with the k hot standby blocks.

4. The method of claim 3, wherein the hard disk on which the hot standby block is located is a hot standby disk.

5. The method of any of claims 1-4, wherein a failed block in the first failed block group corresponds to at least two failed hard disks in the storage system.

6. The method of claim 5, wherein when k hard disk failures occur again in the storage system after preferentially reconstructing data in k failed blocks in the first failed block group, reconstructing data in failed blocks in the first failed block group based on the erasure coding mechanism.

7. The method according to claim 6, wherein after preferentially reconstructing data in k failed blocks in the first failed block group, k hard disk failures occur again in the storage system, the total number of currently failed hard disks in the storage system is M, and the data in the failed blocks in the first failed block group is reconstructed based on the erasure coding mechanism, where M is greater than the number of blocks that can be erased by the erasure coding mechanism.

8. The method according to any of claims 1-7, wherein the erasure coding mechanism is a redundant array of independent disks mechanism.

9. The method of any one of claims 1 to 8, wherein the hard disk is a solid state disk.

10. The method of any of claims 1-8, wherein the hard disk is a mechanical hard disk.

11. An apparatus for reconstructing data in a storage system, comprising: a determination module and a reconstruction module;

the determining module is configured to determine a faulty block group of a plurality of block groups of the storage system; the block group comprises a plurality of blocks, the blocks respectively correspond to different hard disks in the storage system, and erasure code mechanisms are arranged among the blocks in the block group; the fault block group comprises a first fault block group and a second fault block group; the number C1 of the faulty blocks in the first faulty block group is greater than the number C2 of the faulty blocks in the second faulty block group; wherein, C1 and C2 are both positive integers, and C1 is less than or equal to the number of blocks that can be erased by the erasure correcting code mechanism;

the reconstruction module is configured to preferentially reconstruct data in k faulty blocks in the first faulty block group, where k is a difference between C1 and C2.

12. The data reconstruction apparatus in a storage system according to claim 11,

the determining module is specifically configured to determine a failed hard disk in the storage system; and determining a fault block group in the plurality of block groups according to the fault hard disk.

13. The data reconstruction apparatus in a storage system according to claim 10 or 11, further comprising: the device comprises a storage module and an updating module;

the reconstruction module is specifically configured to reconstruct, based on the erasure code mechanism and according to data in healthy blocks in the first faulty block group, data in k faulty blocks in the first faulty block group;

the storage module is configured to store data in k fault blocks in the reconstructed first fault block group to k hot standby blocks;

the updating module is configured to replace the k failed blocks in the first failed block group with the k hot spare blocks.

14. The data reconstruction apparatus in a storage system according to claim 13,

the hard disk where the hot standby block is located is a hot standby disk.

15. The data reconstruction apparatus in a storage system according to any one of claims 10 to 14,

and the fault block in the first fault block group corresponds to at least two fault hard disks in the storage system.

16. The data reconstruction apparatus in a storage system according to claim 15,

and the reconstruction module is used for reconstructing data in the fault block in the first fault block group based on the erasure code mechanism when k hard disk faults occur again in the storage system.

17. The data reconstruction device in a storage system according to claim 16,

after preferentially reconstructing data in k fault blocks in the first fault block group, k hard disk faults occur in the storage system again, the total number of current fault hard disks in the storage system is M, and the data in the fault blocks in the first fault block group are reconstructed based on the erasure code mechanism, wherein M is greater than the number of blocks which can be erased by the erasure code mechanism.

18. The data reconstruction apparatus in a storage system according to any one of claims 10 to 17,

the erasure code mechanism is an independent hard disk redundant array mechanism.

19. The data reconstruction apparatus in a storage system according to any one of claims 10 to 18,

the hard disk is a solid state hard disk.

20. The data reconstruction apparatus in a storage system according to any one of claims 10 to 18,

the hard disk is a mechanical hard disk.

21. A storage system comprising a controller and a hard disk, the controller comprising a processor configured to:

determining a failed block group of a plurality of block groups of the storage system; the block group comprises a plurality of blocks, the blocks respectively correspond to different hard disks in the hard disks, and erasure code mechanisms are arranged among the blocks in the block group; the fault block group comprises a first fault block group and a second fault block group; the number of the fault blocks C1 in the first fault block group is greater than the number of the fault blocks C2 in the second fault block group; wherein, C1 and C2 are both positive integers, and C1 is less than or equal to the number of blocks that can be erased by the erasure correcting code mechanism;

22. The memory system of claim 21, wherein the processor is specifically configured to:

determining a failed hard disk in the storage system;

23. The memory system according to claim 21 or 22, wherein the processor is specifically configured to:

storing the data in k fault blocks in the reconstructed first fault block group into k hot standby blocks;

24. The storage system according to claim 23, wherein the hard disk on which the hot standby block is located is a hot standby disk.

25. The storage system according to any of claims 21-24, wherein a failed block in the first failed block group corresponds to at least two failed hard disks in the storage system.

26. The storage system according to claim 25, wherein the processor is further configured to reconstruct data in a failed block of the first failed block group based on the erasure coding mechanism when k hard disk failures occur again in the storage system after preferentially reconstructing data in k failed blocks of the first failed block group.

27. The storage system according to claim 26, wherein the processor is further configured to, after preferentially reconstructing data in k failed blocks in the first failed block group, re-occur k hard disk failures in the storage system, where a total number of currently failed hard disks in the storage system is M, and reconstruct data in the failed blocks in the first failed block group based on the erasure coding mechanism, where M is greater than a number of blocks that can be erased by the erasure coding mechanism.

28. The storage system according to any one of claims 21 to 27, wherein the erasure coding mechanism is a redundant array of independent hard disks mechanism.

29. The storage system according to any one of claims 21 to 28, wherein the hard disk is a solid state disk.

30. The storage system according to any of claims 21 to 28, wherein said hard disk is a mechanical hard disk.

31. A computer storage medium comprising computer instructions that, when executed on a computing device, cause the computing device to perform the method of any of claims 1 to 10.