CN116521433A - Data protection method, system, equipment and storage medium - Google Patents
Data protection method, system, equipment and storage medium Download PDFInfo
- Publication number
- CN116521433A CN116521433A CN202310416038.6A CN202310416038A CN116521433A CN 116521433 A CN116521433 A CN 116521433A CN 202310416038 A CN202310416038 A CN 202310416038A CN 116521433 A CN116521433 A CN 116521433A
- Authority
- CN
- China
- Prior art keywords
- data
- original data
- error correction
- decoding
- original
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 75
- 238000012937 correction Methods 0.000 claims abstract description 90
- 230000008569 process Effects 0.000 claims abstract description 15
- 238000004590 computer program Methods 0.000 claims description 14
- 230000006870 function Effects 0.000 description 10
- 230000001360 synchronised effect Effects 0.000 description 9
- 230000005291 magnetic effect Effects 0.000 description 7
- 238000011084 recovery Methods 0.000 description 6
- 238000012545 processing Methods 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000008439 repair process Effects 0.000 description 4
- 230000003068 static effect Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000005294 ferromagnetic effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 238000012552 review Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
- G06F11/1004—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's to protect a block of data words, e.g. CRC or checksum
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Signal Processing For Digital Recording And Reproducing (AREA)
Abstract
The invention provides a data protection method, a system, equipment and a storage medium, belonging to the technical field of distributed storage, wherein the method comprises the following steps: when the original data is written into the storage device, encoding the original data and generating at least one error correction copy; when the original data written into the storage device is read, the original data positioned in the storage device is checked through the check block, and if the original data positioned in the storage device is damaged, an error correction copy is read and decoded to obtain the original data. Based on the method, a data protection system, a device and a storage medium are also provided. The invention carries out HDB3 coding on the data before the data is subjected to disc-falling storage, stores the coded data, and can recover the original data by decoding when the coded data has a self-error correction function and is damaged, thereby realizing the correction and protection of the data. The encoded data itself is damaged and can be repaired by itself in the decoding process, thereby increasing the reliability of the data.
Description
Technical Field
The present invention relates to the field of distributed storage technologies, and in particular, to a data protection method, system, device, and storage medium.
Background
In the digital information age, various electronic data have been throughout society, and the preservation of mass data generated by various processes and records is an important problem, and storage and protection are required by means of a storage system. Distributed storage is a data storage technology, uses disk space on each machine in an enterprise through a network, and forms a virtual storage device by using the scattered storage resources, and the scattered storage of data is a critical point in a distributed storage system at each corner of the enterprise, so that data protection is particularly important.
The existing protection method of the data in the distributed storage system mainly protects the data through multiple copies of backup. If the three copy storage pool is to store three copies of data under different fault domains, and when writing data, each 4k data is calculated to obtain a check block for judging the accuracy of the stored data. If the data is damaged, the data with errors are synchronized from other fault domains, so that the data are corrected and protected. Most of error correction methods of a storage system are data protection through copies, and a consistency check algorithm is also a common method for data check. The consistency check only scans the metadata information, so that the speed is relatively high; the depth consistency check not only scans the metadata, but also scans the stored data, which is relatively slow. It can be seen that in order to protect the accuracy of data, a distributed storage system consumes a lot of time to check and verify the stored data to protect the correctness and reliability of the data. However, the large copy data not only occupies a large amount of space, but also occupies a large memory in the process of copy reading operation, and has a long operation time and a large influence on front-end business. And the existing checking method has another disadvantage, and in a special case, when all copies are damaged, the data is lost.
Disclosure of Invention
In order to solve the technical problems, the invention provides a data protection method, a system, a device and a storage medium, wherein an error correction copy is generated through an HDB3 coding algorithm, the data is automatically repaired through a decoding algorithm when the data is damaged or misoperation occurs, and the coding and decoding flow is simple and easy to realize
In order to achieve the above purpose, the present invention adopts the following technical scheme:
a data protection method, comprising the steps of:
when the original data is written into the storage device, encoding the original data and generating at least one error correction copy;
when the original data written into the storage device is read, the original data positioned in the storage device is checked through the check block, and if the original data positioned in the storage device is damaged, an error correction copy is read and decoded to obtain the original data.
Further, before writing the original data into the storage device, the method further comprises:
dividing the original data into data blocks with preset sizes, and calculating each data block to obtain a check block; the check blocks are stored as metadata.
Further, the process of encoding the original data and generating at least one error correction copy includes: the original data is encoded with an error correction code, and at least one error correction copy is generated based on the encoded data.
Further, the error correction code is encoded with HDB 3.
Further, the process of reading the error correction copy and decoding the error correction copy to obtain the original data includes: and decoding the error correction copy by adopting an error correction code decoding algorithm to obtain original data.
Further, the method for judging that the original data positioned in the storage device is damaged comprises the following steps: obtaining coded data through error correction copy reading operation, judging whether the coded data has alternate symbol polarity errors, if so, recovering through error correction code coding, and obtaining original data through error correction code decoding; if polarity alternation errors do not occur, the original data are obtained through direct decoding.
Further, after the decoding to obtain the original data, the method further includes: and reading the check block from the metadata base, and checking the decoded original data again.
The invention also provides a data protection system, which comprises an encoding module and a decoding module;
the encoding module is used for encoding the original data and generating at least one error correction copy when the original data is written into the storage device;
and the decoding module is used for firstly checking the original data positioned in the storage equipment through the checking block when the original data written in the storage equipment is read, and if the original data positioned in the storage equipment is damaged, reading error correction copies and decoding the error correction copies to obtain the original data.
The invention also proposes a device comprising:
a memory for storing a computer program;
and a processor for implementing the method steps when executing the computer program.
The invention also proposes a readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method steps.
The effects provided in the summary of the invention are merely effects of embodiments, not all effects of the invention, and one of the above technical solutions has the following advantages or beneficial effects:
the invention provides a data protection method, a system, equipment and a storage medium, wherein the method comprises the following steps: when the original data is written into the storage device, encoding the original data and generating at least one error correction copy; when the original data written into the storage device is read, the original data positioned in the storage device is checked through the check block, and if the original data positioned in the storage device is damaged, an error correction copy is read and decoded to obtain the original data. Based on a data protection method, a data protection system, a device and a storage medium are also provided. The invention carries out HDB3 coding on the data before the data is subjected to disc-falling storage, stores the coded data, and can recover the original data by decoding when the coded data has a self-error correction function and is damaged, thereby realizing the correction and protection of the data. The HDB3 coding algorithm has simple coding and decoding, and compared with the original data, the coded data has no check bit, and the storage space is not needed to be used too much. In a special case, when all data occur, original data can be obtained through decoding, so that the safety of the data is effectively improved.
In the invention, the original data is encoded by the HDB3 encoding and decoding algorithm and then stored, and the encoded data has no added check bit redundancy and has the same size as the original data, thus not increasing the space occupation. The HDB3 encoding algorithm may encode any fast-size data without limitation of the data size. The encoded data itself is damaged, and the encoded data can be self-repaired in the decoding process, so that the reliability of the data is improved.
The HDB3 coding and decoding algorithm exists independently, the use of other functions of the distributed storage system is not affected, the operation is simple, the complexity is low, and unnecessary burden on the system is not caused.
Drawings
Fig. 1 is a flowchart of a data protection method according to embodiment 1 of the present invention;
FIG. 2 is a flowchart of checking original data by a check block according to embodiment 1 of the present invention;
fig. 3 is a schematic diagram of a data protection system according to embodiment 2 of the present invention;
fig. 4 is a schematic diagram of a data protection device according to embodiment 3 of the present invention.
Detailed Description
In order to clearly illustrate the technical features of the present solution, the present invention will be described in detail below with reference to the following detailed description and the accompanying drawings. The following disclosure provides many different embodiments, or examples, for implementing different structures of the invention. In order to simplify the present disclosure, components and arrangements of specific examples are described below. Furthermore, the present invention may repeat reference numerals and/or letters in the various examples. This repetition is for the purpose of simplicity and clarity and does not in itself dictate a relationship between the various embodiments and/or configurations discussed. It should be noted that the components illustrated in the figures are not necessarily drawn to scale. Descriptions of well-known components and processing techniques and processes are omitted so as to not unnecessarily obscure the present invention.
Example 1
The embodiment 1 of the invention provides a data protection method, which is characterized in that original data of system writing operation is obtained, the original data is encoded through error correction code encoding, encoded data is obtained, and at least one error correction copy is generated according to the encoded data. When the system is found to have data damage or data loss through the reading operation, the decoding module is called through the reading operation to decode the error correction copy, so that the damaged data is repaired by the original data, and the original data can be obtained through decoding when the error correction copy is damaged, so that the safety of the data is improved.
HDB3 code: high Density Bipolar of Order 3code, three-order high-density bipolar code, i.e., error correction code.
Fig. 1 is a flowchart of a data protection method according to embodiment 1 of the present invention;
and acquiring a write request sent by the client, performing error correction code encoding on the original data after the cluster calls a write flow to acquire the original data, and generating at least one error correction copy according to the encoded data.
The method further comprises the following steps before the original data is written into the storage device:
dividing the original data into data blocks with preset sizes, and calculating each data block to obtain a check block; the check blocks are stored as metadata. In the distributed storage system, the data is subjected to disk-drop storage mainly through a writing flow. The write operation calculates a check block every 4k data before the original data is written into the disk, and the check block is stored as metadata. The check block is mainly used for checking disk data, and when data on the disk is damaged in transmission, the calculated check value is different from the check value stored in writing, and the data is considered to be damaged. After the check block is generated, the original data is subjected to HDB3 coding to generate an error correction copy. While the write operations have different block sizes, such as writing 4k, 8k, 64k, and 1M data blocks, and HDB3 encoding can be performed on 4k, 8k, 64k, and 1M data, respectively; the HDB3 encoding algorithm is not limited by the data size.
The error correction code in this application employs HDB3 codes.
The full name of AMI code is a mark-alternating inversion code, and the rule is that "1" in the message code is alternately transformed into "+1" and "—1", while "0" remains unchanged.
For example:
raw data 10 10 10 00 10 1 1 1
AMI code-1+1-0-1 0+0-1+1-1.
Whether or not a data error occurs is judged by the alternate occurrence of "+1" and "+1", but the AMI code also has a great disadvantage. When a long concatenated 0 string appears in the original data, if two error data "+1" "-1" appear in this concatenated "0", the AMI code cannot find error information.
The HDB 3code is a three-order high-density bipolar code, which is an improved generation of the AMI code, and the aim of the improvement is to overcome the defect of the AMI code in order to keep the merits of the AMI code, so that the number of the consecutive 0 codes is not more than three.
(1) When the number of continuous 0 is not more than 3, the rule is the same as AMI, namely 0 is unchanged, and 1 is changed into-1 and +1 alternately;
(2) When the number of continuous 0 exceeds 3, each 4 0 is defined as a subsection, B00V is defined, B can be-1, 0 and +1, and V can be-1 and +1;
(3) The specific value of V satisfies the following condition: v and the preceding adjacent non-0 symbols have the same polarity, and the polarity between adjacent V must alternate;
(4) B may be-1, 0, +1 so that V satisfies both requirements in (3);
for example:
raw data 10 00 0 10 00 0 1 10 00 00 00 0 1 1
AMI code-1 00 0 0+1 00 0 0-1+1 00 00 00 0 0-1+1
HDB3 code: -1 00 0-V+100+V-1+1-B0 0-V+B00+V-1+1
Wherein, V and B are converted from 0 in the original data, thus ensuring that the decoding algorithm is not influenced when the error occurs in the continuous 0 string to obtain the original data.
For example, in the case of a 4+2 erasure pool scenario, the stored data is divided into 4 blocks, and the encoding generates 2 parity blocks to be stored together, but recovery of the 4 block copies is affected by the parity copies. So that at most only two copies are allowed to go wrong, and once the range of the two copies is exceeded, data loss occurs.
Under the condition of the HDB3 error correction copy provided by the invention, the 2-block check block of the 4+2 erasure pool is subjected to HDB3 coding, so that at most, two copies and two check copies can be allowed to be in error, and the safe recovery of data can be ensured. Because the HDB3 decoding can recover the data with errors, the data safety is effectively improved.
When the original data written into the storage device is read, the original data positioned in the storage device is checked through the check block, and if the original data positioned in the storage device is damaged, an error correction copy is read and decoded to obtain the original data.
The HDB3 decoding algorithm first checks the read data and starts decoding if no sign polarity alternation error occurs. By the coding algorithm, it can be found that V always has the same polarity as the previous non-zero data, and then it can be judged that V symbol and the three symbols before it are necessarily consecutive "0" symbols, so as to recover four consecutive "0". The symbol B is changed to 0, and then all-1 is changed to +1, so that the original data can be obtained. Thus, the problem that the error data is difficult to find when the number of continuous '0's is large is avoided.
Due to the specificity of the encoded data, it can be determined whether the data is erroneous by the polarities of the symbol V and the symbol B. When the data between the symbol V and the symbol B shows 1, the data is damaged, and the damaged data is recovered by the encoded data. When the polarity of positive and negative "1" is alternately wrong, the position of "1" can be judged and recovered by the symbol V and the symbol B.
In the reading operation, judging whether the data is damaged or not through the check block, and if the data is damaged, obtaining the original data from the error correction copy through decoding and repairing the damaged data. Also, when the error correction copy is damaged, the original data can be recovered by decoding.
FIG. 2 is a flowchart of checking original data by a check block according to embodiment 1 of the present invention;
when in reading operation, firstly, original data positioned on a disk are read, and if the original data are not damaged after being judged by a check block, the original data are returned; when the original data is damaged and the error correction copy is required to be read, the encoded data is obtained through the error correction copy reading operation, whether the data has alternate symbol polarity errors or not is judged, if so, the original data is obtained through normal decoding after recovery through the HDB3 encoding property; if polarity alternation errors do not occur, the original data are obtained through direct normal decoding. And then reading the check block from the metadata base, checking the decoded original data again, and returning the data after no data damage. The decoding operation is connected with the reading operation and the data consistency check module, and when the data consistency check finds that the data is damaged, the error correction copy can be called to repair the data.
In the data protection method provided by the embodiment 1 of the invention, before the data is subjected to disc-falling storage, the data is subjected to HDB3 coding, the coded data is stored, and when the coded data has a self-error correction function and is damaged, the original data can be restored through decoding, so that the correction and protection of the data are realized. The HDB3 coding algorithm has simple coding and decoding, and compared with the original data, the coded data has no check bit, and the storage space is not needed to be used too much. In a special case, when all data occur, original data can be obtained through decoding, so that the safety of the data is effectively improved.
According to the data protection method provided by the embodiment 1 of the invention, the original data is encoded by the HDB3 encoding and decoding algorithm and then stored, the encoded data is not added with check bit redundancy, and the encoded data has the same size as the original data and does not increase space occupation. The HDB3 encoding algorithm may encode any fast-size data without limitation of the data size. The encoded data itself is damaged, and the encoded data can be self-repaired in the decoding process, so that the reliability of the data is improved.
According to the data protection method provided by the embodiment 1 of the invention, the adopted HDB3 encoding and decoding algorithm exists independently, the use of other functions of the distributed storage system is not affected, the operation is simple, the complexity is low, and unnecessary burden on the system is not caused.
Example 2
Based on the embodiment 1 of the present invention, a data protection method is provided, and the embodiment 2 of the present invention also provides a data protection system, as shown in fig. 3, which is a schematic diagram of the data protection system provided in the embodiment 2 of the present invention;
the system comprises an encoding module and a decoding module;
the encoding module is used for encoding the original data and generating at least one error correction copy when the original data is written into the storage device.
And the decoding module is used for firstly checking the original data positioned in the storage equipment through the checking block when the original data written in the storage equipment is read, and if the original data positioned in the storage equipment is damaged, reading error correction copies and decoding the error correction copies to obtain the original data.
The coding module comprises: the method further comprises the following steps before the original data is written into the storage device:
dividing the original data into data blocks with preset sizes, and calculating each data block to obtain a check block; the check blocks are stored as metadata. In the distributed storage system, the data is subjected to disk-drop storage mainly through a writing flow. The write operation calculates a check block every 4k data before the original data is written into the disk, and the check block is stored as metadata. The check block is mainly used for checking disk data, and when data on the disk is damaged in transmission, the calculated check value is different from the check value stored in writing, and the data is considered to be damaged. After the check block is generated, the original data is subjected to HDB3 coding to generate an error correction copy. While the write operations have different block sizes, such as writing 4k, 8k, 64k, and 1M data blocks, and HDB3 encoding can be performed on 4k, 8k, 64k, and 1M data, respectively; the HDB3 encoding algorithm is not limited by the data size.
In the decoding module, firstly, original data positioned on a magnetic disk is read during reading operation, and if the original data is not damaged after judging through a check block, the original data is returned; when the original data is damaged and the error correction copy is required to be read, the encoded data is obtained through the error correction copy reading operation, whether the data has alternate symbol polarity errors or not is judged, if so, the original data is obtained through normal decoding after recovery through the HDB3 encoding property; if polarity alternation errors do not occur, the original data are obtained through direct normal decoding. And then reading the check block from the metadata base, checking the decoded original data again, and returning the data after no data damage. The decoding operation is connected with the reading operation and the data consistency check module, and when the data consistency check finds that the data is damaged, the error correction copy can be called to repair the data.
In the data protection system provided by the embodiment 2 of the invention, before the data is subjected to disc-falling storage, the data is subjected to HDB3 coding, the coded data is stored, and when the coded data has a self-error correction function and is damaged, the original data can be restored through decoding, so that the correction and protection of the data are realized. The HDB3 coding algorithm has simple coding and decoding, and compared with the original data, the coded data has no check bit, and the storage space is not needed to be used too much. In a special case, when all data occur, original data can be obtained through decoding, so that the safety of the data is effectively improved.
According to the data protection system provided by the embodiment 2 of the invention, the original data is encoded by the HDB3 encoding algorithm and then stored, the encoded data is not added with check bit redundancy, and the encoded data has the same size as the original data and does not increase space occupation. The HDB3 encoding algorithm may encode any fast-size data without limitation of the data size. The encoded data itself is damaged, and the encoded data can be self-repaired in the decoding process, so that the reliability of the data is improved.
The data protection system provided by the embodiment 2 of the invention adopts the independent HDB3 coding and decoding algorithm, does not influence the use of other functions of the distributed storage system, and has simple operation, low complexity and no unnecessary burden on the system.
Example 3
The present invention also proposes a device, as shown in fig. 4, which is a schematic diagram of a data protection device according to embodiment 3 of the present invention, where the system includes:
a memory for storing a computer program;
the processor is used for realizing the following steps when executing the computer program:
fig. 1 is a flowchart of a data protection method according to embodiment 1 of the present invention;
and acquiring a write request sent by the client, performing error correction code encoding on the original data after the cluster calls a write flow to acquire the original data, and generating at least one error correction copy according to the encoded data.
The method further comprises the following steps before the original data is written into the storage device:
dividing the original data into data blocks with preset sizes, and calculating each data block to obtain a check block; the check blocks are stored as metadata. In the distributed storage system, the data is subjected to disk-drop storage mainly through a writing flow. The write operation calculates a check block every 4k data before the original data is written into the disk, and the check block is stored as metadata. The check block is mainly used for checking disk data, and when data on the disk is damaged in transmission, the calculated check value is different from the check value stored in writing, and the data is considered to be damaged. After the check block is generated, the original data is subjected to HDB3 coding to generate an error correction copy. While the write operations have different block sizes, such as writing 4k, 8k, 64k, and 1M data blocks, and HDB3 encoding can be performed on 4k, 8k, 64k, and 1M data, respectively; the HDB3 encoding algorithm is not limited by the data size.
When the original data written into the storage device is read, the original data positioned in the storage device is checked through the check block, and if the original data positioned in the storage device is damaged, an error correction copy is read and decoded to obtain the original data.
In the reading operation, judging whether the data is damaged or not through the check block, and if the data is damaged, obtaining the original data from the error correction copy through decoding and repairing the damaged data. Also, when the error correction copy is damaged, the original data can be recovered by decoding.
FIG. 2 is a flowchart of checking original data by a check block according to embodiment 1 of the present invention;
when in reading operation, firstly, original data positioned on a disk are read, and if the original data are not damaged after being judged by a check block, the original data are returned; when the original data is damaged and the error correction copy is required to be read, the encoded data is obtained through the error correction copy reading operation, whether the data has alternate symbol polarity errors or not is judged, if so, the original data is obtained through normal decoding after recovery through the HDB3 encoding property; if polarity alternation errors do not occur, the original data are obtained through direct normal decoding. And then reading the check block from the metadata base, checking the decoded original data again, and returning the data after no data damage. The decoding operation is connected with the reading operation and the data consistency check module, and when the data consistency check finds that the data is damaged, the error correction copy can be called to repair the data.
In the data protection device provided by the embodiment 3 of the invention, before the data is subjected to disc-falling storage, the data is subjected to HDB3 coding, the coded data is stored, and when the coded data has a self-error correction function and is damaged, the original data can be restored through decoding, so that the correction and protection of the data are realized. The HDB3 coding algorithm has simple coding and decoding, and compared with the original data, the coded data has no check bit, and the storage space is not needed to be used too much. In a special case, when all data occur, original data can be obtained through decoding, so that the safety of the data is effectively improved.
According to the data protection device provided by the embodiment 3 of the invention, the original data is encoded by the HDB3 encoding and decoding algorithm and then stored, the encoded data is not added with check bit redundancy, and the encoded data has the same size as the original data and does not increase space occupation. The HDB3 encoding algorithm may encode any fast-size data without limitation of the data size. The encoded data itself is damaged, and the encoded data can be self-repaired in the decoding process, so that the reliability of the data is improved.
The data protection device provided by the embodiment 3 of the invention adopts the independent HDB3 encoding and decoding algorithm, does not influence the use of other functions of the distributed storage system, and has simple operation, low complexity and no unnecessary burden on the system.
It is necessary to explain that: the technical scheme of the invention also provides electronic equipment, which comprises: a communication interface capable of information interaction with other devices such as a network device and the like; and the processor is connected with the communication interface to realize information interaction with other devices, and is used for executing a data protection method provided by one or more of the technical schemes when running the computer program, and the computer program is stored on the memory. Of course, in practice, the various components in the electronic device are coupled together by a bus system. It will be appreciated that a bus system is used to enable connected communications between these components. The bus system includes a power bus, a control bus, and a status signal bus in addition to the data bus. The memory in the embodiments of the present application is used to store various types of data to support the operation of the electronic device. Examples of such data include: any computer program for operating on an electronic device. It will be appreciated that the memory can be either volatile memory or nonvolatile memory, and can include both volatile and nonvolatile memory. Wherein the nonvolatile Memory may be Read Only Memory (ROM), programmable Read Only Memory (PROM, programmable Read-Only Memory), erasable programmable Read Only Memory (EPROM, erasable Programmable Read-Only Memory), electrically erasable programmable Read Only Memory (EEPROM, electrically Erasable Programmable Read-Only Memory), magnetic random access Memory (FRAM, ferromagnetic random access Memory), flash Memory (Flash Memory), magnetic surface Memory, optical disk, or compact disk Read Only Memory (CD-ROM, compact Disc Read-Only Memory); the magnetic surface memory may be a disk memory or a tape memory. The volatile memory may be random access memory (RAM, random AccessMemory), which acts as external cache memory. By way of example, and not limitation, many forms of RAM are available, such as static random access memory (SRAM, static Random Access Memory), synchronous static random access memory (SSRAM, synchronous Static Random Access Memory), dynamic random access memory (DRAM, dynamic Random Access Memory), synchronous dynamic random access memory (SDRAM, synchronousDynamic Random Access Memory), double data rate synchronous dynamic random access memory (ddr sdram, double Data Rate Synchronous Dynamic Random Access Memory), enhanced synchronous dynamic random access memory (ESDRAM, enhanced Synchronous Dynamic Random Access Memory), synchronous link dynamic random access memory (SLDRAM, syncLink Dynamic Random Access Memory), direct memory bus random access memory (DRRAM, direct Rambus Random Access Memory). The memory described in the embodiments of the present application is intended to comprise, without being limited to, these and any other suitable types of memory. The method disclosed in the embodiments of the present application may be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or by instructions in the form of software. The processor may be a general purpose processor, a DSP (Digital Signal Processing, meaning a chip capable of implementing digital signal processing techniques), or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. The processor may implement or perform the methods, steps, and logic blocks disclosed in embodiments of the present application. The general purpose processor may be a microprocessor or any conventional processor or the like. The steps of the method disclosed in the embodiments of the present application may be directly embodied in a hardware decoding processor or implemented by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium having a memory, and the processor reads the program in the memory and performs the steps of the method in combination with its hardware. The processor executes the program to implement the corresponding flow in each method of the embodiments of the present application, and for brevity, will not be described in detail herein.
Example 4
The invention also provides a readable storage medium, the readable storage medium stores a computer program, and the computer program when executed by a processor realizes the following steps:
fig. 1 is a flowchart of a data protection method according to embodiment 1 of the present invention;
and acquiring a write request sent by the client, performing error correction code encoding on the original data after the cluster calls a write flow to acquire the original data, and generating at least one error correction copy according to the encoded data.
The method further comprises the following steps before the original data is written into the storage device:
dividing the original data into data blocks with preset sizes, and calculating each data block to obtain a check block; the check blocks are stored as metadata. In the distributed storage system, the data is subjected to disk-drop storage mainly through a writing flow. The write operation calculates a check block every 4k data before the original data is written into the disk, and the check block is stored as metadata. The check block is mainly used for checking disk data, and when data on the disk is damaged in transmission, the calculated check value is different from the check value stored in writing, and the data is considered to be damaged. After the check block is generated, the original data is subjected to HDB3 coding to generate an error correction copy. While the write operations have different block sizes, such as writing 4k, 8k, 64k, and 1M data blocks, and HDB3 encoding can be performed on 4k, 8k, 64k, and 1M data, respectively; the HDB3 encoding algorithm is not limited by the data size.
When the original data written into the storage device is read, the original data positioned in the storage device is checked through the check block, and if the original data positioned in the storage device is damaged, an error correction copy is read and decoded to obtain the original data.
In the reading operation, judging whether the data is damaged or not through the check block, and if the data is damaged, obtaining the original data from the error correction copy through decoding and repairing the damaged data. Also, when the error correction copy is damaged, the original data can be recovered by decoding.
FIG. 2 is a flowchart of checking original data by a check block according to embodiment 1 of the present invention;
when in reading operation, firstly, original data positioned on a disk are read, and if the original data are not damaged after being judged by a check block, the original data are returned; when the original data is damaged and the error correction copy is required to be read, the encoded data is obtained through the error correction copy reading operation, whether the data has alternate symbol polarity errors or not is judged, if so, the original data is obtained through normal decoding after recovery through the HDB3 encoding property; if polarity alternation errors do not occur, the original data are obtained through direct normal decoding. And then reading the check block from the metadata base, checking the decoded original data again, and returning the data after no data damage. The decoding operation is connected with the reading operation and the data consistency check module, and when the data consistency check finds that the data is damaged, the error correction copy can be called to repair the data.
The storage medium for protecting data provided in embodiment 4 of the present invention performs HDB3 encoding on data before the data is subjected to disc-drop storage, stores the encoded data, and has a self-error correction function, when the data is damaged, can recover the original data by decoding, thereby realizing correction and protection on the data. The HDB3 coding algorithm has simple coding and decoding, and compared with the original data, the coded data has no check bit, and the storage space is not needed to be used too much. In a special case, when all data occur, original data can be obtained through decoding, so that the safety of the data is effectively improved.
The storage medium for data protection proposed in embodiment 4 of the present invention encodes the original data by using the HDB3 encoding/decoding algorithm, and stores the encoded data without increasing check bit redundancy, and the original data has the same size without increasing space occupation. The HDB3 encoding algorithm may encode any fast-size data without limitation of the data size. The encoded data itself is damaged, and the encoded data can be self-repaired in the decoding process, so that the reliability of the data is improved.
The storage medium for data protection provided by the embodiment 4 of the invention adopts the independent HDB3 encoding and decoding algorithm, does not influence the use of other functions of the distributed storage system, has simple operation and low complexity, and does not cause unnecessary burden to the system.
The embodiments of the present application also provide a storage medium, i.e. a computer storage medium, in particular a computer readable storage medium, for example comprising a memory storing a computer program executable by a processor for performing the steps of the aforementioned method. The computer readable storage medium may be FRAM, ROM, PROM, EPROM, EEPROM, flash Memory, magnetic surface Memory, optical disk, or CD-ROM.
Those of ordinary skill in the art will appreciate that: all or part of the steps for implementing the above method embodiments may be implemented by hardware associated with program instructions, where the foregoing program may be stored in a computer readable storage medium, and when executed, the program performs steps including the above method embodiments; and the aforementioned storage medium includes: a removable storage device, ROM, RAM, magnetic or optical disk, or other medium capable of storing program code. Alternatively, the integrated units described above may be stored in a computer readable storage medium if implemented in the form of software functional modules and sold or used as a stand-alone product. Based on such understanding, the technical solutions of the embodiments of the present application may be essentially or partly contributing to the prior art, and the computer software product may be stored in a storage medium, and include several instructions to cause an electronic device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a removable storage device, ROM, RAM, magnetic or optical disk, or other medium capable of storing program code.
The description of the relevant parts in the processing device and the storage medium for data protection provided in the embodiments of the present application may refer to the detailed description of the corresponding parts in the data protection method provided in embodiment 1 of the present application, which is not repeated herein.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements is inherent to. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element. In addition, the parts of the above technical solutions provided in the embodiments of the present application, which are consistent with the implementation principles of the corresponding technical solutions in the prior art, are not described in detail, so that redundant descriptions are avoided.
While the specific embodiments of the present invention have been described above with reference to the drawings, the scope of the present invention is not limited thereto. Other modifications and variations to the present invention will be apparent to those of skill in the art upon review of the foregoing description. It is not necessary here nor is it exhaustive of all embodiments. On the basis of the technical scheme of the invention, various modifications or variations which can be made by the person skilled in the art without the need of creative efforts are still within the protection scope of the invention.
Claims (10)
1. A method of protecting data, comprising the steps of:
when the original data is written into the storage device, encoding the original data and generating at least one error correction copy;
when the original data written into the storage device is read, the original data positioned in the storage device is checked through the check block, and if the original data positioned in the storage device is damaged, an error correction copy is read and decoded to obtain the original data.
2. The method of claim 1, further comprising, prior to writing the original data to the storage device:
dividing the original data into data blocks with preset sizes, and calculating each data block to obtain a check block; the check blocks are stored as metadata.
3. A method of protecting data according to claim 1, wherein the process of encoding the original data and generating at least one error correction copy comprises: the original data is encoded with an error correction code, and at least one error correction copy is generated based on the encoded data.
4. A data protection method according to claim 3, wherein the error correction code is encoded using HDB 3.
5. A method of protecting data according to claim 3, wherein the process of reading and decoding the error correction copy to obtain the original data comprises: and decoding the error correction copy by adopting an error correction code decoding algorithm to obtain original data.
6. The method for protecting data according to claim 1, wherein the method for judging that the original data located in the storage device is damaged comprises: obtaining coded data through error correction copy reading operation, judging whether the coded data has alternate symbol polarity errors, if so, recovering through error correction code coding, and obtaining original data through error correction code decoding; if polarity alternation errors do not occur, the original data are obtained through direct decoding.
7. The method for protecting data according to claim 6, wherein after the decoding, the method further comprises: and reading the check block from the metadata base, and checking the decoded original data again.
8. The data protection system is characterized by comprising an encoding module and a decoding module;
the encoding module is used for encoding the original data and generating at least one error correction copy when the original data is written into the storage device;
and the decoding module is used for firstly checking the original data positioned in the storage equipment through the checking block when the original data written in the storage equipment is read, and if the original data positioned in the storage equipment is damaged, reading error correction copies and decoding the error correction copies to obtain the original data.
9. An apparatus, comprising:
a memory for storing a computer program;
a processor for implementing the method steps of any one of claims 1 to 7 when executing said computer program.
10. A readable storage medium, characterized in that it has stored thereon a computer program which, when executed by a processor, implements the method steps of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310416038.6A CN116521433A (en) | 2023-04-13 | 2023-04-13 | Data protection method, system, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310416038.6A CN116521433A (en) | 2023-04-13 | 2023-04-13 | Data protection method, system, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116521433A true CN116521433A (en) | 2023-08-01 |
Family
ID=87396841
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310416038.6A Pending CN116521433A (en) | 2023-04-13 | 2023-04-13 | Data protection method, system, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116521433A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118034993A (en) * | 2024-04-11 | 2024-05-14 | 牛芯半导体(深圳)有限公司 | Method and device for protecting stored data, electronic equipment and storage medium |
-
2023
- 2023-04-13 CN CN202310416038.6A patent/CN116521433A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118034993A (en) * | 2024-04-11 | 2024-05-14 | 牛芯半导体(深圳)有限公司 | Method and device for protecting stored data, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR101306645B1 (en) | Error correction decoding by trial and error | |
| EP2908254A1 (en) | Data redundancy implementation method and device | |
| EP1500200B1 (en) | Method and apparatus for embedding an additional layer of error correction into an error correcting code | |
| EP0176218B1 (en) | Error correcting system | |
| US20200350930A1 (en) | Content aware decoding method and system | |
| WO1993018589A1 (en) | Data recovery after error correction failure | |
| IE71652B1 (en) | Byte write error method and apparatus | |
| US9208027B2 (en) | Address error detection | |
| US11301326B2 (en) | Method and apparatus for performing dynamic recovery management regarding redundant array of independent disks | |
| CN116521433A (en) | Data protection method, system, equipment and storage medium | |
| JP4790790B2 (en) | Error detection and correction circuit and semiconductor memory | |
| US5265104A (en) | Data storage system including redundant storage devices | |
| CN112000512B (en) | Data restoration method and related device | |
| US6959412B2 (en) | Error correction coding utilizing numerical base conversion for modulation coding | |
| US6519736B1 (en) | Generating special uncorrectable error codes for failure isolation | |
| CN115729746A (en) | Data storage protection method based on CRC and ECC | |
| CN113377569A (en) | Method, apparatus and computer program product for recovering data | |
| US6463563B1 (en) | Single symbol correction double symbol detection code employing a modular H-matrix | |
| US5249212A (en) | Object reuse protection with error correction | |
| US6460157B1 (en) | Method system and program products for error correction code conversion | |
| US20170161141A1 (en) | Method and apparatus for correcting data in multiple ecc blocks of raid memory | |
| KR20140125987A (en) | Encoder, decoder and semiconductor device including the same | |
| CN116028253A (en) | Solid state disk error correction optimization method, system, equipment and medium | |
| CN108572882B (en) | Data storage method and storage device | |
| CN115269258A (en) | Data recovery method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |