CN112115001B - Data backup method and device, computer storage medium and electronic equipment - Google Patents
Data backup method and device, computer storage medium and electronic equipment Download PDFInfo
- Publication number
- CN112115001B CN112115001B CN202010990027.5A CN202010990027A CN112115001B CN 112115001 B CN112115001 B CN 112115001B CN 202010990027 A CN202010990027 A CN 202010990027A CN 112115001 B CN112115001 B CN 112115001B
- Authority
- CN
- China
- Prior art keywords
- data
- copies
- copy
- backed
- writing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000012795 verification Methods 0.000 claims description 33
- 238000007789 sealing Methods 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 5
- 230000009919 sequestration Effects 0.000 claims 1
- 239000002699 waste material Substances 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 10
- 238000004364 calculation method Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 7
- 230000010354 integration Effects 0.000 description 6
- 238000013508 migration Methods 0.000 description 6
- 230000005012 migration Effects 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 230000005540 biological transmission Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 5
- 238000012423 maintenance Methods 0.000 description 5
- 238000007726 management method Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 239000011159 matrix material Substances 0.000 description 4
- 238000012937 correction Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data backup method, a data backup device, a computer storage medium and electronic equipment, wherein the method comprises the following steps: writing the data to be backed up into R copies; wherein each copy includes N data blocks; after the data to be backed up is written, calculating check data according to N data blocks in the target copy, and writing the check data into M check blocks; the M check blocks and the data blocks of the R copies are stored in the same strip; r is a positive integer greater than or equal to 2, and N and M are positive integers; deleting data in other copies except the target copy in the stripe. By implementing the embodiment of the application, bandwidth waste can be avoided, and the method is compatible with reading and writing of small files.
Description
Technical Field
The present application relates to the field of cloud storage, and in particular, to a data backup method and apparatus, a computer storage medium, and an electronic device.
Background
With the continuous development of information technology, cloud storage systems occupy a significant position. To prevent data loss, affecting user usage, a multi-copy mode is typically employed to implement redundant backup of data. With the increasing amount of data generated by the internet, the cost of implementing redundant backup of data by adopting a multi-copy mode is higher and higher. The mainstream cloud storage system adopts an erasure code mode to realize redundancy backup. Current erasure code patterns include both continuous and bar layouts. The continuous layout can cause bandwidth waste, and the bar layout is not compatible with the reading and writing of small files.
Disclosure of Invention
The embodiment of the application provides a data backup method, a data backup device, a computer storage medium and electronic equipment, which can avoid bandwidth waste and are compatible with small file reading and writing.
In a first aspect, an embodiment of the present application provides a data backup method, including:
Writing the data to be backed up into R copies; wherein each copy includes N data blocks;
After the data to be backed up is written, calculating check data according to N data blocks in the target copy, and writing the check data into M check blocks; the M check blocks and the data blocks of the R copies are stored in the same strip; r is a positive integer greater than or equal to 2, and N and M are positive integers;
deleting data in other copies of the stripe than the target copy
In a second aspect, an embodiment of the present application provides a data backup apparatus, including:
The backup module is used for writing the data to be backed up into R copies; wherein each copy includes N data blocks;
the verification module is used for calculating verification data according to N data blocks in the target copy after the data to be backed up is written, and writing the verification data into M verification blocks; the M check blocks and the data blocks of the R copies are stored in the same strip; r is a positive integer greater than or equal to 2, and N and M are positive integers;
And the deleting module is used for deleting the data in other copies except the target copy in the stripe.
In a third aspect, embodiments of the present application provide a computer storage medium storing a plurality of instructions adapted to be loaded by a processor and to perform the method steps provided in the first aspect of embodiments of the present application.
In a fourth aspect, an embodiment of the present application provides an electronic device, including: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by a processor and to perform the method steps provided in the first aspect of the embodiments of the present application.
The technical scheme provided by the embodiments of the application has the beneficial effects that at least:
According to the embodiment of the application, the data to be backed up can be written into a plurality of copies at the same time, the verification data is calculated according to the target copy, the verification data and the data contained in each copy are stored in the same stripe, and the data in other copies except the target copy in the stripe are deleted after the verification data calculation is completed. The embodiment of the application can pre-allocate the data block storage space of each stripe in advance, realize the integration of a multi-copy system and an erasure code system, facilitate the operation and maintenance of the unified management of resources by a cloud storage system, avoid the migration of data from the multi-copy system to the erasure code system, save the transmission bandwidth, greatly save the cost of the cloud storage system, and also can be compatible with the reading and writing of small files, and avoid the overlarge time delay of uploading and downloading the small files.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a data backup system according to an embodiment of the present application;
fig. 2 is a schematic structural diagram of a cloud storage system with a continuous layout according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of a cloud storage system with a strip layout according to an embodiment of the present application;
FIG. 4 is a schematic flow chart of a data backup method according to an embodiment of the present application;
Fig. 5 is a schematic structural diagram of a cloud storage system according to an embodiment of the present application;
FIG. 6 is a flowchart illustrating another data backup method according to an embodiment of the present application;
FIG. 7 is a schematic diagram of a data backup device according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the application as detailed in the accompanying claims.
In the description of the present application, it should be understood that the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. The specific meaning of the above terms in the present application will be understood in specific cases by those of ordinary skill in the art. Furthermore, in the description of the present application, unless otherwise indicated, "a plurality" means two or more. "and/or", describes an association relationship of an association object, and indicates that there may be three relationships, for example, a and/or B, and may indicate: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship.
Fig. 1 schematically illustrates a structure of a data backup system according to an embodiment of the present application. As shown in fig. 1, the data backup system may include at least: control terminal 110 and a plurality of storage terminals (120 a, 120b, 120 c). The number of control terminals 110 and storage terminals is not limited to the number shown in the figure. The number of control terminals may be plural, and the number of storage terminals may be larger or smaller. The embodiment of the application does not limit the number of the control terminals and the storage terminals.
The control terminal 110 is used to control which storage terminal or terminals the data to be stored is stored in, which disk of a certain storage terminal, and so on. The control terminal can also be used for setting the size of the data block and the check block in the storage terminal, the sealing condition of the data block and the like. After the data is stored, the control terminal can also be used for issuing an instruction for deleting the redundant copies.
The storage terminal is used for receiving the control instruction of the control terminal and writing the data to be stored into a specific magnetic disk. The storage terminal is also used for receiving an instruction for controlling the terminal to set the size of the data block and an instruction for sealing the database, and sealing the data block under the condition that the data block meets the sealing condition. The storage terminal is also used for receiving a deleting instruction of the control terminal, and deleting more copies after the data backup is finished.
The control terminal and the storage terminal related to the embodiment of the application can be a mobile phone, a tablet computer, a desktop, a laptop, a notebook, an Ultra-mobile Personal Computer (UMPC), a handheld computer, a netbook, a Personal digital assistant (Personal DIGITAL ASSISTANT, PDA), a server and the like.
The data backup method provided in the following embodiments of the present application is executed by the control terminal.
The erasure coding techniques according to embodiments of the present application are described next. The most common code of erasure coding technology is RS (Reed-Solomon) code, the implementation principle of the RS code is that an original data block is evenly divided into N data blocks, then a proper check matrix is selected, a reversible cauchy matrix is generally adopted, M check block data are generated in a matrix operation mode, N+M block data are respectively written into different storage nodes, so that data damage of M storage nodes at most can be tolerated, and original data can be recovered through matrix operation.
To ensure that the backup data is reliable, the data is typically replicated into multiple copies, with the data of each copy being identical. The data for each copy is stored in a plurality of data blocks, wherein a data block is a plurality of memory slices allocated in the same stripe (stripe).
The cloud storage system with continuous layout provided by the embodiment of the application is described next. The cloud storage system can adopt a multi-copy mode to realize redundant backup of large file data. Specifically, original data (data to be backed up) can be written into a plurality of large files, then data blocks with specific sizes are taken from the large files each time to perform erasure coding calculation to obtain check blocks, and finally the data blocks and the check blocks are written into an erasure coding system together.
Fig. 2 is a schematic structural diagram of a cloud storage system with a continuous layout according to an embodiment of the present application. As shown in fig. 2, the cloud storage system is three pairs of cloud storage systems. First, the original data (data to be backed up) can be written into a large file in the three-copy system, and then transferred to the erasure code system. The erasure code system adopts an N4M2 mode, wherein N represents the number of databases as 4, and M represents the data of the check block as 2. If the size of each data block in one stripe is 128M, 128 x 4 = 512M data is obtained from a large file every migration, then the data are equally divided into 4 data blocks (D1, D2, D3, D4) and 2 check blocks (P1, P2) are generated through RS codes, and finally the data blocks and the check blocks are written into the erasure coding system together.
The embodiment of the application provides a bar layout cloud storage system. By adopting a strip layout mode, each uploaded data can be equally divided into data blocks with the same size, 0 is supplemented behind the data which cannot be equally divided, and the check block is calculated in real time by adopting the RS code. These same-sized data blocks and parity blocks together form a stripe, and the data blocks and parity blocks of the same stripe are written to different storage nodes.
Fig. 3 shows a schematic structural diagram of a cloud storage system with a stripe layout according to an embodiment of the present application. As shown in fig. 3, the cloud storage system adopts an N4M2 mode, original data (data to be backed up) of 4M can be equally divided into 4 data blocks (D1, D2, D3, D4), 2 check blocks (P1, P2) are generated by real-time calculation, and 6 data blocks are written into different storage nodes. The generated data block sizes are different for different sizes of raw data.
The data backup method provided by the embodiment of the application can be used for redundant backup of data. As shown in fig. 4, the data backup method may include the following steps:
s401: and writing the data to be backed up into R copies.
Wherein each copy includes N data blocks. The capacity size of each data block is the same.
Referring to fig. 5, the process of allocating stripes is illustrated in an R3N4M2 mode, where R represents the number of copies of the multi-copy mode as 3, N represents the number of data blocks of the erasure coding mode as 4, and M represents the number of check blocks of the erasure coding mode as 2. A total of 14 data blocks of block0-block13 are allocated for one stripe at a time, wherein block0, block4 and block8 store identical data, and form 3-copy groups with each other, and each column is three copies with each other except for the last row. Wherein 4 data blocks of block0, block1, block2, block3 are stored in the first copy.
The R copies are respectively stored in different servers and/or different storage spaces of the same server.
Possibly, each copy may be stored in a different server, ensuring that data read-write is not affected after one of the servers is bad.
Possibly, each copy may also be stored in a different storage space of the same server, and when the number of devices is limited, data security may also be ensured.
Possibly, partial copies may be stored in different servers, and the remaining partial copies may be stored in different storage spaces of the same server.
S402: after the writing of the backup data is completed, calculating check data according to N data blocks in the target copy, and writing the check data into M check blocks.
Wherein the M check blocks are stored in the same stripe as the R copies of the data blocks. R is a positive integer greater than or equal to 2, and N and M are positive integers. For example, the embodiment may include 3 copies, and the backup data is written into the 3 copies respectively, and then the writing is completed. And selecting one copy from the 3 copies as a target copy to calculate the check data. The storage space (check block) of the check data is a storage space allocated in advance. After the check data is calculated, the check data is directly written into the check block.
S403: deleting data in other copies except the target copy in the stripe.
Specifically, if one of the copies is selected as the target copy to calculate the check block in S402, after the check block calculation is completed, the data in the other copies may be deleted to release the storage space. For example, reference may be made again to fig. 5. If the 3 rd copy is selected as the target copy, the correction data is calculated according to the data block8-block11 in the 3 rd copy, and after the correction data is written into the block12-block13, the data in the 1 st copy and the second copy in the stripe, namely the data in the block0-block7, can be deleted.
According to the embodiment of the application, the data to be backed up can be written into a plurality of copies at the same time, the verification data is calculated according to the target copy, the verification data and the data contained in each copy are stored in the same stripe, and the data in other copies except the target copy in the stripe are deleted after the verification data calculation is completed. The embodiment of the application can pre-allocate the data block storage space of each stripe in advance, realize the integration of a multi-copy system and an erasure code system, facilitate the operation and maintenance of the unified management of resources by a cloud storage system, avoid the migration of data from the multi-copy system to the erasure code system, save the transmission bandwidth, greatly save the cost of the cloud storage system, and also can be compatible with the reading and writing of small files, and avoid the overlarge time delay of uploading and downloading the small files.
The embodiment of the application provides another data backup method. As shown in fig. 6, the data backup method may include the following steps:
S601: the capacity size of each data block is set.
Specifically, the capacity size of the data block is uniformly set by the control terminal. Illustratively, the capacity size of the data block is, for example, but not limited to, 128M.
S602: and writing the data to be backed up into R copies according to the capacity of the data block.
Specifically, the data to be backed up is written into the first data block of each of the R copies according to the capacity size of the data block. And when the first data block meets the sealing condition, writing the rest data to be backed up into the next data block of each copy until the data to be backed up is completely written.
Specifically, the capacity size of each block is set in advance, for example, the capacity of the block is set to 128M. For allocating block blocks to each stripe, taking R3N4M2 as an example, 14 blocks need to be allocated, and the actual writable capacity of the multi-copy block in each stripe is 4×128=512M capacity. Each object file needs to write 3 blocks at the same time, returns success after waiting for the success of the simultaneous writing of 3 blocks, for example, according to the model of R3N4M2, needs to write block0, block4 and block8 at the same time, writes block1, block5 and block9 at the same time after the writing is finished, and so on until all data blocks are written. In the same copy, one block is written fully and then the next block is written, for example, taking the first copy as an example, the writing sequence is from block0 to block3, block0 is written first, and block1 is written after block0 is written fully until the copy meets the sealing condition.
It will be appreciated that the write speeds of each copy are not necessarily exactly the same, and that there may be some nodes that fail, resulting in a copy stored in that node failing to write or a delay.
Specifically, the condition for the data block may be that the remaining capacity of the data block is lower than a preset threshold. Wherein a threshold value such as, but not limited to, 128kb is also preset.
S603: and after the data to be backed up is written, determining a target copy according to a preset rule.
Specifically, determining the target copy according to the preset rule may be, for example, selecting, as the target copy, a copy in which the data is successfully written according to a default copy arrangement sequence.
For example, the default copy arrangement may be copy 1, copy 2, copy 3. If the 1 st copy fails to write data due to errors in the process of writing data and the 2 nd copy and the 3 rd copy succeed in writing data, the target copy can be selected from the 2 nd copy and the 3 rd copy. As can be seen from the default ranking, the 2 nd copy takes precedence over the 3 rd copy, and then the 2 nd copy can be selected as the target copy.
It will be appreciated that the default copy arrangement described above is merely illustrative, and that other copy arrangements are possible in specific implementations, and embodiments of the present application are not limited in this respect.
S604: and calculating check data according to the N data blocks in the target copy, and writing the check data into the M check blocks.
Specifically, the M check blocks are stored in the same stripe as the R copies of the data blocks. R is a positive integer greater than or equal to 2, and N and M are positive integers. For example, the embodiment may include 3 copies, and the backup data is written into the 3 copies respectively, and then the writing is completed. And selecting one copy from the 3 copies as a target copy to calculate the check data. The storage space (check block) of the check data is a storage space allocated in advance. After the check data is calculated, the check data is directly written into the check block.
For example, refer again to fig. 5. After the target copy is selected as the 3 rd copy, check data is generated using the original data in blocks 8, 9, 10, 11 of the 3 rd copy, and the check data is written into blocks 12 and 13.
S605: deleting data in other copies of the stripe except the target copy.
Specifically, S605 corresponds to S403, and will not be described here.
According to the embodiment of the application, the data to be backed up can be written into a plurality of copies at the same time, the verification data is calculated according to the target copy, the verification data and the data contained in each copy are stored in the same stripe, and the data in other copies except the target copy in the stripe are deleted after the verification data calculation is completed. The embodiment of the application can pre-allocate the data block storage space of each stripe in advance, realize the integration of a multi-copy system and an erasure code system, facilitate the operation and maintenance of the unified management of resources by a cloud storage system, avoid the migration of data from the multi-copy system to the erasure code system, save the transmission bandwidth, greatly save the cost of the cloud storage system, and also can be compatible with the reading and writing of small files, and avoid the overlarge time delay of uploading and downloading the small files.
Fig. 7 is a schematic structural diagram of a data backup device according to an embodiment of the present application. As shown in fig. 7, the data backup apparatus 70 may include at least: backup module 710, verification module 720, and deletion module 730. Wherein:
backup module 710, configured to write data to be backed up into R copies; wherein each copy includes N data blocks;
the verification module 720 is configured to calculate verification data according to N data blocks in the target copy after the writing of the data to be backed up is completed, and write the verification data into M verification blocks; the M check blocks and the data blocks of the R copies are stored in the same strip; r is a positive integer greater than or equal to 2, and N and M are positive integers;
and a deleting module 730, configured to delete data in other copies of the stripe except the target copy.
In one possible embodiment, the data backup apparatus 70 further comprises: a setting module, configured to set a capacity size of each data block before the backup module 710 writes the data to be backed up into R copies;
the backup module 710 is specifically configured to write the data to be backed up into R copies according to the capacity size of the data block.
In one possible embodiment, the backup module 710 is specifically configured to write the data to be backed up into the first data block of each of the R copies according to the capacity size of the data block;
and when the first data block meets the sealing condition, writing the rest data to be backed up into the next data block of each copy until the whole writing of the data to be backed up is completed.
In a possible embodiment, the sealing condition is that the remaining capacity is below a preset threshold.
In a possible embodiment, the R copies are stored in different servers, and/or different storage spaces of the same server, respectively.
In one possible embodiment, the verification module 720 includes: a determining unit and a checking unit. Wherein:
The determining unit is specifically configured to determine a target copy according to a preset rule after the data to be backed up is written;
and the verification unit is specifically used for calculating verification data according to the N data blocks in the target copy and writing the verification data into the M verification blocks.
In a possible embodiment, the determining unit is specifically configured to select, as the target copy, a copy in which the data is successfully written according to a default copy arrangement order.
According to the embodiment of the application, the data to be backed up can be written into a plurality of copies at the same time, the verification data is calculated according to the target copy, the verification data and the data contained in each copy are stored in the same stripe, and the data in other copies except the target copy in the stripe are deleted after the verification data calculation is completed. The embodiment of the application can pre-allocate the data block storage space of each stripe in advance, realize the integration of a multi-copy system and an erasure code system, facilitate the operation and maintenance of the unified management of resources by a cloud storage system, avoid the migration of data from the multi-copy system to the erasure code system, save the transmission bandwidth, greatly save the cost of the cloud storage system, and also can be compatible with the reading and writing of small files, and avoid the overlarge time delay of uploading and downloading the small files.
It should be noted that, in the data backup apparatus provided in the foregoing embodiment, only the division of the foregoing functional modules is used for illustration when the data backup method is executed, and in practical application, the foregoing functional allocation may be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules, so as to complete all or part of the functions described above. In addition, the data backup device and the data backup method provided in the foregoing embodiments belong to the same concept, which embody the detailed implementation process in the method embodiment, and are not described herein again.
The foregoing embodiment numbers of the present application are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
Referring to fig. 8, a schematic structural diagram of an electronic device is provided in an embodiment of the present application. As shown in fig. 8, the electronic device 80 may include: at least one processor 801, at least one network interface 804, a user interface 803, memory 805, at least one communication bus 802.
Wherein a communication bus 802 is used to enable connected communication between these components.
The user interface 803 may include a Display screen (Display) and a Camera (Camera), and the optional user interface 803 may further include a standard wired interface and a wireless interface.
The network interface 804 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface), among others.
Wherein the processor 801 may include one or more processing cores. The processor 801 utilizes various interfaces and lines to connect various portions of the overall electronic device 80, perform various functions of the electronic device 80, and process data by executing or executing instructions, programs, code sets, or instruction sets stored in the memory 805, and invoking data stored in the memory 805. Alternatively, the processor 801 may be implemented in at least one hardware form of digital signal Processing (DIGITAL SIGNAL Processing, DSP), field-Programmable gate array (Field-Programmable GATE ARRAY, FPGA), programmable logic array (Programmable Logic Array, PLA). The processor 801 may integrate one or a combination of several of a central processing unit (Central Processing Unit, CPU), an image processor (Graphics Processing Unit, GPU), and a modem, etc. The CPU mainly processes an operating system, a user interface, an application program and the like; the GPU is used for rendering and drawing the content required to be displayed by the display screen; the modem is used to handle wireless communications. It will be appreciated that the modem may not be integrated into the processor 801 and may be implemented on a single chip.
The Memory 805 may include a random access Memory (Random Access Memory, RAM) or a Read-Only Memory (Read-Only Memory). Optionally, the memory 805 includes a non-transitory computer readable medium (non-transitory computer-readable storage medium). Memory 805 may be used to store instructions, programs, code, sets of codes, or instruction sets. The memory 805 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the above-described respective method embodiments, etc.; the storage data area may store data or the like referred to in the above respective method embodiments. The memory 805 may also optionally be at least one storage device located remotely from the aforementioned processor 801. As shown in FIG. 8, an operating system, network communication modules, user interface modules, and data backup applications may be included in memory 805, which is a type of computer storage media.
In the electronic device 80 shown in fig. 8, the user interface 803 is mainly used for providing an input interface for a user, and acquiring data input by the user; and the processor 801 may be used to invoke the data backup application stored in the memory 805 and specifically perform the following operations:
Writing the data to be backed up into R copies; wherein each copy includes N data blocks;
After the data to be backed up is written, calculating check data according to N data blocks in the target copy, and writing the check data into M check blocks; the M check blocks and the data blocks of the R copies are stored in the same strip; r is a positive integer greater than or equal to 2, and N and M are positive integers;
deleting data in other copies except the target copy in the stripe.
In one possible embodiment, before the processor 801 writes the data to be backed up into R copies, it is further configured to perform: setting the capacity of each data block;
When the processor 801 writes the data to be backed up into R copies, specific execution is performed: and writing the data to be backed up into R copies according to the capacity of the data block.
In one possible embodiment, when the processor 801 writes data to be backed up into R copies according to the capacity size of the data block, the specific implementation is that:
Writing the data to be backed up into a first data block of each copy in R copies according to the capacity of the data block;
and when the first data block meets the sealing condition, writing the rest data to be backed up into the next data block of each copy until the whole writing of the data to be backed up is completed.
In a possible embodiment, the sealing condition is that the remaining capacity is below a preset threshold.
In a possible embodiment, the R copies are stored in different servers, and/or different storage spaces of the same server, respectively.
In one possible embodiment, after the writing of the data to be backed up is completed, the processor 801 calculates check data according to N data blocks in the target copy, and when writing the check data into M check blocks, specifically performs:
after the data to be backed up is written, determining a target copy according to a preset rule;
And calculating check data according to the N data blocks in the target copy, and writing the check data into M check blocks.
In one possible embodiment, when the processor 801 determines the target copy according to a preset rule, the specific implementation is that: and selecting the copy successfully written by the data as a target copy according to the default copy arrangement sequence.
According to the embodiment of the application, the data to be backed up can be written into a plurality of copies at the same time, the verification data is calculated according to the target copy, the verification data and the data contained in each copy are stored in the same stripe, and the data in other copies except the target copy in the stripe are deleted after the verification data calculation is completed. The embodiment of the application can pre-allocate the data block storage space of each stripe in advance, realize the integration of a multi-copy system and an erasure code system, facilitate the operation and maintenance of the unified management of resources by a cloud storage system, avoid the migration of data from the multi-copy system to the erasure code system, save the transmission bandwidth, greatly save the cost of the cloud storage system, and also can be compatible with the reading and writing of small files, and avoid the overlarge time delay of uploading and downloading the small files.
Embodiments of the present application also provide a computer-readable storage medium having instructions stored therein, which when executed on a computer or processor, cause the computer or processor to perform one or more of the steps of the embodiments described above with reference to fig. 4-5. The above-described constituent modules of the data backup apparatus may be stored in the computer-readable storage medium if implemented in the form of software functional units and sold or used as independent products.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted across a computer-readable storage medium. The computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital subscriber line (Digital Subscriber Line, DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital versatile disk (DIGITAL VERSATILE DISC, DVD)), or a semiconductor medium (e.g., a Solid state disk (Solid STATE DISK, SSD)), or the like.
Those skilled in the art will appreciate that implementing all or part of the above-described embodiment methods may be accomplished by way of a computer program, which may be stored in a computer-readable storage medium, instructing relevant hardware, and which, when executed, may comprise the embodiment methods as described above. And the aforementioned storage medium includes: a Read Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk or an optical disk, or the like. The technical features in the present examples and embodiments may be arbitrarily combined without conflict.
The above-described embodiments are merely illustrative of the preferred embodiments of the present application and are not intended to limit the scope of the present application, and various modifications and improvements made by those skilled in the art to the technical solution of the present application should fall within the scope of protection defined by the claims of the present application without departing from the design spirit of the present application.
Claims (7)
1. A method of data backup, comprising:
Writing the data to be backed up into R copies; wherein each copy includes N data blocks;
After the data to be backed up is written, selecting a copy which is successfully written with the data from the R copies as a target copy according to a preset rule, calculating check data according to N data blocks in the target copy, and writing the check data into M check blocks; the M check blocks and the data blocks of the R copies are stored in the same strip; r is a positive integer greater than or equal to 2, and N and M are positive integers;
deleting data in other copies except the target copy in the stripe;
Before the data to be backed up is written into the R copies, the method further includes: setting the capacity of each data block; the writing the data to be backed up into the R copies comprises the following steps: writing the data to be backed up into R copies according to the capacity of the data block;
The writing the data to be backed up into R copies according to the capacity of the data block includes: writing the data to be backed up into a first data block of each copy in R copies according to the capacity of the data block; and when the first data block meets the sealing condition, writing the rest data to be backed up into the next data block of each copy until the whole writing of the data to be backed up is completed.
2. The method of claim 1, wherein the sequestration condition is a residual capacity below a preset threshold.
3. The method of claim 1, wherein the R copies are stored on different servers, and/or different storage spaces of the same server, respectively.
4. The method of claim 1, wherein selecting, as the target copy, a copy from the R copies to which data was successfully written according to a preset rule, comprises: and selecting the copy successfully written with the data from the R copies as a target copy according to a default copy arrangement sequence.
5. A data backup apparatus, comprising:
The backup module is used for writing the data to be backed up into R copies; wherein each copy includes N data blocks;
The verification module is used for selecting a copy successfully written with data from the R copies as a target copy according to a preset rule after the data to be backed up is written, calculating verification data according to N data blocks in the target copy, and writing the verification data into M verification blocks; the M check blocks and the data blocks of the R copies are stored in the same strip; r is a positive integer greater than or equal to 2, and N and M are positive integers;
the deleting module is used for deleting the data in other copies except the target copy in the stripe;
The setting module is used for setting the capacity of each data block before the backup module writes the data to be backed up into R copies;
The backup module is further used for writing the data to be backed up into R copies according to the capacity of the data block;
The backup module is further specifically configured to write data to be backed up into a first data block of each of the R copies according to the capacity of the data block; and when the first data block meets the sealing condition, writing the rest data to be backed up into the next data block of each copy until the whole writing of the data to be backed up is completed.
6. A computer storage medium storing a plurality of instructions adapted to be loaded by a processor and to perform the method steps of any of claims 1-4.
7. An electronic device, comprising: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by a processor and to perform the method steps of any of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010990027.5A CN112115001B (en) | 2020-09-18 | 2020-09-18 | Data backup method and device, computer storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010990027.5A CN112115001B (en) | 2020-09-18 | 2020-09-18 | Data backup method and device, computer storage medium and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112115001A CN112115001A (en) | 2020-12-22 |
| CN112115001B true CN112115001B (en) | 2024-06-07 |
Family
ID=73800306
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010990027.5A Active CN112115001B (en) | 2020-09-18 | 2020-09-18 | Data backup method and device, computer storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112115001B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115686342A (en) * | 2021-07-22 | 2023-02-03 | 华为技术有限公司 | Data storage method and device in storage system |
| CN115599314B (en) * | 2022-12-12 | 2023-03-31 | 深圳市泛联信息科技有限公司 | Data redundancy strategy changing method and device, storage node and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101630282A (en) * | 2009-07-29 | 2010-01-20 | 国网电力科学研究院 | Data backup method based on Erasure coding and copying technology |
| CN106227617A (en) * | 2016-07-15 | 2016-12-14 | 乐视控股（北京）有限公司 | Self-repair method and storage system based on correcting and eleting codes algorithm |
| CN106528330A (en) * | 2016-10-29 | 2017-03-22 | 华为技术有限公司 | Data backup method, node and data backup system |
| CN106649891A (en) * | 2017-02-24 | 2017-05-10 | 深圳市中博睿存信息技术有限公司 | Distributed data storage method and system |
| CN107729536A (en) * | 2017-10-31 | 2018-02-23 | 新华三技术有限公司 | A kind of date storage method and device |
| CN109783016A (en) * | 2018-12-25 | 2019-05-21 | 西安交通大学 | A kind of elastic various dimensions redundancy approach in distributed memory system |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2014118776A1 (en) * | 2013-01-31 | 2014-08-07 | Technion Research & Development Foundation Limited | Management and recovery of distributed storage of replicas |
| US11301324B2 (en) * | 2019-02-15 | 2022-04-12 | Sanmina Corporation | Method and apparatus for consistent and highly available data storage using local and fabric attached non-volatile memory storage devices |
-
2020
- 2020-09-18 CN CN202010990027.5A patent/CN112115001B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101630282A (en) * | 2009-07-29 | 2010-01-20 | 国网电力科学研究院 | Data backup method based on Erasure coding and copying technology |
| CN106227617A (en) * | 2016-07-15 | 2016-12-14 | 乐视控股（北京）有限公司 | Self-repair method and storage system based on correcting and eleting codes algorithm |
| CN106528330A (en) * | 2016-10-29 | 2017-03-22 | 华为技术有限公司 | Data backup method, node and data backup system |
| CN106649891A (en) * | 2017-02-24 | 2017-05-10 | 深圳市中博睿存信息技术有限公司 | Distributed data storage method and system |
| CN107729536A (en) * | 2017-10-31 | 2018-02-23 | 新华三技术有限公司 | A kind of date storage method and device |
| CN109783016A (en) * | 2018-12-25 | 2019-05-21 | 西安交通大学 | A kind of elastic various dimensions redundancy approach in distributed memory system |
Non-Patent Citations (1)
| Title |
|---|
| 云存储环境下分组校验纠删码冗余算法研究;曾赛峰 等;《湖南工程学院学报(自然科学版)》;20161231;第26卷(第4期);第42-45页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112115001A (en) | 2020-12-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11327689B1 (en) | Storage unit including memories of different operational speeds for optimizing data storage functions | |
| US10042706B2 (en) | Optimizing secondary storage in a dispersed storage network | |
| US20170123848A1 (en) | Multi-task processing in a distributed storage network | |
| US10289319B2 (en) | Varying rebuild task priorities | |
| CN109062512A (en) | A kind of distributed storage cluster, data read-write method, system and relevant apparatus | |
| WO2021115021A1 (en) | Data processing method and apparatus, terminal device and computer-readable storage medium | |
| US10423476B2 (en) | Aggressive searching for missing data in a DSN memory that has had migrations | |
| CN113176858B (en) | Data processing method, storage system and storage device | |
| CN112115001B (en) | Data backup method and device, computer storage medium and electronic equipment | |
| US10996894B2 (en) | Application storage segmentation reallocation | |
| WO2015035768A1 (en) | System expansion method and apparatus for redundant array of independent disks (raid) | |
| US11385823B2 (en) | Method, electronic device and computer program product for rebuilding disk array | |
| CN112463058B (en) | Fragmented data sorting method and device and storage node | |
| CN110442298B (en) | Storage equipment abnormality detection method and device and distributed storage system | |
| US10282135B2 (en) | Strong consistency write threshold | |
| CN109840051B (en) | Data storage method and device of storage system | |
| CN115878052B (en) | RAID array inspection method, inspection device and electronic equipment | |
| CN115878381A (en) | Data recovery method and device based on SRM disc, storage medium and electronic device | |
| US9998147B1 (en) | Method for using write intents in a distributed storage network | |
| US20220066879A1 (en) | Metadata Based Listing in a Distributed Storage System | |
| US10838660B2 (en) | Identifying and processing predefined dispersed storage network workflows | |
| US10511665B2 (en) | Efficient resource reclamation after deletion of slice from common file | |
| US10489247B2 (en) | Generating time-ordered globally unique revision numbers | |
| US10379961B2 (en) | Ensuring metadata and index consistency using write intents | |
| US10523241B2 (en) | Object fan out write operation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |