CN111813342A - Data recovery method, device, equipment and computer readable storage medium - Google Patents
Data recovery method, device, equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN111813342A CN111813342A CN202010673842.9A CN202010673842A CN111813342A CN 111813342 A CN111813342 A CN 111813342A CN 202010673842 A CN202010673842 A CN 202010673842A CN 111813342 A CN111813342 A CN 111813342A
- Authority
- CN
- China
- Prior art keywords
- data
- data block
- file
- thread
- aggregation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000011084 recovery Methods 0.000 title claims abstract description 78
- 238000000034 method Methods 0.000 title claims abstract description 36
- 230000002776 aggregation Effects 0.000 claims abstract description 88
- 238000004220 aggregation Methods 0.000 claims abstract description 88
- 238000012217 deletion Methods 0.000 claims abstract description 52
- 230000037430 deletion Effects 0.000 claims abstract description 52
- 238000004590 computer program Methods 0.000 claims description 10
- 230000000694 effects Effects 0.000 abstract description 3
- 238000004064 recycling Methods 0.000 description 5
- 238000004458 analytical method Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 230000006872 improvement Effects 0.000 description 3
- 239000003550 marker Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0652—Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data recovery method, which comprises the following steps: analyzing the received data deletion request to obtain target data blocks to be deleted in the aggregated file; calling a deletion thread to delete each target data block; calling a recovery thread to perform aggregation operation on each residual data block in the aggregation file so as to complete data recovery operation; and each residual data block is a data block in the aggregate file except each target data block. By applying the technical scheme provided by the embodiment of the invention, the data recovery efficiency is greatly improved, and the influence on the front-end service performance is reduced. The invention also discloses a data recovery device, equipment and a storage medium, and has corresponding technical effects.
Description
Technical Field
The present invention relates to the field of storage technologies, and in particular, to a data recovery method, apparatus, device, and computer-readable storage medium.
Background
In the cloud computing era, storage of massive small file data blocks (KB level) is a common problem in the industry, and it is a common practice to aggregate small file data blocks to reduce frequency of bottom delivery so as to improve performance of small file data blocks IO.
A series of problems exist after the small file data blocks are aggregated, wherein the small file data block deletion problem and the small file data block recovery problem are more prominent problems. Which is related to the storage efficiency of the front-end service usage.
When deleting small file data blocks, the current common processing mode firstly marks the small file data blocks inside the aggregation file, does not delete the small file data in real time, avoids influencing the system performance, adopts a garbage recovery thread to read when the data volume of the small file data blocks needing to be deleted reaches the fixed water level of the aggregation file, then splices the small file data blocks which do not need to be deleted inside into other aggregation files, and deletes the small file data blocks needing to be deleted. The method has very low data recovery efficiency, and the front-end service performance is influenced when the data are read, spliced and the like.
To sum up, how to effectively solve the problems of low recovery efficiency, influence on the front-end service performance and the like of the existing data recovery mode is a problem that needs to be solved urgently by technical personnel in the field at present.
Disclosure of Invention
The invention aims to provide a data recovery method, which greatly improves the data recovery efficiency and reduces the influence on the front-end service performance; another object of the present invention is to provide a data recovery device, an apparatus and a computer-readable storage medium.
In order to solve the technical problems, the invention provides the following technical scheme:
a method of data recovery, comprising:
analyzing the received data deletion request to obtain target data blocks to be deleted in the aggregated file;
calling a deletion thread to delete each target data block;
calling a recovery thread to perform aggregation operation on the residual data blocks in the aggregation file so as to complete data recovery operation; and each residual data block is a data block in the aggregate file except each target data block.
In a specific embodiment of the present invention, parsing a received data deletion request to obtain target data blocks to be deleted in an aggregated file includes:
analyzing the received data deletion request to obtain target data blocks to be deleted respectively corresponding to each data strip in the aggregation file;
calling a deletion thread to delete each target data block, wherein the deletion operation comprises the following steps:
distributing corresponding deleting threads for each data stripe respectively;
and calling each deleting thread to delete each target data block.
In a specific embodiment of the present invention, allocating a corresponding delete thread to each data stripe includes:
calculating a proportion value of the total data amount of each target data block in the data amount of the aggregation file;
and respectively distributing corresponding deleting threads for each data stripe according to the proportion value.
In a specific embodiment of the present invention, allocating a corresponding delete thread to each data stripe according to the ratio includes:
judging whether the proportional value exceeds a preset proportional value or not;
if so, allocating a special deleting thread for each data stripe;
and if not, allocating a special deletion thread for each storage area storing the data strip.
In a specific embodiment of the present invention, after the calling the recycle thread to perform the aggregation operation on each remaining data block inside the aggregation file, the method further includes:
judging whether the data volume of the aggregated residual file obtained by aggregation exceeds a preset value or not;
and if not, storing the residual aggregation file into a KV database.
In an embodiment of the present invention, after storing the aggregation residual file in the KV database, the method further includes:
and when detecting that the data volume obtained by performing data addition on the aggregation residual file exceeds the preset value, migrating the added whole aggregation residual file to a disk.
A data recovery device, comprising:
the to-be-deleted data block acquisition module is used for analyzing the received data deletion request to obtain each to-be-deleted target data block in the aggregated file;
the data block deleting module is used for calling a deleting thread to delete each target data block;
the data recovery module is used for calling a recovery thread to perform aggregation operation on the residual data blocks in the aggregation file so as to complete data recovery operation; and each residual data block is a data block in the aggregate file except each target data block.
In a specific embodiment of the present invention, the to-be-deleted data block obtaining module is specifically a module that analyzes the received data deletion request to obtain each target data block to be deleted, which corresponds to each data stripe in the aggregate file, respectively;
the data block deleting module comprises:
a delete thread allocation submodule for allocating corresponding delete threads to each data stripe respectively;
and the data block deleting submodule is used for calling each deleting thread to delete each target data block.
A data recovery apparatus comprising:
a memory for storing a computer program;
a processor for implementing the steps of the data recovery method as described above when executing the computer program.
A computer-readable storage medium having stored thereon a computer program which, when executed by a processor, carries out the steps of the data reclamation method as described above.
By applying the method provided by the embodiment of the invention, the received data deletion request is analyzed to obtain each target data block to be deleted in the aggregated file; calling a deletion thread to delete each target data block; calling a recovery thread to perform aggregation operation on each residual data block in the aggregation file so as to complete data recovery operation; and each residual data block is a data block in the aggregate file except each target data block. The target data block to be deleted is obtained through analysis, the target data block is directly deleted, and aggregation operation is performed on each remaining data block in the original aggregation file, so that a series of operations of reading the data block to be deleted, splicing the data block to other aggregation files, deleting the data block at last and the like are not needed, the data recovery efficiency is greatly improved, and the influence on the front-end service performance is reduced.
Accordingly, embodiments of the present invention further provide a data recovery apparatus, a device, and a computer-readable storage medium corresponding to the data recovery method, which have the above technical effects and are not described herein again.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flow chart of an implementation of a data recovery method according to an embodiment of the present invention;
FIG. 2 is a flow chart of another embodiment of a data recovery method according to the present invention;
FIG. 3 is a block diagram of a data recovery device according to an embodiment of the present invention;
fig. 4 is a block diagram of a data recovery apparatus according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the disclosure, the invention will be described in further detail with reference to the accompanying drawings and specific embodiments. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The first embodiment is as follows:
referring to fig. 1, fig. 1 is a flowchart of an implementation of a data recovery method according to an embodiment of the present invention, where the method may include the following steps:
s101: and analyzing the received data deletion request to obtain each target data block to be deleted in the aggregation file.
When the data blocks in the aggregated file need to be deleted, a data deletion request is sent to the data processing center, wherein the data deletion request comprises data identification information of each target data block to be deleted in the aggregated file. After the data deletion request is received, analyzing the received data deletion request to obtain each target data block to be deleted in the aggregation file.
S102: and calling a deletion thread to delete each target data block.
And after analyzing to obtain each target data block to be deleted in the aggregation file, calling a deletion thread to delete each target data block. Specifically, after analyzing each target data block to be deleted in the aggregate file, the delete marker information may be set for each target data block, and a delete thread may be invoked to delete each target data block by scanning the delete marker information.
S103: and calling a recovery thread to perform aggregation operation on each residual data block in the aggregation file so as to complete data recovery operation.
And each residual data block is a data block in the aggregate file except each target data block.
Data recovery operations need to be performed on the remaining data blocks in the aggregate file except for each target data block. And after the deleting thread is called to delete each target data block, calling a recovery thread to perform aggregation operation on each residual data block in the aggregation file, thereby completing the data recovery operation. The target data block to be deleted is obtained through analysis, the target data block is directly deleted, and aggregation operation is performed on each remaining data block in the original aggregation file, so that a series of operations of reading the data block to be deleted, splicing the data block to other aggregation files, deleting the data block at last and the like are not needed, the data recovery efficiency is greatly improved, and the influence on the front-end service performance is reduced.
By applying the method provided by the embodiment of the invention, the received data deletion request is analyzed to obtain each target data block to be deleted in the aggregated file; calling a deletion thread to delete each target data block; calling a recovery thread to perform aggregation operation on each residual data block in the aggregation file so as to complete data recovery operation; and each residual data block is a data block in the aggregate file except each target data block. The target data block to be deleted is obtained through analysis, the target data block is directly deleted, and aggregation operation is performed on each remaining data block in the original aggregation file, so that a series of operations of reading the data block to be deleted, splicing the data block to other aggregation files, deleting the data block at last and the like are not needed, the data recovery efficiency is greatly improved, and the influence on the front-end service performance is reduced.
It should be noted that, based on the first embodiment, the embodiment of the present invention further provides a corresponding improvement scheme. In the following embodiments, steps that are the same as or correspond to those in the first embodiment may be referred to each other, and corresponding advantageous effects may also be referred to each other, which are not described in detail in the following modified embodiments.
Example two:
referring to fig. 2, fig. 2 is a flowchart of another implementation of a data recovery method according to an embodiment of the present invention, where the method may include the following steps:
s201: and analyzing the received data deletion request to obtain target data blocks to be deleted respectively corresponding to each data stripe in the aggregation file.
When the aggregate file falls to the storage pool layer, the aggregate file is cut into ordered data objects, that is, each data block included in the aggregate file is stored in a data stripe manner. The data deletion request comprises target data blocks to be deleted respectively corresponding to each data stripe in the aggregate file, and the received data deletion request is analyzed to obtain the target data blocks to be deleted respectively corresponding to each data stripe in the aggregate file.
S202: and calculating a proportion value of the total data amount of each target data block to the data amount of the aggregation file.
And after analyzing to obtain each target data block to be deleted, calculating a proportion value of the total data amount of each target data block to the data amount of the aggregation file.
S203: and judging whether the proportional value exceeds a preset proportional value, if so, executing the step S204, and if not, executing the step S205.
A proportion value is preset to be used as a boundary condition for dividing the distribution mode of the deleted threads. After calculating a ratio of the total data amount of each target data block to the data amount of the aggregate file, determining whether the ratio exceeds a preset ratio, if so, indicating that the data amount of the target data block to be deleted in the aggregate file is large, executing step S204, otherwise, indicating that the data amount of the target data block to be deleted in the aggregate file is small, and executing step S205.
It should be noted that the preset proportional value may be set and adjusted according to actual conditions, which is not limited in the embodiment of the present invention.
S204: a dedicated delete thread is assigned to each data stripe.
And after determining that the proportion value of the total data amount of each target data block to the data amount of the aggregation file exceeds a preset proportion value, allocating a special deletion thread for each data stripe. Therefore, the parallel deletion of each target data block in each data stripe is realized, and the data deletion efficiency is greatly improved.
S205: and allocating a special deleting thread for each storage area storing the data strips.
After determining that the ratio of the total data amount of each target data block to the data amount of the aggregate file does not exceed the preset ratio, allocating a dedicated deletion thread to each storage area in which a data stripe is stored, for example, taking each disk as a storage area, each disk storing a plurality of data stripes, and allocating a dedicated deletion thread to each disk respectively by taking the disk as a unit.
S206: and calling each deleting thread to delete each target data block.
And after the distribution of the deleting threads is completed, calling each deleting thread to delete each target data block. The underlying layer may be called for deletion by the falsocete function. By carrying out deletion thread allocation according to the proportion value of the total data amount of the target data block to be deleted to the data amount of the aggregated file, the issuing of the deletion command is controlled at a stable speed, the conflict with the read-write request is avoided, and the influence on the front-end service access is reduced.
S207: and calling a recovery thread to perform aggregation operation on each residual data block in the aggregation file.
And each residual data block is a data block in the aggregate file except each target data block.
S208: and judging whether the data volume of the aggregated residual file obtained by aggregation exceeds a preset value, if not, executing the step S209, and if so, not processing.
And presetting a data quantity threshold as a boundary condition for dividing the storage mode of the aggregation residual file. After the aggregation operation is performed on each residual data block, it is determined whether the data size of the aggregated residual file obtained by aggregation exceeds a preset value, if so, it is determined that the data size of the aggregated residual file is relatively large, no processing is performed, otherwise, it is determined that the data size of the aggregated residual file is relatively small, and step S209 is performed.
It should be noted that the preset value of the data amount can be set and adjusted according to actual situations, which is not limited in the embodiment of the present invention, and may be set to be 100KB, for example.
S209: and storing the aggregation residual file into the KV database to finish the data recovery operation.
And when the data volume of the aggregated residual file obtained by aggregation is determined not to exceed a preset value, storing the aggregated residual file into a KV database (rocksdb), thereby completing the data recovery operation. And transferring the aggregation residual file with small data volume into the KV database to release the storage space of the disk.
S210: and when detecting that the data volume obtained by performing data addition on the aggregation residual file exceeds a preset value, migrating the whole aggregation residual file obtained by adding to a disk.
After the aggregation residual file is stored in the KV database, the aggregation residual file which is generated in the subsequent data recovery processing and does not exceed the preset value can be added into the KV database, when the fact that the data volume obtained by performing data addition on the aggregation residual file exceeds the preset value is detected, the added whole aggregation residual file is migrated to a magnetic disk, data marks in the KV database are deleted, the data marks can be recorded in attribute information of the aggregation residual file, and direct positioning during subsequent searching is facilitated. The storage space of each layer of the system is properly utilized by storing the whole aggregated residual file with large data volume obtained by recycling to a proper storage position.
Corresponding to the above method embodiments, embodiments of the present invention further provide a data recycling apparatus, and the data recycling apparatus described below and the data recycling method described above may be referred to correspondingly.
Referring to fig. 3, fig. 3 is a block diagram of a data recovery apparatus according to an embodiment of the present invention, where the apparatus may include:
a to-be-deleted data block obtaining module 31, configured to parse the received data deletion request to obtain each to-be-deleted target data block in the aggregated file;
the data block deleting module 32 is configured to invoke a deleting thread to delete each target data block;
the data recovery module 33 is configured to invoke a recovery thread to perform aggregation operation on each remaining data block inside the aggregation file to complete data recovery operation; and each residual data block is a data block in the aggregate file except each target data block.
The device provided by the embodiment of the invention is applied to analyze the received data deletion request to obtain each target data block to be deleted in the aggregated file; calling a deletion thread to delete each target data block; calling a recovery thread to perform aggregation operation on each residual data block in the aggregation file so as to complete data recovery operation; and each residual data block is a data block in the aggregate file except each target data block. The target data block to be deleted is obtained through analysis, the target data block is directly deleted, and aggregation operation is performed on each remaining data block in the original aggregation file, so that a series of operations of reading the data block to be deleted, splicing the data block to other aggregation files, deleting the data block at last and the like are not needed, the data recovery efficiency is greatly improved, and the influence on the front-end service performance is reduced.
In a specific embodiment of the present invention, the to-be-deleted data block obtaining module 31 is specifically a module that analyzes the received data deletion request to obtain each to-be-deleted target data block corresponding to each data stripe in the aggregate file;
the data block deletion module 32 includes:
the deleting thread distribution submodule is used for respectively distributing corresponding deleting threads for each data strip;
and the data block deleting submodule is used for calling each deleting thread to delete each target data block.
In a specific embodiment of the present invention, the delete thread assignment submodule includes:
the proportion value calculation unit is used for calculating the proportion value of the total data amount of each target data block in the data amount of the aggregation file;
and the deleting thread distribution unit is used for respectively distributing corresponding deleting threads for the data stripes according to the proportion value.
In one embodiment of the present invention, the delete thread allocating unit includes:
the judging subunit is used for judging whether the proportional value exceeds a preset proportional value;
the first deleting thread allocating subunit is used for allocating a special deleting thread to each data stripe when the proportion value is determined to exceed the preset proportion value;
and the second deleting thread allocating subunit is used for allocating a special deleting thread to each storage area in which the data strips are stored when the proportion value is determined not to exceed the preset proportion value.
In one embodiment of the present invention, the apparatus may further include:
the judging module is used for judging whether the data volume of the aggregated residual file obtained by aggregation exceeds a preset value or not after the recovery thread is called to perform aggregation operation on each residual data block in the aggregated file;
and the file storage module is used for storing the aggregated residual files into the KV database when the data volume of the aggregated residual files obtained by aggregation is determined not to exceed a preset value.
In one embodiment of the present invention, the apparatus may further include:
and the file migration module is used for migrating the added whole residual aggregation file to a disk when detecting that the data volume obtained by adding data to the residual aggregation file exceeds a preset value after the residual aggregation file is stored in the KV database.
Corresponding to the above method embodiment, referring to fig. 4, fig. 4 is a schematic diagram of a data recycling apparatus provided by the present invention, and the apparatus may include:
a memory 41 for storing a computer program;
the processor 42, when executing the computer program stored in the memory 41, may implement the following steps:
analyzing the received data deletion request to obtain target data blocks to be deleted in the aggregated file; calling a deletion thread to delete each target data block; calling a recovery thread to perform aggregation operation on each residual data block in the aggregation file so as to complete data recovery operation; and each residual data block is a data block in the aggregate file except each target data block.
For the introduction of the device provided by the present invention, please refer to the above method embodiment, which is not described herein again.
Corresponding to the above method embodiment, the present invention further provides a computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, implementing the steps of:
analyzing the received data deletion request to obtain target data blocks to be deleted in the aggregated file; calling a deletion thread to delete each target data block; calling a recovery thread to perform aggregation operation on each residual data block in the aggregation file so as to complete data recovery operation; and each residual data block is a data block in the aggregate file except each target data block.
The computer-readable storage medium may include: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
For the introduction of the computer-readable storage medium provided by the present invention, please refer to the above method embodiments, which are not described herein again.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other. The device, the apparatus and the computer-readable storage medium disclosed in the embodiments correspond to the method disclosed in the embodiments, so that the description is simple, and the relevant points can be referred to the description of the method.
The principle and the implementation of the present invention are explained in the present application by using specific examples, and the above description of the embodiments is only used to help understanding the technical solution and the core idea of the present invention. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.
Claims (10)
1. A method of data recovery, comprising:
analyzing the received data deletion request to obtain target data blocks to be deleted in the aggregated file;
calling a deletion thread to delete each target data block;
calling a recovery thread to perform aggregation operation on the residual data blocks in the aggregation file so as to complete data recovery operation; and each residual data block is a data block in the aggregate file except each target data block.
2. The data recovery method according to claim 1, wherein parsing the received data deletion request to obtain each target data block to be deleted in the aggregate file comprises:
analyzing the received data deletion request to obtain target data blocks to be deleted respectively corresponding to each data strip in the aggregation file;
calling a deletion thread to delete each target data block, wherein the deletion operation comprises the following steps:
distributing corresponding deleting threads for each data stripe respectively;
and calling each deleting thread to delete each target data block.
3. The data recovery method of claim 2, wherein assigning a corresponding delete thread to each of the data stripes comprises:
calculating a proportion value of the total data amount of each target data block in the data amount of the aggregation file;
and respectively distributing corresponding deleting threads for each data stripe according to the proportion value.
4. The data recovery method of claim 3, wherein allocating a corresponding delete thread to each of the data stripes according to the ratio comprises:
judging whether the proportional value exceeds a preset proportional value or not;
if so, allocating a special deleting thread for each data stripe;
and if not, allocating a special deletion thread for each storage area storing the data strip.
5. The data recovery method according to any one of claims 1 to 4, further comprising, after invoking a recovery thread to perform an aggregation operation on each remaining data block inside the aggregation file:
judging whether the data volume of the aggregated residual file obtained by aggregation exceeds a preset value or not;
and if not, storing the residual aggregation file into a KV database.
6. The data recovery method of claim 5, further comprising, after storing the aggregated residual file in a KV database:
and when detecting that the data volume obtained by performing data addition on the aggregation residual file exceeds the preset value, migrating the added whole aggregation residual file to a disk.
7. A data recovery device, comprising:
the to-be-deleted data block acquisition module is used for analyzing the received data deletion request to obtain each to-be-deleted target data block in the aggregated file;
the data block deleting module is used for calling a deleting thread to delete each target data block;
the data recovery module is used for calling a recovery thread to perform aggregation operation on the residual data blocks in the aggregation file so as to complete data recovery operation; and each residual data block is a data block in the aggregate file except each target data block.
8. The data recovery device according to claim 7, wherein the to-be-deleted data block obtaining module is specifically a module that parses the received data deletion request to obtain each target data block to be deleted that corresponds to each data stripe in the aggregate file;
the data block deleting module comprises:
a delete thread allocation submodule for allocating corresponding delete threads to each data stripe respectively;
and the data block deleting submodule is used for calling each deleting thread to delete each target data block.
9. A data recovery apparatus, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the data recovery method according to any one of claims 1 to 6 when executing the computer program.
10. A computer-readable storage medium, having stored thereon a computer program which, when being executed by a processor, carries out the steps of the data recovery method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010673842.9A CN111813342A (en) | 2020-07-14 | 2020-07-14 | Data recovery method, device, equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010673842.9A CN111813342A (en) | 2020-07-14 | 2020-07-14 | Data recovery method, device, equipment and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111813342A true CN111813342A (en) | 2020-10-23 |
Family
ID=72842838
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010673842.9A Withdrawn CN111813342A (en) | 2020-07-14 | 2020-07-14 | Data recovery method, device, equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111813342A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113625959A (en) * | 2021-06-30 | 2021-11-09 | 济南浪潮数据技术有限公司 | Data processing method and device |
| CN114115744A (en) * | 2021-11-30 | 2022-03-01 | 新华三大数据技术有限公司 | Control method and device for data recovery task, electronic equipment and storage medium |
| CN114415958A (en) * | 2022-01-20 | 2022-04-29 | 京东科技信息技术有限公司 | Disk data processing method and system, storage medium and electronic equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108958653A (en) * | 2018-06-26 | 2018-12-07 | 郑州云海信息技术有限公司 | A kind of space reclamation method, system and relevant apparatus based on bottom aggregate file |
| CN109739933A (en) * | 2019-01-02 | 2019-05-10 | 郑州云海信息技术有限公司 | Memory space method for releasing, device, terminal and computer readable storage medium |
| CN110119387A (en) * | 2019-05-23 | 2019-08-13 | 苏州浪潮智能科技有限公司 | A kind of file delet method, device, equipment and readable storage medium storing program for executing |
-
2020
- 2020-07-14 CN CN202010673842.9A patent/CN111813342A/en not_active Withdrawn
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108958653A (en) * | 2018-06-26 | 2018-12-07 | 郑州云海信息技术有限公司 | A kind of space reclamation method, system and relevant apparatus based on bottom aggregate file |
| CN109739933A (en) * | 2019-01-02 | 2019-05-10 | 郑州云海信息技术有限公司 | Memory space method for releasing, device, terminal and computer readable storage medium |
| CN110119387A (en) * | 2019-05-23 | 2019-08-13 | 苏州浪潮智能科技有限公司 | A kind of file delet method, device, equipment and readable storage medium storing program for executing |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113625959A (en) * | 2021-06-30 | 2021-11-09 | 济南浪潮数据技术有限公司 | Data processing method and device |
| CN113625959B (en) * | 2021-06-30 | 2023-12-29 | 济南浪潮数据技术有限公司 | Data processing method and device |
| CN114115744A (en) * | 2021-11-30 | 2022-03-01 | 新华三大数据技术有限公司 | Control method and device for data recovery task, electronic equipment and storage medium |
| CN114415958A (en) * | 2022-01-20 | 2022-04-29 | 京东科技信息技术有限公司 | Disk data processing method and system, storage medium and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111813342A (en) | Data recovery method, device, equipment and computer readable storage medium | |
| CN109271343A (en) | A kind of data merging method and device applied in key assignments storage system | |
| CN111061752B (en) | Data processing method and device and electronic equipment | |
| CN106383666B (en) | Data storage method and device | |
| CN105404691A (en) | File storage method and apparatus | |
| CN113867644B (en) | Disk array optimization method, disk array optimization device, computer equipment and storage medium | |
| CN112465032A (en) | Distribution method and device of training data labeling tasks and computing equipment | |
| CN103631589B (en) | Method and device for recognizing application | |
| CN107229421B (en) | Method and device for creating video data storage system, method and device for writing file into video data storage system and method and device for reading video data storage system | |
| CN112347131A (en) | Urban rail project demand identification and coverage method and device | |
| CN113157600A (en) | Space allocation method of shingled hard disk, file storage system and server | |
| CN110618946A (en) | Stack memory allocation method, device, equipment and storage medium | |
| CN110647498A (en) | File storage method, device, equipment and computer readable storage medium | |
| CN105243011A (en) | Memory leak positioning method and apparatus | |
| CN112765170B (en) | Embedded time sequence data management method and device | |
| CN115617802A (en) | Method and device for quickly generating full snapshot, electronic equipment and storage medium | |
| CN113608679A (en) | File storage method, device, equipment and computer readable storage medium | |
| US12001397B2 (en) | Method, apparatus and device for deleting distributed system file, and storage medium | |
| US20220365677A1 (en) | Storage management device, storage management method, and program | |
| CN106959888B (en) | Task processing method and device in cloud storage system | |
| CN109086223B (en) | Method and device for controlling garbage recovery | |
| CN111176571A (en) | Method, device, equipment and medium for managing local object | |
| CN112967454A (en) | Resource distribution method, device, cabinet, electronic equipment and storage medium | |
| CN114743203A (en) | Document processing method, device, equipment and storage medium | |
| CN107885839B (en) | Method and device for reading information in Word file |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20201023 |
|
| WW01 | Invention patent application withdrawn after publication |