CN114968111A - Data deleting method, device, equipment and computer readable storage medium - Google Patents
Data deleting method, device, equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN114968111A CN114968111A CN202210606581.8A CN202210606581A CN114968111A CN 114968111 A CN114968111 A CN 114968111A CN 202210606581 A CN202210606581 A CN 202210606581A CN 114968111 A CN114968111 A CN 114968111A
- Authority
- CN
- China
- Prior art keywords
- deleted
- time
- deletion
- value pair
- key
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 238000012217 deletion Methods 0.000 claims abstract description 122
- 230000037430 deletion Effects 0.000 claims abstract description 122
- 239000012634 fragment Substances 0.000 claims abstract description 62
- 238000012795 verification Methods 0.000 claims description 48
- 230000014759 maintenance of location Effects 0.000 claims description 28
- 238000004590 computer program Methods 0.000 claims description 10
- 238000004064 recycling Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 238000007726 management method Methods 0.000 description 3
- 230000000717 retained effect Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/162—Delete operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02W—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO WASTEWATER TREATMENT OR WASTE MANAGEMENT
- Y02W90/00—Enabling technologies or technologies with a potential or indirect contribution to greenhouse gas [GHG] emissions mitigation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data deletion method, a device, equipment and a computer readable storage medium, relating to the field of due deletion, wherein the method comprises the following steps: when the object in the object storage bucket moves to a recycle bin, establishing an index fragment of the object storage bucket; storing object information and deletion time of an object to the index fragment through an omap key value pair, wherein keys in the omap key value pair comprise deletion time and object names, and are sorted according to the deletion time; and determining the object to be deleted according to the deletion time, and deleting the object to be deleted and the corresponding omap key value pair. When the object is deleted, an index fragment is established for the object storage bucket, object information and deletion time of the object are stored in the index fragment in an omap key value pair mode, the key comprises the deletion time and the object name of the object, the key is sequenced according to the deletion time, the object to be deleted is determined according to the deletion time of the object, all the objects in the recycle bin do not need to be traversed, and traversing operation is reduced.
Description
Technical Field
The present application relates to the field of due deletion, and in particular, to a data deletion method, apparatus, device, and computer-readable storage medium.
Background
The distributed object Storage is suitable for storing unstructured data such as pictures and videos, has the characteristics of high-speed direct access to disks by SAN (Storage Area Network and SAN Protocols, Storage Area Network and Protocols thereof) and NAS (Network Attached Storage) distributed sharing, and provides a Storage system structure with high performance, high reliability, cross-platform and safe data sharing. With the increasing size of unstructured data of the internet, distributed object storage is widely developed and applied.
The document data temporarily deleted by the user is stored in the recycle bin, and the document data in the recycle bin can be recovered. The existing object storage is realized mainly by means of creating hidden directories in multiple versions or buckets when the recycle bin function is realized, and for objects in the recycle bin, namely historical version objects, in the multiple versions, due deletion of the recycle bin objects is realized by setting a life cycle strategy of the historical version due time for the buckets; for the way the hidden directory is created within the bucket, the lifecycle policy is deleted by setting the expiration prefixed with the recycle bin directory. That is, at present, a user can only delete an object through the object lifecycle, scan all objects in the recycling bin through the lifecycle thread, check the time of the object added to the recycling bin and compare the time with the current time, detect whether the object in the recycling bin is expired, and delete the object if the object is expired. Because all objects in the bucket recycle bin need to be traversed, a large amount of query operations on the index pool can be caused, and the performance of the cluster metadata pool is seriously influenced.
Therefore, how to solve the above technical problems should be a great concern to those skilled in the art.
Disclosure of Invention
The application aims to provide a data deleting method, a data deleting device, data deleting equipment and a computer readable storage medium, so that query operation is reduced, and system stability is improved.
In order to solve the above technical problem, the present application provides a data deletion method, including:
when an object in an object bucket moves to a recycle bin, establishing an index fragment of the object bucket;
storing the object information and the deletion time of the object to the index fragment through an omp key value pair, wherein a key in the omp key value pair comprises the deletion time and the object name, and the key is sorted according to the deletion time;
and determining an object to be deleted according to the deletion time, and deleting the object to be deleted and the corresponding omap key value pair.
Optionally, the determining an object to be deleted according to the deletion time, and deleting the object to be deleted and the corresponding omap key-value pair include:
obtaining the retention time and the current time of the object in the recycle bin;
determining an expiration time node of the object in the recycle bin according to the current time and the retention time;
traversing the key on the index tile before the expiration time node with a thread;
and deleting the object to be deleted and the omap key-value pair corresponding to the traversed key.
Optionally, the scan cycle of the thread is in the order of minutes.
Optionally, before deleting the object to be deleted and the omap key-value pair corresponding to the traversed key, the method further includes:
judging whether verification information in the index fragment is consistent with target verification information in a head object, wherein the verification information is located in object metadata of value in the omap key value pair, the target verification information is located in metadata of the head object, and the verification information corresponds to the traversed key;
and if the verification information is consistent with the target verification information, executing a step of deleting the object to be deleted and the omap key-value pair corresponding to the traversed key.
Optionally, the obtaining the retention time of the object in the recycle bin includes:
acquiring bucket metadata;
determining the retention time from the bucket metadata.
Optionally, the determining an object to be deleted according to the deletion time, and deleting the object to be deleted and the corresponding omap key-value pair include:
acquiring the object name and the bucket name of the object to be deleted;
determining the deletion time of the object to be deleted in the head object according to the object name and the bucket name;
determining corresponding target deletion time in the index fragment according to the deletion time in the head object;
and deleting the object to be deleted and the omap key value pair corresponding to the target deletion time.
Optionally, the determining, according to the object name and the bucket name, the deletion time when the object to be deleted is located in the head object includes:
determining the head object according to the object name and the bucket name;
determining metadata for the object from the head object;
determining the deletion time located in the head object from the metadata.
Optionally, when a plurality of recycle bins delete an object at the same time, the creating the index fragment of the object bucket includes:
and establishing the index fragment corresponding to each object bucket.
The present application further provides a data deleting device, including:
the system comprises an establishing module, a searching module and a judging module, wherein the establishing module is used for establishing an index fragment of an object storage bucket when an object in the object storage bucket moves to a recycle bin;
the storage module is used for storing the object information and the deletion time of the object to the index fragment through an omap key value pair, wherein keys in the omap key value pair comprise the deletion time and an object name, and the keys are sorted according to the deletion time;
and the deleting module is used for determining the object to be deleted according to the deleting time and deleting the object to be deleted and the corresponding omap key value pair.
The present application further provides a data deletion apparatus, including:
a memory for storing a computer program;
a processor for implementing the steps of any of the above-mentioned data deletion methods when executing the computer program.
The present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the steps of any one of the data deleting methods.
The application provides a data deleting method, which comprises the following steps: when an object in an object storage bucket moves to a recycle bin, establishing an index fragment of the object storage bucket; storing the object information and the deletion time of the object to the index fragment through an omap key value pair, wherein keys in the omap key value pair comprise the deletion time and an object name, and the keys are sorted according to the deletion time; and determining an object to be deleted according to the deletion time, and deleting the object to be deleted and the corresponding omap key value pair.
Therefore, according to the data deletion method, when an object in the object storage bucket moves to a recycle bin, an index fragment is established for the object storage bucket, object information and deletion time of the object are stored in the index fragment in an omp key value pair mode, keys in the omp key value pair comprise deletion time and object names of the object, the keys are sorted according to the deletion time, the object to be deleted is determined according to the deletion time of the object, the object to be deleted and the omp key value pair corresponding to the object to be deleted are deleted together, all objects in the recycle bin do not need to be traversed, traversal operation of a large number of metadata pools is reduced, pressure of the metadata pools is reduced, cluster IO is reduced, and system stability is improved.
In addition, the application also provides a device, equipment and a computer readable storage medium with the advantages.
Drawings
For a clearer explanation of the embodiments or technical solutions of the prior art of the present application, the drawings needed for the description of the embodiments or prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a flowchart of a data deleting method according to an embodiment of the present disclosure;
fig. 2 is a flowchart illustrating an automatic delete operation according to an embodiment of the present application;
FIG. 3 is a flow chart illustrating another embodiment of an automatic delete operation;
FIG. 4 is a flowchart illustrating a manual delete operation according to an embodiment of the present disclosure;
fig. 5 is a block diagram of a data deleting device according to an embodiment of the present application;
fig. 6 is a block diagram of a data deleting device according to an embodiment of the present application.
Detailed Description
In order that those skilled in the art will better understand the disclosure, the following detailed description will be given with reference to the accompanying drawings. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
As described in the background section, currently, a user can only delete an object through the object lifecycle, scan all objects in the recycling bin through the lifecycle thread, check the time when the object joins the recycling bin and compare the current time, detect whether the object in the recycling bin is expired, and delete the object if the object is expired. Because all objects in the bucket recycle bin need to be traversed, a large amount of query operations on the index pool can be caused, and the performance of the cluster metadata pool is seriously influenced.
In view of the above, the present application provides a data deleting method, please refer to fig. 1, which includes:
step S101: when the object in the object bucket moves to a recycle bin, the index fragment of the object bucket is established.
Object-based Storage (OBS) is a computer data Storage architecture that manages data as objects, as opposed to other Storage architectures (e.g., file systems that manage data as file hierarchies) and block Storage that manages data as blocks and blocks within sectors. Each object typically includes the data itself, a variable amount of metadata, and a globally unique identifier.
An object bucket is a container in an OBS that stores objects. The object storage provides a flat storage mode based on a bucket and objects, all the objects in the bucket are in the same logic level, and a multi-level tree directory structure in a file system is removed. Each bucket has the attributes of the storage category, the access authority, the region to which the bucket belongs and the like, and a user can create buckets with different storage categories and access authorities in different regions and configure more high-level attributes to meet the storage requirements of different scenes.
When the object in the object bucket moves to the recycle bin, the object is deleted and placed in the recycle bin. One object bucket corresponds to one index fragment, and the index fragment is used for recording the objects in the object bucket and related information of the objects. The index shards are built in a metadata pool.
Step S102: and storing the object information and the deletion time of the object to the index fragment through an omap key value pair, wherein keys in the omap key value pair comprise the deletion time and the object name, and the keys are sorted according to the deletion time.
The object information includes object names and object metadata, and the object names correspond to the objects one to one. The omap key-value pair includes a key (key) including a deletion time and an object name and a value (key value) including object metadata. The time is used as a prefix in the key, and the key sorting according to the deletion time comprises forward sorting according to the deletion time or reverse sorting according to the deletion time, and the key sorting can be specifically set by self, and is not specifically limited in the application. Wherein, the deletion time is the time when the object moves to the recycle bin.
Step S103: and determining an object to be deleted according to the deletion time, and deleting the object to be deleted and the corresponding omap key value pair.
The objects are stored in the data pool, the related information of the objects in the data pool is recorded on the index fragment, and the objects in the data pool and the corresponding information on the index fragment are deleted together when the deletion is carried out, so that the consistency of the data pool and the index fragment is ensured. Specific deletion operations include manual deletion and automatic deletion, which are described below.
Automatic delete object and omap key-value pair operations:
referring to fig. 2, the determining an object to be deleted according to the deletion time, and deleting the object to be deleted and the corresponding omap key-value pair includes:
step S201: and acquiring the retention time and the current time of the object in the recycle bin.
The retention time, i.e., the time that an object in the recycle bin can be stored in the recycle bin, exceeds which the object expires, and does not exceed the retention time, the object can continue to be stored in the recycle bin. The current time is the current time node.
The obtaining the retention time of the object in the recycle bin comprises:
acquiring bucket metadata;
determining the retention time from the bucket metadata.
The bucket metadata includes the retention time of the object, so the retention time of the object can be obtained after the bucket metadata is obtained.
Step S202: and determining an expiration time node of the object in the recycle bin according to the current time and the retention time.
Subtracting the retention time from the current time to obtain an expiration time node of the object in the recycle bin, wherein the objects which are moved to the recycle bin before the expiration time node are all expired and invalid and need to be deleted; the object moved to the recycle bin after the expiration time node is still stored in the recycle bin without deletion operation. For example, if the retention time of the objects in the recycle bin is seven days, and the current time is 5 months, 29 am, 8 am, the expiration time node is 5 months, 22 am, 8 am, i.e., the objects moved into the recycle bin before 5 months, 22 am, 8 am are all expired and need to be deleted, and the objects moved into the recycle bin after 5 months, 22 am, 8 am continue to be retained.
Step S203: traversing the key on the index tile before the expiration time node with a thread.
The keys are arranged according to the time sequence, so that all the keys do not need to be traversed when the threads traverse, the keys with deletion time before the expiration time node only need to be traversed, and the mark of the traversal end is the expiration time node, so that a large amount of unnecessary traversal operation is reduced.
In the prior art, the lifecycle of an object in a recycle bin is an execution cycle of days, so that an expiration time can only be in units of days, and in order to implement finer cycle management on the object in the recycle bin, in an embodiment of the present application, a scan cycle of a thread is in a minute level, which can implement lifecycle management of an object in the recycle bin at an hour level, and meet a requirement of a user for finer granularity, for example, the scan cycle of the thread may be 10 minutes or 20 minutes, and may specifically be set by a user. Accordingly, the object retention time within the recycle bin may be several hours. For example, if the retention time of the objects in the recycle bin is 5 hours, and the current time is 8 am on 29 th month in 5 months, the expiration time node is 3 am on 29 th month in 5 months, that is, the objects moved to the recycle bin before 3 am on 29 th month in 5 months are all expired and expired, and all the objects need to be deleted, and the objects moved to the recycle bin after 3 am on 29 th month in 5 months continue to be retained.
Step S204: and deleting the object to be deleted and the omap key-value pair corresponding to the traversed key.
The keys comprise deletion time and object names, the objects to be deleted can be determined through the object names in the traversed keys, then the objects to be deleted are deleted from the data pool, the omap key value pairs of the traversed keys on the index fragments are deleted, and the information of the objects recorded on the index fragments is consistent with the objects in the data pool.
Referring to fig. 3 when performing automatic object deletion and omap key value pair deletion, in an embodiment of the present application, determining an object to be deleted according to the deletion time, and deleting the object to be deleted and the corresponding omap key value pair includes:
step S301: and acquiring the retention time and the current time of the object in the recycle bin.
Step S302: and determining an expiration time node of the object in the recycle bin according to the current time and the retention time.
Step S303: traversing the key on the index tile before the expiration time node with a thread.
Step S304: and judging whether verification information in the index fragment is consistent with target verification information in the head object, wherein the verification information is located in object metadata of value in the omap key value pair, the target verification information is located in metadata of the head object, and the verification information corresponds to the traversed key.
The head object records metadata of the object, the verification information of the object in the metadata is called target verification information, the omap key value pair comprises a key and a value, the value comprises the verification information of the object, namely, each key corresponds to one verification information, and each object name corresponds to one verification information.
It should be noted that, the specific category of the verification information in the present application is not limited, and can be selected by oneself. For example, the verification information may be a tag, or MD5 fingerprint information, or the like.
Step S305: and if the verification information is consistent with the target verification information, executing a step of deleting the object to be deleted and the omap key-value pair corresponding to the traversed key.
Step S306: and deleting the object to be deleted and the omap key-value pair corresponding to the traversed key.
Step S307: and when the verification information is inconsistent with the target verification information, stopping the deleting operation.
It should be noted that, please refer to step S201, step S202, step S203, and step S204 above in step S301, step S302, step S303, and step S306, which are not described in detail here.
In the embodiment, before the object to be deleted and the omp key value pair are deleted, whether the verification information in the index fragment is consistent with the target verification information in the head object is verified, and the deletion operation is performed only when the verification information in the index fragment is consistent with the target verification information in the head object, so that the object in the recycle bin can be prevented from being deleted by mistake.
Manual delete object and omap key value pair operations:
referring to fig. 4, the determining an object to be deleted according to the deletion time, and deleting the object to be deleted and the corresponding omap key-value pair includes:
step S401: and acquiring the object name and the bucket name of the object to be deleted.
The bucket name is also the ID of the bucket, the bucket name is unique, and the object to be deleted stored in the data pool can be determined through the object name and the bucket name.
Step S402: and determining the deletion time of the object to be deleted in the head object according to the object name and the bucket name.
The head object includes an object name, a bucket name, and object metadata, which is recorded in xattr of the head object.
As one possible implementation, the determining the deletion time that the object to be deleted is located in a head object according to the object name and the bucket name comprises:
step S4021: determining the head object according to the object name and the bucket name;
step S4022: determining metadata for the object from the head object;
step S4023: determining the deletion time located in the head object from the metadata.
The object metadata in the head object and the object metadata in the value are not completely the same, and the object metadata in the head object and the object metadata in the value comprise the deletion time of the object, some metadata of self-definition, object attribute information, manifest (tail object of the recorded object), access control information and the like; the object metadata in value includes the size of the object, etc.
Step S403: and determining the corresponding target deletion time in the index fragment according to the deletion time in the head object.
The object metadata in the head object and the keys of the omap key value pair both comprise the deletion time of the object, so that the corresponding target deletion time in the omap key value pair can be found when the deletion time in the object metadata of the head object is obtained.
Step S404: and deleting the object to be deleted and the omap key value pair corresponding to the target deletion time.
And finding the corresponding target deletion time in the omap key value pair, namely finding the corresponding key, thereby obtaining the object name corresponding to the object to be deleted, deleting the object to be deleted according to the object name, deleting the corresponding omap key value pair according to the target deletion time, and keeping the information of the object recorded on the index fragment consistent with the object in the data pool.
According to the data deleting method, when an object in an object storage bucket moves to a recycle bin, an index fragment is established for the object storage bucket, object information and deleting time of the object are stored in the index fragment in an omp key value pair mode, keys in the omp key value pair include the deleting time and object name of the object, the keys are sorted according to the deleting time, the object to be deleted is determined according to the deleting time of the object, the object to be deleted and the omp key value pair corresponding to the object to be deleted are deleted together, all objects in the recycle bin do not need to be traversed, traversing operation of a large number of metadata pools is reduced, pressure of the metadata pools is reduced, cluster IO is reduced, and system stability is improved.
On the basis of any of the foregoing embodiments, in an embodiment of the present application, when multiple recycle bins delete an object at the same time, the creating an index fragment of the object bucket includes:
and establishing the index fragment corresponding to each object bucket.
The object buckets correspond to the index fragments one to one, and the method for deleting the object in each recycle bin is the same, which can specifically refer to the above explanation.
The data deleting method in the present application is explained in a specific case.
Step 1, when the object in the object storage bucket is deleted, the object in the object storage bucket is moved to a recycle bin, and an index fragment corresponding to the object storage bucket is newly built in a metadata pool.
And 2, mapping the object information (object name and object metadata) to an index fragment, storing the object information on the index fragment by using an omap key value pair, wherein omap key is deletion time + object name, and value is object metadata information and is used for traversing the recycle bin object and checking the object for use during deletion. The purpose of sorting the index records according to time is achieved by using keys and taking time as a prefix.
Step 3, manual deletion: according to the object name and the bucket name, acquiring head object information, checking the time of the object moving to the recycle bin, acquiring a key of the index fragment at the recycle bin according to the time, and deleting the object and the index information;
and (3) automatic deletion: and acquiring bucket metadata information, checking the retention time of the objects in the recycle bin, and calculating when the objects added into the recycle bin are invalid. And the recycle bin object deleting thread scans the object on the recycle bin index fragment of the object storage bucket by taking the calculated time as an end mark of the pass duration to obtain the invalidated recycle bin object, compares whether the verification information in the index is consistent with the object metadata information, if so, executes deleting operation on the object and deletes the index record at the same time, and if not, does not delete the index record.
In the following, the data deleting device provided in the embodiment of the present application is introduced, and the data deleting device described below and the data deleting method described above may be referred to correspondingly.
Fig. 5 is a block diagram of a data deleting device according to an embodiment of the present application, where the data deleting device shown in fig. 5 may include:
the system comprises an establishing module 100, a fragment storing module and a fragment storing module, wherein the establishing module is used for establishing an index fragment of an object bucket when an object in the object bucket moves to a recycle bin;
a storage module 200, configured to store the object information and the deletion time of the object to the index segment through an omp key value pair, where a key in the omp key value pair includes the deletion time and an object name, and the keys are sorted according to the deletion time;
and the deleting module 300 is configured to determine an object to be deleted according to the deleting time, and delete the object to be deleted and the corresponding omap key value pair.
The data deleting device of this embodiment is configured to implement the foregoing data deleting method, and therefore a specific implementation manner of the data deleting device may be found in the foregoing embodiment portions of the data deleting method, for example, the establishing module 100, the saving module 200, and the deleting module 300, which are respectively configured to implement steps S101, S102, and S103 in the data deleting method.
According to the data deleting device, when an object in the object storage bucket moves to a recycle bin, an index fragment is established for the object storage bucket, object information and deleting time of the object are stored in the index fragment in an omp key value pair mode, keys in the omp key value pair include the deleting time and object name of the object, the keys are sorted according to the deleting time, then the object to be deleted is determined according to the deleting time of the object, the object to be deleted and the omp key value pair corresponding to the object to be deleted are deleted together, all objects in the recycle bin do not need to be traversed, traversing operation of a large number of metadata pools is reduced, pressure of the metadata pools is reduced, cluster IO is reduced, and system stability is improved.
Optionally, when performing an automatic deletion operation, the deletion module 300 includes:
the first acquisition unit is used for acquiring the retention time and the current time of the object in the recycle bin;
a first determining unit, configured to determine an expiration time node of the object in the recycle bin according to the current time and the retention time;
a traversing unit, configured to traverse the key before the expiration time node on the index tile by using a thread;
and the first deleting unit is used for deleting the object to be deleted and the omap key-value pair corresponding to the traversed key.
Optionally, when the traversal unit traverses the key located before the expiration time node on the index tile by using a thread, a scan cycle of the thread is in the order of minutes.
In the prior art, the lifecycle of an object in a recycle bin takes days as an execution cycle, so that the expiration time can only take days as a unit, the scanning cycle of the process in the embodiment is in the minute level, the management of the lifecycle of the recycle bin object in the hour level can be realized, and the requirement of a user on finer granularity is met. For example, the scan cycle of the thread may be 10 minutes, or 20 minutes, etc., and may be set by itself. Accordingly, the object retention time within the recycle bin may be several hours. For example, if the retention time of the objects in the recycle bin is 5 hours, and the current time is 8 am on 29 th month in 5 months, the expiration time node is 3 am on 29 th month in 5 months, that is, the objects moved to the recycle bin before 3 am on 29 th month in 5 months are all expired and expired, and all the objects need to be deleted, and the objects moved to the recycle bin after 3 am on 29 th month in 5 months continue to be retained.
Optionally, in an embodiment of the present application, the deleting module 300 further includes:
a determining unit, configured to determine whether verification information in the index fragment is consistent with target verification information in a head object, where the verification information is located in object metadata of a value in the omap key value pair, the target verification information is located in metadata of the head object, and the verification information corresponds to the traversed key;
the execution unit is used for executing the step of deleting the object to be deleted and the omap key value pair corresponding to the traversed key if the verification information is consistent with the target verification information;
and a stopping unit for stopping the deleting operation when the verification information is inconsistent with the target verification information.
In the embodiment, before deleting the object to be deleted and the omap key value pair, whether the verification information in the index fragment is consistent with the target verification information in the head object is verified, and the deletion operation is performed only when the verification information in the index fragment is consistent with the target verification information in the head object, so that the object in the recycle bin can be prevented from being deleted by mistake.
Optionally, the first obtaining unit includes:
an obtaining subunit configured to obtain bucket metadata;
a first determining subunit configured to determine the retention time according to the bucket metadata.
Optionally, when performing a manual deletion operation, the deletion module 300 includes:
a second obtaining unit, configured to obtain the object name and the bucket name of the object to be deleted;
a second determining unit, configured to determine, according to the object name and the bucket name, the deletion time at which the object to be deleted is located in the head object;
a third determining unit, configured to determine, according to the deletion time located in the head object, a target deletion time corresponding to the index fragment;
and the second deleting unit is used for deleting the object to be deleted and the omap key value pair corresponding to the target deleting time.
Optionally, the second determining unit includes:
a second determining subunit, configured to determine the head object according to the object name and the bucket name;
a third determining subunit, configured to determine metadata of the object according to the head object;
a fourth determining subunit, configured to determine the deletion time located in the head object according to the metadata.
Optionally, when a plurality of recycle bins delete an object at the same time, the creating module 100 is specifically configured to create the index fragment corresponding to each object bucket.
The object buckets correspond to the index fragments one to one, and the object deletion method in each recycle bin is the same, which specifically refers to the above explanation.
In the following, the data deleting device provided in the embodiment of the present application is introduced, and the data deleting device described below and the data deleting method described above may be referred to correspondingly.
Fig. 6 is a block diagram of a data deleting device according to an embodiment of the present application, where the data deleting device includes:
a memory 11 for storing a computer program;
a processor 12, configured to implement the steps of the data deleting method according to any of the above embodiments when the computer program is executed.
According to the data deleting device, when an object in an object storage bucket moves to a recycle bin, an index fragment is established for the object storage bucket, object information and deleting time of the object are stored in the index fragment in an omp key value pair mode, keys in the omp key value pair include the deleting time and object name of the object, the keys are sorted according to the deleting time, then the object to be deleted is determined according to the deleting time of the object, the object to be deleted and the omp key value pair corresponding to the object to be deleted are deleted together, all objects in the recycle bin do not need to be traversed, traversing operation of a large number of metadata pools is reduced, pressure of the metadata pools is reduced, cluster IO is reduced, and system stability is improved.
The following describes a computer-readable storage medium provided in an embodiment of the present application, and the computer-readable storage medium described below and the data deleting method described above may be referred to correspondingly.
A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the steps of the data deletion method according to any one of the above embodiments.
When an object in an object storage bucket moves to a recycle bin, an index fragment is established for the object storage bucket, object information and deletion time of the object are stored in the index fragment in an omp key value pair mode, keys in the omp key value pair include deletion time and object names of the object, the keys are sorted according to the deletion time, then the object to be deleted is determined according to the deletion time of the object, the object to be deleted and the omp key value pair corresponding to the object to be deleted are deleted together, all the objects in the recycle bin do not need to be traversed, traversal operation of a large number of metadata pools is reduced, pressure of the metadata pools is reduced, cluster IO is reduced, and system stability is improved.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The data deleting method, device, equipment and computer readable storage medium provided by the present application are described in detail above. The principles and embodiments of the present application are explained herein using specific examples, which are provided only to help understand the method and the core idea of the present application. It should be noted that, for those skilled in the art, it is possible to make several improvements and modifications to the present application without departing from the principle of the present application, and such improvements and modifications also fall within the scope of the claims of the present application.
Claims (11)
1. A method for deleting data, comprising:
when an object in an object storage bucket moves to a recycle bin, establishing an index fragment of the object storage bucket;
storing the object information and the deletion time of the object to the index fragment through an omap key value pair, wherein keys in the omap key value pair comprise the deletion time and an object name, and the keys are sorted according to the deletion time;
and determining an object to be deleted according to the deletion time, and deleting the object to be deleted and the corresponding omap key value pair.
2. The data deletion method of claim 1, wherein said determining an object to be deleted according to said deletion time and deleting said object to be deleted and said omap key-value pair corresponding thereto comprises:
obtaining the retention time and the current time of the object in the recycle bin;
determining an expiration time node of the object in the recycle bin according to the current time and the retention time;
traversing the key on the index tile before the expiration time node with a thread;
and deleting the object to be deleted and the omap key-value pair corresponding to the traversed key.
3. The data deletion method of claim 2, wherein the scan cycle of the thread is in the order of minutes.
4. The data deletion method of claim 2, wherein before said deleting the object to be deleted and the omap key-value pair corresponding to the traversed key, further comprising:
judging whether verification information in the index fragment is consistent with target verification information in the head object, wherein the verification information is located in object metadata of value in the omp key value pair, the target verification information is located in metadata of the head object, and the verification information corresponds to the traversed key;
and if the verification information is consistent with the target verification information, executing a step of deleting the object to be deleted and the omap key value pair corresponding to the traversed key.
5. The data deletion method of claim 2, wherein said obtaining a retention time of the object within the recycle bin comprises:
acquiring bucket metadata;
determining the retention time from the bucket metadata.
6. The data deletion method of claim 1, wherein said determining an object to be deleted according to said deletion time and deleting said object to be deleted and said omap key-value pair corresponding thereto comprises:
acquiring the object name and the bucket name of the object to be deleted;
determining the deletion time of the object to be deleted in the head object according to the object name and the bucket name;
determining corresponding target deletion time in the index fragment according to the deletion time in the head object;
and deleting the object to be deleted and the omap key value pair corresponding to the target deletion time.
7. The data deletion method of claim 6, wherein said determining the deletion time for the object to be deleted to be in a head object based on the object name and the bucket name comprises:
determining the head object according to the object name and the bucket name;
determining metadata for the object from the head object;
determining the deletion time located in the head object from the metadata.
8. The data deletion method of any one of claims 1 to 7, wherein when multiple recycle bins delete an object simultaneously, the creating the index shard of the object bucket comprises:
and establishing the index fragment corresponding to each object bucket.
9. A data deletion apparatus, comprising:
the system comprises an establishing module, a searching module and a judging module, wherein the establishing module is used for establishing an index fragment of an object storage bucket when an object in the object storage bucket moves to a recycle bin;
the storage module is used for storing the object information and the deletion time of the object to the index fragment through an omap key value pair, wherein keys in the omap key value pair comprise the deletion time and an object name, and the keys are sorted according to the deletion time;
and the deleting module is used for determining the object to be deleted according to the deleting time and deleting the object to be deleted and the corresponding omap key value pair.
10. A data deletion apparatus, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the data deletion method of any one of claims 1 to 8 when executing the computer program.
11. A computer-readable storage medium, having stored thereon a computer program which, when being executed by a processor, carries out the steps of the data deletion method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210606581.8A CN114968111A (en) | 2022-05-31 | 2022-05-31 | Data deleting method, device, equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210606581.8A CN114968111A (en) | 2022-05-31 | 2022-05-31 | Data deleting method, device, equipment and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114968111A true CN114968111A (en) | 2022-08-30 |
Family
ID=82958483
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210606581.8A Pending CN114968111A (en) | 2022-05-31 | 2022-05-31 | Data deleting method, device, equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114968111A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115934363A (en) * | 2023-03-07 | 2023-04-07 | 苏州浪潮智能科技有限公司 | Method, system and device for emptying bucket of distributed object storage system |
| CN116089068A (en) * | 2022-12-22 | 2023-05-09 | 博上(山东)网络科技有限公司 | Resource recycling method, system, equipment and readable storage medium |
-
2022
- 2022-05-31 CN CN202210606581.8A patent/CN114968111A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116089068A (en) * | 2022-12-22 | 2023-05-09 | 博上(山东)网络科技有限公司 | Resource recycling method, system, equipment and readable storage medium |
| CN115934363A (en) * | 2023-03-07 | 2023-04-07 | 苏州浪潮智能科技有限公司 | Method, system and device for emptying bucket of distributed object storage system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| RU2672719C2 (en) | Extended storage without locks for multiple access methods | |
| US9183236B2 (en) | Low level object version tracking using non-volatile memory write generations | |
| US8626717B2 (en) | Database backup and restore with integrated index reorganization | |
| KR101084816B1 (en) | Systems and methods for versioning based triggers | |
| JP4117265B2 (en) | Method and system for managing file system versions | |
| CN114968111A (en) | Data deleting method, device, equipment and computer readable storage medium | |
| US10013312B2 (en) | Method and system for a safe archiving of data | |
| US11226934B2 (en) | Storage system garbage collection and defragmentation | |
| US7769719B2 (en) | File system dump/restore by node numbering | |
| US20060224550A1 (en) | Policy Based Resource Management for Legacy Data | |
| CN107943718A (en) | A kind of method and apparatus for clearing up cache file | |
| CN107066349A (en) | A kind of method and system of cluster file system data protection | |
| US11841826B2 (en) | Embedded reference counts for file clones | |
| US11822806B2 (en) | Using a secondary storage system to implement a hierarchical storage management plan | |
| CN110008174A (en) | A kind of catalogue deriving method, system and relevant apparatus | |
| CN102819570B (en) | A kind of data access method, Apparatus and system | |
| CN108959614A (en) | A kind of snapshot management method, system, device, equipment and readable storage medium storing program for executing | |
| CN114253950B (en) | Method and device for managing database | |
| CN103645967A (en) | Read-only snapshot rollback method and device | |
| Zhang et al. | Bi-directional Log-Structured Merge Tree | |
| Zhu | Research on data recovery technology based on YAFFS2 file system's hash linked list and time series analysis | |
| US20230385304A1 (en) | Snapshots for distributed object-based datastores | |
| CN117009439B (en) | Data processing method, device, electronic equipment and storage medium | |
| US11100088B1 (en) | Bulk file verification | |
| CN110851445B (en) | Method for safely storing data based on block chain technology |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |