CN109086220B - Method and device for recycling storage space - Google Patents

Method and device for recycling storage space Download PDF

Info

Publication number
CN109086220B
CN109086220B CN201810643419.7A CN201810643419A CN109086220B CN 109086220 B CN109086220 B CN 109086220B CN 201810643419 A CN201810643419 A CN 201810643419A CN 109086220 B CN109086220 B CN 109086220B
Authority
CN
China
Prior art keywords
file
posix
storage space
recovered
small
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201810643419.7A
Other languages
Chinese (zh)
Other versions
CN109086220A (en
Inventor
李杰辉
王文涛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing QIYI Century Science and Technology Co Ltd
Original Assignee
Beijing QIYI Century Science and Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing QIYI Century Science and Technology Co Ltd filed Critical Beijing QIYI Century Science and Technology Co Ltd
Priority to CN201810643419.7A priority Critical patent/CN109086220B/en
Publication of CN109086220A publication Critical patent/CN109086220A/en
Application granted granted Critical
Publication of CN109086220B publication Critical patent/CN109086220B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G06F12/0223—User address space allocation, e.g. contiguous or non contiguous base addressing
    • G06F12/023—Free address space management
    • G06F12/0253—Garbage collection, i.e. reclamation of unreferenced memory
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The embodiment of the invention provides a method and a device for recycling storage space, wherein the method comprises the following steps: acquiring POSIX files of a portable operating system interface to be processed, when a deletion instruction is received, marking a first target small file in the POSIX files as deleted, and recovering the storage space of the first target small file to obtain pre-recovered POSIX files, when the pre-recovered POSIX files meet a preset recovery condition, judging whether each small file in the pre-recovered POSIX files is marked as deleted, if the small file is not marked as deleted, copying the small file to the preset storage space, and recovering the storage space of the small file. Based on the processing, the file space generated by deleting the small files can be combined, the number of file holes is reduced, and further the influence of deleting the small files on the system performance of the server is reduced.

Description

Method and device for recycling storage space
Technical Field
The present invention relates to the field of data storage technologies, and in particular, to a method and apparatus for recovering storage space.
Background
With the rapid development of the internet, a huge number (hundreds of millions) of files such as pictures and documents are generated in the internet. These files are typically small (typically below 100 KB), and may be referred to as small files. The method of merging and storing is to store the content of huge small files into POSIX (Portable Operating System Interface of UNIX, portable operating system interface) files. A common practice currently to reclaim these small file storage spaces in POSIX files is to use Copy-Commit methods. For example, hasstack of Facebook, ambry of LinkedIn and TFS of panning (Taobao File System, panning file system).
In the prior art, after the server stores the content of the small files with huge quantity into the POSIX file, the offset of the small files in the large files, namely index information, is required to be recorded, and then the index information is added into the index file. When the server receives a deletion request for a small file, the small file can be deleted from the POSIX file according to the index information of the small file, and the storage space occupied by the small file can be recovered.
However, the inventors have found that in implementing the present invention, the prior art has at least the following problems:
as each small file is deleted, the storage space corresponding to the small file becomes a file hole. When more small files in the POSIX file are deleted, more file holes exist in the POSIX file. The server needs to allocate system resources to maintain each file hole, which in turn reduces the system performance of the server.
Disclosure of Invention
The embodiment of the invention aims to provide a method and a device for recycling storage space, which can reduce the influence on the system performance of a server. The specific technical scheme is as follows:
in order to achieve the above object, an embodiment of the present invention discloses a method for reclaiming a storage space, the method comprising:
acquiring a POSIX file of a portable operating system interface to be processed;
when a deleting instruction is received, marking a first target small file in the POSIX file as deleted, and recovering the storage space of the first target small file to obtain a pre-recovered POSIX file;
when the pre-recovery POSIX file meets a preset recovery condition, judging whether each small file in the pre-recovery POSIX file is marked as deleted or not;
if the small file is not identified as deleted, the small file is copied to a preset storage space and the storage space of the small file is reclaimed.
Optionally, the determining, for each small file in the pre-reclaimed POSIX file, whether the small file is identified as deleted includes:
segmenting the pre-recovered POSIX file according to a preset mode to obtain a plurality of file segments, wherein the file segments comprise a plurality of small files;
calculating the hole density of each file segment in the plurality of file segments, wherein the hole density represents the ratio of the recovered storage space in the file segment to the storage space in the file segment;
sorting the file segments according to the hole density of each file segment in the file segments and the size of the hole density;
and acquiring small files contained in the file segments to be processed according to the corresponding sequence of the file segments, and judging whether each small file is marked as deleted or not according to each acquired small file.
Optionally, the pre-recycling POSIX file meets a preset recycling condition, including:
the read-write delay of the pre-recovered POSIX file is larger than a preset threshold;
or,
reaching the preset recovery time.
Optionally, the segmenting the pre-reclaimed POSIX file according to a preset mode to obtain a plurality of file segments includes:
determining a segmentation mode corresponding to the pre-recovered POSIX file according to the corresponding relation between the storage space occupied by the pre-recovered POSIX file and the segmentation mode, and segmenting the pre-recovered POSIX file according to the determined segmentation mode to obtain a plurality of file segments.
Optionally, determining a segmentation mode corresponding to the pre-recovered POSIX file according to a corresponding relationship between a storage space occupied by the pre-stored pre-recovered POSIX file and the segmentation mode, and segmenting the pre-recovered POSIX file according to the determined segmentation mode to obtain a plurality of file segments, where the method includes:
when the storage space occupied by the pre-recovered POSIX file is in a first preset storage space range, segmenting the pre-recovered POSIX file according to a preset file segment storage space to obtain a plurality of file segments.
Optionally, determining a segmentation mode corresponding to the pre-recovered POSIX file according to a corresponding relationship between a storage space occupied by the pre-stored pre-recovered POSIX file and the segmentation mode, and segmenting the pre-recovered POSIX file according to the determined segmentation mode to obtain a plurality of file segments, where the method includes:
when the storage space occupied by the pre-recovered POSIX file is in a second preset storage space range, segmenting the pre-recovered POSIX file according to the preset number of file segments to obtain a plurality of file segments.
In order to achieve the above object, an embodiment of the present invention provides an apparatus for reclaiming a storage space, including:
the acquisition module is used for acquiring a POSIX file of the portable operating system interface to be processed;
the first recycling module is used for marking a first target small file in the POSIX file as deleted when a deleting instruction is received, recycling the storage space of the first target small file, and obtaining a pre-recycled POSIX file;
the judging module is used for judging whether each small file in the pre-recovered POSIX file is marked as deleted or not according to each small file in the pre-recovered POSIX file when the pre-recovered POSIX file meets a preset recovery condition;
and the second recycling module is used for copying the small file to a preset storage space and recycling the storage space of the small file if the small file is not identified to be deleted.
Optionally, the judging module includes:
the segmentation sub-module is used for segmenting the pre-recovered POSIX file according to a preset mode to obtain a plurality of file segments, wherein the file segments comprise a plurality of small files;
the computing submodule is used for computing the hole density of each file segment in the plurality of file segments, wherein the hole density represents the ratio of the recovered storage space in the file segment to the storage space in the file segment;
the sorting sub-module is used for sorting the file segments according to the hole density of each file segment in the file segments and the hole density;
and the judging sub-module is used for acquiring small files contained in the file segments to be processed according to the corresponding sequence of the file segments, and judging whether each small file is marked as deleted or not according to each acquired small file.
Optionally, the pre-recycling POSIX file meets a preset recycling condition, including:
the read-write delay of the pre-recovered POSIX file is larger than a preset threshold;
or,
reaching the preset recovery time.
Optionally, the segmentation submodule is specifically configured to determine a segmentation mode corresponding to the pre-recovered POSIX file according to a corresponding relationship between a storage space occupied by the pre-recovered POSIX file and the segmentation mode, and segment the pre-recovered POSIX file according to the determined segmentation mode to obtain a plurality of file segments.
Optionally, the segmentation submodule is specifically configured to segment the pre-recovered POSIX file according to a preset file segment storage space when the storage space occupied by the pre-recovered POSIX file is within a first preset storage space range, so as to obtain a plurality of file segments.
Optionally, the segmentation submodule is specifically configured to segment the pre-recovered POSIX file according to a preset number of file segments to obtain a plurality of file segments when a storage space occupied by the pre-recovered POSIX file is within a second preset storage space range.
In order to achieve the above object, the embodiment of the present invention further discloses an electronic device, where the electronic device includes a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication between each other through the communication bus:
a memory for storing a computer program;
a processor for performing the method steps as described in any one of the above when executing a program stored on a memory.
In a fourth aspect, to achieve the above object, an embodiment of the present invention further discloses a computer-readable storage medium, in which a computer program is stored, which when executed by a processor, implements the method steps as described in any of the above.
In yet another aspect of the invention, embodiments of the invention also provide a computer program product comprising instructions which, when run on a computer, cause the computer to perform any of the method steps described above.
According to the method and the device for recovering the storage space, the POSIX file to be processed can be obtained, when a deleting instruction is received, the first target small file in the POSIX file is marked as deleted, the storage space of the first target small file is recovered, the pre-recovered POSIX file is obtained, when the pre-recovered POSIX file meets the preset recovering condition, whether each small file in the pre-recovered POSIX file is marked as deleted or not is judged, if the small file is not marked as deleted, the small file is copied to the preset storage space, and the storage space of the small file is recovered. Based on the processing, the file holes generated by deleting the small files can be combined, the number of the file holes is reduced, and further the influence of deleting the small files on the system performance of the server is reduced.
Of course, it is not necessary for any one product or method of practicing the invention to achieve all of the advantages set forth above at the same time.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.
FIG. 1 is a flowchart of a method for reclaiming storage space according to an embodiment of the present invention;
FIG. 2 is a block diagram of a POSIX file according to an embodiment of the present invention;
FIG. 3 is a block diagram of a POSIX file according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a process for copying a pre-reclaimed POSIX file according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a process for recycling pre-recycled POSIX files according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of a process for segmenting a pre-reclaimed POSIX file according to an embodiment of the present invention;
FIG. 7 is a block diagram of an apparatus for reclaiming storage space according to an embodiment of the present invention;
fig. 8 is a block diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be described below with reference to the accompanying drawings in the embodiments of the present invention.
In the prior art, when a small file in a POSIX file is deleted, a storage space corresponding to the small file becomes a file hole. When more small files in the POSIX file are deleted, more file holes exist in the POSIX file. The server needs to allocate system resources to maintain each file hole, which in turn reduces the system performance of the server.
The embodiment of the invention provides a method and a device for recovering storage space, which can be applied to a server, and when the storage space is recovered by the server, file holes generated by deleting small files can be combined, so that the number of the file holes is reduced, and the influence of deleting the small files on the system performance of the server is further reduced.
Referring to fig. 1, fig. 1 is a flowchart of a method for reclaiming storage space according to an embodiment of the present invention, where the method may include the following steps:
s100, acquiring a POSIX file of the portable operating system interface to be processed.
Wherein the POSIX file is composed of a large number of small files. During the running process, the application program can generate a huge number (hundreds of millions) of files such as pictures and documents, and the files are usually smaller (generally below 100 KB) and can be called small files. The server may store these small files in the POSIX file while recording the offset of the small file in the POSIX file, i.e., the index information of the small file, in the index file. When an application program needs to read a small file, the server firstly searches the offset of the small file in the index file, then obtains the small file according to the searched offset, and sends the small file to the application program.
In an implementation, when an application program needs to process a POSIX file, the application program may send a small file processing request to a server, and after receiving the small file processing request, the server may obtain the POSIX file to which the small file belongs, obtain the small file from the POSIX file, and send the small file to the application program. For example, when a user uses a chat application of a terminal, the user may send a read request of a small file (e.g., a picture) to a server through the chat application, and after receiving the read request, the server may obtain the small file from the POSIX file and send the small file to the chat application. After receiving the doclet, the chat application may present the user with, for example, a contact avatar, the chat's picture content, and remark information. As shown in fig. 2, the POSIX file may include a plurality of small files, and may further include available storage space, where the available storage space is free storage space for storing small files.
S110, when a deleting instruction is received, the first target small file in the POSIX file is marked as deleted, and the storage space of the first target small file is recovered, so that the pre-recovered POSIX file is obtained.
In an implementation, when a user needs to delete a small file (a first target file), the user may select the small file (i.e., the first target small file) that needs to be deleted, such as a certain picture or a certain document, in the application. The user can click on a deletion option in the application program, the application program can send a deletion instruction to the server, and the server can receive a deletion request of the corresponding first target small file. Then, the server may search index information of the first target small file in index information corresponding to the POSIX file, and further add marking information in the index information of the first target small file to mark the first target small file as deleted. Then, the server can release the storage space occupied by the first target small file, so as to recycle the storage space of the first target small file. The POSIX file after deleting the first target small file is a pre-reclaimed POSIX file, as shown in FIG. 3, when deleting a certain small file, the occupied storage space of the small file is reclaimed to form a file hole.
It should be noted that, after the storage space occupied by a small file is reclaimed, the storage space can be used to store other new small files. For example, when a user deletes some picture files, the server may reclaim the storage space of those picture files. When a user newly adds a part of the picture file, the server may store the newly added part of the picture file in the reclaimed storage space.
S120, when the pre-recovery POSIX file meets the preset recovery condition, judging whether each small file in the pre-recovery POSIX file is marked as deleted or not.
When the number of small files deleted by the user is large and the storage space of the small files is discontinuous, a large number of file holes exist in the pre-reclaimed POSIX file. Because the server needs to allocate system resources to maintain each file hole, these file holes can affect the time that the server reads small files in the pre-reclaimed POSIX file. Based on this, the preset reclamation condition may be that the read-write delay of the preset reclamation POSIX file is greater than a preset threshold. For example, the preset threshold may be 100ms, and when the read-write delay of the server on the pre-reclaimed POSIX file is greater than 100ms, the server may determine that the POSIX file meets the preset reclaiming condition. Alternatively, the technician may empirically set a fixed recovery time as a preset recovery condition for the pre-recovery POSIX file.
In practice, when the preset recycling condition of the POSIX file is reached, the server may perform secondary recycling on the small file in the pre-recycled POSIX file. The server can copy small files in the pre-reclaimed POSIX file from the storage space address of the beginning of the pre-reclaimed POSIX file, and can recycle a certain section of storage space in the storage space of the pre-reclaimed POSIX file. Specifically, for each doclet in the pre-reclaimed POSIX file, the server may determine whether the doclet is identified as deleted.
S130, if the small file is not identified as deleted, copying the small file to a preset storage space, and recycling the storage space of the small file.
In an implementation, for a small file, if the server determines that the small file (which may be referred to as a second target small file) is not identified as deleted, the server may copy the second target small file into a preset available memory space and release the memory space occupied by the second target small file in the pre-reclaimed POSIX file. The preset available storage space may be an available storage space at the tail end of the pre-reclaimed POSIX file, or may be other available storage space that is not utilized in the server. Based on the above processing, the server may incorporate discrete file holes, thereby reducing the number of file holes. Fig. 4 is a schematic diagram of a processing procedure of copying a pre-reclaimed POSIX file according to an embodiment of the present invention.
As shown in fig. 5, a schematic diagram of a process for recycling pre-recycled POSIX files is provided for the present invention, and specific steps may include S121, S122, S123, and S124.
S121, segmenting the pre-recovered POSIX file according to a preset segmentation mode to obtain a plurality of file segments.
Wherein each file segment may include a plurality of small files.
In implementation, as shown in fig. 6, a schematic diagram of a processing procedure for segmenting a pre-reclaimed POSIX file is provided in an embodiment of the present invention. The server can store a preset segmentation mode, and the server can segment the pre-recovered POSIX file according to the preset segmentation mode to obtain a plurality of file segments. The segmentation mode can be to segment according to the storage space of the file segments, or segment according to the number of the file segments. For example, the pre-reclaimed POSIX file size is 10G, if the segments are made with a storage space of 2G for each file segment, the server can divide the pre-reclaimed POSIX file into 5 segments; if the segmentation is performed according to the total number of file segments being 2 segments, the server can divide the pre-reclaimed POSIX file into two file segments with 5G storage space.
S122, for each file segment in the plurality of file segments, calculating the hole density of the file segment.
The hole density may represent a ratio of the reclaimed memory space in the file segment to the memory space in the file segment.
In implementation, a portion of the reclaimed storage space exists in each file segment, and the server can calculate the ratio of the data volume of the reclaimed storage space to the data volume of the storage space of the file segment to obtain the hole density of the file segment. For example, a file segment has a size of 5G, where the recovered storage space is 2G, and the server calculates the hole density of the file segment to be 2G/5 g=0.4.
S123, sorting the file segments according to the size of the hole density according to the hole density of each file segment.
In implementation, the server establishes index information for each file segment according to the hole density of each file segment obtained by calculation from the big to the small.
S124, according to the sequence corresponding to the file segments, acquiring small files contained in the file segments to be processed, and judging whether each small file is marked as deleted or not according to each acquired small file.
In implementation, the server may process each file segment sequentially in order of the hole density from high to low. For each doclet in a file segment to be processed, the server may determine whether the doclet is identified as deleted, and if so, the server may reclaim the storage space of the doclet.
The invention provides a method for segmenting a pre-recovered POSIX file, which comprises the following specific processing steps:
determining a segmentation mode corresponding to the pre-recovered POSIX file according to the corresponding relation between the storage space occupied by the pre-stored pre-recovered POSIX file and the segmentation mode, and segmenting the pre-recovered POSIX file according to the determined segmentation mode to obtain a plurality of file segments.
In implementation, the server may store in advance a correspondence between a storage space occupied by the pre-reclaimed POSIX file and a segmentation mode, where the correspondence may be set by a technician according to experience. The server can query in the corresponding relation according to the storage space occupied by the pre-recovered POSIX file to obtain the segmentation mode of the pre-recovered POSIX file. Then, the server segments the pre-recovered POSIX file according to the segment mode obtained by query to obtain a plurality of file segments. For example, when the pre-reclaimed POSIX file is 10G, the server may segment the pre-reclaimed POSIX file by 1G per file segment, resulting in 10 file segments. When the pre-reclaimed POSIX file is 20G, the server can pre-reclaim the POSIX file for segmentation according to a mode of 5 file segments, so that 5 file segments with the storage space of 4G are obtained.
The invention provides a method for segmenting the pre-recovered POSIX file, which is concretely characterized by the following two cases.
In the first case, when the storage space occupied by the pre-recovered POSIX file is in the first preset storage space range, segmenting the pre-recovered POSIX file according to the preset file segment storage space to obtain a plurality of file segments.
The first preset storage space range may be set empirically by a technician.
In implementation, after determining the size of the storage space occupied by the pre-recovered POSIX file, the server may query in the corresponding relationship between the storage space occupied by the pre-recovered POSIX file and the segmentation method, determine the storage space range corresponding to the storage space occupied by the pre-recovered POSIX file, and further determine the segmentation method corresponding to the storage space range. If the storage space occupied by the pre-reclaimed POSIX file is within the first storage space range, the server may determine the segmentation manner to segment the storage space occupied by the file segments. The server may segment the pre-reclaimed POSIX file in this segmented fashion. For example, when the storage space corresponding to the storage space occupied by the pre-recovered POSIX file is 10-20G, the server may segment the pre-recovered POSIX file according to the file segment size of 1G of each segment, to obtain multiple file segments.
And secondly, when the storage space occupied by the pre-recovered POSIX file is in a second preset storage space range, segmenting the pre-recovered POSIX file according to the preset number of file segments to obtain a plurality of file segments.
The lower limit of the second preset storage space range may be greater than the upper limit of the first preset storage space range in one case, and in particular, the second preset storage space range may be set by a technician according to experience.
In implementation, after determining the size of the storage space occupied by the pre-recovered POSIX file, the server may query in the corresponding relationship between the storage space occupied by the pre-recovered POSIX file and the segmentation method, determine the storage space range corresponding to the storage space occupied by the pre-recovered POSIX file, and further determine the segmentation method corresponding to the storage space range. If the storage space occupied by the pre-reclaimed POSIX is within the second storage space range, the server may determine the segmentation method to segment by the occupied storage space of the file segment. The server may segment the pre-reclaimed POSIX file in this segmented fashion. For example, when the storage space corresponding to the storage space occupied by the pre-recovered POSIX file is 20-100G, the server may segment the pre-recovered POSIX file according to the number of file segments being 5 segments, to obtain 5 file segments.
According to the method for recovering the storage space, the POSIX file of the portable operating system interface to be processed can be obtained, when a deleting instruction is received, the first target small file in the POSIX file is marked as deleted, the storage space of the first target small file is recovered, the pre-recovered POSIX file is obtained, when the pre-recovered POSIX file meets the preset recovering condition, whether each small file in the pre-recovered POSIX file is marked as deleted or not is judged, and if the small file is not marked as deleted, the small file is copied to the preset storage space, and the storage space of the small file is recovered. Based on the processing, the file holes generated by deleting the small files can be combined, the number of the file holes is reduced, and further the influence of deleting the small files on the system performance of the server is reduced.
Corresponding to the method embodiment of fig. 1, referring to fig. 7, fig. 7 is a block diagram of an apparatus for reclaiming storage space according to an embodiment of the present invention, where the apparatus may include:
the acquiring module 700 is configured to acquire a POSIX file of a portable operating system interface to be processed;
the first recycling module 710 is configured to identify, when a deletion instruction is received, a first target small file in the POSIX file as deleted, and recycle a storage space of the first target small file, to obtain a pre-recycled POSIX file;
a judging module 720, configured to judge, for each small file in the pre-reclaimed POSIX file, whether the small file is identified as deleted when the pre-reclaimed POSIX file meets a preset reclaiming condition;
and a second reclamation module 730, configured to copy the small file to a preset storage space and reclaim the storage space of the small file if the small file is not identified as deleted.
Optionally, the determining module 720 includes:
the segmentation sub-module is used for segmenting the pre-recovered POSIX file according to a preset mode to obtain a plurality of file segments, wherein the file segments comprise a plurality of small files;
the computing submodule is used for computing the hole density of each file segment in the plurality of file segments, wherein the hole density represents the ratio of the recovered storage space in the file segment to the storage space in the file segment;
the sorting sub-module is used for sorting the file segments according to the hole density of each file segment in the file segments and the hole density;
and the judging sub-module is used for acquiring small files contained in the file segments to be processed according to the corresponding sequence of the file segments, and judging whether each small file is marked as deleted or not according to each acquired small file.
Optionally, the pre-recycling POSIX file meets a preset recycling condition, including:
the read-write delay of the pre-recovered POSIX file is larger than a preset threshold;
or,
reaching the preset recovery time.
Optionally, the segmentation submodule is specifically configured to determine a segmentation mode corresponding to the pre-recovered POSIX file according to a corresponding relationship between a storage space occupied by the pre-recovered POSIX file and the segmentation mode, and segment the pre-recovered POSIX file according to the determined segmentation mode to obtain a plurality of file segments.
Optionally, the segmentation submodule is specifically configured to segment the pre-recovered POSIX file according to a preset file segment storage space when the storage space occupied by the pre-recovered POSIX file is within a first preset storage space range, so as to obtain a plurality of file segments.
Optionally, the segmentation submodule is specifically configured to segment the pre-recovered POSIX file according to a preset number of file segments to obtain a plurality of file segments when a storage space occupied by the pre-recovered POSIX file is within a second preset storage space range.
According to the device for recovering the storage space, disclosed by the embodiment of the invention, the POSIX file of the portable operating system interface to be processed can be obtained, when a deletion instruction is received, the first target small file in the POSIX file is marked as deleted, the storage space of the first target small file is recovered, the pre-recovered POSIX file is obtained, when the pre-recovered POSIX file meets the preset recovery condition, whether each small file in the pre-recovered POSIX file is marked as deleted or not is judged, and if the small file is not marked as deleted, the small file is copied to the preset storage space, and the storage space of the small file is recovered. Based on the processing, the file holes generated by deleting the small files can be combined, the number of the file holes is reduced, and further the influence of deleting the small files on the system performance of the server is reduced.
The embodiment of the present invention further provides an electronic device, as shown in fig. 8, including a processor 801, a communication interface 802, a memory 803, and a communication bus 804, where the processor 801, the communication interface 802, and the memory 803 complete communication with each other through the communication bus 804,
a memory 803 for storing a computer program;
the processor 801, when executing the program stored in the memory 803, implements the following steps:
acquiring a POSIX file of a portable operating system interface to be processed;
when a deleting instruction is received, marking a first target small file in the POSIX file as deleted, and recovering the storage space of the first target small file to obtain a pre-recovered POSIX file;
when the pre-recovery POSIX file meets a preset recovery condition, judging whether each small file in the pre-recovery POSIX file is marked as deleted or not;
if the small file is not identified as deleted, the small file is copied to a preset storage space and the storage space of the small file is reclaimed.
The communication bus mentioned by the above electronic device may be a peripheral component interconnect standard (Peripheral Component Interconnect, abbreviated as PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, abbreviated as EISA) bus, or the like. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.
The communication interface is used for communication between the electronic device and other devices.
The memory may include random access memory (Random Access Memory, RAM) or non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor.
The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; but also digital signal processors (Digital Signal Processing, DSP for short), application specific integrated circuits (Application Specific Integrated Circuit, ASIC for short), field-programmable gate arrays (Field-Programmable Gate Array, FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
In yet another embodiment of the present invention, a computer readable storage medium having instructions stored therein, which when run on a computer, cause the computer to perform the method of any of the above embodiments is also provided.
In a further embodiment of the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the method of any of the above embodiments.
In the embodiment of the invention, a POSIX file of a portable operating system interface to be processed can be obtained, when a deletion instruction is received, a first target small file in the POSIX file is marked as deleted, and the storage space of the first target small file is recovered, so that a pre-recovered POSIX file is obtained, when the pre-recovered POSIX file meets a preset recovery condition, whether each small file in the pre-recovered POSIX file is marked as deleted or not is judged, and if the small file is not marked as deleted, the small file is copied to the preset storage space, and the storage space of the small file is recovered. Based on the processing, the file holes generated by deleting the small files can be combined, the number of the file holes is reduced, and further the influence of deleting the small files on the system performance of the server is reduced.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present invention, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, by wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), etc.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for the apparatus, the electronic device, the computer readable storage medium, and the computer program product embodiments, the description is relatively simple, as relevant to the method embodiments being referred to in the section of the description of the method embodiments.
The foregoing description is only of the preferred embodiments of the present invention and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims (11)

1. A method of reclaiming storage space, the method comprising:
acquiring a POSIX file of a portable operating system interface to be processed;
when a deleting instruction is received, marking a first target small file in the POSIX file as deleted, and recovering the storage space of the first target small file to obtain a pre-recovered POSIX file;
when the pre-recovery POSIX file meets a preset recovery condition, judging whether each small file in the pre-recovery POSIX file is marked as deleted or not;
if the small file is not identified as deleted, copying the small file to a preset storage space, and recovering the storage space of the small file;
the determining, for each small file in the pre-reclaimed POSIX file, whether the small file is identified as deleted includes:
segmenting the pre-recovered POSIX file according to a preset mode to obtain a plurality of file segments, wherein the file segments comprise a plurality of small files;
calculating the hole density of each file segment in the plurality of file segments, wherein the hole density represents the ratio of the recovered storage space in the file segment to the storage space in the file segment;
sorting the file segments according to the hole density of each file segment in the file segments and the size of the hole density;
and acquiring small files contained in the file segments to be processed according to the corresponding sequence of the file segments, and judging whether each small file is marked as deleted or not according to each acquired small file.
2. The method of claim 1, wherein the pre-reclaimed POSIX file meets a pre-set reclamation condition comprising:
the read-write delay of the pre-recovered POSIX file is larger than a preset threshold;
or,
reaching the preset recovery time.
3. The method of claim 1, wherein segmenting the pre-reclaimed POSIX file in a preset manner results in a plurality of file segments, comprising:
determining a segmentation mode corresponding to the pre-recovered POSIX file according to the corresponding relation between the storage space occupied by the pre-recovered POSIX file and the segmentation mode, and segmenting the pre-recovered POSIX file according to the determined segmentation mode to obtain a plurality of file segments.
4. The method according to claim 3, wherein determining the segmentation method corresponding to the pre-reclaimed POSIX file according to the corresponding relation between the storage space occupied by the pre-reclaimed POSIX file and the segmentation method, and segmenting the pre-reclaimed POSIX file according to the determined segmentation method, to obtain a plurality of file segments, includes:
when the storage space occupied by the pre-recovered POSIX file is in a first preset storage space range, segmenting the pre-recovered POSIX file according to a preset file segment storage space to obtain a plurality of file segments.
5. The method according to claim 3, wherein determining the segmentation method corresponding to the pre-reclaimed POSIX file according to the corresponding relation between the storage space occupied by the pre-reclaimed POSIX file and the segmentation method, and segmenting the pre-reclaimed POSIX file according to the determined segmentation method, to obtain a plurality of file segments, includes:
when the storage space occupied by the pre-recovered POSIX file is in a second preset storage space range, segmenting the pre-recovered POSIX file according to the preset number of file segments to obtain a plurality of file segments.
6. An apparatus for reclaiming storage space, the apparatus comprising:
the acquisition module is used for acquiring a POSIX file of the portable operating system interface to be processed;
the first recycling module is used for marking a first target small file in the POSIX file as deleted when a deleting instruction is received, recycling the storage space of the first target small file, and obtaining a pre-recycled POSIX file;
the judging module is used for judging whether each small file in the pre-recovered POSIX file is marked as deleted or not according to each small file in the pre-recovered POSIX file when the pre-recovered POSIX file meets a preset recovery condition;
the second recovery module is used for copying the small file to a preset storage space and recovering the storage space of the small file if the small file is not marked as deleted;
the judging module comprises:
the segmentation sub-module is used for segmenting the pre-recovered POSIX file according to a preset mode to obtain a plurality of file segments, wherein the file segments comprise a plurality of small files;
the computing submodule is used for computing the hole density of each file segment in the plurality of file segments, wherein the hole density represents the ratio of the recovered storage space in the file segment to the storage space in the file segment;
the sorting sub-module is used for sorting the file segments according to the hole density of each file segment in the file segments and the hole density;
and the judging sub-module is used for acquiring small files contained in the file segments to be processed according to the corresponding sequence of the file segments, and judging whether each small file is marked as deleted or not according to each acquired small file.
7. The apparatus of claim 6, wherein the pre-reclaimed POSIX file satisfies a pre-set reclamation condition comprising:
the read-write delay of the pre-recovered POSIX file is larger than a preset threshold;
or,
reaching the preset recovery time.
8. The device of claim 6, wherein the segmentation submodule is specifically configured to determine a segmentation mode corresponding to a pre-recovered POSIX file according to a corresponding relationship between a storage space occupied by the pre-recovered POSIX file and the segmentation mode, and segment the pre-recovered POSIX file according to the determined segmentation mode to obtain a plurality of file segments.
9. The apparatus of claim 8, wherein the segmentation submodule is specifically configured to segment the pre-reclaimed POSIX file according to a preset file segment storage space to obtain a plurality of file segments when the storage space occupied by the pre-reclaimed POSIX file is within a first preset storage space range.
10. The apparatus of claim 8, wherein the segmentation submodule is specifically configured to segment the pre-reclaimed POSIX file according to a preset number of file segments to obtain a plurality of file segments when a storage space occupied by the pre-reclaimed POSIX file is within a second preset storage space range.
11. The electronic equipment is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are communicated with each other through the communication bus;
a memory for storing a computer program;
a processor for carrying out the method steps of any one of claims 1-5 when executing a program stored on a memory.
CN201810643419.7A 2018-06-21 2018-06-21 Method and device for recycling storage space Active CN109086220B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810643419.7A CN109086220B (en) 2018-06-21 2018-06-21 Method and device for recycling storage space

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810643419.7A CN109086220B (en) 2018-06-21 2018-06-21 Method and device for recycling storage space

Publications (2)

Publication Number Publication Date
CN109086220A CN109086220A (en) 2018-12-25
CN109086220B true CN109086220B (en) 2023-04-28

Family

ID=64840108

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810643419.7A Active CN109086220B (en) 2018-06-21 2018-06-21 Method and device for recycling storage space

Country Status (1)

Country Link
CN (1) CN109086220B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112328549A (en) * 2020-10-29 2021-02-05 无锡先进技术研究院 Small file storage method, electronic device and storage medium
CN112925745B (en) * 2021-03-25 2023-08-08 北京奇艺世纪科技有限公司 Data cleaning method and device, electronic equipment and computer readable storage medium

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100235365A1 (en) * 2009-03-13 2010-09-16 Newby Jr Marvon M PS9110 Linear time sorting and constant time searching algorithms
CN105138282A (en) * 2015-08-06 2015-12-09 上海七牛信息技术有限公司 Storage space recycling method and storage system
CN106446044B (en) * 2016-08-31 2020-08-11 北京小米移动软件有限公司 Storage space recovery method and device
CN108090128B (en) * 2017-11-16 2021-11-26 北京奇艺世纪科技有限公司 Recovery method and device for merged storage space and electronic equipment

Also Published As

Publication number Publication date
CN109086220A (en) 2018-12-25

Similar Documents

Publication Publication Date Title
CN107786638B (en) Data processing method, device and system
US11316925B2 (en) Video data storage method and device in cloud storage system
CN109241003B (en) File management method and device
CN113435328B (en) Video clip processing method and device, electronic equipment and readable storage medium
US11409766B2 (en) Container reclamation using probabilistic data structures
CN109086220B (en) Method and device for recycling storage space
CN111163072A (en) Method and device for determining characteristic value in machine learning model and electronic equipment
CN111782707A (en) Data query method and system
CN111191591B (en) Watermark detection and video processing method and related equipment
CN114995759A (en) Barrel fragment processing method, device, equipment and medium
CN106919554B (en) Method and device for identifying invalid words in document
CN110083482B (en) Method and device for performing erasure code processing on file storage system and electronic equipment
CN108090128B (en) Recovery method and device for merged storage space and electronic equipment
CN111177082A (en) PDF file duplicate removal storage method and system
CN110990611B (en) Picture caching method and device, electronic equipment and storage medium
CN112417259B (en) Media resource processing method, device, equipment and storage medium
CN109871361B (en) File uploading method and device, electronic equipment and readable storage medium
CN115080504A (en) File management method, terminal and storage medium
CN111159438B (en) Data storage and retrieval method, electronic device and storage medium
CN113419792A (en) Event processing method and device, terminal equipment and storage medium
CN112783843A (en) Data reading method and device and electronic equipment
CN111104528A (en) Picture obtaining method and device and client
CN111708908A (en) Video tag adding method and device, electronic equipment and computer-readable storage medium
CN112486404B (en) Method, apparatus and computer program product for managing memory blocks
CN117729176B (en) Method and device for aggregating application program interfaces based on network address and response body

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant