CN112749135A - Method, apparatus and computer program product for balancing storage space of a file system - Google Patents

Method, apparatus and computer program product for balancing storage space of a file system Download PDF

Info

Publication number
CN112749135A
CN112749135A CN201911046051.7A CN201911046051A CN112749135A CN 112749135 A CN112749135 A CN 112749135A CN 201911046051 A CN201911046051 A CN 201911046051A CN 112749135 A CN112749135 A CN 112749135A
Authority
CN
China
Prior art keywords
storage space
metadata
proportion
threshold
file system
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201911046051.7A
Other languages
Chinese (zh)
Other versions
CN112749135B (en
Inventor
赵朝俊
张扬
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
EMC Corp
Original Assignee
EMC IP Holding Co LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by EMC IP Holding Co LLC filed Critical EMC IP Holding Co LLC
Priority to CN201911046051.7A priority Critical patent/CN112749135B/en
Priority to US16/877,910 priority patent/US11681447B2/en
Publication of CN112749135A publication Critical patent/CN112749135A/en
Application granted granted Critical
Publication of CN112749135B publication Critical patent/CN112749135B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0628Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0629Configuration or reconfiguration of storage systems
    • G06F3/0631Configuration or reconfiguration of storage systems by allocating resources to storage systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/16File or folder operations, e.g. details of user interfaces specifically adapted to file systems
    • G06F16/164File meta data generation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/17Details of further file system functions
    • G06F16/1727Details of free space management performed by the file system
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/14Details of searching files based on file metadata
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/17Details of further file system functions
    • G06F16/172Caching, prefetching or hoarding of files
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0602Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0614Improving the reliability of storage systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0628Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0653Monitoring storage devices or systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0628Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0659Command handling arrangements, e.g. command buffers, queues, command scheduling
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0668Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671In-line storage system
    • G06F3/0673Single storage device

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Library & Information Science (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

Embodiments of the present disclosure provide a method, apparatus, and computer program product for balancing storage space of a file system. In one embodiment, a determination is made whether a first allocated proportion of a first metadata storage space for a first type of metadata of one or more types of metadata of a file system is greater than a first threshold proportion. If the first allocation ratio is greater than a first threshold ratio, storage space is reserved in the storage space of the file system for the first type of metadata.

Description

Method, apparatus and computer program product for balancing storage space of a file system
Technical Field
Embodiments of the present disclosure relate generally to computer technology and, more particularly, relate to a method, apparatus, and computer program product for balancing storage space of a file system.
Background
Both metadata and data are widely used in modern file systems. There are different types of metadata that describe various characteristics of a file. An inode (inode) is an important type of metadata, and each inode corresponds to a file system object in a file system and is used for storing file information such as access rights, owner, timestamp and storage location of a file. In addition to inodes, there are other types of metadata in a file system. For example, the virtual block may serve as a metadata block for built-in data compression and deduplication.
For a thin file system, storage space is allocated on demand. However, the storage space for some metadata (e.g., inodes) is statically allocated when the file system is initially created. However, as the storage space allocation of the file system changes following different client workloads, the data layout may have changed significantly. For example, if a large number of small files and directories are created, the consumption rate of the inode block will be fast. The data block consumption rate is relatively slow. Therefore, there will be an imbalance between the storage space of the inode and the data block. As time passes, the situation becomes worse.
Disclosure of Invention
In general, embodiments of the present disclosure propose methods, apparatuses, and computer program products for balancing storage space of a file system.
In a first aspect, embodiments of the present disclosure provide a method for balancing storage space of a file system. In the method, it is determined whether a first allocation proportion of a first metadata storage space of a first type of metadata of the one or more types of metadata of the file system is greater than a first threshold proportion. If the first allocation proportion is higher than the first threshold proportion, reserving storage space in the storage space of the file system for the first type of metadata.
In a second aspect, embodiments of the present disclosure provide an apparatus for balancing storage space of a file system. The apparatus includes a processor and a memory storing computer-executable instructions. The computer-executable instructions, when executed by the processor, cause the device to perform acts comprising: determining whether a first allocation proportion of a first metadata storage space for a first type of metadata of the one or more types of metadata of the file system is greater than a first threshold proportion; and in accordance with a determination that the first allocation percentage is above the first threshold percentage, reserving storage space in the storage space of the file system for the first type of metadata.
In a third aspect, embodiments of the present disclosure provide a computer program product. The computer program product is tangibly stored on a non-transitory computer-readable medium and includes machine-executable instructions. The machine executable instructions, when executed, cause a machine to perform a method according to the first aspect.
It should be understood that the statements herein reciting aspects are not intended to limit the critical or essential features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.
Drawings
The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. In the drawings, like or similar reference characters designate like or similar elements, and wherein:
FIG. 1 illustrates an imbalance between metadata blocks and storage space of data blocks;
FIG. 2 illustrates an example environment in which embodiments of the present disclosure may be implemented;
FIG. 3 illustrates an example setting of a threshold proportion of storage allocation for a file system according to certain embodiments of the present disclosure;
FIG. 4 illustrates an example process of allocating metadata storage space in a storage space of a file system in accordance with certain embodiments of the present disclosure;
FIG. 5 illustrates an example process of freeing storage reserved for metadata in accordance with certain embodiments of the present disclosure;
FIG. 6 illustrates a flowchart of an example process for balancing metadata storage space and data storage space, according to certain embodiments of the present disclosure;
FIG. 7 illustrates a flow diagram of an example method of balancing storage space of a file system in accordance with certain embodiments of the present disclosure; and
FIG. 8 illustrates a block diagram of a device suitable for implementing embodiments of the present disclosure.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.
The terms "include" and variations thereof as used herein are inclusive and open-ended, i.e., "including but not limited to. The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment". Relevant definitions for other terms will be given in the following description.
The terms "first," "second," and the like, as used herein, may be used herein to describe various elements, which should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of example embodiments. As used herein, the term "and/or" includes any and all combinations of one or more of the listed terms.
As previously mentioned, the storage space for certain metadata (e.g., inodes) is statically allocated when the file system is initially created. Thus, as the allocation and usage of storage space of the file system changes, an imbalance between the storage space of the metadata block and the data block may occur. However, the conventional static metadata allocation manner does not consider such an unbalanced state between the metadata block and the data block usage, thereby resulting in a waste of storage space of the file system. For example, a metadata block, such as an inode, has been exhausted, but a large number of data blocks remain. This will result in a Data Unavailable (DU) situation, but there is still available space visible to the customer at this time.
Fig. 1 illustrates this imbalance of metadata blocks and storage space of data blocks. In fig. 1, a curve 105 and a curve 110 represent consumption of a metadata block and a data block over time, respectively. As shown in fig. 1, the consumption rate is unbalanced between the metadata block and the data block. The metadata block will be exhausted at time 115 (labeled T15), but there are still many data blocks available at this time.
This imbalance will cause the client to enter a Data Unavailable (DU) state. Although file systems are not completely unavailable from Graphical User Interfaces (GUIs) or Command Line Interfaces (CLIs), clients are unable to create any new files and/or directories. At this time, even if there is available space in the file system, the storage administrator must extend the file system to increase the available space of the metadata block, which seriously decreases the storage efficiency.
The embodiment of the disclosure provides a self-balancing scheme between a metadata storage space and a data storage space of a file system, so as to avoid a DU situation and improve the storage efficiency of the file system. The scheme reserves storage space for certain type of metadata of the file system when the allocation proportion of the corresponding metadata storage space is high. In this way, the usage of metadata storage space and data storage space can be balanced by storage space multiplexing without extending the file system.
According to the scheme, the metadata block allocation is dynamic, so that the client workload load mode change can be automatically adapted, and the early file system layout planning work of a storage administrator is reduced. In addition, the scheme can also avoid the condition that DU (data Unit) occurs in the file system due to the unbalance of the metadata storage space and the data storage space, improve the storage space efficiency of the file system and save the cost of expanding the storage array of the file system.
FIG. 2 illustrates an example environment 200 in which embodiments of the present disclosure may be implemented.
Included in environment 200 is a Space Manager (SM)205, which is responsible for dynamically managing metadata storage space and data storage space in the storage space of the file system. The space manager 205 may be implemented in software, firmware, hardware, or any combination thereof.
In the space manager 205, respective metadata storage spaces 210, 215, and 220 are configured for the three types of metadata of the file system. For ease of discussion, the three types of metadata will be referred to as first type of metadata (labeled metadata 1), second type of metadata (labeled metadata 2), and third type of metadata (labeled metadata 3), respectively, while the corresponding metadata storage spaces are referred to as first metadata storage space 210, second metadata storage space 215, and third metadata storage space 220, respectively.
It should be understood that three types of metadata and corresponding three metadata storage spaces are shown in fig. 2 for illustrative purposes only. In practice, any number of metadata types and corresponding metadata storage spaces may be included in the file system.
As shown in FIG. 2, in addition to metadata storage space, data storage space 225 is allocated for file data in space manager 205. Metadata storage spaces 210, 215, and 220 and data storage space 225 may be of any size, and may include any number of storage blocks, for example.
In accordance with embodiments of the present disclosure, to avoid having data blocks still available when the metadata storage space is exhausted, i.e., still showing to clients that there is storage space available in the file system, a Reservation Agent (RA)230 is introduced to the file system. Reservation agent 230, as with the space manager 205 type, may also be implemented in software, firmware, hardware, or any combination thereof.
Reservation agent 230 is invoked or initiated when necessary (e.g., when a metadata block will run out before a data block) to reserve storage space for the metadata from space manager 205. The reserved storage space will only be used for storing metadata and not for storing data. In other words, the reserved storage space will first meet the space allocation requirements of the metadata block. Thereby, the DU problem of the client is prevented from occurring.
The reserved storage space may be of any size. In some embodiments, the Reserved memory space may comprise one memory Slice (e.g., 256MB), which may also be referred to as a Metadata Reserved Slice (MRS). An example data structure for an MRS is discussed below.
Metadata Reserved Slice{
SliceId/unique Identification (ID) of slice
Seqid/sequence ID, which is monotonically incremented by reservation agent 230 when an MRS is reserved
Metadata ID for which ID/MRS is reserved
Duration of time reserved for Duration/MRS, which is zeroed when MRS is allocated
}
In an embodiment of the present disclosure, if the allocation ratio of the first metadata storage space 210 exceeds a threshold ratio 235 (denoted as Rh), the reservation agent 230 will reserve storage space 240 for the corresponding first metadata, denoted as Ri. For convenience of discussion, the threshold ratio 235 will be referred to as a first threshold ratio 235, and the specific value thereof may be determined according to actual requirements, and may be fixed or dynamically configured. As an example, the first threshold proportion 235 may be set to 90%.
The allocation ratio of the metadata storage space may be determined by a ratio of the allocated space in the metadata storage space. For example, whether to create the reserved storage space may be determined by the following equation:
allocated metadata storage space/metadata storage space > Rh (1)
In some embodiments, the determination of the allocation proportion of metadata storage space may be triggered by the use of the file system. For example, the comparison of the allocation ratio of metadata storage space to the first threshold ratio 235 may be triggered based on the allocation ratio of storage space of the file system being greater than or equal to the threshold ratio 245 (referred to as the second threshold ratio 245). Similar to the first threshold ratio 235, the second threshold ratio 245 may be determined by the fraction of allocated space in the storage space of the file system.
In some embodiments, multiple threshold ratios may be set in the space manager 205. For example, as shown in FIG. 2, a lower threshold proportion 250 and a higher threshold proportion 255 may be provided in addition to the second threshold proportion 245, referred to for ease of discussion as a third threshold proportion 250 and a fourth threshold proportion 255, respectively.
FIG. 3 illustrates an example setting 300 of a threshold proportion of storage allocation for a file system according to some embodiments of the present disclosure.
As shown in fig. 3, three threshold proportions, i.e., a second threshold proportion 250, a third threshold proportion 245, and a fourth threshold proportion 255, which are also referred to as a Low Water Mark (LWM), a Middle Water Mark (MWM), and a High Water Mark (HWM), are set according to the proportion of the allocated storage space 310 in the storage space 305 of the file system. Similar to the first threshold ratio 235, the specific values of the second threshold ratio 245, the third threshold ratio 250, and the fourth threshold ratio 255 may also be determined according to actual requirements. As an example, the second threshold percentage 245 may be set to 90%, the third threshold percentage 250 may be set to 80%, and the fourth threshold percentage 255 may be set to 95%.
When the proportion of allocated storage 310 reaches different threshold proportions, the space manager 205 will allocate storage blocks of the file system in different ways. For example, when the proportion of allocated storage space 310 is less than or equal to the third threshold proportion 250 (i.e., the low watermark), storage space will be allocated for each of the metadata 210, 215, and 220 from the storage space 305 in a conventional manner, as shown in FIG. 4.
When the proportion of allocated storage 310 is greater than or equal to the second threshold proportion 245 (i.e., the mid-water mark), the reservation agent 230 will be initialized or invoked to reserve storage units for metadata, as shown in FIG. 2. Reservation agent 230 may compare the allocation ratio of the respective storage spaces 210, 215, and 220 for each type of metadata to first threshold ratio 235. If the allocation ratio of the corresponding storage space for a certain type of metadata exceeds first threshold ratio 235, reservation agent 230 reserves storage space for that type of metadata, e.g., one 256MB slice. In the example shown in fig. 2, the allocation proportion of the first metadata storage space 210 for only the first metadata exceeds the first threshold proportion 235, so storage space 240 is reserved for only the first metadata.
In some embodiments, different types of metadata may be assigned different priorities. In the case where the allocation proportions of the respective storage spaces of the plural types of metadata all exceed the first threshold proportion 235, the storage space may be preferentially reserved for the metadata with a high priority. The priority may be determined by a current allocation state of a corresponding storage space of each type of metadata, a consumption speed of a corresponding storage space of each type of metadata, and the like. The priority of metadata whose storage space is most likely to be exhausted first may be set to be high.
For example, reservation agent 230 may create the reserved storage space when the current allocation ratio of the storage space of the file system reaches second threshold ratio 245. Reservation agent 230 will then check the current allocation status and consumption rate of the metadata and data blocks. If an imbalance condition occurs between a certain metadata block and a data block, a storage block may be first allocated for the unbalanced metadata in the reserved storage space. In this way, the use efficiency of the storage space can be further improved.
In some embodiments, after reserving a storage space (e.g., storage space 240) for a certain type of metadata (e.g., first metadata 210), reservation agent 230 may determine whether the reservation condition is satisfied. For example, whether equation (1) above is still true. If the reservation condition is not satisfied, the reserved storage space (e.g., storage space 240) will be released.
In addition to the above formula (1), any other suitable reservation condition may be adopted according to actual requirements. In some embodiments, the reserved storage space may be determined to be reserved based on the fraction of the file system's storage space 305 that is allocated storage space 310. For example, if the proportion of allocated storage 310 is again below the third threshold proportion 250 (i.e., the low watermark), all of the storage reserved for metadata will be returned to the space manager 205. At this point, reservation agent 230 will stop and allocate storage space for the metadata from storage space 305 of the file system in a conventional manner.
To further facilitate rebalancing between metadata storage space and data storage space, in some embodiments, data storage space that has not been accessed within a predetermined period of time may be reclaimed from the allocated data storage space of the file system. For example, as shown in FIG. 2, reservation agent 230 may scan allocated data storage space 225 and reclaim data storage space 260 that has not been accessed within a predetermined period of time for subsequent reservation for metadata use. As an example, reservation agent 230 may reclaim Least Recently Used (LRU) free background data blocks. These data blocks are allocated in the disk but are not used by the user.
In this way, the available space in the data block will be proactively returned to the space manager 205, freeing space in the file system from the data block to the metadata block. This reduces the fragmentation level of the file system and improves the storage efficiency of the file system. By reserving storage space for metadata and reclaiming free data storage space for subsequent reservation for metadata use, automatic rebalancing of allocation states between metadata storage space and data storage space can be achieved without extending the storage space of the file system.
If the reserved storage space has been fully utilized and the allocated proportion of storage space that still has metadata exceeds the first threshold proportion 235, new storage space may continue to be reserved for the metadata. The processing method described above may be adopted, and details are not described here.
When the proportion of allocated storage 310 is greater than or equal to the fourth threshold proportion 255 (i.e. the high watermark), this indicates that space in the space manager 205 is limited. At this point, the reserved storage space may be released. One specific example is discussed below with reference to fig. 5.
In this example, as shown in FIG. 5, the allocation of the second metadata storage space 215 for the second type of metadata (labeled metadata 2) and the allocation of the third metadata storage space 220 for the third type of metadata (labeled metadata 3) both exceed the first threshold proportion 235 (e.g., by 90%). Thus, storage space 505 and 510, denoted Rm and Rv, respectively, is also reserved for these two types of metadata.
If the reservation agent 230 monitors that the proportion of allocated storage space of the file system exceeds the fourth threshold proportion 255 (e.g., exceeds 95%), this indicates that the storage space of the file system is limited. At this point, reservation agent 230 may determine whether there is a reclaimable data storage space, e.g., whether there is a data storage space that has not been accessed within a predetermined period of time. As an example, reservation agent 230 may scan the data space at predetermined time intervals (e.g., 2 minutes) to find recoverable data blocks.
If a data block will run out before a metadata block, reservation agent 230 will not be able to reclaim any blocks from the data space to space manager 205. In the space manager 205, there is no space or block available other than the reserved storage space. In this case, reservation agent 230 may release the reserved memory space.
In the example shown in fig. 5, reservation agent 230 selects the memory space to be freed from the plurality of reserved memory spaces 240, 505, and 510. For example, a storage space with a low probability of being exhausted in the near future may be selected for release according to the respective generation states of the reserved storage spaces. As an example, the generation status of the reserved storage space may be determined according to the following formula (2):
MGSiSeqid/Duration x (1-free space/allocated space) (2) where MGS is presenti(i ═ 1,2,3 …) represents the generation state of the i-th MRS.
At the same time, reservation agent 230 may raise an alert to indicate that the storage space of the file system is limited and/or that the metadata storage space has a higher usage. Thus, a storage administrator or user is notified that a rebalancing of the storage space of the file system cannot be achieved, so that the storage administrator or client can take corresponding measures.
FIG. 6 illustrates a flow diagram of an example process 600 for balancing metadata storage space and data storage space, in accordance with certain embodiments of the present disclosure.
As shown in fig. 6, process 600 begins at block 605. At block 610, the current allocation ratio of storage space of the file system is determined. At block 615, it is determined whether the current allocation proportion of storage space is greater than or equal to a second threshold proportion 245 (e.g., the mid-water mark). If so, process 600 proceeds to block 620, where reservation agent 230 is invoked. At block 625, the current allocation proportion of the storage space of the file system is checked. If the current allocation of storage space is greater than or equal to the second threshold proportion 245 and less than the fourth threshold proportion 255 (e.g., the high watermark), the process 600 proceeds to block 630 where it is determined whether the current allocation of metadata storage space is greater than the first threshold proportion 235. If so, storage space is reserved for the corresponding type of metadata at block 635 and allocated data storage space that has not been accessed for a predetermined period of time is reclaimed.
At block 640, the latest allocation proportion of the storage space of the file system is determined. The latest allocation ratio of the storage space is then checked at block 645. If the latest allocation ratio of the storage space is still greater than or equal to the second threshold ratio 245 and less than the fourth threshold ratio 255, then returning to block 630, the determination of the allocation ratio of the metadata storage space continues. If the latest allocation proportion of storage space is less than or equal to the third threshold proportion 250 (e.g., the low watermark), then at block 650, the reservation agent 230 is stopped. Next, at block 655, metadata storage space is allocated from the storage space of the file system. At block 660, the process 600 ends.
If it is determined at block 645 that the most recent allocation proportion of storage space is greater than or equal to the fourth threshold proportion 255, the process 600 proceeds to block 660, where the corresponding reserved storage space is released according to the generation status of the reserved storage space and the administrator is notified for subsequent expansion. Reservation agent 230 is then stopped at block 670. The process 600 then ends at block 660.
Table 1 below shows the metadata blocks and data block usage change states in a 100GB reduced file system.
TABLE 1
Figure BDA0002254170480000101
At time T1, the allocation of storage space for the file system is 78% less than a third threshold proportion 250 (e.g., 80%). At this time, the allocation of the metadata blocks and the data blocks will be performed in the storage space of the file system in a conventional manner.
At time T2, a number of new small files or directories have been created, the allocation proportion of the storage space of the file system is 90%, and the second threshold proportion 245 is reached. At this point, reservation agent 230 will be initialized or invoked and reserve the MRS for the metadata. At the same time, reservation agent 230 will begin scanning LRU free back end data blocks for reclamation.
At time T3, the metadata block is increased by 4GB by the MRS. At the same time, reservation agent 230 will reduce the free backend data blocks to satisfy 1) the new allocation request, and 2) reclaim the LRU free backend data blocks to the MRS for use by the metadata. In this way, there is a rebalancing between the metadata blocks and the data blocks. There are still free back-end data blocks in the file system and the metadata blocks will not run out.
At time T4, the allocation of storage space for the file system is 96%, exceeding a fourth threshold proportion of 255 (e.g., 95%). At this point, there are no free backend data blocks already, meaning that the rebalancing request of reservation agent 230 cannot be supported. Thus, an alert will be sent to the client to indicate that the storage space of the file system is limited and that the usage of the metadata storage space is high.
FIG. 7 illustrates a flow diagram of an example method 700 of balancing storage space of a file system in accordance with certain embodiments of the present disclosure. The method 700 may be implemented in the environment 200 as shown in fig. 2. For ease of discussion, the method 700 will be described below in conjunction with FIG. 2.
As shown in FIG. 7, at block 705, a determination is made whether an allocation proportion of a first metadata storage space for a first type of metadata of the one or more types of metadata for the file system (referred to as a "first allocation proportion") is greater than a first threshold proportion. At block 710, if it is determined that the first allocation proportion is above a first threshold proportion, storage space is reserved in the storage space of the file system for the first type of metadata.
In some embodiments, data storage space that has not been accessed within a predetermined period of time may be determined among the allocated data storage space of the file system. The unaccessed data storage space may then be reclaimed for reservation for one or more types of metadata.
In some embodiments, it may again be determined whether the current allocation proportion of the first metadata storage space for the first type of metadata is greater than the first threshold proportion. The reserved storage space may be released if it is determined that the current allocation ratio is less than or equal to the first threshold ratio.
In some embodiments, it may be determined whether an allocation proportion of a second metadata storage space for a second type of metadata of the one or more types of metadata (referred to as a "second allocation proportion") is greater than a first threshold proportion. If it is determined that the second allocation proportion is greater than the first threshold proportion, a first priority of the first type of metadata is compared with a second priority of the second type of metadata. Based on the first priority being higher than the second priority, storage space may be preferentially reserved in the storage space of the file system for the first type of metadata.
In some embodiments, it may be determined whether the allocation proportion of the storage space of the file system (referred to as the "third allocation proportion") is greater than or equal to a second threshold proportion. If it is determined that the third allocation percentage is greater than or equal to the first threshold percentage, then it is determined whether the first allocation percentage is greater than the first threshold percentage.
In some embodiments, it may be determined whether a third allocated proportion of storage space of the file system is less than or equal to a third threshold proportion, wherein the third threshold proportion is less than the second threshold proportion. The reserved storage space may be released if it is determined that the third allocation proportion is less than or equal to the third threshold proportion.
In some embodiments, it may be determined whether a third allocation proportion of storage space of the file system is greater than or equal to a fourth threshold proportion, wherein the fourth threshold proportion is greater than the second threshold proportion. If it is determined that the third allocation proportion is greater than or equal to the fourth threshold proportion, the reserved storage space may be released.
In some embodiments, if it is determined that the third allocation percentage is greater than or equal to a fourth threshold percentage, it may be determined whether there is data storage space in the allocated data storage space of the file system that has not been accessed for a predetermined period of time. If it is determined that there is no unaccessed data storage space, the reserved storage space may be freed. In some embodiments, an alert may also be raised to indicate that the storage space of the file system is limited.
It should be understood that the operations and features described above in connection with fig. 2 through 5 are equally applicable to the method 700 and have the same effect, and detailed description is omitted.
FIG. 8 shows a schematic block diagram of a device 800 that may be used to implement embodiments of the present disclosure.
As shown in fig. 8, device 800 includes a controller or processor, or Central Processing Unit (CPU)801, which may perform various appropriate actions and processes in accordance with programs stored in a Read Only Memory (ROM)802 and/or a Random Access Memory (RAM) 803. The ROM 802 and/or RAM 803 may store various programs and data required for operation of the device 800. The CPU 801, ROM 802, and RAM 803 are connected to each other via a bus 804. In particular, device 800 also includes one or more special-purpose processing units (not shown), which may also be connected to bus 804.
An input/output (I/O) interface 805 is also connected to bus 804. A number of components in the device 800 are connected to the I/O interface 805, including: an input unit 806, such as a keyboard, a mouse, or the like; an output unit 807 such as various types of displays, speakers, and the like; a storage unit 808, such as a magnetic disk, optical disk, or the like; and a communication unit 809 such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks. In particular, in embodiments of the present disclosure, the communication unit 809 supports communication with clients or other devices.
In some embodiments, the CPU 801 may be configured to perform the various processes and processes described above, such as the method 700. For example, in some embodiments, the method 700 may be implemented as a computer software program tangibly embodied on a computer-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and/or installed onto device 800 via ROM 802 and/or communications unit 809. When loaded into RAM 803 and executed by CPU 801, a computer program may perform one or more of the steps of method 700 described above. Alternatively, in other embodiments, the CPU 801 may also be configured in any other suitable manner to implement the processes/methods described above.
In particular, according to embodiments of the present disclosure, the processes described above with reference to fig. 1-7 may be implemented as a computer program product, which may be tangibly stored on a non-transitory computer-readable storage medium and include computer-executable instructions that, when executed, cause a device to implement various aspects in accordance with the present disclosure.
The computer readable storage medium may be a tangible device that may store instructions for use by an instruction execution device. The computer readable storage medium may include, for example, but is not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific, non-exhaustive examples of the computer readable storage medium include: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.
The computer program instructions for carrying out operations of the present disclosure may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, the electronic circuitry that can execute the computer-readable program instructions implements aspects of the present disclosure by utilizing the state information of the computer-readable program instructions to personalize the electronic circuitry, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA).
Various aspects of the present disclosure are described herein with reference to block diagrams and/or flowchart illustrations of apparatus, methods, and computer program products according to embodiments of the disclosure. It will be understood that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by computer-readable program instructions.
Various embodiments of the present disclosure have been described for purposes of illustration, but the present disclosure is not intended to be limited to the disclosed embodiments. All such modifications and variations are within the scope of the disclosure as defined in the appended claims without departing from the spirit thereof.

Claims (19)

1. A method of balancing storage space of a file system, comprising:
determining whether a first allocation proportion of a first metadata storage space for a first type of metadata of the one or more types of metadata of the file system is greater than a first threshold proportion; and
in accordance with a determination that the first allocation percentage is greater than the first threshold percentage, reserving storage space for the first type of metadata in the storage space of the file system.
2. The method of claim 1, further comprising:
determining, among the allocated data storage spaces of the file system, data storage spaces that have not been accessed within a predetermined period of time; and
reclaiming the unaccessed data storage space for reservation for the one or more types of metadata.
3. The method of claim 1, further comprising:
determining whether a current allocation proportion of a first metadata storage space for a first type of metadata is greater than the first threshold proportion; and
in accordance with a determination that the current allocation ratio is less than or equal to the first threshold ratio, releasing the reserved storage space.
4. The method of claim 1, wherein reserving the storage space for the first type of metadata comprises:
determining whether a second allocation proportion of a second metadata storage space for a second type of metadata of the one or more types of metadata is greater than the first threshold proportion;
in accordance with a determination that the second allocation proportion is greater than the first threshold proportion, comparing a first priority of the first type of metadata to a second priority of the second type of metadata; and
preferentially reserving the storage space for the first type of metadata in the storage space of the file system based on the first priority being higher than the second priority.
5. The method of claim 1, wherein determining whether the first allocation proportion is greater than the first threshold proportion comprises:
determining whether a third allocation proportion of the storage space of the file system is greater than or equal to the second threshold proportion; and
in accordance with a determination that the third allocation percentage is greater than or equal to a first threshold percentage, determining whether the first allocation percentage is greater than the first threshold percentage.
6. The method of claim 5, further comprising:
determining whether the third allocation proportion of the storage space of the file system is less than or equal to a third threshold proportion, the third threshold proportion being less than the second threshold proportion;
in accordance with a determination that the third allocation proportion is less than or equal to the third threshold proportion, releasing the reserved storage space.
7. The method of claim 5, further comprising:
determining whether the third allocation proportion of the storage space of the file system is greater than or equal to a fourth threshold proportion, the fourth threshold proportion being greater than the second threshold proportion;
releasing the reserved storage space in accordance with a determination that the third allocation percentage is greater than or equal to the fourth threshold percentage.
8. The method of claim 7, further comprising:
causing an alert to indicate that the storage space of the file system is limited.
9. The method of claim 7, wherein releasing the reserved storage space comprises:
in accordance with a determination that the third allocation percentage is greater than or equal to the fourth threshold percentage, determining whether there is data storage space in the allocated data storage spaces of the file system that has not been accessed within a predetermined period of time; and
and releasing the reserved storage space according to the fact that the unaccessed data storage space does not exist.
10. An apparatus for balancing storage space of a file system, comprising:
a processor, and
a memory storing computer-executable instructions that, when executed by the processor, cause the device to perform acts comprising:
determining whether a first allocation proportion of a first metadata storage space for a first type of metadata of the one or more types of metadata of the file system is greater than a first threshold proportion; and
in accordance with a determination that the first allocation percentage is greater than the first threshold percentage, reserving storage space for the first type of metadata in the storage space of the file system.
11. The apparatus of claim 10, wherein the actions further comprise:
determining, among the allocated data storage spaces of the file system, data storage spaces that have not been accessed within a predetermined period of time; and
reclaiming the unaccessed data storage space for reservation for the one or more types of metadata.
12. The apparatus of claim 10, wherein the actions further comprise:
determining whether a current allocation proportion of a first metadata storage space for a first type of metadata is greater than the first threshold proportion; and
in accordance with a determination that the current allocation ratio is less than or equal to the first threshold ratio, releasing the reserved storage space.
13. The apparatus of claim 10, wherein reserving the storage space for the first type of metadata comprises:
determining whether a second allocation proportion of a second metadata storage space for a second type of metadata of the one or more types of metadata is greater than the first threshold proportion;
in accordance with a determination that the second allocation proportion is greater than the first threshold proportion, comparing a first priority of the first type of metadata to a second priority of the second type of metadata; and
preferentially reserving the storage space for the first type of metadata in the storage space of the file system based on the first priority being higher than the second priority.
14. The apparatus of claim 10, wherein determining whether the first allocation proportion is greater than the first threshold proportion comprises:
determining whether a third allocation proportion of the storage space of the file system is greater than or equal to the second threshold proportion; and
in accordance with a determination that the third allocation percentage is greater than or equal to a first threshold percentage, determining whether the first allocation percentage is greater than the first threshold percentage.
15. The apparatus of claim 14, wherein the actions further comprise:
determining whether the third allocation proportion of the storage space of the file system is less than or equal to a third threshold proportion, the third threshold proportion being less than the second threshold proportion;
in accordance with a determination that the third allocation proportion is less than or equal to the third threshold proportion, releasing the reserved storage space.
16. The apparatus of claim 14, wherein the actions further comprise:
determining whether the third allocation proportion of the storage space of the file system is greater than or equal to a fourth threshold proportion, the fourth threshold proportion being greater than the second threshold proportion;
releasing the reserved storage space in accordance with a determination that the third allocation percentage is greater than or equal to the fourth threshold percentage.
17. The apparatus of claim 16, wherein the actions further comprise:
causing an alert to indicate that the storage space of the file system is limited.
18. The apparatus of claim 18, wherein releasing the reserved storage space comprises:
in accordance with a determination that the third allocation percentage is greater than or equal to the fourth threshold percentage, determining whether there is data storage space in the allocated data storage spaces of the file system that has not been accessed within a predetermined period of time; and
and releasing the reserved storage space according to the fact that the unaccessed data storage space does not exist.
19. A computer program product tangibly stored on a non-transitory computer-readable medium and comprising machine executable instructions that, when executed, cause a machine to perform the method of claims 1 to 9.
CN201911046051.7A 2019-10-30 2019-10-30 Method, apparatus and computer program product for balancing storage space of a file system Active CN112749135B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201911046051.7A CN112749135B (en) 2019-10-30 2019-10-30 Method, apparatus and computer program product for balancing storage space of a file system
US16/877,910 US11681447B2 (en) 2019-10-30 2020-05-19 Method, device and computer program product of balance of storage space for file system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911046051.7A CN112749135B (en) 2019-10-30 2019-10-30 Method, apparatus and computer program product for balancing storage space of a file system

Publications (2)

Publication Number Publication Date
CN112749135A true CN112749135A (en) 2021-05-04
CN112749135B CN112749135B (en) 2024-06-07

Family

ID=75640640

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911046051.7A Active CN112749135B (en) 2019-10-30 2019-10-30 Method, apparatus and computer program product for balancing storage space of a file system

Country Status (2)

Country Link
US (1) US11681447B2 (en)
CN (1) CN112749135B (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11295031B2 (en) 2019-10-08 2022-04-05 International Business Machines Corporation Event log tamper resistance
US11392348B2 (en) * 2020-02-13 2022-07-19 International Business Machines Corporation Ordering records for timed meta-data generation in a blocked record environment
US11237738B2 (en) 2020-03-16 2022-02-01 EMC IP Holding Company, LLC Managing operation of a data storage system with reservation of metadata space
US11372846B2 (en) * 2020-07-17 2022-06-28 International Business Machines Corporation Generating and utilizing pre-allocated storage space
US20240202080A1 (en) * 2022-12-20 2024-06-20 Dell Products L.P. Prioritization in cloud migration for deduplication systems

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2002039279A2 (en) * 2000-11-07 2002-05-16 Surgient Networks, Inc. Systems and methods for resource monitoring in information storage environments
US20020091722A1 (en) * 2000-03-03 2002-07-11 Surgient Networks, Inc. Systems and methods for resource management in information storage environments
US6928459B1 (en) * 2000-07-18 2005-08-09 International Business Machines Corporation Plurality of file systems using weighted allocation to allocate space on one or more storage devices
US8443153B1 (en) * 2010-01-06 2013-05-14 Netapp, Inc. Dynamic balancing of performance with block sharing in a storage system
US20160173603A1 (en) * 2014-12-10 2016-06-16 International Business Machines Corporation Method and apparatus for managing file system of unified storage system
US20190235782A1 (en) * 2018-01-31 2019-08-01 EMC IP Holding Company LLC Techniques for rebalancing storage between subspaces

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9355121B1 (en) * 2013-06-28 2016-05-31 Emc Corporation Segregating data and metadata in a file system
US9870366B1 (en) * 2015-09-18 2018-01-16 EMC IP Holding Company LLC Processing storage capacity events in connection with file systems
US9678683B1 (en) * 2016-11-01 2017-06-13 Red Hat, Inc. Lazy persistent storage volume provisioning
US10168910B2 (en) * 2017-03-09 2019-01-01 International Business Machines Corporation Storage area network reclamation for virtual environments
US10936195B2 (en) 2017-10-30 2021-03-02 EMC IP Holding Company LLC Data storage system using in-memory structure for reclaiming space from internal file system to pool storage
US10235286B1 (en) 2017-10-30 2019-03-19 EMC IP Holding Company LLC Data storage system dynamically re-marking slices for reclamation from internal file system to pool storage
US10866869B2 (en) * 2019-01-16 2020-12-15 Vmware, Inc. Method to perform crash and failure recovery for a virtualized checkpoint protected storage system

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020091722A1 (en) * 2000-03-03 2002-07-11 Surgient Networks, Inc. Systems and methods for resource management in information storage environments
US6928459B1 (en) * 2000-07-18 2005-08-09 International Business Machines Corporation Plurality of file systems using weighted allocation to allocate space on one or more storage devices
WO2002039279A2 (en) * 2000-11-07 2002-05-16 Surgient Networks, Inc. Systems and methods for resource monitoring in information storage environments
US8443153B1 (en) * 2010-01-06 2013-05-14 Netapp, Inc. Dynamic balancing of performance with block sharing in a storage system
US20160173603A1 (en) * 2014-12-10 2016-06-16 International Business Machines Corporation Method and apparatus for managing file system of unified storage system
US20190235782A1 (en) * 2018-01-31 2019-08-01 EMC IP Holding Company LLC Techniques for rebalancing storage between subspaces

Also Published As

Publication number Publication date
CN112749135B (en) 2024-06-07
US20210132829A1 (en) 2021-05-06
US11681447B2 (en) 2023-06-20

Similar Documents

Publication Publication Date Title
CN112749135B (en) Method, apparatus and computer program product for balancing storage space of a file system
KR101357397B1 (en) Method for tracking memory usages of a data processing system
US10572285B2 (en) Method and apparatus for elastically scaling virtual machine cluster
CN110753131A (en) Microservice distributed current limiting method and device, storage medium and electronic equipment
CN110389905B (en) Resource release method, resource allocation method, device and computer program product
CN107209716B (en) Memory management device and method
EP3865992A2 (en) Distributed block storage system, method, apparatus and medium
KR102469927B1 (en) Apparatus for managing disaggregated memory and method for the same
US8458719B2 (en) Storage management in a data processing system
CN111464331B (en) Control method and system for thread creation and terminal equipment
CN110209354B (en) Method, apparatus, device and medium for processing data
CN114168490A (en) Method for determining memory recovery threshold and related equipment
CN110471769B (en) Resource management method and device for virtual machine
US10896062B2 (en) Inter-process memory management
US11403026B2 (en) Method, device and computer program product for managing storage system
CN111124256A (en) Method, apparatus and computer program product for managing storage
WO2022048674A1 (en) Server cabinet-based virtual machine management method and apparatus
CN112291326B (en) Load balancing method, load balancing device, storage medium and electronic equipment
CN114518962A (en) Memory management method and device
CN114253663A (en) Virtual machine resource scheduling method and device
CN113391757B (en) Node expansion method and device and migration node
CN114327862A (en) Memory allocation method and device, electronic equipment and storage medium
CN115840635A (en) Computing resource management method, electronic device, and program product
CN111143073A (en) Management method, equipment and storage medium of virtualized resources
CN116431327B (en) Task current limiting processing method and fort machine

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant