CN112749135A - Method, apparatus and computer program product for balancing storage space of a file system - Google Patents
Method, apparatus and computer program product for balancing storage space of a file system Download PDFInfo
- Publication number
- CN112749135A CN112749135A CN201911046051.7A CN201911046051A CN112749135A CN 112749135 A CN112749135 A CN 112749135A CN 201911046051 A CN201911046051 A CN 201911046051A CN 112749135 A CN112749135 A CN 112749135A
- Authority
- CN
- China
- Prior art keywords
- storage space
- metadata
- proportion
- threshold
- file system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 43
- 238000004590 computer program Methods 0.000 title claims abstract description 13
- 238000013500 data storage Methods 0.000 claims description 36
- 239000003795 chemical substances by application Substances 0.000 description 28
- 230000008569 process Effects 0.000 description 15
- 238000010586 diagram Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 6
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 230000001902 propagating effect Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000013467 fragmentation Methods 0.000 description 1
- 238000006062 fragmentation reaction Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
- G06F3/0631—Configuration or reconfiguration of storage systems by allocating resources to storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/164—File meta data generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1727—Details of free space management performed by the file system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0653—Monitoring storage devices or systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0659—Command handling arrangements, e.g. command buffers, queues, command scheduling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Library & Information Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Embodiments of the present disclosure provide a method, apparatus, and computer program product for balancing storage space of a file system. In one embodiment, a determination is made whether a first allocated proportion of a first metadata storage space for a first type of metadata of one or more types of metadata of a file system is greater than a first threshold proportion. If the first allocation ratio is greater than a first threshold ratio, storage space is reserved in the storage space of the file system for the first type of metadata.
Description
Technical Field
Embodiments of the present disclosure relate generally to computer technology and, more particularly, relate to a method, apparatus, and computer program product for balancing storage space of a file system.
Background
Both metadata and data are widely used in modern file systems. There are different types of metadata that describe various characteristics of a file. An inode (inode) is an important type of metadata, and each inode corresponds to a file system object in a file system and is used for storing file information such as access rights, owner, timestamp and storage location of a file. In addition to inodes, there are other types of metadata in a file system. For example, the virtual block may serve as a metadata block for built-in data compression and deduplication.
For a thin file system, storage space is allocated on demand. However, the storage space for some metadata (e.g., inodes) is statically allocated when the file system is initially created. However, as the storage space allocation of the file system changes following different client workloads, the data layout may have changed significantly. For example, if a large number of small files and directories are created, the consumption rate of the inode block will be fast. The data block consumption rate is relatively slow. Therefore, there will be an imbalance between the storage space of the inode and the data block. As time passes, the situation becomes worse.
Disclosure of Invention
In general, embodiments of the present disclosure propose methods, apparatuses, and computer program products for balancing storage space of a file system.
In a first aspect, embodiments of the present disclosure provide a method for balancing storage space of a file system. In the method, it is determined whether a first allocation proportion of a first metadata storage space of a first type of metadata of the one or more types of metadata of the file system is greater than a first threshold proportion. If the first allocation proportion is higher than the first threshold proportion, reserving storage space in the storage space of the file system for the first type of metadata.
In a second aspect, embodiments of the present disclosure provide an apparatus for balancing storage space of a file system. The apparatus includes a processor and a memory storing computer-executable instructions. The computer-executable instructions, when executed by the processor, cause the device to perform acts comprising: determining whether a first allocation proportion of a first metadata storage space for a first type of metadata of the one or more types of metadata of the file system is greater than a first threshold proportion; and in accordance with a determination that the first allocation percentage is above the first threshold percentage, reserving storage space in the storage space of the file system for the first type of metadata.
In a third aspect, embodiments of the present disclosure provide a computer program product. The computer program product is tangibly stored on a non-transitory computer-readable medium and includes machine-executable instructions. The machine executable instructions, when executed, cause a machine to perform a method according to the first aspect.
It should be understood that the statements herein reciting aspects are not intended to limit the critical or essential features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.
Drawings
The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. In the drawings, like or similar reference characters designate like or similar elements, and wherein:
FIG. 1 illustrates an imbalance between metadata blocks and storage space of data blocks;
FIG. 2 illustrates an example environment in which embodiments of the present disclosure may be implemented;
FIG. 3 illustrates an example setting of a threshold proportion of storage allocation for a file system according to certain embodiments of the present disclosure;
FIG. 4 illustrates an example process of allocating metadata storage space in a storage space of a file system in accordance with certain embodiments of the present disclosure;
FIG. 5 illustrates an example process of freeing storage reserved for metadata in accordance with certain embodiments of the present disclosure;
FIG. 6 illustrates a flowchart of an example process for balancing metadata storage space and data storage space, according to certain embodiments of the present disclosure;
FIG. 7 illustrates a flow diagram of an example method of balancing storage space of a file system in accordance with certain embodiments of the present disclosure; and
FIG. 8 illustrates a block diagram of a device suitable for implementing embodiments of the present disclosure.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.
The terms "include" and variations thereof as used herein are inclusive and open-ended, i.e., "including but not limited to. The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment". Relevant definitions for other terms will be given in the following description.
The terms "first," "second," and the like, as used herein, may be used herein to describe various elements, which should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of example embodiments. As used herein, the term "and/or" includes any and all combinations of one or more of the listed terms.
As previously mentioned, the storage space for certain metadata (e.g., inodes) is statically allocated when the file system is initially created. Thus, as the allocation and usage of storage space of the file system changes, an imbalance between the storage space of the metadata block and the data block may occur. However, the conventional static metadata allocation manner does not consider such an unbalanced state between the metadata block and the data block usage, thereby resulting in a waste of storage space of the file system. For example, a metadata block, such as an inode, has been exhausted, but a large number of data blocks remain. This will result in a Data Unavailable (DU) situation, but there is still available space visible to the customer at this time.
Fig. 1 illustrates this imbalance of metadata blocks and storage space of data blocks. In fig. 1, a curve 105 and a curve 110 represent consumption of a metadata block and a data block over time, respectively. As shown in fig. 1, the consumption rate is unbalanced between the metadata block and the data block. The metadata block will be exhausted at time 115 (labeled T15), but there are still many data blocks available at this time.
This imbalance will cause the client to enter a Data Unavailable (DU) state. Although file systems are not completely unavailable from Graphical User Interfaces (GUIs) or Command Line Interfaces (CLIs), clients are unable to create any new files and/or directories. At this time, even if there is available space in the file system, the storage administrator must extend the file system to increase the available space of the metadata block, which seriously decreases the storage efficiency.
The embodiment of the disclosure provides a self-balancing scheme between a metadata storage space and a data storage space of a file system, so as to avoid a DU situation and improve the storage efficiency of the file system. The scheme reserves storage space for certain type of metadata of the file system when the allocation proportion of the corresponding metadata storage space is high. In this way, the usage of metadata storage space and data storage space can be balanced by storage space multiplexing without extending the file system.
According to the scheme, the metadata block allocation is dynamic, so that the client workload load mode change can be automatically adapted, and the early file system layout planning work of a storage administrator is reduced. In addition, the scheme can also avoid the condition that DU (data Unit) occurs in the file system due to the unbalance of the metadata storage space and the data storage space, improve the storage space efficiency of the file system and save the cost of expanding the storage array of the file system.
FIG. 2 illustrates an example environment 200 in which embodiments of the present disclosure may be implemented.
Included in environment 200 is a Space Manager (SM)205, which is responsible for dynamically managing metadata storage space and data storage space in the storage space of the file system. The space manager 205 may be implemented in software, firmware, hardware, or any combination thereof.
In the space manager 205, respective metadata storage spaces 210, 215, and 220 are configured for the three types of metadata of the file system. For ease of discussion, the three types of metadata will be referred to as first type of metadata (labeled metadata 1), second type of metadata (labeled metadata 2), and third type of metadata (labeled metadata 3), respectively, while the corresponding metadata storage spaces are referred to as first metadata storage space 210, second metadata storage space 215, and third metadata storage space 220, respectively.
It should be understood that three types of metadata and corresponding three metadata storage spaces are shown in fig. 2 for illustrative purposes only. In practice, any number of metadata types and corresponding metadata storage spaces may be included in the file system.
As shown in FIG. 2, in addition to metadata storage space, data storage space 225 is allocated for file data in space manager 205. Metadata storage spaces 210, 215, and 220 and data storage space 225 may be of any size, and may include any number of storage blocks, for example.
In accordance with embodiments of the present disclosure, to avoid having data blocks still available when the metadata storage space is exhausted, i.e., still showing to clients that there is storage space available in the file system, a Reservation Agent (RA)230 is introduced to the file system. Reservation agent 230, as with the space manager 205 type, may also be implemented in software, firmware, hardware, or any combination thereof.
The reserved storage space may be of any size. In some embodiments, the Reserved memory space may comprise one memory Slice (e.g., 256MB), which may also be referred to as a Metadata Reserved Slice (MRS). An example data structure for an MRS is discussed below.
Metadata Reserved Slice{
SliceId/unique Identification (ID) of slice
Seqid/sequence ID, which is monotonically incremented by reservation agent 230 when an MRS is reserved
Metadata ID for which ID/MRS is reserved
Duration of time reserved for Duration/MRS, which is zeroed when MRS is allocated
}
In an embodiment of the present disclosure, if the allocation ratio of the first metadata storage space 210 exceeds a threshold ratio 235 (denoted as Rh), the reservation agent 230 will reserve storage space 240 for the corresponding first metadata, denoted as Ri. For convenience of discussion, the threshold ratio 235 will be referred to as a first threshold ratio 235, and the specific value thereof may be determined according to actual requirements, and may be fixed or dynamically configured. As an example, the first threshold proportion 235 may be set to 90%.
The allocation ratio of the metadata storage space may be determined by a ratio of the allocated space in the metadata storage space. For example, whether to create the reserved storage space may be determined by the following equation:
allocated metadata storage space/metadata storage space > Rh (1)
In some embodiments, the determination of the allocation proportion of metadata storage space may be triggered by the use of the file system. For example, the comparison of the allocation ratio of metadata storage space to the first threshold ratio 235 may be triggered based on the allocation ratio of storage space of the file system being greater than or equal to the threshold ratio 245 (referred to as the second threshold ratio 245). Similar to the first threshold ratio 235, the second threshold ratio 245 may be determined by the fraction of allocated space in the storage space of the file system.
In some embodiments, multiple threshold ratios may be set in the space manager 205. For example, as shown in FIG. 2, a lower threshold proportion 250 and a higher threshold proportion 255 may be provided in addition to the second threshold proportion 245, referred to for ease of discussion as a third threshold proportion 250 and a fourth threshold proportion 255, respectively.
FIG. 3 illustrates an example setting 300 of a threshold proportion of storage allocation for a file system according to some embodiments of the present disclosure.
As shown in fig. 3, three threshold proportions, i.e., a second threshold proportion 250, a third threshold proportion 245, and a fourth threshold proportion 255, which are also referred to as a Low Water Mark (LWM), a Middle Water Mark (MWM), and a High Water Mark (HWM), are set according to the proportion of the allocated storage space 310 in the storage space 305 of the file system. Similar to the first threshold ratio 235, the specific values of the second threshold ratio 245, the third threshold ratio 250, and the fourth threshold ratio 255 may also be determined according to actual requirements. As an example, the second threshold percentage 245 may be set to 90%, the third threshold percentage 250 may be set to 80%, and the fourth threshold percentage 255 may be set to 95%.
When the proportion of allocated storage 310 reaches different threshold proportions, the space manager 205 will allocate storage blocks of the file system in different ways. For example, when the proportion of allocated storage space 310 is less than or equal to the third threshold proportion 250 (i.e., the low watermark), storage space will be allocated for each of the metadata 210, 215, and 220 from the storage space 305 in a conventional manner, as shown in FIG. 4.
When the proportion of allocated storage 310 is greater than or equal to the second threshold proportion 245 (i.e., the mid-water mark), the reservation agent 230 will be initialized or invoked to reserve storage units for metadata, as shown in FIG. 2. Reservation agent 230 may compare the allocation ratio of the respective storage spaces 210, 215, and 220 for each type of metadata to first threshold ratio 235. If the allocation ratio of the corresponding storage space for a certain type of metadata exceeds first threshold ratio 235, reservation agent 230 reserves storage space for that type of metadata, e.g., one 256MB slice. In the example shown in fig. 2, the allocation proportion of the first metadata storage space 210 for only the first metadata exceeds the first threshold proportion 235, so storage space 240 is reserved for only the first metadata.
In some embodiments, different types of metadata may be assigned different priorities. In the case where the allocation proportions of the respective storage spaces of the plural types of metadata all exceed the first threshold proportion 235, the storage space may be preferentially reserved for the metadata with a high priority. The priority may be determined by a current allocation state of a corresponding storage space of each type of metadata, a consumption speed of a corresponding storage space of each type of metadata, and the like. The priority of metadata whose storage space is most likely to be exhausted first may be set to be high.
For example, reservation agent 230 may create the reserved storage space when the current allocation ratio of the storage space of the file system reaches second threshold ratio 245. Reservation agent 230 will then check the current allocation status and consumption rate of the metadata and data blocks. If an imbalance condition occurs between a certain metadata block and a data block, a storage block may be first allocated for the unbalanced metadata in the reserved storage space. In this way, the use efficiency of the storage space can be further improved.
In some embodiments, after reserving a storage space (e.g., storage space 240) for a certain type of metadata (e.g., first metadata 210), reservation agent 230 may determine whether the reservation condition is satisfied. For example, whether equation (1) above is still true. If the reservation condition is not satisfied, the reserved storage space (e.g., storage space 240) will be released.
In addition to the above formula (1), any other suitable reservation condition may be adopted according to actual requirements. In some embodiments, the reserved storage space may be determined to be reserved based on the fraction of the file system's storage space 305 that is allocated storage space 310. For example, if the proportion of allocated storage 310 is again below the third threshold proportion 250 (i.e., the low watermark), all of the storage reserved for metadata will be returned to the space manager 205. At this point, reservation agent 230 will stop and allocate storage space for the metadata from storage space 305 of the file system in a conventional manner.
To further facilitate rebalancing between metadata storage space and data storage space, in some embodiments, data storage space that has not been accessed within a predetermined period of time may be reclaimed from the allocated data storage space of the file system. For example, as shown in FIG. 2, reservation agent 230 may scan allocated data storage space 225 and reclaim data storage space 260 that has not been accessed within a predetermined period of time for subsequent reservation for metadata use. As an example, reservation agent 230 may reclaim Least Recently Used (LRU) free background data blocks. These data blocks are allocated in the disk but are not used by the user.
In this way, the available space in the data block will be proactively returned to the space manager 205, freeing space in the file system from the data block to the metadata block. This reduces the fragmentation level of the file system and improves the storage efficiency of the file system. By reserving storage space for metadata and reclaiming free data storage space for subsequent reservation for metadata use, automatic rebalancing of allocation states between metadata storage space and data storage space can be achieved without extending the storage space of the file system.
If the reserved storage space has been fully utilized and the allocated proportion of storage space that still has metadata exceeds the first threshold proportion 235, new storage space may continue to be reserved for the metadata. The processing method described above may be adopted, and details are not described here.
When the proportion of allocated storage 310 is greater than or equal to the fourth threshold proportion 255 (i.e. the high watermark), this indicates that space in the space manager 205 is limited. At this point, the reserved storage space may be released. One specific example is discussed below with reference to fig. 5.
In this example, as shown in FIG. 5, the allocation of the second metadata storage space 215 for the second type of metadata (labeled metadata 2) and the allocation of the third metadata storage space 220 for the third type of metadata (labeled metadata 3) both exceed the first threshold proportion 235 (e.g., by 90%). Thus, storage space 505 and 510, denoted Rm and Rv, respectively, is also reserved for these two types of metadata.
If the reservation agent 230 monitors that the proportion of allocated storage space of the file system exceeds the fourth threshold proportion 255 (e.g., exceeds 95%), this indicates that the storage space of the file system is limited. At this point, reservation agent 230 may determine whether there is a reclaimable data storage space, e.g., whether there is a data storage space that has not been accessed within a predetermined period of time. As an example, reservation agent 230 may scan the data space at predetermined time intervals (e.g., 2 minutes) to find recoverable data blocks.
If a data block will run out before a metadata block, reservation agent 230 will not be able to reclaim any blocks from the data space to space manager 205. In the space manager 205, there is no space or block available other than the reserved storage space. In this case, reservation agent 230 may release the reserved memory space.
In the example shown in fig. 5, reservation agent 230 selects the memory space to be freed from the plurality of reserved memory spaces 240, 505, and 510. For example, a storage space with a low probability of being exhausted in the near future may be selected for release according to the respective generation states of the reserved storage spaces. As an example, the generation status of the reserved storage space may be determined according to the following formula (2):
MGSiSeqid/Duration x (1-free space/allocated space) (2) where MGS is presenti(i ═ 1,2,3 …) represents the generation state of the i-th MRS.
At the same time, reservation agent 230 may raise an alert to indicate that the storage space of the file system is limited and/or that the metadata storage space has a higher usage. Thus, a storage administrator or user is notified that a rebalancing of the storage space of the file system cannot be achieved, so that the storage administrator or client can take corresponding measures.
FIG. 6 illustrates a flow diagram of an example process 600 for balancing metadata storage space and data storage space, in accordance with certain embodiments of the present disclosure.
As shown in fig. 6, process 600 begins at block 605. At block 610, the current allocation ratio of storage space of the file system is determined. At block 615, it is determined whether the current allocation proportion of storage space is greater than or equal to a second threshold proportion 245 (e.g., the mid-water mark). If so, process 600 proceeds to block 620, where reservation agent 230 is invoked. At block 625, the current allocation proportion of the storage space of the file system is checked. If the current allocation of storage space is greater than or equal to the second threshold proportion 245 and less than the fourth threshold proportion 255 (e.g., the high watermark), the process 600 proceeds to block 630 where it is determined whether the current allocation of metadata storage space is greater than the first threshold proportion 235. If so, storage space is reserved for the corresponding type of metadata at block 635 and allocated data storage space that has not been accessed for a predetermined period of time is reclaimed.
At block 640, the latest allocation proportion of the storage space of the file system is determined. The latest allocation ratio of the storage space is then checked at block 645. If the latest allocation ratio of the storage space is still greater than or equal to the second threshold ratio 245 and less than the fourth threshold ratio 255, then returning to block 630, the determination of the allocation ratio of the metadata storage space continues. If the latest allocation proportion of storage space is less than or equal to the third threshold proportion 250 (e.g., the low watermark), then at block 650, the reservation agent 230 is stopped. Next, at block 655, metadata storage space is allocated from the storage space of the file system. At block 660, the process 600 ends.
If it is determined at block 645 that the most recent allocation proportion of storage space is greater than or equal to the fourth threshold proportion 255, the process 600 proceeds to block 660, where the corresponding reserved storage space is released according to the generation status of the reserved storage space and the administrator is notified for subsequent expansion. Reservation agent 230 is then stopped at block 670. The process 600 then ends at block 660.
Table 1 below shows the metadata blocks and data block usage change states in a 100GB reduced file system.
TABLE 1
At time T1, the allocation of storage space for the file system is 78% less than a third threshold proportion 250 (e.g., 80%). At this time, the allocation of the metadata blocks and the data blocks will be performed in the storage space of the file system in a conventional manner.
At time T2, a number of new small files or directories have been created, the allocation proportion of the storage space of the file system is 90%, and the second threshold proportion 245 is reached. At this point, reservation agent 230 will be initialized or invoked and reserve the MRS for the metadata. At the same time, reservation agent 230 will begin scanning LRU free back end data blocks for reclamation.
At time T3, the metadata block is increased by 4GB by the MRS. At the same time, reservation agent 230 will reduce the free backend data blocks to satisfy 1) the new allocation request, and 2) reclaim the LRU free backend data blocks to the MRS for use by the metadata. In this way, there is a rebalancing between the metadata blocks and the data blocks. There are still free back-end data blocks in the file system and the metadata blocks will not run out.
At time T4, the allocation of storage space for the file system is 96%, exceeding a fourth threshold proportion of 255 (e.g., 95%). At this point, there are no free backend data blocks already, meaning that the rebalancing request of reservation agent 230 cannot be supported. Thus, an alert will be sent to the client to indicate that the storage space of the file system is limited and that the usage of the metadata storage space is high.
FIG. 7 illustrates a flow diagram of an example method 700 of balancing storage space of a file system in accordance with certain embodiments of the present disclosure. The method 700 may be implemented in the environment 200 as shown in fig. 2. For ease of discussion, the method 700 will be described below in conjunction with FIG. 2.
As shown in FIG. 7, at block 705, a determination is made whether an allocation proportion of a first metadata storage space for a first type of metadata of the one or more types of metadata for the file system (referred to as a "first allocation proportion") is greater than a first threshold proportion. At block 710, if it is determined that the first allocation proportion is above a first threshold proportion, storage space is reserved in the storage space of the file system for the first type of metadata.
In some embodiments, data storage space that has not been accessed within a predetermined period of time may be determined among the allocated data storage space of the file system. The unaccessed data storage space may then be reclaimed for reservation for one or more types of metadata.
In some embodiments, it may again be determined whether the current allocation proportion of the first metadata storage space for the first type of metadata is greater than the first threshold proportion. The reserved storage space may be released if it is determined that the current allocation ratio is less than or equal to the first threshold ratio.
In some embodiments, it may be determined whether an allocation proportion of a second metadata storage space for a second type of metadata of the one or more types of metadata (referred to as a "second allocation proportion") is greater than a first threshold proportion. If it is determined that the second allocation proportion is greater than the first threshold proportion, a first priority of the first type of metadata is compared with a second priority of the second type of metadata. Based on the first priority being higher than the second priority, storage space may be preferentially reserved in the storage space of the file system for the first type of metadata.
In some embodiments, it may be determined whether the allocation proportion of the storage space of the file system (referred to as the "third allocation proportion") is greater than or equal to a second threshold proportion. If it is determined that the third allocation percentage is greater than or equal to the first threshold percentage, then it is determined whether the first allocation percentage is greater than the first threshold percentage.
In some embodiments, it may be determined whether a third allocated proportion of storage space of the file system is less than or equal to a third threshold proportion, wherein the third threshold proportion is less than the second threshold proportion. The reserved storage space may be released if it is determined that the third allocation proportion is less than or equal to the third threshold proportion.
In some embodiments, it may be determined whether a third allocation proportion of storage space of the file system is greater than or equal to a fourth threshold proportion, wherein the fourth threshold proportion is greater than the second threshold proportion. If it is determined that the third allocation proportion is greater than or equal to the fourth threshold proportion, the reserved storage space may be released.
In some embodiments, if it is determined that the third allocation percentage is greater than or equal to a fourth threshold percentage, it may be determined whether there is data storage space in the allocated data storage space of the file system that has not been accessed for a predetermined period of time. If it is determined that there is no unaccessed data storage space, the reserved storage space may be freed. In some embodiments, an alert may also be raised to indicate that the storage space of the file system is limited.
It should be understood that the operations and features described above in connection with fig. 2 through 5 are equally applicable to the method 700 and have the same effect, and detailed description is omitted.
FIG. 8 shows a schematic block diagram of a device 800 that may be used to implement embodiments of the present disclosure.
As shown in fig. 8, device 800 includes a controller or processor, or Central Processing Unit (CPU)801, which may perform various appropriate actions and processes in accordance with programs stored in a Read Only Memory (ROM)802 and/or a Random Access Memory (RAM) 803. The ROM 802 and/or RAM 803 may store various programs and data required for operation of the device 800. The CPU 801, ROM 802, and RAM 803 are connected to each other via a bus 804. In particular, device 800 also includes one or more special-purpose processing units (not shown), which may also be connected to bus 804.
An input/output (I/O) interface 805 is also connected to bus 804. A number of components in the device 800 are connected to the I/O interface 805, including: an input unit 806, such as a keyboard, a mouse, or the like; an output unit 807 such as various types of displays, speakers, and the like; a storage unit 808, such as a magnetic disk, optical disk, or the like; and a communication unit 809 such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks. In particular, in embodiments of the present disclosure, the communication unit 809 supports communication with clients or other devices.
In some embodiments, the CPU 801 may be configured to perform the various processes and processes described above, such as the method 700. For example, in some embodiments, the method 700 may be implemented as a computer software program tangibly embodied on a computer-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and/or installed onto device 800 via ROM 802 and/or communications unit 809. When loaded into RAM 803 and executed by CPU 801, a computer program may perform one or more of the steps of method 700 described above. Alternatively, in other embodiments, the CPU 801 may also be configured in any other suitable manner to implement the processes/methods described above.
In particular, according to embodiments of the present disclosure, the processes described above with reference to fig. 1-7 may be implemented as a computer program product, which may be tangibly stored on a non-transitory computer-readable storage medium and include computer-executable instructions that, when executed, cause a device to implement various aspects in accordance with the present disclosure.
The computer readable storage medium may be a tangible device that may store instructions for use by an instruction execution device. The computer readable storage medium may include, for example, but is not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific, non-exhaustive examples of the computer readable storage medium include: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.
The computer program instructions for carrying out operations of the present disclosure may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, the electronic circuitry that can execute the computer-readable program instructions implements aspects of the present disclosure by utilizing the state information of the computer-readable program instructions to personalize the electronic circuitry, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA).
Various aspects of the present disclosure are described herein with reference to block diagrams and/or flowchart illustrations of apparatus, methods, and computer program products according to embodiments of the disclosure. It will be understood that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by computer-readable program instructions.
Various embodiments of the present disclosure have been described for purposes of illustration, but the present disclosure is not intended to be limited to the disclosed embodiments. All such modifications and variations are within the scope of the disclosure as defined in the appended claims without departing from the spirit thereof.
Claims (19)
1. A method of balancing storage space of a file system, comprising:
determining whether a first allocation proportion of a first metadata storage space for a first type of metadata of the one or more types of metadata of the file system is greater than a first threshold proportion; and
in accordance with a determination that the first allocation percentage is greater than the first threshold percentage, reserving storage space for the first type of metadata in the storage space of the file system.
2. The method of claim 1, further comprising:
determining, among the allocated data storage spaces of the file system, data storage spaces that have not been accessed within a predetermined period of time; and
reclaiming the unaccessed data storage space for reservation for the one or more types of metadata.
3. The method of claim 1, further comprising:
determining whether a current allocation proportion of a first metadata storage space for a first type of metadata is greater than the first threshold proportion; and
in accordance with a determination that the current allocation ratio is less than or equal to the first threshold ratio, releasing the reserved storage space.
4. The method of claim 1, wherein reserving the storage space for the first type of metadata comprises:
determining whether a second allocation proportion of a second metadata storage space for a second type of metadata of the one or more types of metadata is greater than the first threshold proportion;
in accordance with a determination that the second allocation proportion is greater than the first threshold proportion, comparing a first priority of the first type of metadata to a second priority of the second type of metadata; and
preferentially reserving the storage space for the first type of metadata in the storage space of the file system based on the first priority being higher than the second priority.
5. The method of claim 1, wherein determining whether the first allocation proportion is greater than the first threshold proportion comprises:
determining whether a third allocation proportion of the storage space of the file system is greater than or equal to the second threshold proportion; and
in accordance with a determination that the third allocation percentage is greater than or equal to a first threshold percentage, determining whether the first allocation percentage is greater than the first threshold percentage.
6. The method of claim 5, further comprising:
determining whether the third allocation proportion of the storage space of the file system is less than or equal to a third threshold proportion, the third threshold proportion being less than the second threshold proportion;
in accordance with a determination that the third allocation proportion is less than or equal to the third threshold proportion, releasing the reserved storage space.
7. The method of claim 5, further comprising:
determining whether the third allocation proportion of the storage space of the file system is greater than or equal to a fourth threshold proportion, the fourth threshold proportion being greater than the second threshold proportion;
releasing the reserved storage space in accordance with a determination that the third allocation percentage is greater than or equal to the fourth threshold percentage.
8. The method of claim 7, further comprising:
causing an alert to indicate that the storage space of the file system is limited.
9. The method of claim 7, wherein releasing the reserved storage space comprises:
in accordance with a determination that the third allocation percentage is greater than or equal to the fourth threshold percentage, determining whether there is data storage space in the allocated data storage spaces of the file system that has not been accessed within a predetermined period of time; and
and releasing the reserved storage space according to the fact that the unaccessed data storage space does not exist.
10. An apparatus for balancing storage space of a file system, comprising:
a processor, and
a memory storing computer-executable instructions that, when executed by the processor, cause the device to perform acts comprising:
determining whether a first allocation proportion of a first metadata storage space for a first type of metadata of the one or more types of metadata of the file system is greater than a first threshold proportion; and
in accordance with a determination that the first allocation percentage is greater than the first threshold percentage, reserving storage space for the first type of metadata in the storage space of the file system.
11. The apparatus of claim 10, wherein the actions further comprise:
determining, among the allocated data storage spaces of the file system, data storage spaces that have not been accessed within a predetermined period of time; and
reclaiming the unaccessed data storage space for reservation for the one or more types of metadata.
12. The apparatus of claim 10, wherein the actions further comprise:
determining whether a current allocation proportion of a first metadata storage space for a first type of metadata is greater than the first threshold proportion; and
in accordance with a determination that the current allocation ratio is less than or equal to the first threshold ratio, releasing the reserved storage space.
13. The apparatus of claim 10, wherein reserving the storage space for the first type of metadata comprises:
determining whether a second allocation proportion of a second metadata storage space for a second type of metadata of the one or more types of metadata is greater than the first threshold proportion;
in accordance with a determination that the second allocation proportion is greater than the first threshold proportion, comparing a first priority of the first type of metadata to a second priority of the second type of metadata; and
preferentially reserving the storage space for the first type of metadata in the storage space of the file system based on the first priority being higher than the second priority.
14. The apparatus of claim 10, wherein determining whether the first allocation proportion is greater than the first threshold proportion comprises:
determining whether a third allocation proportion of the storage space of the file system is greater than or equal to the second threshold proportion; and
in accordance with a determination that the third allocation percentage is greater than or equal to a first threshold percentage, determining whether the first allocation percentage is greater than the first threshold percentage.
15. The apparatus of claim 14, wherein the actions further comprise:
determining whether the third allocation proportion of the storage space of the file system is less than or equal to a third threshold proportion, the third threshold proportion being less than the second threshold proportion;
in accordance with a determination that the third allocation proportion is less than or equal to the third threshold proportion, releasing the reserved storage space.
16. The apparatus of claim 14, wherein the actions further comprise:
determining whether the third allocation proportion of the storage space of the file system is greater than or equal to a fourth threshold proportion, the fourth threshold proportion being greater than the second threshold proportion;
releasing the reserved storage space in accordance with a determination that the third allocation percentage is greater than or equal to the fourth threshold percentage.
17. The apparatus of claim 16, wherein the actions further comprise:
causing an alert to indicate that the storage space of the file system is limited.
18. The apparatus of claim 18, wherein releasing the reserved storage space comprises:
in accordance with a determination that the third allocation percentage is greater than or equal to the fourth threshold percentage, determining whether there is data storage space in the allocated data storage spaces of the file system that has not been accessed within a predetermined period of time; and
and releasing the reserved storage space according to the fact that the unaccessed data storage space does not exist.
19. A computer program product tangibly stored on a non-transitory computer-readable medium and comprising machine executable instructions that, when executed, cause a machine to perform the method of claims 1 to 9.
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911046051.7A CN112749135B (en) | 2019-10-30 | 2019-10-30 | Method, apparatus and computer program product for balancing storage space of a file system |
US16/877,910 US11681447B2 (en) | 2019-10-30 | 2020-05-19 | Method, device and computer program product of balance of storage space for file system |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911046051.7A CN112749135B (en) | 2019-10-30 | 2019-10-30 | Method, apparatus and computer program product for balancing storage space of a file system |
Publications (2)
Publication Number | Publication Date |
---|---|
CN112749135A true CN112749135A (en) | 2021-05-04 |
CN112749135B CN112749135B (en) | 2024-06-07 |
Family
ID=75640640
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201911046051.7A Active CN112749135B (en) | 2019-10-30 | 2019-10-30 | Method, apparatus and computer program product for balancing storage space of a file system |
Country Status (2)
Country | Link |
---|---|
US (1) | US11681447B2 (en) |
CN (1) | CN112749135B (en) |
Families Citing this family (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11295031B2 (en) | 2019-10-08 | 2022-04-05 | International Business Machines Corporation | Event log tamper resistance |
US11392348B2 (en) * | 2020-02-13 | 2022-07-19 | International Business Machines Corporation | Ordering records for timed meta-data generation in a blocked record environment |
US11237738B2 (en) | 2020-03-16 | 2022-02-01 | EMC IP Holding Company, LLC | Managing operation of a data storage system with reservation of metadata space |
US11372846B2 (en) * | 2020-07-17 | 2022-06-28 | International Business Machines Corporation | Generating and utilizing pre-allocated storage space |
US20240202080A1 (en) * | 2022-12-20 | 2024-06-20 | Dell Products L.P. | Prioritization in cloud migration for deduplication systems |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2002039279A2 (en) * | 2000-11-07 | 2002-05-16 | Surgient Networks, Inc. | Systems and methods for resource monitoring in information storage environments |
US20020091722A1 (en) * | 2000-03-03 | 2002-07-11 | Surgient Networks, Inc. | Systems and methods for resource management in information storage environments |
US6928459B1 (en) * | 2000-07-18 | 2005-08-09 | International Business Machines Corporation | Plurality of file systems using weighted allocation to allocate space on one or more storage devices |
US8443153B1 (en) * | 2010-01-06 | 2013-05-14 | Netapp, Inc. | Dynamic balancing of performance with block sharing in a storage system |
US20160173603A1 (en) * | 2014-12-10 | 2016-06-16 | International Business Machines Corporation | Method and apparatus for managing file system of unified storage system |
US20190235782A1 (en) * | 2018-01-31 | 2019-08-01 | EMC IP Holding Company LLC | Techniques for rebalancing storage between subspaces |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9355121B1 (en) * | 2013-06-28 | 2016-05-31 | Emc Corporation | Segregating data and metadata in a file system |
US9870366B1 (en) * | 2015-09-18 | 2018-01-16 | EMC IP Holding Company LLC | Processing storage capacity events in connection with file systems |
US9678683B1 (en) * | 2016-11-01 | 2017-06-13 | Red Hat, Inc. | Lazy persistent storage volume provisioning |
US10168910B2 (en) * | 2017-03-09 | 2019-01-01 | International Business Machines Corporation | Storage area network reclamation for virtual environments |
US10936195B2 (en) | 2017-10-30 | 2021-03-02 | EMC IP Holding Company LLC | Data storage system using in-memory structure for reclaiming space from internal file system to pool storage |
US10235286B1 (en) | 2017-10-30 | 2019-03-19 | EMC IP Holding Company LLC | Data storage system dynamically re-marking slices for reclamation from internal file system to pool storage |
US10866869B2 (en) * | 2019-01-16 | 2020-12-15 | Vmware, Inc. | Method to perform crash and failure recovery for a virtualized checkpoint protected storage system |
-
2019
- 2019-10-30 CN CN201911046051.7A patent/CN112749135B/en active Active
-
2020
- 2020-05-19 US US16/877,910 patent/US11681447B2/en active Active
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020091722A1 (en) * | 2000-03-03 | 2002-07-11 | Surgient Networks, Inc. | Systems and methods for resource management in information storage environments |
US6928459B1 (en) * | 2000-07-18 | 2005-08-09 | International Business Machines Corporation | Plurality of file systems using weighted allocation to allocate space on one or more storage devices |
WO2002039279A2 (en) * | 2000-11-07 | 2002-05-16 | Surgient Networks, Inc. | Systems and methods for resource monitoring in information storage environments |
US8443153B1 (en) * | 2010-01-06 | 2013-05-14 | Netapp, Inc. | Dynamic balancing of performance with block sharing in a storage system |
US20160173603A1 (en) * | 2014-12-10 | 2016-06-16 | International Business Machines Corporation | Method and apparatus for managing file system of unified storage system |
US20190235782A1 (en) * | 2018-01-31 | 2019-08-01 | EMC IP Holding Company LLC | Techniques for rebalancing storage between subspaces |
Also Published As
Publication number | Publication date |
---|---|
CN112749135B (en) | 2024-06-07 |
US20210132829A1 (en) | 2021-05-06 |
US11681447B2 (en) | 2023-06-20 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN112749135B (en) | Method, apparatus and computer program product for balancing storage space of a file system | |
KR101357397B1 (en) | Method for tracking memory usages of a data processing system | |
US10572285B2 (en) | Method and apparatus for elastically scaling virtual machine cluster | |
CN110753131A (en) | Microservice distributed current limiting method and device, storage medium and electronic equipment | |
CN110389905B (en) | Resource release method, resource allocation method, device and computer program product | |
CN107209716B (en) | Memory management device and method | |
EP3865992A2 (en) | Distributed block storage system, method, apparatus and medium | |
KR102469927B1 (en) | Apparatus for managing disaggregated memory and method for the same | |
US8458719B2 (en) | Storage management in a data processing system | |
CN111464331B (en) | Control method and system for thread creation and terminal equipment | |
CN110209354B (en) | Method, apparatus, device and medium for processing data | |
CN114168490A (en) | Method for determining memory recovery threshold and related equipment | |
CN110471769B (en) | Resource management method and device for virtual machine | |
US10896062B2 (en) | Inter-process memory management | |
US11403026B2 (en) | Method, device and computer program product for managing storage system | |
CN111124256A (en) | Method, apparatus and computer program product for managing storage | |
WO2022048674A1 (en) | Server cabinet-based virtual machine management method and apparatus | |
CN112291326B (en) | Load balancing method, load balancing device, storage medium and electronic equipment | |
CN114518962A (en) | Memory management method and device | |
CN114253663A (en) | Virtual machine resource scheduling method and device | |
CN113391757B (en) | Node expansion method and device and migration node | |
CN114327862A (en) | Memory allocation method and device, electronic equipment and storage medium | |
CN115840635A (en) | Computing resource management method, electronic device, and program product | |
CN111143073A (en) | Management method, equipment and storage medium of virtualized resources | |
CN116431327B (en) | Task current limiting processing method and fort machine |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant |