CN117520278A - Multi-client high-precision directory quota control method for distributed file system - Google Patents
Multi-client high-precision directory quota control method for distributed file system Download PDFInfo
- Publication number
- CN117520278A CN117520278A CN202311591254.0A CN202311591254A CN117520278A CN 117520278 A CN117520278 A CN 117520278A CN 202311591254 A CN202311591254 A CN 202311591254A CN 117520278 A CN117520278 A CN 117520278A
- Authority
- CN
- China
- Prior art keywords
- quota
- directory
- file system
- information
- checking
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 27
- 238000012795 verification Methods 0.000 claims description 45
- 238000010586 diagram Methods 0.000 description 7
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1727—Details of free space management performed by the file system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/176—Support for shared access to files; File sharing support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a high-precision directory quota control method for multiple clients of a distributed file system, which comprises the steps of firstly creating multiple directory quotas, storing quota information of different directories in different metadata servers, collecting self quota checking information by each file system client when the multiple file system clients request to write data into a target quota directory, then sending the self quota checking information to the target metadata server, and sequentially carrying out quota checking on all quota checking information and quota information of the target quota directory after the target metadata server receives the quota checking information sent by all the file system clients, wherein each file system client writes the data into a storage server or returns errors to a user according to a quota checking result. The method can effectively reduce the load of the metadata server, avoid errors when a user operates the client, and avoid that the written catalog capacity can not exceed the quota limit threshold when the multiple clients concurrently perform data operation on the same catalog.
Description
Technical Field
The invention relates to the technical field of computer storage, in particular to a multi-client high-precision directory quota control method for a distributed file system.
Background
With the advent of the big data age, the demand of service applications for storage space is increasing, and the storage requirement of current service applications cannot be met by the data storage capability of single machine storage, so that the storage system is developing towards the distributed direction. Distributed file system cluster storage systems typically include tens of nodes, with total storage up to hundreds of TBs, and typically such large storage is allocated to multiple users simultaneously, and when allocated, the storage is distributed to different users in the form of directories.
In order to prevent the storage space from being used excessively by individual users, affecting the normal use of other users, it is necessary to use the file system to make directory quotas, limiting the maximum usable capacity allocated to directories used by individual users.
As shown in fig. 1, fig. 1 is a schematic diagram of an implementation architecture of a directory quota method in the prior art, a user 1 performs operations such as file reading and writing on an allocated directory through a file system client 1; the file system client 1 can acquire quota information of the user 1 from the metadata server, monitor data operation of the user 1 by using the quota information, count operation quota information of the user 1, and send the operation quota information to the metadata server; the metadata server is used for updating quota information of the user according to the operation quota information and sharing the updated quota information to the file system client side 2; when the user 2 performs operations such as file reading and writing through the file system client 2, the user is constrained by updated quota information. In the directory quota method in the prior art, the limitation of the maximum usable capacity of the directory used by a single user is realized by synchronizing the latest quota information on the metadata server by each file system client.
However, in the directory quota method in the prior art, all directory quota information is stored on one metadata server, and the metadata server has a larger load; moreover, it may happen that each file system client fails to synchronize the latest quota information on the metadata server in time, resulting in errors when the user operates the client; in addition, the catalog quota information is stored by the metadata server, but the quota limitation is finally carried out by each client, the change of the quota information caused by the data operation of each client cannot be transmitted to other clients in real time, and when multiple clients concurrently carry out the data operation on the same catalog, the actual capacity of the catalog exceeds the threshold value of the quota limitation.
Disclosure of Invention
Based on the foregoing, it is necessary to provide a method for controlling a high-precision directory quota of a distributed file system, which can effectively reduce the load of a metadata server, avoid errors when a user operates a client, and avoid that the capacity of a directory written by multiple clients when the multiple clients concurrently perform data operation on the same directory does not exceed a quota limiting threshold.
The invention provides a distributed file system multi-client high-precision directory quota control method, wherein a plurality of clients share a plurality of metadata servers and a plurality of storage servers, and each user accesses the metadata servers and the storage servers through one file system client, and the method comprises the following steps:
creating a plurality of directory quotas and storing quota information of different directories into different metadata servers;
when a plurality of users request to write data into a target quota directory through respective corresponding file system clients, each file system client collects self quota checking information;
all file system clients send self quota checking information to a target metadata server, wherein the target metadata server is the metadata server where the target quota directory is located;
after receiving the quota checking information sent by all file system clients, the target metadata server sequentially performs quota checking on all the quota checking information and the quota information of the target quota directory according to a quota checking information receiving sequence, wherein when performing quota checking each time, the target metadata server returns a quota checking result of the current quota checking information and the quota information of the target quota directory to the file system client corresponding to the current quota checking information;
and each file system client writes the data into the storage server or returns an error to the user according to the quota checking result, wherein when the quota checking result is successful, the file system client writes the data into the storage server, otherwise, the file system client returns the error to the user.
In one embodiment, creating multiple directory quotas adds quota information for multiple different directories.
In one embodiment, the quota information of each directory includes a directory path, a user to which the directory belongs, a hard threshold and a soft threshold of the directory.
In one embodiment, the quota checking information includes data amount of data to be written, the user to which the quota checking information belongs, operation type, and metadata server information where the target quota directory is located.
In one embodiment, when quota verification is performed each time, the target metadata server locks all the received quota verification information, if the quota verification is successful, unlocks all the quota verification information after the quota capacity of the quota information of the target quota directory is increased, then returns a success result of the quota verification to the file system client corresponding to the quota verification information of the current quota verification, and if the quota verification is unsuccessful, directly returns a failure result of the quota verification to the file system client corresponding to the quota verification information of the current quota verification.
In one embodiment, the quota checking is to determine whether the sum of the data amount of the existing data and the data amount of the data to be written in the target quota directory exceeds a hard threshold of the target quota directory, if so, the quota checking fails, and if not, the quota checking is successful.
In one embodiment, the distributed file system is provided with a quota setting interface for creating, deleting, modifying, and querying directory quotas.
In one embodiment, quota information for different directories is saved to different metadata servers according to load balancing.
In one embodiment, a quota checking module is provided in each metadata server, and the quota checking module is configured to perform quota checking on quota checking information and quota information of the target quota directory.
The beneficial effects of the invention are as follows:
(1) According to the method and the device, the quota information of different catalogues is stored in different metadata servers, so that the load of a single metadata server is effectively reduced.
(2) And each file system client does not store quota information of the catalogue, so that errors caused by delay in synchronizing information between the metadata server and the file system clients are avoided.
(3) Quota verification is performed in the metadata server, so that the situation that modification of quota information of a directory by one file system client cannot be timely synchronized to other file system clients can be avoided, and errors are avoided when a user operates the clients.
(4) When a plurality of users request to write data into a target quota directory through respective corresponding file system clients, the target metadata server sequentially carries out quota verification on all quota verification information and quota information of the target quota directory according to a quota verification information receiving sequence, and the quota verification is carried out by using uniform and real-time quota verification information, so that accuracy of the quota verification information of each file system client can be ensured, and capacity of the target quota directory after data writing can be ensured not to exceed a quota limiting threshold.
Drawings
FIG. 1 is a schematic diagram of an implementation architecture of a directory quota method in the prior art;
FIG. 2 is a schematic diagram of the architecture of a distributed file system of the present invention;
FIG. 3 is a schematic flow chart of a method for controlling high-precision directory quota of multiple clients in a distributed file system according to an embodiment of the present invention;
fig. 4 is a schematic diagram of a quota checking flow provided in an embodiment of the invention.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
As shown in fig. 2, fig. 2 is a schematic architecture diagram of a distributed file system of the present invention, which includes a plurality of clients sharing a plurality of metadata servers and a plurality of storage servers, each user accessing the metadata servers and the storage servers through one file system client.
In one embodiment, as shown in fig. 3, fig. 3 is one of flow diagrams of a distributed file system multi-client high-precision directory quota control method provided by the embodiment of the present invention, where the distributed file system multi-client high-precision directory quota control method includes:
s301, creating a plurality of catalogue quotas and storing the quotas of different catalogues into different metadata servers.
In this embodiment, creating a plurality of directory quotas adds quota information to a plurality of different directories, and the directory to which quota information has been added is referred to as a quota directory. The quota information of each directory comprises a directory path, a user to which the directory belongs, a hard threshold and a soft threshold of the directory, wherein the hard threshold of the directory is the maximum capacity of data stored in the directory.
The quota information of different catalogues is stored in different metadata servers, so that the load of a single metadata server can be effectively reduced, the quota information of the catalogues is stored in the metadata server instead of each file system client, and errors caused by delay in synchronizing information between the metadata server and the file system clients can be avoided.
S302, when a plurality of users respectively request to write data into a target quota directory through respective corresponding file system clients, each file system client collects self quota checking information.
The quota directory refers to a directory to which quota information has been added.
In this embodiment, the quota checking information includes data amount of data to be written, the user to which the quota checking information belongs, operation type, and metadata server information where the target quota directory is located. The types of the self quota checking information collected by each file system client are consistent, and the uniformity is achieved.
S303, all file system clients send self quota checking information to a target metadata server, wherein the target metadata server is the metadata server where the target quota directory is located.
S304, after the target metadata server receives the quota checking information sent by all the file system clients, the target metadata server sequentially performs quota checking on all the quota checking information and the quota information of the target quota directory according to the quota checking information receiving sequence, wherein when performing quota checking each time, the target metadata server returns a quota checking result of the current quota checking information and the quota information of the target quota directory to the file system client corresponding to the current quota checking information.
And when quota verification is carried out each time, the real-time quota verification information sent by the file system clients is verified, and the quota verification information has uniformity, so that the accuracy of the quota verification information sent by each file system client is ensured, and the capacity of the target quota directory after data writing is ensured not to exceed a quota limit threshold.
Quota verification is performed in the metadata server, so that the situation that modification of quota information of a directory by one file system client cannot be timely synchronized to other file system clients can be avoided, and errors are avoided when a user operates the clients.
In this embodiment, the quota checking is to determine whether the sum of the data amount of the existing data and the data amount of the data to be written in the target quota directory exceeds the hard threshold of the target quota directory, if so, the quota checking fails, and if not, the quota checking is successful.
S305, each file system client writes the data into the storage server or returns errors to the user according to the quota checking result, wherein when the quota checking result is successful, the file system client writes the data into the storage server, otherwise, the file system client returns the errors to the user.
In one embodiment, when quota verification is performed each time, the target metadata server locks all the received quota verification information, if the quota verification is successful, unlocks all the quota verification information after the quota capacity of the quota information of the target quota directory is increased, then returns a success result of the quota verification to the file system client corresponding to the quota verification information of the current quota verification, and if the quota verification is unsuccessful, directly returns a failure result of the quota verification to the file system client corresponding to the quota verification information of the current quota verification.
In one embodiment, a quota checking module is provided in each metadata server, and the quota checking module is configured to perform quota checking on quota checking information and quota information of the target quota directory.
It should be noted that, the quota checking module in this embodiment is also suitable for checking the quota when the single user writes data into the metadata server through the file system client, directly checking the quota checking information of the file system client of the single user with the quota information of the target quota directory in the metadata server, where the checking is successful, the file system client of the single user writes the data requested to be written by the user into the target quota directory, and if the data fails, returns the quota failure result to the file system client.
In one embodiment, a directory quota check procedure is illustrated. For example, the user 1 and the user 2 simultaneously request to write data to the target quota directory in the metadata server 1 through the corresponding file system clients 1 and 2, the directory quota checking flow is shown in fig. 4, and fig. 4 is a schematic diagram of a quota checking flow provided in an embodiment of the present invention. After the metadata server 1 receives the quota checking information sent by the file system clients 1 and 2, quota checking is sequentially performed in a quota checking module on the metadata server according to the order of receiving the quota checking information, the quota checking of the quota checking information 1 is successful, the file system client 1 writes data into a storage server, the quota checking of the quota checking information 2 fails, the data writing fails, a failure result is returned to the file system client 2, and the file system client 2 returns an error to the user 2.
In one embodiment, the quota checking is to determine whether the sum of the data amount of the existing data and the data amount of the data to be written in the target quota directory exceeds a hard threshold of the target quota directory, if so, the quota checking fails, and if not, the quota checking is successful.
In one embodiment, the distributed file system is provided with a quota setting interface for creating, deleting, modifying, and querying directory quotas. The user can synchronize operations such as adding, deleting, modifying and checking any directory quota to all clients in real time before the operations are completed, so as to ensure that all file system clients store the latest quota information.
In one embodiment, quota information for different directories is saved to different metadata servers according to load balancing.
Specifically, load balancing refers to keeping the quota of the directory in each metadata server as consistent as possible, i.e. the amount of data that can be saved is consistent.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The foregoing examples illustrate only a few embodiments of the invention and are described in detail herein without thereby limiting the scope of the invention. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the invention, which are all within the scope of the invention. Accordingly, the scope of the invention should be assessed as that of the appended claims.
Claims (9)
1. A distributed file system multi-client high precision directory quota control method, wherein a plurality of clients share a plurality of metadata servers and a plurality of storage servers, each user accessing the metadata servers and the storage servers through one file system client, the method comprising:
creating a plurality of directory quotas and storing quota information of different directories into different metadata servers;
when a plurality of users request to write data into a target quota directory through respective corresponding file system clients, each file system client collects self quota checking information;
all file system clients send self quota checking information to a target metadata server, wherein the target metadata server is the metadata server where a target quota directory is located;
after receiving the quota checking information sent by all file system clients, the target metadata server sequentially performs quota checking on all the quota checking information and the quota information of the target quota directory according to a quota checking information receiving sequence, wherein when performing quota checking each time, the target metadata server returns a quota checking result of the current quota checking information and the quota information of the target quota directory to the file system client corresponding to the current quota checking information;
and each file system client writes the data into the storage server or returns an error to the user according to the quota checking result, wherein when the quota checking result is successful, the file system client writes the data into the storage server, otherwise, the file system client returns the error to the user.
2. The method of claim 1, wherein creating a plurality of directory quotas adds quota information for a plurality of different directories.
3. The method for controlling the quota of the high-precision directory of the multiple clients of the distributed file system according to claim 2, wherein the quota information of each directory comprises a directory path, a user to which the directory belongs, a hard threshold and a soft threshold of the directory.
4. The method for controlling a multi-client high-precision directory quota of a distributed file system according to claim 3, wherein the quota checking information comprises data volume of data to be written, user, operation type and metadata server information of a target quota directory.
5. The method for controlling the quota of the multi-client high-precision directory of the distributed file system according to claim 4, wherein each time the quota verification is performed, the target metadata server locks all the received quota verification information, if the quota verification is successful, unlocks all the quota verification information after the quota information of the target quota directory increases the quota capacity, returns a success result of the quota verification to the file system client corresponding to the quota verification information of the current quota verification, and if the quota verification is unsuccessful, directly returns a failure result of the quota verification to the file system client corresponding to the quota verification information of the current quota verification.
6. The method for controlling a quota of a multi-client high precision directory of a distributed file system according to claim 5, wherein the quota check is to determine whether a sum of a data amount of existing data in a target quota directory and a data amount of data to be written exceeds a hard threshold of the target quota directory, if so, the quota check fails, and if not, the quota check is successful.
7. A method of controlling a multi-client high precision directory quota in a distributed file system as in any one of claims 1 to 6 wherein the distributed file system is provided with a quota setting interface for creating, deleting, modifying and querying directory quotas.
8. The method for high-precision directory quota control for multiple clients of a distributed file system as recited in claim 7 wherein quota information for different directories is stored in different metadata servers according to load balancing.
9. The method for controlling the quota of the multi-client high-precision directory of the distributed file system according to claim 8, wherein a quota checking module is arranged in each metadata server, and the quota checking module is used for performing quota checking on quota checking information and quota information of a target quota directory.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311591254.0A CN117520278A (en) | 2023-11-27 | 2023-11-27 | Multi-client high-precision directory quota control method for distributed file system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311591254.0A CN117520278A (en) | 2023-11-27 | 2023-11-27 | Multi-client high-precision directory quota control method for distributed file system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117520278A true CN117520278A (en) | 2024-02-06 |
Family
ID=89751059
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311591254.0A Pending CN117520278A (en) | 2023-11-27 | 2023-11-27 | Multi-client high-precision directory quota control method for distributed file system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117520278A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118034611A (en) * | 2024-04-09 | 2024-05-14 | 联想凌拓科技有限公司 | Method, device, equipment and medium for managing quota of file |
-
2023
- 2023-11-27 CN CN202311591254.0A patent/CN117520278A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118034611A (en) * | 2024-04-09 | 2024-05-14 | 联想凌拓科技有限公司 | Method, device, equipment and medium for managing quota of file |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11153380B2 (en) | Continuous backup of data in a distributed data store | |
| CN101334797B (en) | Distributed file systems and its data block consistency managing method | |
| US7685459B1 (en) | Parallel backup | |
| US9547706B2 (en) | Using colocation hints to facilitate accessing a distributed data storage system | |
| CN112084258A (en) | Data synchronization method and device | |
| CN108762668B (en) | Method and device for processing write conflict | |
| CN111177143B (en) | Key value data storage method and device, storage medium and electronic equipment | |
| US11409711B2 (en) | Barriers for dependent operations among sharded data stores | |
| US12013758B2 (en) | Methods and systems for power failure resistance for a distributed storage system | |
| CN117520278A (en) | Multi-client high-precision directory quota control method for distributed file system | |
| CN113806300B (en) | Data storage method, system, device, equipment and storage medium | |
| CN113051102B (en) | File backup method, device, system, storage medium and computer equipment | |
| CN111399760B (en) | NAS cluster metadata processing method and device, NAS gateway and medium | |
| CN115756955A (en) | Data backup and data recovery method and device and computer equipment | |
| CN109726211B (en) | Distributed time sequence database | |
| US20100043010A1 (en) | Data processing method, cluster system, and data processing program | |
| CN114490540B (en) | Data storage method, medium, device and computing equipment | |
| CN112711606A (en) | Database access method and device, computer equipment and storage medium | |
| CN103685359B (en) | Data processing method and device | |
| CN110442573A (en) | A kind of method and device of distributed fault-tolerance key assignments storage | |
| CN115390754A (en) | Hard disk management method and device | |
| CN116594551A (en) | Data storage method and device | |
| CN111221857B (en) | Method and apparatus for reading data records from a distributed system | |
| CN111399753A (en) | Method and device for writing pictures | |
| CN108023914B (en) | Memory data sharing system, and memory data writing and reading method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |