CN108271420B - Method for managing files, file system and server system - Google Patents
Method for managing files, file system and server system Download PDFInfo
- Publication number
- CN108271420B CN108271420B CN201680003120.4A CN201680003120A CN108271420B CN 108271420 B CN108271420 B CN 108271420B CN 201680003120 A CN201680003120 A CN 201680003120A CN 108271420 B CN108271420 B CN 108271420B
- Authority
- CN
- China
- Prior art keywords
- data
- metadata
- file
- area
- management unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
Abstract
A method of managing files, a file system and a server system. The method comprises the following steps: receiving a file writing request; determining data to be stored of the file and metadata of the file according to the file writing request; determining a plurality of data fragments of the data to be stored of the file; determining local metadata for each of the plurality of data slices; storing each of the plurality of data slices, the local metadata of each of the plurality of data slices, and the metadata of the file to a data area; the metadata of the file is stored to a metadata area. The scheme can effectively protect the metadata.
Description
Technical Field
The present invention relates to the field of information technology, and more particularly, to a method of managing files, a file system, and a server system.
Background
The distributed file system is a relatively complex system, and the managed data can be generally reduced to two clusters of data and metadata. Data is the actual data seen by the user, and basically refers to the actual content of the file. The metadata is also called data of data, and mainly includes directory structure, attribute information of files and directories, mapping relationship between directories, and the like. The metadata is used for realizing functions of quickly searching data and the like, and is internal management information of the file system.
In the current file system mechanism, to find a file or find its information, layer by layer directory search is started from a root directory, and finally a file is found and then the file is opened. The entire hierarchical relationship is recorded by the metadata. If the metadata cluster crashes, even if the data are not damaged, the data cannot be organized, the data cannot be associated with the directory or even the file name, and functions of searching the file and the like cannot be realized.
Currently, to ensure reliability of metadata clusters, various complex logical or physical overheads are employed by each file system. For example, a metadata backup scheme is adopted to ensure the reliability of metadata, and a database or another cluster is adopted to backup part of important metadata or all metadata in the distributed file system; or, the change of the metadata is recorded in a log or snapshot mode, and the metadata at the latest time point can be played back or directly rolled back when a problem occurs, so that the lost content is reduced. The above scheme is complex or has high overhead, and therefore, a more concise metadata protection mechanism is required.
Disclosure of Invention
The embodiment of the invention provides a method for managing files, a file system and a server system, which can effectively protect metadata.
In a first aspect, a method for managing files in a file system is provided, the file system managing a storage area comprising a data area and a metadata area; the method comprises the following steps:
receiving a file writing request;
determining data to be stored of the file and metadata of the file according to the file writing request;
determining a plurality of data fragments of the data to be stored of the file;
determining local metadata for each of the plurality of data slices;
storing each of the plurality of data slices, the local metadata of each of the plurality of data slices, and the metadata of the file to the data area;
the metadata of the file is stored to the metadata area.
The method for managing the file of the embodiment of the invention not only stores the metadata of the file in the metadata area managed by the file system, but also stores the metadata of the file in the data area managed by the file system, thus, when the metadata area fails, the metadata can be recovered through the data area, and the metadata can be effectively protected.
In some possible implementations, storing each of the plurality of data slices, the local metadata of each of the plurality of data slices, and the metadata of the file to the data area includes:
expanding a local metadata region in the data region corresponding to each data slice in the plurality of data slices;
storing each data fragment in the plurality of data fragments to a corresponding data fragment area in the data area;
storing the local metadata of each data fragment in the plurality of data fragments into a sub-region before expansion in a corresponding local metadata region;
and storing the metadata of the file into an expansion sub-area in the local metadata area corresponding to each data slice in the plurality of data slices.
In some possible implementations, the extended sub-region is 512 bytes.
In some possible implementations, storing each of the plurality of data slices, the local metadata of each of the plurality of data slices, and the metadata of the file to the data area includes:
storing each data fragment in the plurality of data fragments to a corresponding data fragment area in the data area;
storing the local metadata of each of the plurality of data slices to a corresponding first sub-region in a local metadata region in the data region;
and storing the metadata of the file into a second sub-area in the local metadata area in the data area corresponding to each data slice in the plurality of data slices.
In some possible implementations, the data slices, the local metadata of the data slices, and the metadata of the file are downloaded together.
According to the method for managing the file, disclosed by the embodiment of the invention, in the data area, the metadata of the file and the local metadata of the data fragment are stored in the local metadata area together, and the increased I/O (input/output) cost is low, so that the reliability of the metadata can be improved with low cost.
In some possible implementations, the method further includes:
and when the metadata area fails, restoring the metadata area according to the metadata of the file in the data area.
In some possible implementations, restoring the metadata area according to the metadata of the file in the data area includes:
reading metadata of files in a local metadata area corresponding to each data fragment in the data area;
and recovering the directory hierarchical relationship in the metadata area according to the read metadata of the file.
In some possible implementations, when a metadata area fails, the data slice list may be scanned, and the metadata of the file in the local metadata area of each data slice may be read according to the data slice list.
In some possible implementations, if the metadata of the file is stored in the last 512 bytes of the local metadata region, the metadata of the file in the 512 bytes is read.
In some possible implementations, the metadata of the file need only be scanned for a local metadata region in the data region.
In the metadata area recovery flow of the embodiment of the invention, only the specific area in the data area needs to be scanned, so the recovery flow is simpler.
In some possible implementations, each time metadata for a file is obtained, a record is added at the directory level of the metadata region.
In some possible implementations, a hash table may be used to record relationships that are not temporarily concatenated.
In some possible implementations, determining a plurality of data slices of the data to be stored for the file includes:
the plurality of data fragments are determined according to a protection mode of the file system.
In some possible implementations, the protection mode of the file system is an EC protection mode.
In a second aspect, there is provided a file system that manages a storage area including a data area and a metadata area; the file system includes: a file management unit, a data management unit and a metadata management unit;
the file management unit is used for receiving a file writing request; determining data to be stored of the file and metadata of the file according to the file writing request; determining a plurality of data fragments of the data to be stored of the file; sending the plurality of data fragments and the metadata of the file to the data management unit; sending the metadata of the file to the metadata management unit;
the data management unit is used for determining local metadata of each data slice in the plurality of data slices; storing each of the plurality of data slices, the local metadata of each of the plurality of data slices, and the metadata of the file to the data area;
the metadata management unit is used for storing the metadata of the file in the metadata area.
The file system of the embodiment of the invention stores the metadata of the file in the metadata area managed by the file system and also stores the metadata of the file in the data area managed by the file system, so that when the metadata area fails, the metadata can be recovered through the data area, and the metadata can be effectively protected.
In some possible implementations, the data management unit is specifically configured to,
expanding a local metadata region in the data region corresponding to each data slice in the plurality of data slices;
storing each data fragment in the plurality of data fragments to a corresponding data fragment area in the data area;
storing the local metadata of each data fragment in the plurality of data fragments into a sub-region before expansion in a corresponding local metadata region;
and storing the metadata of the file into an expansion sub-area in the local metadata area corresponding to each data slice in the plurality of data slices.
In some possible implementations, the data management unit is specifically configured to,
storing each data fragment in the plurality of data fragments to a corresponding data fragment area in the data area;
storing the local metadata of each of the plurality of data slices to a corresponding first sub-region in a local metadata region in the data region;
and storing the metadata of the file into a second sub-area in the local metadata area in the data area corresponding to each data slice in the plurality of data slices.
In some possible implementations, upon failure of the metadata region,
the data management unit is also used for reading the metadata of the file in the data area and sending the read metadata of the file to the metadata management unit;
the metadata management unit is also used for recovering the metadata area according to the metadata of the file sent by the data management unit.
In some possible implementations, the data management unit is specifically configured to read metadata of a file in a local metadata area corresponding to each data slice in the data area, and send the read metadata of the file to the metadata management unit;
the metadata management unit is specifically configured to restore a directory hierarchical relationship in the metadata area according to the metadata of the file sent by the data management unit.
In some possible implementations, the file management unit is specifically configured to determine the plurality of data fragments according to a protection mode of the file system.
In a third aspect, a server system is provided, including:
the file system of the second aspect or any possible implementation of the second aspect; and
a storage resource comprising a storage area managed by the file system.
In a fourth aspect, there is provided a server system comprising:
a storage resource including a storage area managed by a file system, the storage area including a data area and a metadata area;
a management unit configured to perform the method of the first aspect or any one of the possible implementations of the first aspect.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the embodiments of the present invention will be briefly described below, and the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained according to the drawings.
Fig. 1 is a schematic diagram of a server system to which the technical solution of the embodiment of the present invention is applicable.
Fig. 2 is an architecture diagram of a system in accordance with an embodiment of the present invention.
FIG. 3 is a schematic flow chart diagram of a method of managing files in an embodiment of the present invention.
FIG. 4 is a flow chart of writing a file according to an embodiment of the present invention.
FIG. 5 is a flow diagram of recovering metadata according to an embodiment of the present invention.
FIG. 6 is a schematic block diagram of a file system of an embodiment of the present invention.
FIG. 7 is a schematic block diagram of a server system of one embodiment of the present invention.
Fig. 8 is a schematic block diagram of a server system of another embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other examples obtained based on the examples of the present invention shall fall within the scope of protection of the present invention.
The technical scheme of the embodiment of the invention can be applied to various file systems, including distributed file systems and non-distributed file systems.
In the embodiment of the invention, the metadata of the file is additionally stored in the data area besides the metadata area. The data area and the metadata area may belong to different fault domains, e.g. located in different logical locations, or located in different physical locations. Accordingly, protection of metadata of a file can be enhanced. For example, the data fragment, the local metadata of the data fragment, and the metadata of the file are issued to a storage medium (e.g., a hard disk, a solid state disk, etc.) in the same IO request. With each data slice being stored to the data area, metadata for a file may also be stored to the data area. Therefore, the greater the number of data slices, the greater the number of times the metadata is backed up.
Because the metadata of the file is downloaded along with the data fragmentation, the method has very little influence on the system performance and can also greatly improve the reliability of the metadata of the file.
Fig. 1 is a schematic diagram of a server system to which the technical solution of the embodiment of the present invention can be applied.
As shown in fig. 1, the server system 100 may include at least one server 110, and each server 110 includes a storage resource 101, such as a disk, a solid state disk, or other storage resource, providing a storage space. The server 110 may further include a management unit 102, where the management unit 102 implements a file system function to provide a file system access service.
For a distributed file system, the data of a file may be stored in a distributed manner to the storage resources 101 of any server 110; for a non-distributed file system, the data of the file is stored into the storage resources 101 of the corresponding server 110.
Each server 110 may be a data and/or metadata node, i.e., may store only data or metadata, or may store both data and metadata. The area for storing data of all the servers 110 constitutes a data area for file system management, and the area for storing metadata of all the servers constitutes a metadata area for file system management. The data area and the metadata area are logically separated and may be physically located in the same node (e.g., a server) or may be located in different nodes.
It should be understood that in various embodiments of the present invention, the various management units may be functional modules implemented by the processor, for example, may be programs running in the processor; the various management units may also be hardware, e.g. may be one or more processors. The various storage resources may be hardware, such as disks, solid state disks, or volatile memory; the various storage resources may also be logical storage spaces, such as logical storage spaces generated by virtualizing physical storage resources and managed by a management unit. For brevity, further description is omitted below.
FIG. 2 illustrates a logical system architecture diagram of an embodiment of the present invention.
As shown in FIG. 2, system 200 includes metadata cluster 210 and data cluster 220.
The metadata cluster 210 includes a metadata area 211 for storing metadata of files. The metadata region 211 may be located on one or more nodes. The metadata cluster 210 also includes one or more metadata management units 212 for managing metadata of files in the metadata cluster. Similarly, one or more metadata management units 212 may also be located on one or more nodes.
The data cluster 220 includes a data area 221 for storing metadata for the file in addition to data and local metadata for the file. The local metadata may also be referred to as local file system metadata, and includes information such as the size and creation time of the data slice.
The metadata management unit 212 is a functional unit, and may be located in a single server or a distributed file system composed of a plurality of servers. The metadata management unit 212 may be software, and may be collectively composed of distributed software running on a plurality of servers, for example, in a distributed manner; or may be hardware, e.g. consisting of a plurality of server processors, which may run the distributed software. The metadata area 211 accepts management by the metadata management unit 212 and provides a storage space for data read/written by the metadata management unit 212. The metadata area 211 may be a physical storage medium, such as hard disks of a plurality of servers or a solid state disk; or a logical storage space, for example a storage space logically composed of the memories of a plurality of servers.
The data management unit 222 and the data area 221 in the data cluster 220 are similar to the metadata management unit 212 and the metadata area 211, respectively, and are not described herein again.
Alternatively, the local metadata and the metadata of the file may be stored in one block in a local metadata area in the data area 221. For example, the local metadata region may be expanded, the expanded region is referred to as an expanded sub-region of the local metadata region, the local metadata may be stored in the original region, which is referred to as a sub-region before expansion, and the metadata of the file may be stored in the expanded sub-region. In addition, if the space provided by the local metadata region is originally sufficient to store the local metadata and the metadata of the file, or the local metadata region has been expanded in advance, the local metadata and the metadata of the file may be stored in different sub-regions in the local metadata region, that is, in a first sub-region and a second sub-region in the local metadata region, respectively, for example, the first sub-region may be a predetermined byte in the front of the local metadata region, and the second sub-region may be a predetermined byte in the back of the local metadata region.
Similarly, the data region 221 may be located on one or more nodes. For example, each node provides a portion of storage space that collectively make up the data region 221. The data cluster 220 also includes one or more data management units 222 for managing data of files, local metadata, and metadata of files in the data cluster. Similarly, one or more data management units 222 may also be located on one or more nodes. For example, a plurality of nodes each run data management software, the entirety of which constitutes the data management unit 222.
In some implementations, the metadata management unit 212 and the data management unit 222 are functional units in a file system. For example, it may be a functional unit in the management unit 102 in fig. 1. It should be understood that the functional units in the file system may also include other functional units for managing files, and the present invention is not limited thereto.
The metadata area 211 and the data area 221 are storage areas managed by the file system.
The file system and the storage area managed by the file system may be provided in a server system, for example, the server system 100 in fig. 1.
FIG. 3 shows a schematic flow chart of a method 300 of managing files of an embodiment of the present invention.
In an embodiment of the present invention, the storage area managed by the file system includes a data area and a metadata area, for example, the data area 221 and the metadata area 211 in fig. 2.
The method 300 may be performed by a server system or a file system in a server system, for example, by a management unit in a server system or a file system, which in turn may be embodied as one or more management units. For convenience of description, the following description will be given taking a server system as an example.
A write file request is received 310.
When writing a file, the client sends a request to write the file to the server system. The write file request may include identification information of the file, information of the data to be stored and the access address, and the like.
The data to be stored of the file is real data of the file, namely data bearing file content.
The identification information of the file indicates which file is to be written, and may be, for example, a directory name and a file name.
The information of the access address indicates an address to which data to be stored is written, and may be specifically a first address and an address space length, for example.
And 320, determining the data to be stored of the file and the metadata of the file according to the file writing request.
After receiving a write file request, the server system determines the data to be stored of the file and the metadata of the file. The data to be stored of the file can be directly carried in the file writing request; the metadata of the file can be generated according to the information in the file writing request, and is equivalent to be indirectly carried in the file writing request. The metadata of the file includes information such as the directory structure of the file, the attributes of the file and directory, and the like.
A plurality of data slices of the file for which data is to be stored is determined 330.
Alternatively, the plurality of data slices may be determined according to a protection mode of the file system.
The determining of the data slice of the data to be stored of the file may be logically dividing the data to be stored into a plurality of data slices. For example, an Erasure Code (EC) protection scheme may be employed to divide the file into a plurality of data slices.
It should be understood that the present invention is not limited to the protection mode, that is, other protection modes may be adopted.
Optionally, the step 310 and the step 330 may be specifically executed by a file management unit in the system, and then the file management unit sends the plurality of data fragments and the metadata of the file to the data management unit in the system.
At 340, local metadata for each of the plurality of data slices is determined.
When writing a file, two kinds of data need to be processed, one is the data of the file to be stored, and the other is the metadata of the file. After the data to be stored of the file is divided into data fragments, the data fragments are stored in a data area managed by a file system. In addition, each data slice also has corresponding local metadata, and the local metadata comprises information of the size, the creation time and the like of the data slice. The local metadata is also stored in the data area, and specifically may be stored in a local metadata area in the data area.
After obtaining a plurality of data fragments of a file, the server system determines local metadata of each data fragment. Specifically, if no corresponding local metadata exists before, the local metadata is generated according to the new data fragment; and if the local metadata exists, modifying the local metadata according to the new data fragment.
350, storing each of the plurality of data slices, the local metadata of each of the plurality of data slices, and the metadata of the file to the data area.
In this step, each data slice, the local metadata of each data slice, and a chunk of metadata of the file are stored to the data area.
Alternatively, the local metadata of each data slice and the metadata of the file may be stored together in a local metadata region in the data region.
Specifically, the data area may also be divided into two parts, one part is used for storing the data slice and may be referred to as a data slice area; the other part is used for storing metadata and may be referred to as a local metadata area. It is to be understood that other expressions may be used for the above two regions, and the specific expressions should not be construed as limiting the present invention.
In an embodiment of the present invention, metadata of a file and local metadata of each data slice are stored in a local metadata area in one block. The local metadata area in the embodiment of the present invention is large compared to the case where only local metadata of a data slice is stored. In other words, the technical solution of the embodiment of the present invention adopts a larger local metadata region or expands the local metadata region.
Optionally, as an embodiment of the present invention, a local metadata area in the data area corresponding to each data slice in the plurality of data slices may be expanded;
storing each data fragment in the plurality of data fragments to a corresponding data fragment area in the data area;
storing the local metadata of each data fragment in the plurality of data fragments into a sub-region before expansion in a corresponding local metadata region;
and storing the metadata of the file into an expansion sub-area in the local metadata area corresponding to each data slice in the plurality of data slices.
Specifically, in this embodiment, a local metadata region corresponding to a data fragment is expanded, and metadata of a file is stored in an expanded sub-region in the local metadata region. That is, the local metadata is stored in the original area, and the metadata of the file is stored in the extended sub-area.
For example, the local metadata area corresponding to each data slice may be extended by 512 bytes, and the extended area is filled with metadata of the file.
It should be understood that the 512 bytes are only examples, and should not be construed as limiting the present invention, and the specific implementation can be reasonably set according to the size of the metadata of the file.
Optionally, as another embodiment of the present invention, each data slice in the plurality of data slices may be stored to a corresponding data slice area in the data area;
storing the local metadata of each of the plurality of data slices to a corresponding first sub-region in a local metadata region in the data region;
and storing the metadata of the file into a second sub-area in the local metadata area in the data area corresponding to each data slice in the plurality of data slices.
Specifically, in the present embodiment, the local metadata region may be set large enough in advance, so that the metadata of the file can be directly stored to a specific sub-region in the local metadata region.
For example, the local metadata region may be divided into two sub-regions, the former sub-region storing local metadata and the latter sub-region storing metadata of a file.
In specific implementation, before downloading the data fragments, the local metadata of the data fragments, and the metadata of the file, the corresponding data may be processed in the cache, and then downloaded together, that is, written into the disk. In this way, the metadata of the file and the local metadata are stored in the local metadata area together, without increasing the I/O overhead to a certain extent.
Alternatively, the above steps 340 and 350 may be specifically executed by a data management unit in the system. Then, the data management unit sends a write success message to the file management unit, wherein the write success message indicates that the data fragments and the metadata of the file are successfully written. The file management unit sends the metadata of the file to a metadata management unit in the system to request the metadata management unit to update the metadata of the file.
And 360, storing the metadata of the file in the metadata area.
The server system updates the metadata of the file in a metadata area managed by the file system. Optionally, this step may be performed after the data area is successfully written, or may be performed in parallel with the writing process of the data area, which is not limited in the present invention.
Alternatively, the step 360 may be specifically executed by a metadata management unit in the system. Then, the metadata management unit transmits a write success message indicating that the metadata of the file is successfully written to the file management unit. And the file management unit sends a file writing success message to the client.
The method for managing the file of the embodiment of the invention not only stores the metadata of the file in the metadata area managed by the file system, but also stores the metadata of the file in the data area managed by the file system, thus, when the metadata area fails, the metadata can be recovered through the data area, and the metadata can be effectively protected.
Further, in the method for managing a file according to the embodiment of the present invention, in the data area, the metadata of the file and the local metadata of the data fragment are stored in the local metadata area together, and the increased I/O overhead is small, so that the reliability of the metadata can be improved with a small overhead.
The storage flow is described above, and the metadata recovery flow is described below.
Optionally, as shown in fig. 3, the method 300 may further include:
when the metadata area fails, the metadata area is restored 370 based on the metadata of the file in the data area.
Since the metadata of the file is also stored in the data area, when the metadata area fails, the metadata of the metadata area can be restored by scanning the metadata of the data area.
Specifically, the metadata of the file in the local metadata area corresponding to each data fragment in the data area may be read; and recovering the directory hierarchical relationship in the metadata area according to the read metadata of the file.
For example, when a failure occurs in the metadata area, the data slice list may be scanned, and the metadata of the file in the local metadata area of each data slice may be read according to the data slice list. For example, if the metadata of a file is stored in the last 512 bytes of the local metadata area, the metadata of the file in the 512 bytes is read.
Because the data region is fully available, the metadata of the file need only be scanned for the local metadata region in the data region.
Each time metadata of a file is obtained, a record is added at the directory level of the metadata area. Alternatively, a hash (hash) table may be used to record a record relationship that is not concatenated temporarily. And after all the records are collected, the directory hierarchy relationship of the metadata area is established.
Optionally, the scanning process may be executed by the data management unit, and the metadata of the read file is sent to the metadata management unit; the directory hierarchy relationship establishing process may be performed by the metadata management unit, and after the establishment is completed, a recovery success message indicating that the metadata region is completely recovered may be sent to the file management unit.
It can be seen from the foregoing technical solutions that, in the metadata region recovery flow in the embodiment of the present invention, only a specific region in the data region needs to be scanned, so the recovery flow is relatively simple.
The following describes the technical solution of the embodiment of the present invention in detail from the perspective of each management unit with reference to fig. 4 and 5. It should be understood that the management units in fig. 4 and 5 are functionally divided management units, and they may be physically located on the same node or located on different nodes. In addition, the mutual relationship and functions of the management units in fig. 4 and 5, the relationship with other components in the system, and the like can refer to the description in the foregoing various embodiments.
FIG. 4 is a flow chart of writing a file according to one embodiment of the invention.
401, the client sends a write file request.
The client sends the file writing request to the server system, and specifically to the file management unit. The write file request may include identification information of the file, information of the data to be stored and the access address, and the like.
The file management unit determines 402 the data to be stored of the file and the metadata of the file.
The metadata of the file includes information such as the directory structure of the file, the attributes of the file and directory, and the like.
The file management unit determines a plurality of data slices of the data to be stored of the file 403.
Alternatively, the file may be divided into a plurality of data slices according to a protection mode of the file system, for example, an EC protection mode.
404, the file management unit sends the plurality of data slices and the metadata of the file to the data management unit.
The data management unit writes 405 the received data to the cache.
That is, the metadata of the data slice and the file is written into the cache.
The data management unit determines 406 local metadata.
The local metadata includes information of the size of the data slice, creation time, and the like. And the data management unit determines the local metadata of each data fragment according to the plurality of data fragments.
The data management unit fills in metadata of the file in the local metadata region 407.
For example, the data management unit expands the local metadata area of each data slice by 512 bytes, and fills metadata of a file in the expanded area; alternatively, the data management unit fills the metadata of the file in the last 512 bytes of the local metadata area expanded in advance.
408, the data management unit downloads the data fragments, the local metadata of the data fragments, and the metadata of the file together.
In this way, the data management unit stores the data fragments, the local metadata of the data fragments, and the metadata of the file together in the data area.
409, the data management unit responds to the success of the data fragment of the file management unit and the metadata writing of the file.
The file management unit sends a request to the metadata management unit to update the metadata of the file 410.
The metadata management unit stores the metadata of the file in the metadata area 411.
The metadata management unit returns a metadata write success message to the file management unit 412.
413, the file management unit returns a file write success message to the client.
FIG. 5 is a flow diagram of recovering metadata, according to one embodiment of the invention.
501, the data management unit scans the data slice list.
And when the metadata area fails, recovering the metadata of the metadata area according to the metadata of the data area. The data management unit scans the data slice list and performs the following operation for each data slice.
502, the data management unit reads the metadata of the file in the local metadata area of each slice according to the data slice list.
For example, if the metadata of a file is stored in the last 512 bytes of the local metadata area, the metadata of the file in the 512 bytes is read.
Because the data region is fully available, the metadata of the file need only be scanned for the local metadata region in the data region.
The data management unit sends 503 the metadata of the read file to the metadata management unit.
The metadata management unit restores the directory hierarchy based on the metadata of the received file 504.
The metadata management unit adds a record at the directory level of the metadata area every time it receives metadata of a file. Alternatively, a hash table may be used to record a record relationship that is not concatenated temporarily. And after all the records are collected, the directory hierarchy relationship of the metadata area is established.
505, the metadata management unit sends a recovery success message to the file management unit.
After the metadata area is restored, the metadata management unit notifies the file management unit that the metadata area is restored, so that the file service is restored to provide.
It should be understood that, in various embodiments of the present invention, the sequence numbers of the above-mentioned processes do not mean the execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
It should also be understood that the specific examples in the embodiments of the present invention are for the purpose of helping those skilled in the art better understand the embodiments of the present invention, and do not limit the scope of the embodiments of the present invention.
Having described the method of managing files according to an embodiment of the present invention in detail above, a file system and a server system according to an embodiment of the present invention will be described below.
FIG. 6 illustrates a schematic block diagram of a file system 600 of an embodiment of the present invention.
The storage area managed by the file system 600 includes a data area and a metadata area. As shown in fig. 6, the file system 600 may include: a file management unit 610, a data management unit 620, and a metadata management unit 630.
The file management unit 610 is configured to receive a request for writing a file; determining data to be stored of the file and metadata of the file according to the file writing request; determining a plurality of data fragments of the data to be stored of the file; sending the plurality of data fragments and the metadata of the file to the data management unit 620; the metadata of the file is transmitted to the metadata management unit 630.
The data management unit 620 is configured to determine local metadata of each of the plurality of data slices; storing each of the plurality of data slices, the local metadata of each of the plurality of data slices, and the metadata of the file to the data area.
The metadata management unit 630 is configured to store metadata of the file in the metadata area.
The file system of the embodiment of the invention stores the metadata of the file in the metadata area managed by the file system and also stores the metadata of the file in the data area managed by the file system, so that when the metadata area fails, the metadata can be recovered through the data area, and the metadata can be effectively protected.
Optionally, in an embodiment of the present invention, the data management unit 620 is specifically configured to,
expanding a local metadata region in the data region corresponding to each data slice in the plurality of data slices;
storing each data fragment in the plurality of data fragments to a corresponding data fragment area in the data area;
storing the local metadata of each data fragment in the plurality of data fragments into a sub-region before expansion in a corresponding local metadata region;
and storing the metadata of the file into an expansion sub-area in the local metadata area corresponding to each data slice in the plurality of data slices.
Optionally, in an embodiment of the present invention, the data management unit 620 is specifically configured to,
storing each data fragment in the plurality of data fragments to a corresponding data fragment area in the data area;
storing the local metadata of each of the plurality of data slices to a corresponding first sub-region in a local metadata region in the data region;
and storing the metadata of the file into a second sub-area in the local metadata area in the data area corresponding to each data slice in the plurality of data slices.
Alternatively, in one embodiment of the invention, upon failure of the metadata region,
the data management unit 620 is further configured to read metadata of a file in the data area, and send the read metadata of the file to the metadata management unit 630;
the metadata management unit 630 is further configured to restore the metadata area according to the metadata of the file sent by the data management unit 620.
Optionally, in an embodiment of the present invention, the data management unit 620 is specifically configured to read metadata of a file in a local metadata area corresponding to each data slice in the data area, and send the read metadata of the file to the metadata management unit 630;
the metadata management unit 630 is specifically configured to restore the directory hierarchical relationship in the metadata area according to the metadata of the file sent by the data management unit 620.
Optionally, in an embodiment of the present invention, the file management unit 610 is specifically configured to determine the multiple data fragments according to a protection mode of the file system.
The file system 600 in the embodiment of the present invention may execute the method for managing files in the embodiment of the present invention, and the above and other operations and/or functions of each unit in the file system 600 respectively implement corresponding processes of the foregoing methods, reference may be made to the descriptions in the foregoing embodiments, and for brevity, no further description is given here.
FIG. 7 shows a schematic block diagram of a server system 700 of one embodiment of the invention. As shown in fig. 7, the server system 700 may include:
the file system 600 of the foregoing embodiment of the invention; and
a storage resource 710, the storage resource 710 comprising a storage area managed by the file system 600.
For a detailed description of the file system 600 according to the embodiment of the present invention, reference may be made to the foregoing embodiments, and for brevity, detailed descriptions thereof are omitted here.
Fig. 8 shows a schematic block diagram of a server system 800 according to another embodiment of the invention. As shown in fig. 8, the server system 800 may include:
a storage resource 810, the storage resource 810 comprising a file system managed storage area, the storage area comprising a data area and a metadata area;
a management unit 820 for:
receiving a file writing request;
determining data to be stored of the file and metadata of the file according to the file writing request;
determining a plurality of data fragments of the data to be stored of the file;
determining local metadata for each of the plurality of data slices;
storing each of the plurality of data slices, the local metadata of each of the plurality of data slices, and the metadata of the file to the data area;
the metadata of the file is stored to the metadata area.
The server system of the embodiment of the invention stores the metadata of the file in the metadata area managed by the file system and also stores the metadata of the file in the data area managed by the file system, so that when the metadata area fails, the metadata can be recovered through the data area, and the metadata can be effectively protected.
Optionally, in an embodiment of the present invention, the management unit 820 is specifically configured to,
expanding a local metadata region in the data region corresponding to each data slice in the plurality of data slices;
storing each data fragment in the plurality of data fragments to a corresponding data fragment area in the data area;
storing the local metadata of each data fragment in the plurality of data fragments into a sub-region before expansion in a corresponding local metadata region;
and storing the metadata of the file into an expansion sub-area in the local metadata area corresponding to each data slice in the plurality of data slices.
Optionally, in an embodiment of the present invention, the management unit 820 is specifically configured to,
storing each data fragment in the plurality of data fragments to a corresponding data fragment area in the data area;
storing the local metadata of each of the plurality of data slices to a corresponding first sub-region in a local metadata region in the data region;
and storing the metadata of the file into a second sub-area in the local metadata area in the data area corresponding to each data slice in the plurality of data slices.
Optionally, in an embodiment of the present invention, the management unit 820 is further configured to,
and when the metadata area fails, restoring the metadata area according to the metadata of the file in the data area.
Optionally, in an embodiment of the present invention, the management unit 820 is specifically configured to,
reading metadata of files in a local metadata area corresponding to each data fragment in the data area;
and recovering the directory hierarchical relationship in the metadata area according to the read metadata of the file.
Optionally, in an embodiment of the present invention, the management unit 820 is specifically configured to,
the plurality of data fragments are determined according to a protection mode of the file system.
The management unit 820 in the server system 800 according to the embodiment of the present invention may execute each process in the foregoing method embodiments, and for the corresponding detailed description, reference may be made to each foregoing embodiment, which is not described herein again for brevity.
It should be understood that, in the embodiment of the present invention, the term "and/or" is only one kind of association relation describing an associated object, and means that three kinds of relations may exist. For example, a and/or B, may represent: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.
Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may also be an electric, mechanical or other form of connection.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment of the present invention.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention essentially or partially contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
While the invention has been described with reference to specific embodiments, the invention is not limited thereto, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (16)
1. A method of managing files in a file system, wherein a storage area managed by the file system includes a data area and a metadata area; the method comprises the following steps:
receiving a file writing request;
determining data to be stored of a file and metadata of the file according to the file writing request;
determining a plurality of data fragments of the data to be stored of the file;
determining local metadata for each of the plurality of data slices;
storing each of the plurality of data slices, local metadata for each of the plurality of data slices, and metadata for the file to the data area;
storing metadata of the file to the metadata area;
when the metadata area fails, restoring the metadata area according to the metadata of the file in the data area;
wherein the data area and the metadata area belong to different fault domains and are located at different logical locations or different physical locations.
2. The method of claim 1, wherein storing the local metadata of each of the plurality of data slices, and the metadata of the file to the data area comprises:
expanding a local metadata region in the data region corresponding to each data slice in the plurality of data slices;
storing each data slice in the plurality of data slices to a corresponding data slice area in the data area;
storing the local metadata of each data fragment in the plurality of data fragments to a sub-region before expansion in a corresponding local metadata region;
and storing the metadata of the file into an expansion sub-area in a local metadata area corresponding to each data slice in the plurality of data slices.
3. The method of claim 1, wherein storing the local metadata of each of the plurality of data slices, and the metadata of the file to the data area comprises:
storing each data slice in the plurality of data slices to a corresponding data slice area in the data area;
storing local metadata of each of the plurality of data slices to a corresponding first sub-region of a local metadata region in the data region;
storing the metadata of the file to a second sub-region in a local metadata region in the data region corresponding to each of the plurality of data slices.
4. The method according to claim 1, wherein the restoring the metadata area according to the metadata of the file in the data area comprises:
reading metadata of files in a local metadata area corresponding to each data fragment in the data area;
and recovering the directory hierarchical relationship in the metadata area according to the read metadata of the file.
5. The method according to any one of claims 1 to 4, wherein the determining a plurality of data slices of the data to be stored of the file comprises:
determining the plurality of data fragments according to a protection mode of the file system.
6. A file system, wherein a storage area managed by the file system includes a data area and a metadata area; the file system includes: a file management unit, a data management unit and a metadata management unit;
the file management unit is used for receiving a file writing request; determining data to be stored of a file and metadata of the file according to the file writing request; determining a plurality of data fragments of the data to be stored of the file; sending the plurality of data fragments and the metadata of the file to the data management unit; sending the metadata of the file to the metadata management unit;
the data management unit is used for determining local metadata of each data slice in the plurality of data slices; storing each of the plurality of data slices, local metadata for each of the plurality of data slices, and metadata for the file to the data area;
the metadata management unit is used for storing the metadata of the file into the metadata area;
the data management unit is further configured to, when the metadata area fails, read metadata of a file in the data area, and send the read metadata of the file to the metadata management unit;
the metadata management unit is further configured to restore the metadata area according to the metadata of the file sent by the data management unit;
wherein the data area and the metadata area belong to different fault domains and are located at different logical locations or different physical locations.
7. File system in accordance with claim 6, wherein the data management unit is specifically configured to,
expanding a local metadata region in the data region corresponding to each data slice in the plurality of data slices;
storing each data slice in the plurality of data slices to a corresponding data slice area in the data area;
storing the local metadata of each data fragment in the plurality of data fragments to a sub-region before expansion in a corresponding local metadata region;
and storing the metadata of the file into an expansion sub-area in a local metadata area corresponding to each data slice in the plurality of data slices.
8. File system in accordance with claim 6, wherein the data management unit is specifically configured to,
storing each data slice in the plurality of data slices to a corresponding data slice area in the data area;
storing local metadata of each of the plurality of data slices to a corresponding first sub-region of a local metadata region in the data region;
storing the metadata of the file to a second sub-region in a local metadata region in the data region corresponding to each of the plurality of data slices.
9. The file system according to claim 6, wherein the data management unit is specifically configured to read metadata of a file in a local metadata area corresponding to each data slice in the data area, and send the read metadata of the file to the metadata management unit;
the metadata management unit is specifically configured to restore a directory hierarchical relationship in the metadata area according to the metadata of the file sent by the data management unit.
10. The file system according to any of claims 6 to 9, wherein the file management unit is specifically configured to determine the plurality of data fragments according to a protection mode of the file system.
11. A server system, comprising:
the file system of any one of claims 6 to 10; and
a storage resource comprising a storage area managed by the file system.
12. A server system, comprising:
a storage resource comprising a storage area managed by a file system, the storage area comprising a data area and a metadata area;
a management unit to:
receiving a file writing request;
determining data to be stored of a file and metadata of the file according to the file writing request;
determining a plurality of data fragments of the data to be stored of the file;
determining local metadata for each of the plurality of data slices;
storing each of the plurality of data slices, local metadata for each of the plurality of data slices, and metadata for the file to the data area;
storing metadata of the file to the metadata area;
the management unit is further configured to, when the metadata area fails, restore the metadata area according to metadata of a file in the data area;
wherein the data area and the metadata area belong to different fault domains and are located at different logical locations or different physical locations.
13. The server system according to claim 12, wherein the management unit is specifically configured to,
expanding a local metadata region in the data region corresponding to each data slice in the plurality of data slices;
storing each data slice in the plurality of data slices to a corresponding data slice area in the data area;
storing the local metadata of each data fragment in the plurality of data fragments to a sub-region before expansion in a corresponding local metadata region;
and storing the metadata of the file into an expansion sub-area in a local metadata area corresponding to each data slice in the plurality of data slices.
14. The server system according to claim 12, wherein the management unit is specifically configured to,
storing each data slice in the plurality of data slices to a corresponding data slice area in the data area;
storing local metadata of each of the plurality of data slices to a corresponding first sub-region of a local metadata region in the data region;
storing the metadata of the file to a second sub-region in a local metadata region in the data region corresponding to each of the plurality of data slices.
15. The server system according to claim 12, wherein the management unit is specifically configured to,
reading metadata of files in a local metadata area corresponding to each data fragment in the data area;
and recovering the directory hierarchical relationship in the metadata area according to the read metadata of the file.
16. The server system according to any of claims 12 to 15, wherein the management unit is specifically configured to,
determining the plurality of data fragments according to a protection mode of the file system.
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2016/104385 WO2018081960A1 (en) | 2016-11-02 | 2016-11-02 | File management method, file system, and server system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108271420A CN108271420A (en) | 2018-07-10 |
| CN108271420B true CN108271420B (en) | 2020-11-27 |
Family
ID=62075382
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201680003120.4A Active CN108271420B (en) | 2016-11-02 | 2016-11-02 | Method for managing files, file system and server system |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN108271420B (en) |
| WO (1) | WO2018081960A1 (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113986944B (en) * | 2021-12-29 | 2022-03-25 | 天地伟业技术有限公司 | Writing method and system of fragment data and electronic equipment |
| CN114328421B (en) * | 2022-03-17 | 2022-06-10 | 联想凌拓科技有限公司 | Metadata service architecture management method, computer system, electronic device and medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103019626A (en) * | 2012-12-17 | 2013-04-03 | 华为技术有限公司 | Storage system, method and device for controlling cluster metadata |
| CN103608783A (en) * | 2011-06-08 | 2014-02-26 | 微软公司 | Storage architecture for backup application |
| CN104166524A (en) * | 2014-08-19 | 2014-11-26 | 浪潮电子信息产业股份有限公司 | Processing method of metadata and data |
| CN105786655A (en) * | 2016-03-08 | 2016-07-20 | 成都云祺科技有限公司 | Repeated data deleting method for virtual machine backup data |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8112531B2 (en) * | 2004-07-14 | 2012-02-07 | Nokia Corporation | Grouping of session objects |
| CN101656094B (en) * | 2009-09-25 | 2012-04-18 | 杭州华三通信技术有限公司 | Data storage method and storage device |
| CN102088389B (en) * | 2009-12-02 | 2015-01-28 | 中兴通讯股份有限公司 | Distributed content access scheduling device and content reading method |
| CN102024059B (en) * | 2010-12-31 | 2012-07-18 | 成都市华为赛门铁克科技有限公司 | Method and device for protecting redundant array of independent disk in file system |
| CN105159607A (en) * | 2015-08-28 | 2015-12-16 | 浪潮(北京)电子信息产业有限公司 | Discrete storage based high-speed writing method |
-
2016
- 2016-11-02 CN CN201680003120.4A patent/CN108271420B/en active Active
- 2016-11-02 WO PCT/CN2016/104385 patent/WO2018081960A1/en active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103608783A (en) * | 2011-06-08 | 2014-02-26 | 微软公司 | Storage architecture for backup application |
| CN103019626A (en) * | 2012-12-17 | 2013-04-03 | 华为技术有限公司 | Storage system, method and device for controlling cluster metadata |
| CN104166524A (en) * | 2014-08-19 | 2014-11-26 | 浪潮电子信息产业股份有限公司 | Processing method of metadata and data |
| CN105786655A (en) * | 2016-03-08 | 2016-07-20 | 成都云祺科技有限公司 | Repeated data deleting method for virtual machine backup data |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2018081960A1 (en) | 2018-05-11 |
| CN108271420A (en) | 2018-07-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11520670B2 (en) | Method and apparatus for restoring data from snapshots | |
| CN110249321B (en) | System and method for capturing change data from a distributed data source for use by heterogeneous targets | |
| CN108170768B (en) | Database synchronization method, device and readable medium | |
| CN106547859B (en) | Data file storage method and device under multi-tenant data storage system | |
| US11138156B2 (en) | Continuous data management system and operating method thereof | |
| US8683228B2 (en) | System and method for WORM data storage | |
| US10956388B2 (en) | Eventual consistency in a deduplicated cloud storage system | |
| CN108319602B (en) | Database management method and database system | |
| US20150213100A1 (en) | Data synchronization method and system | |
| CN106951375B (en) | Method and device for deleting snapshot volume in storage system | |
| US11093387B1 (en) | Garbage collection based on transmission object models | |
| US20070061540A1 (en) | Data storage system using segmentable virtual volumes | |
| CN109582213B (en) | Data reconstruction method and device and data storage system | |
| CN105630632A (en) | Virtual machine recovery method and virtual machine management device | |
| JP6968876B2 (en) | Expired backup processing method and backup server | |
| US10628298B1 (en) | Resumable garbage collection | |
| US7549037B1 (en) | Efficient off-host backup of a file set clone | |
| JP2019523955A (en) | Data storage system and method for performing data storage | |
| CN109710185A (en) | Data processing method and device | |
| CN102360321A (en) | Terminal program quick backup and recovery method based on cloud architecture | |
| CN105278882A (en) | Disk management method of distributed file system | |
| CN103501319A (en) | Low-delay distributed storage system for small files | |
| US20230376385A1 (en) | Reducing bandwidth during synthetic restores from a deduplication file system | |
| JP7215971B2 (en) | METHOD AND APPARATUS FOR PROCESSING DATA LOCATION IN STORAGE DEVICE, COMPUTER DEVICE AND COMPUTER-READABLE STORAGE MEDIUM | |
| CN108271420B (en) | Method for managing files, file system and server system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |