CN111046001B - Method, device and equipment for creating files in batch and storage medium - Google Patents
Method, device and equipment for creating files in batch and storage medium Download PDFInfo
- Publication number
- CN111046001B CN111046001B CN201911385174.3A CN201911385174A CN111046001B CN 111046001 B CN111046001 B CN 111046001B CN 201911385174 A CN201911385174 A CN 201911385174A CN 111046001 B CN111046001 B CN 111046001B
- Authority
- CN
- China
- Prior art keywords
- creation
- target client
- files
- client
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 53
- 238000013475 authorization Methods 0.000 claims abstract description 30
- 238000004590 computer program Methods 0.000 claims description 11
- 238000012790 confirmation Methods 0.000 claims description 6
- 238000012545 processing Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/164—File meta data generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
- Storage Device Security (AREA)
Abstract
The invention discloses a method, a device, equipment and a storage medium for creating files in batches, wherein the method comprises the following steps: receiving an authorization request sent by a target client, and determining a directory corresponding to the authorization request as an appointed directory; judging whether the appointed directory is authorized to other clients except the target client, if so, waiting until the batch creation of all files corresponding to the client with the authority to the appointed directory is completed, releasing the appointed directory, authorizing the appointed directory to the target client, and if not, authorizing the appointed directory to the target client; receiving a plurality of creation requests sent by a target client, realizing simultaneous creation of a plurality of corresponding files based on the received creation requests, storing metadata of each file obtained by creation, and releasing a specified directory after batch creation of all files corresponding to the target client is completed. The file creating method and the file creating device can improve file creating performance and improve performance of a distributed file system.
Description
Technical Field
The present invention relates to the field of distributed file system technology, and more particularly, to a method, an apparatus, a device, and a storage medium for creating files in batch.
Background
The distributed cluster storage is widely applied in various scenes of current big data (such as high performance, video monitoring, broadcast and television media resources and the like), mass data exist in a distributed file system, the premise of big data calculation is to organize and store the mass data, and the inventor researches and discovers that one of indexes influencing the performance of the distributed file system is the file creation performance, so that how to improve the file creation performance becomes an important target.
Disclosure of Invention
The invention aims to provide a method, a device, equipment and a storage medium for creating files in batches, which can improve the file creation performance and further improve the performance of a distributed file system.
In order to achieve the above purpose, the invention provides the following technical scheme:
a method of creating files in batches, comprising:
receiving an authorization request sent by a target client, and determining a directory corresponding to the authorization request as an appointed directory; the target client is any client needing to create batch files;
judging whether the specified directory is authorized to other clients except the target client, if so, waiting until the batch creation of all files corresponding to the client with the authority to the specified directory is completed, releasing the specified directory, and authorizing the specified directory to the target client, and if not, authorizing the specified directory to the target client; wherein the specified directory only allows access to clients having permission to the specified directory;
receiving a plurality of creation requests sent by the target client, realizing the simultaneous creation of a plurality of corresponding files based on the received creation requests, storing the metadata of each file obtained by the creation, and releasing the specified directory after completing the batch creation of all the files corresponding to the target client.
Preferably, before receiving the authorization request sent by the destination client, the method further includes:
receiving a connection request sent by the target client, responding to the connection request to establish connection with the target client, and applying for a file number in a preset range for the target client in advance;
after the simultaneous creation of the plurality of files is realized based on the received creation request, the method further comprises:
respectively allocating file numbers belonging to the preset range to each file obtained by creation; wherein the metadata of the file includes a file number of the file.
Preferably, after the simultaneous creation of the plurality of files is realized based on the received creation request, the method further includes:
setting corresponding file names for each created file respectively, and establishing association between the file name of each created file and a file number; wherein the metadata of the file includes the file name and the file number having an association.
Preferably, the method further comprises the following steps:
when the batch creation of all files corresponding to the target client is not completed, receiving authorization requests sent by other clients except the target client, indicating the target client to forcibly refresh all creation requests cached by the target client so as to obtain all creation requests refreshed by the target client, and executing the step of realizing the simultaneous creation of a plurality of corresponding files based on the received creation requests.
Preferably, the method further comprises the following steps:
if a fault occurs in the process of realizing batch creation of all files corresponding to the target client, receiving each creation request of the target client record, which is retransmitted by the target client, after the fault is repaired; the client records a creation request corresponding to each file to be created.
Preferably, after the target client is implemented to create all files in batch, the method further includes:
and returning confirmation information of successful batch creation of all files corresponding to the target client to indicate the creation request of deleting the record of the target client.
Preferably, after the target client is implemented to create all files in batch, the method further includes:
and sending a marking instruction to the target client to indicate the target client to record a mark of successful creation of the batch files in a log of file creation.
An apparatus for creating files in batches, comprising:
a determination module to: receiving an authorization request sent by a target client, and determining a directory corresponding to the authorization request as an appointed directory; the target client is any client needing to create the batch files;
a determination module configured to: judging whether the specified directory is authorized to other clients except the target client, if so, waiting until the batch creation of all files corresponding to the client with the authority to the specified directory is completed, releasing the specified directory, and authorizing the specified directory to the target client, and if not, authorizing the specified directory to the target client; wherein the specified directory only allows access to clients having rights to the specified directory;
a creation module to: receiving a plurality of creation requests sent by the target client, realizing simultaneous creation of a plurality of corresponding files based on the received creation requests, storing metadata of each file obtained by creation, and releasing the specified directory after completing batch creation of all files corresponding to the target client.
An apparatus for creating files in bulk, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the method for batch creation of files as described in any one of the above when the computer program is executed.
A computer-readable storage medium, having stored thereon a computer program which, when executed by a processor, carries out the steps of the method of bulk creating files according to any one of the preceding claims.
The invention provides a method, a device, equipment and a storage medium for creating files in batches, wherein the method comprises the following steps: receiving an authorization request sent by a target client, and determining a directory corresponding to the authorization request as an appointed directory; the target client is any client needing to create batch files; judging whether the specified directory is authorized to other clients except the target client, if so, waiting until the batch creation of all files corresponding to the client with the authority to the specified directory is completed, releasing the specified directory, and authorizing the specified directory to the target client, and if not, authorizing the specified directory to the target client; wherein the specified directory only allows access to clients having permission to the specified directory; receiving a plurality of creation requests sent by the target client, realizing the simultaneous creation of a plurality of corresponding files based on the received creation requests, storing the metadata of each file obtained by the creation, and releasing the specified directory after completing the batch creation of all the files corresponding to the target client. According to the technical scheme, when the client side needs to create the files in batch, the authority of the appointed directory can be obtained through application, if the appointed directory is not authorized to other client sides currently, the authority of the appointed directory can be obtained, and then the files are created in batch under the appointed directory, otherwise, the authority of the appointed directory is obtained after the access authority of other client sides to the appointed directory is removed, and then the files are created in batch under the appointed directory. Therefore, the files required to be created can be created in batches under the instruction of the client, the file creation performance is improved, and the performance of the distributed file system is further improved; in addition, by setting the exclusive right for the specified directory, the situation that the names of created files are the same and further file creation fails when multiple clients simultaneously achieve file creation under the specified directory can be avoided, effective implementation of batch creation of the files is guaranteed, the situation that when one client achieves file creation under the specified directory, other clients cannot access the information such as the valid files when accessing the specified directory can be avoided, and the effectiveness of the clients in obtaining the information is guaranteed.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a method for creating files in batch according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of an apparatus for creating files in batch according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, a flowchart of a method for creating files in batch according to an embodiment of the present invention is shown, where the method may include:
s11: receiving an authorization request sent by a target client, and determining a directory corresponding to the authorization request as an appointed directory; the target client is any client which needs to create the batch files.
The execution subject of the method for creating files in batches provided by the embodiment of the present invention may be a corresponding device, and the device may be disposed in a client, so that the execution subject of the method may be a server, and the following detailed description will be given by taking the execution subject of the method as a server.
When any client needs to create files in batch, the client needing to create the files in batch can be called as a target client, and therefore the files of any client can be created in batch. When a target client needs to create files in batch, an authorization request can be sent to a server, after the server receives the authorization request sent by the target client, a directory corresponding to the authorization request, that is, a directory identifier carried in the authorization request can be determined, and the directory for which the target client needs to create files in batch, which can be called as an appointed directory. In addition, creating a file in batch in this embodiment, that is, creating multiple files at the same time, and the Server can provide MDS (Meta Data Server metadata service).
S12: judging whether the appointed directory is authorized to other clients except the target client, if so, waiting until the batch creation of all files corresponding to the client with the authority to the appointed directory is completed, releasing the appointed directory, authorizing the appointed directory to the target client, and if not, authorizing the appointed directory to the target client; wherein the specified directory allows access only to clients having rights to the specified directory.
It should be noted that, in this embodiment, a Cap (Capability, metadata operation permission) that monopolizes a directory is introduced, and when a certain client holds the Cap of a certain directory, other clients cannot operate the directory to perform operations such as reading, writing, file creation, and the like; specifically, when a certain directory is an appointed directory, the appointed directory can only be authorized to one client at the same time, the client authorized by the appointed directory is a client having access right to the appointed directory, namely, the client can realize operations such as information reading and writing, file creation and the like under the appointed directory, and other clients not authorized cannot access the appointed directory; because if a plurality of clients create files under a specified directory at the same time, the situation that the names of the files created by different clients are the same may occur, but the situation that the names of the files are the same is actually not allowed, the situation that the names of the created files are the same and further the file creation fails when the plurality of clients simultaneously achieve the file creation under the same directory can be avoided through the setting of the authority of the directory; moreover, if a certain client creates a file or reads and writes under a directory, other clients accessing the directory at the moment cannot access the latest data information, that is, cannot access valid data information, so that through the setting of the permission to the directory, the situation that when one client achieves file creation under a directory, other clients cannot access valid data information such as files and the like under the directory can be avoided. Therefore, in this embodiment, after determining the designated directory, the server may determine whether the designated directory is currently authorized to other clients except the destination client, that is, whether other clients except the destination client have permission to the designated directory, and if so, the designated directory cannot be authorized to the destination client, so that the designated directory is released after the batch creation of all files that the client having permission to the designated directory needs to implement is completed, that is, after the permission of the client having permission to the designated directory is released, the designated directory does not have a client having permission to the designated directory, and at this time, the designated directory may be authorized to the destination client; if not, the designated directory may be directly authorized to the destination client.
In addition, if the specified directory is authorized to other clients except the destination client, the destination client may wait until the specified directory has the authority of the specified directory after being released, or may implement sequential creation of files in a conventional manner, and may specifically perform setting according to actual needs, all of which are within the protection scope of the present invention.
S13: receiving a plurality of creation requests sent by a target client, realizing simultaneous creation of a plurality of corresponding files based on the received creation requests, storing metadata of each file obtained by creation, and releasing a specified directory after batch creation of all files corresponding to the target client is completed.
After the target client side applies for obtaining the authority of the specified directory, the creation requests in a certain time (or certain items) can be packaged simultaneously, and the packaged result is sent to the server, so that the server analyzes the creation requests contained in the packaged result, the batch creation, namely simultaneous creation, of the corresponding files is further realized, the metadata of each file obtained by the creation is stored, and the specified directory is released after all files required to be created by the target client side are created and is available for other client sides to obtain. In addition, in this embodiment, the certain time may be total time required for the server to implement batch creation of the files, and the certain entry may be total number of files required for the server to implement batch creation of the files, so as to implement batch creation of the files according to the requirement of the target client.
According to the technical scheme, when the client needs to create the files in batch, the authority of the appointed directory can be obtained through application, if the appointed directory is not authorized to other clients currently, the authority of the appointed directory can be obtained, and then the files are created in batch under the appointed directory, otherwise, the authority of the appointed directory is obtained after the access authority of other clients to the appointed directory is removed, and then the files are created in batch under the appointed directory. Therefore, the files required to be created can be created in batches under the instruction of the client, the file creation performance is improved, and the performance of the distributed file system is further improved; in addition, by setting the exclusive right for the specified directory, the situation that the names of created files are the same and further file creation fails when multiple clients simultaneously achieve file creation under the specified directory can be avoided, effective implementation of batch creation of the files is guaranteed, the situation that when one client achieves file creation under the specified directory, other clients cannot access the information such as the valid files when accessing the specified directory can be avoided, and the effectiveness of the clients in obtaining the information is guaranteed.
The method for creating files in batch provided by the embodiment of the invention can further include, before receiving an authorization request sent by a destination client:
receiving a connection request sent by a target client, responding to the connection request to establish connection with the target client, and pre-applying a file number in a preset range for the target client;
after the simultaneous creation of the plurality of files is realized based on the received creation request, the method further comprises:
respectively allocating file numbers belonging to a preset range to each file obtained by creation; wherein the metadata of the file includes a file number of the file.
It should be noted that, in this embodiment, in order to achieve simultaneous creation of multiple corresponding files, a file number of each file may be quickly obtained, and then the file creation rate is increased, so in this embodiment, after receiving a connection request sent by a destination client, a connection with the destination client may be established, and a file number (Inode, a file number that is included in a node) in a certain range (a preset range and may be set according to actual needs) is applied for the destination client from a metadata pool, and then allocation of the file number may be directly achieved when creating the file, and it is not necessary to obtain the file number from the metadata pool, and thus the file creation rate is greatly increased. After the server applies for a certain range of file numbers for the target client in advance, the server can inform the target client of all the file numbers distributed for the target client, so that the target client can know the file numbers of the files which are determined to be created based on the file numbers. In addition, if the target client needs to create files in batch and the file numbers allocated to the target client are used up, a certain range of file numbers can be allocated to the target client from the metadata pool, and so on, so as to ensure that the target client has sufficient file numbers to create the files.
The method for creating files in batch provided by the embodiment of the present invention may further include, after the simultaneous creation of a plurality of corresponding files is realized based on the received creation request:
setting corresponding file names for each created file respectively, and establishing association between the file name of each created file and a file number; the metadata of the file includes a file name and a file number having an association.
The creation request sent by the destination client may carry file names (i.e., file names) set by the client for corresponding files, so that the file names obtained by creation are set as the file names carried in the corresponding creation request, and the file name and the file number of each file are associated and stored as metadata, thereby facilitating searching for files and the like. In addition, the information with association may specifically be a CInode (metadata Inode information, which mainly includes file attributes) and a CDentry (metadata directory entry, which mainly includes file names), where the CInode includes a file number, and the CDentry includes a file name, so that the associated CInode and CDentry include an associated file name and file number, and further include the associated CInode and CDentry in the metadata for storage, such as storing in a metadata pool.
The method for creating files in batch provided by the embodiment of the invention can further comprise the following steps:
when the batch creation of all files corresponding to the incomplete target client is not completed, and authorization requests sent by other clients except the target client are received, the target client is instructed to forcedly refresh all creation requests cached by the target client so as to obtain all creation requests refreshed by the target client, and the step of realizing the simultaneous creation of a plurality of corresponding files based on the received creation requests is executed.
It should be noted that, the target client in this embodiment may package and send the creation requests corresponding to all files that the target client needs to create to the server, or may package and send the creation requests included in each batch to the server after dividing all files that the target client needs to create into multiple batches, which may be specifically determined according to actual situations; correspondingly, the application of the destination client for having the authority on the specified directory may include a time period in which the authority on the specified directory needs to be provided and the total number of all files needing to be created in the specified directory, while the incomplete destination client corresponds to the batch creation of all files, specifically, the time period in which the authority on the specified directory does not reach the application of the destination client for having the authority on the specified directory, and/or the number of files created in the specified directory does not reach the total number of all files needing to be created by the destination client. In addition, the destination client can cache the creation request which needs to be sent and is not sent locally; if the target client side does not finish the batch creation of all files corresponding to the target client side, authorization requests sent by other client sides are received, whether the target client side has creation requests which are not sent to the server or not can be judged, if yes, the target client side can be informed to forcibly refresh the cached creation requests to the server, so that the server can realize corresponding file batch creation, file metadata storage and the like, if not, the server can finish the file batch creation, the file metadata storage and the like corresponding to the creation requests which are sent to the server by the target client side, and then the authority of the specified directory is released after the batch creation and the corresponding metadata storage of all files which need to be created by the target client side are realized, so that the use of the specified directory by other client sides can be facilitated.
The method for creating files in batch provided by the embodiment of the invention can further comprise the following steps:
if a fault occurs in the process of realizing batch creation of all files corresponding to the target client, receiving each creation request recorded by the target client and retransmitted by the target client after the fault is repaired; the client records a creation request corresponding to each file to be created.
The client can locally record the creation request to be sent before sending the creation request to the server, and further if the server fails during processing the batch creation request, the server needs to repair the failure, and after the server is repaired and can normally work, the client needs to resend the recorded creation request to the server, and the server realizes batch file creation again after receiving the creation request resent by the client, so that the effective realization of batch file creation by the client is ensured.
The method for creating files in batch provided by the embodiment of the invention can further include the following steps after the target client side is created in batch corresponding to all the files:
and returning confirmation information of successful batch creation of all the files corresponding to the target client to indicate the creation request of deleting the record of the target client.
After the server completes batch creation of all files corresponding to the target client, confirmation information can be returned to the target client, so that the target client can know the created information of all files to be created, and further delete the recorded creation request, thereby avoiding waste of storage space in the target client.
Moreover, after the batch creation of all the files corresponding to the destination client is realized, the method may further include:
and sending a marking instruction to the target client to indicate the target client to record the mark of successful creation of the batch file in the log of file creation.
By sending the marking instruction to the target client, the target client can be instructed to record the mark of successful creation of the batch files in the log after receiving the marking instruction, so as to perform query in the later period.
In a specific implementation scenario, a method for creating files in batch provided in an embodiment of the present application may include:
(1) Inode pre-allocation:
when the server establishes connection with the client, the server applies for a certain range of inodes from the metadata pool and informs the client;
(2) The Cap application:
when a client creates a file under a certain directory, applying for exclusive Cap of the directory to a server, and if no other client uses the directory, the server agrees to grant the exclusive Cap of the directory to the client; and if other clients are using the directory, the server refuses to authorize the exclusive Cap, and after all files of other clients are created, the server agrees to authorize the exclusive Cap of the directory to the client.
(3) Sending in batches:
after a client applies for an exclusive Cap of a directory, a creation request within a certain time (or a certain request entry) is packaged at the same time and sent to a server, and the client records the sent request (for fault processing); if a certain time or an item period is not reached, other clients apply for the exclusive Cap of the directory to the server, and the server informs the client of forcibly refreshing the creation request in the cache and then releases the exclusive Cap;
(4) The server processes the batch requests:
after receiving the batch creation request of the client, the server analyzes the batch creation request, creates a corresponding file, associates the CInode and the CDentry of the file, drops all created metadata into a metadata pool, and then responds (can comprise confirmation information and a marking instruction) to the client to complete the request;
(5) And a finishing stage:
after receiving the server response, the client clears the sent request recorded in the client, and marks that the batch creation service is completed;
(6) And (3) fault repair:
if the server fails during processing of the batch creation request, the client needs to resend the batch creation request to the server for processing again in the server recovery phase.
The method for creating the files in batch in the distributed file system can greatly improve the file creating performance and the metadata creating performance, particularly the creating performance of small file scenes, and further improve the creating performance of the distributed file system.
An embodiment of the present invention further provides a device for creating files in batch, and as shown in fig. 2, the device may specifically include:
a determining module 11, configured to: receiving an authorization request sent by a target client, and determining a directory corresponding to the authorization request as an appointed directory; the target client is any client needing to create the batch files;
a judging module 12, configured to: judging whether the specified directory is authorized to other clients except the target client, if so, waiting until the batch creation of all files corresponding to the client with the authority to the specified directory is completed, releasing the specified directory, and authorizing the specified directory to the target client, and if not, authorizing the specified directory to the target client; wherein, the appointed directory only allows the client with the authority to access the appointed directory;
a creation module 13 configured to: receiving a plurality of creation requests sent by a target client, realizing simultaneous creation of a plurality of corresponding files based on the received creation requests, storing metadata of each file obtained by creation, and releasing a specified directory after batch creation of all files corresponding to the target client is completed.
The apparatus for creating files in batch provided by the embodiment of the present invention may further include:
a connection module for: before receiving an authorization request sent by a target client, receiving a connection request sent by the target client, responding to the connection request to establish connection with the target client, and pre-applying a file number in a preset range for the target client;
an assignment module to: after the simultaneous creation of a plurality of corresponding files is realized based on the received creation request, file numbers belonging to a preset range are respectively distributed to each file obtained by the creation; wherein the metadata of the file includes a file number of the file.
The apparatus for creating files in batch provided by the embodiment of the present invention may further include:
a setup module to: after the simultaneous creation of a plurality of corresponding files is realized based on the received creation request, respectively setting corresponding file names for each created file, and establishing association between the file name of each created file and a file number; the metadata of the file includes a file name and a file number having an association.
The apparatus for creating files in batch provided by the embodiment of the present invention may further include:
an indication module to: when the batch creation of all files corresponding to the target client is not completed, receiving authorization requests sent by other clients except the target client, indicating the target client to forcibly refresh all creation requests cached by the target client so as to obtain all creation requests refreshed by the target client, and executing the step of realizing the simultaneous creation of a plurality of corresponding files based on the received creation requests.
The apparatus for creating files in batch provided by the embodiment of the present invention may further include:
a reconstruction module to: if a fault occurs in the process of realizing batch creation of all files corresponding to the target client, receiving each creation request recorded by the target client and retransmitted by the target client after the fault is repaired; the client records a creation request corresponding to each file to be created.
The apparatus for creating files in batch provided by the embodiment of the present invention may further include:
a return module to: after the batch creation of all files corresponding to the target client is realized, returning confirmation information of successful batch creation of all files corresponding to the target client to indicate the creation request of deleting the record of the target client.
The apparatus for creating files in batch provided by the embodiment of the present invention may further include:
a sending module configured to: after the batch creation of all files corresponding to the target client is realized, a marking instruction is sent to the target client to indicate the target client to record a mark of successful batch file creation in a log of file creation.
An embodiment of the present invention further provides a device for creating files in batch, where the device may include:
a memory for storing a computer program;
a processor for implementing the steps of the method for batch creation of files as described in any one of the above when executing a computer program.
Embodiments of the present invention further provide a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method for creating files in batch as described above may be implemented.
It should be noted that for the description of the relevant parts in the apparatus, the device, and the storage medium for creating files in batches provided in the embodiment of the present invention, reference is made to the detailed description of the corresponding parts in the method for creating files in batches provided in the embodiment of the present invention, and details are not repeated here. In addition, parts of the above technical solutions provided in the embodiments of the present invention that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail, so as to avoid redundant description.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (9)
1. A method for creating files in batches, comprising:
receiving an authorization request sent by a target client, and determining a directory corresponding to the authorization request as an appointed directory; the target client is any client needing to create the batch files;
judging whether the specified directory is authorized to other clients except the target client, if so, waiting until the batch creation of all files corresponding to the client with the authority to the specified directory is completed, releasing the specified directory, and authorizing the specified directory to the target client, and if not, authorizing the specified directory to the target client; wherein the specified directory only allows access to clients having permission to the specified directory;
receiving a plurality of creation requests sent by the target client, realizing the simultaneous creation of a plurality of corresponding files based on the received creation requests, storing metadata of each file obtained by creation, and releasing the specified directory after the batch creation of all the files corresponding to the target client is completed;
when the batch creation of all files corresponding to the target client is not completed, receiving authorization requests sent by other clients except the target client, indicating the target client to forcibly refresh all creation requests cached by the target client so as to obtain all creation requests refreshed by the target client, and executing the step of realizing the simultaneous creation of a plurality of corresponding files based on the received creation requests;
the batch creation of all files corresponding to the unfinished target client comprises the following steps: the time period that the duration with the authority to the specified directory does not reach the time period that the client applies for having the authority to the specified directory, and/or the number of the files created in the specified directory does not reach the total number of all the files required to be created by the target client.
2. The method of claim 1, wherein before receiving the authorization request sent by the destination client, the method further comprises:
receiving a connection request sent by the target client, responding to the connection request to establish connection with the target client, and applying for a file number in a preset range for the target client in advance;
after the simultaneous creation of the plurality of files is realized based on the received creation request, the method further comprises:
respectively allocating file numbers belonging to the preset range to each file obtained by creation; wherein the metadata of the file includes a file number of the file.
3. The method of claim 2, wherein after enabling the simultaneous creation of the respective plurality of files based on the received creation request, further comprising:
setting corresponding file names for each created file respectively, and establishing association between the file name of each created file and a file number; wherein the metadata of the file includes the file name and the file number with an association.
4. The method of claim 1, further comprising:
if a fault occurs in the process of realizing batch creation of all files corresponding to the target client, receiving each creation request of the target client record, which is retransmitted by the target client, after the fault is repaired; the client records a creation request corresponding to each file to be created.
5. The method of claim 4, after the batch creation of all files corresponding to the destination client is realized, further comprising:
and returning confirmation information of successful batch creation of all the files corresponding to the target client to indicate the creation request of deleting the record of the target client.
6. The method according to claim 5, wherein after the batch creation of all files corresponding to the destination client is realized, the method further comprises:
and sending a marking instruction to the target client to indicate the target client to record a mark of successful creation of the batch file in a log of file creation.
7. An apparatus for creating files in batches, comprising:
a determination module to: receiving an authorization request sent by a target client, and determining a directory corresponding to the authorization request as an appointed directory; the target client is any client needing to create batch files;
a determination module configured to: judging whether the specified directory is authorized to other clients except the target client, if so, waiting until the batch creation of all files corresponding to the client with the authority to the specified directory is completed, releasing the specified directory, and authorizing the specified directory to the target client, and if not, authorizing the specified directory to the target client; wherein the specified directory only allows access to clients having rights to the specified directory;
a creation module to: receiving a plurality of creation requests sent by the target client, realizing the simultaneous creation of a plurality of corresponding files based on the received creation requests, storing metadata of each file obtained by creation, and releasing the specified directory after the batch creation of all the files corresponding to the target client is completed;
an indication module to: when the batch creation of all files corresponding to the incomplete target client is not completed, receiving authorization requests sent by other clients except the target client, indicating the target client to forcedly refresh all creation requests cached by the target client so as to obtain all creation requests refreshed by the target client, and executing the step of realizing the simultaneous creation of a plurality of corresponding files based on the received creation requests; the batch creation of all files corresponding to the unfinished target client comprises the following steps: the time period that the time length of the appointed directory does not reach the authority of the target client applies for the time period that the appointed directory has the authority, and/or the total number of the files created under the appointed directory does not reach the total number of all the files required to be created by the target client.
8. An apparatus for batch creating files, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the method of bulk creating files according to any of claims 1 to 6 when executing the computer program.
9. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, carries out the steps of the method of bulk creating files according to any one of claims 1 to 6.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911385174.3A CN111046001B (en) | 2019-12-28 | 2019-12-28 | Method, device and equipment for creating files in batch and storage medium |
| PCT/CN2020/110731 WO2021128892A1 (en) | 2019-12-28 | 2020-08-24 | Method and apparatus for creating files in batches, device, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911385174.3A CN111046001B (en) | 2019-12-28 | 2019-12-28 | Method, device and equipment for creating files in batch and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111046001A CN111046001A (en) | 2020-04-21 |
| CN111046001B true CN111046001B (en) | 2023-03-14 |
Family
ID=70241006
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911385174.3A Active CN111046001B (en) | 2019-12-28 | 2019-12-28 | Method, device and equipment for creating files in batch and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111046001B (en) |
| WO (1) | WO2021128892A1 (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111046001B (en) * | 2019-12-28 | 2023-03-14 | 浪潮电子信息产业股份有限公司 | Method, device and equipment for creating files in batch and storage medium |
| CN112597120B (en) * | 2020-11-30 | 2022-05-27 | 新华三大数据技术有限公司 | Catalog management method and device |
| CN113760853B (en) * | 2021-08-16 | 2024-02-20 | 联想凌拓科技有限公司 | Directory processing method, server and storage medium |
| CN115580610B (en) * | 2022-09-22 | 2024-03-19 | 广州文远知行科技有限公司 | Batch file uploading method, device, equipment and readable storage medium |
| CN116611988B (en) * | 2023-05-23 | 2024-04-26 | 释空(上海)展示制作有限公司 | Image processing method, electronic device, and storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103179185A (en) * | 2012-12-25 | 2013-06-26 | 中国科学院计算技术研究所 | Method and system for creating files in cache of distributed file system client |
| CN104021137A (en) * | 2014-04-21 | 2014-09-03 | 中国科学院计算技术研究所 | Method and system for opening and closing file locally through client side based on catalogue authorization |
| CN104113587A (en) * | 2014-06-23 | 2014-10-22 | 华中科技大学 | Client metadata buffer optimization method of distributed file system |
| CN105512279A (en) * | 2015-12-04 | 2016-04-20 | 华为技术有限公司 | Metadata access method, related equipment and system |
| CN105630810A (en) * | 2014-10-30 | 2016-06-01 | 曙光信息产业股份有限公司 | Method for uploading mass small files in distributed storage system |
| CN106485156A (en) * | 2016-09-22 | 2017-03-08 | 中广核工程有限公司 | A kind of apparatus and method for files in batch mandate |
| CN110138881A (en) * | 2019-06-05 | 2019-08-16 | 安徽三实信息技术服务有限公司 | A kind of distributed memory system and its storage method |
| CN110597773A (en) * | 2019-08-28 | 2019-12-20 | 重庆爱奇艺智能科技有限公司 | Method and apparatus for sharing files between computer device and virtual reality device |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9361229B2 (en) * | 2008-08-25 | 2016-06-07 | International Business Machines Corporation | Distributed shared caching for clustered file systems |
| CN102024016B (en) * | 2010-11-04 | 2013-03-13 | 曙光信息产业股份有限公司 | Rapid data restoration method for distributed file system (DFS) |
| US9307025B1 (en) * | 2011-03-29 | 2016-04-05 | Riverbed Technology, Inc. | Optimized file creation in WAN-optimized storage |
| EP2541473A1 (en) * | 2011-06-27 | 2013-01-02 | Amadeus S.A.S. | Method and system for a pre-shopping reservation system with increased search efficiency |
| US20130218934A1 (en) * | 2012-02-17 | 2013-08-22 | Hitachi, Ltd. | Method for directory entries split and merge in distributed file system |
| US10264071B2 (en) * | 2014-03-31 | 2019-04-16 | Amazon Technologies, Inc. | Session management in distributed storage systems |
| US10268632B2 (en) * | 2015-07-28 | 2019-04-23 | International Business Machines Corporation | Real time and scalable space-time data recovery from existing backup systems |
| CN109408474B (en) * | 2018-11-09 | 2021-11-19 | 郑州云海信息技术有限公司 | File creation method and client |
| CN111046001B (en) * | 2019-12-28 | 2023-03-14 | 浪潮电子信息产业股份有限公司 | Method, device and equipment for creating files in batch and storage medium |
-
2019
- 2019-12-28 CN CN201911385174.3A patent/CN111046001B/en active Active
-
2020
- 2020-08-24 WO PCT/CN2020/110731 patent/WO2021128892A1/en active Application Filing
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103179185A (en) * | 2012-12-25 | 2013-06-26 | 中国科学院计算技术研究所 | Method and system for creating files in cache of distributed file system client |
| CN104021137A (en) * | 2014-04-21 | 2014-09-03 | 中国科学院计算技术研究所 | Method and system for opening and closing file locally through client side based on catalogue authorization |
| CN104113587A (en) * | 2014-06-23 | 2014-10-22 | 华中科技大学 | Client metadata buffer optimization method of distributed file system |
| CN105630810A (en) * | 2014-10-30 | 2016-06-01 | 曙光信息产业股份有限公司 | Method for uploading mass small files in distributed storage system |
| CN105512279A (en) * | 2015-12-04 | 2016-04-20 | 华为技术有限公司 | Metadata access method, related equipment and system |
| CN106485156A (en) * | 2016-09-22 | 2017-03-08 | 中广核工程有限公司 | A kind of apparatus and method for files in batch mandate |
| CN110138881A (en) * | 2019-06-05 | 2019-08-16 | 安徽三实信息技术服务有限公司 | A kind of distributed memory system and its storage method |
| CN110597773A (en) * | 2019-08-28 | 2019-12-20 | 重庆爱奇艺智能科技有限公司 | Method and apparatus for sharing files between computer device and virtual reality device |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2021128892A1 (en) | 2021-07-01 |
| CN111046001A (en) | 2020-04-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111046001B (en) | Method, device and equipment for creating files in batch and storage medium | |
| CN112527489B (en) | Task scheduling method, device, equipment and computer readable storage medium | |
| CN109101341B (en) | Distribution method and equipment of distributed lock | |
| US5751997A (en) | Method and apparatus for transferring archival data among an arbitrarily large number of computer devices in a networked computer environment | |
| KR100625595B1 (en) | Parallel Logging Method of Transaction Processing System | |
| US4961224A (en) | Controlling access to network resources | |
| CN108885671B (en) | Directory deletion method and device and storage server | |
| US8190857B2 (en) | Deleting a shared resource node after reserving its identifier in delete pending queue until deletion condition is met to allow continued access for currently accessing processor | |
| CN105653406A (en) | Method for managing distributed storage system and distributed storage system | |
| US20180295206A1 (en) | Parallel prefetching log/meta stream sub-portions to recreate partition states in a distributed computing system | |
| CN105701156A (en) | Distributed file system management method and device | |
| US7840671B2 (en) | Managing the size and accessibility of a name service | |
| CN113342507B (en) | Distributed lock service realization method and device and computer equipment | |
| CN111176559B (en) | Data writing method and device and dual-activity system | |
| CN116662426A (en) | Database connection establishment method, device, equipment and medium | |
| CN110737635A (en) | data blocking method | |
| CN109828720B (en) | Data storage method, device, server and storage medium | |
| CN108345431B (en) | Data reading method and device | |
| JP3212787B2 (en) | Transfer data management method and data transfer method between host and terminal | |
| CN114764403A (en) | Data migration method, device, equipment and storage medium | |
| CN110019031B (en) | File creation method and file management device | |
| CN110879747B (en) | Resource management method and device | |
| CN114879908A (en) | Quota management method for shared directory and related components | |
| CN117555482A (en) | Method and device for online quick deletion of Ceph storage space | |
| CA1311850C (en) | Controlling access to network resources |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |