CN116383161A - File synchronization method, device and medium - Google Patents
File synchronization method, device and medium Download PDFInfo
- Publication number
- CN116383161A CN116383161A CN202310018869.8A CN202310018869A CN116383161A CN 116383161 A CN116383161 A CN 116383161A CN 202310018869 A CN202310018869 A CN 202310018869A CN 116383161 A CN116383161 A CN 116383161A
- Authority
- CN
- China
- Prior art keywords
- synchronization
- snapshot
- file
- difference
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 63
- 238000003860 storage Methods 0.000 claims abstract description 24
- 230000001360 synchronised effect Effects 0.000 claims abstract description 22
- 238000004590 computer program Methods 0.000 claims description 11
- 230000004048 modification Effects 0.000 claims description 9
- 238000012986 modification Methods 0.000 claims description 9
- 238000009826 distribution Methods 0.000 claims description 8
- 238000012545 processing Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 230000009471 action Effects 0.000 description 3
- 238000007792 addition Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000002688 persistence Effects 0.000 description 2
- 238000009877 rendering Methods 0.000 description 2
- 238000012790 confirmation Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Hardware Redundancy (AREA)
Abstract
The application relates to the field of distributed storage and discloses a file synchronization method, a device and a medium, wherein a target snapshot is obtained by performing synchronous snapshot on files in a main directory; confirming the synchronization type according to the corresponding relation between the target snapshot and the reference snapshot; the reference snapshot is the data backup of the last successful synchronization time point; and according to the synchronization type, calling multithreading to synchronize the files. Compared with the prior art, the method has the advantages that the synchronization efficiency caused by data synchronization by using a single thread is low, by adopting the technical scheme, the file in the main cluster is subjected to snapshot and is compared with the reference snapshot obtained by data backup at the last successful synchronization time point, so that the corresponding relation between the target snapshot and the reference snapshot is confirmed, the synchronization type is confirmed according to the corresponding relation between the target snapshot and the reference snapshot, and the multithreading is called according to the synchronization type to perform data synchronization, so that the synchronization efficiency is improved.
Description
Technical Field
The present disclosure relates to the field of distributed storage, and in particular, to a method, an apparatus, and a medium for file synchronization.
Background
Along with the digitization of information, data gradually becomes the business core of a user, the stability requirement of the user on a storage system carrying data is gradually improved, and although most storage manufacturers can provide storage equipment with higher stability for the user, the storage equipment cannot prevent various natural disasters from damaging the production system in an unrecoverable manner, and various modes for protecting the data are realized, wherein most commonly, the data are copied and stored in a plurality of places, one place is failed to cause data loss, and the data can be recovered through the data at other places, so that the safety of the data is ensured. Remote copy technology has evolved to ensure data storage persistence, recoverability, high availability.
Remote copying requires that data be transferred as quickly as possible, improving the copying efficiency. Concurrent copying can remarkably improve copying efficiency, and the current mainstream mode is completed in a snapshot transmission mode. The definition of the snapshot is: with respect to a fully available copy of a given data set, the copy includes an image of the corresponding data at some point in time (the point in time at which the copy began).
Each remote copy essentially synchronizes a snapshot of the master cluster to the slave cluster, the difference between the master and slave clusters being in essence the point of difference between the two snapshots. Because the distributed file system service has strict time sequence, if a complete sequence mode is used when the master and slave clusters synchronize the difference points, only a single thread can be adopted, and the synchronization efficiency is lower.
It can be seen that how to improve the synchronization efficiency in data synchronization is a problem to be solved by those skilled in the art.
Disclosure of Invention
The application aims to provide a file synchronization method, a device and a medium, which are used for improving synchronization efficiency during data synchronization.
In order to solve the above technical problems, the present application provides a file synchronization method, including:
performing synchronous snapshot on files in the main catalog to obtain a target snapshot;
confirming a synchronization type according to the corresponding relation between the target snapshot and the reference snapshot; the reference snapshot is the data backup of the last synchronization success time point;
and according to the synchronization type, calling multithreading to synchronize the files.
Preferably, the determining the synchronization type according to the correspondence between the target snapshot and the reference snapshot includes:
confirming a difference point between the target snapshot and the reference snapshot; wherein, the difference point is an index node of the changed file;
and confirming the synchronization type according to the distribution condition of the difference points in the target snapshot and the reference snapshot.
Preferably, the method further comprises:
if the difference point fails to synchronize, recording the difference point which fails to synchronize;
and after the synchronization is completed, retrying the synchronization of the difference points with failed synchronization until all the difference points are completed.
Preferably, said determining the synchronization type according to the distribution situation of the difference point between the target snapshot and the reference snapshot includes:
if the difference point does not exist on the reference snapshot but exists on the target snapshot, confirming that the synchronization type is a new operation;
if the difference point exists on the reference snapshot but does not exist on the target snapshot, confirming that the synchronization type is a deleting operation;
and if the difference point exists on the reference snapshot and the target snapshot at the same time, confirming that the synchronization type is a modification operation.
Preferably, after all the difference points are synchronized, the method further comprises:
snapshot the files in the master catalog of the slave cluster to obtain a slave catalog snapshot;
judging whether the name of the slave catalog snapshot is consistent with the name of the target snapshot;
if the synchronization is consistent, the success of the synchronization is confirmed.
Preferably, if the name of the slave directory snapshot and the name of the target snapshot are inconsistent, the method further includes:
recording the difference points with inconsistent names, and sending alarm information.
Preferably, the method further comprises:
and storing the recorded difference points to a cloud server.
In order to solve the above technical problem, the present application further provides a file synchronization device, including:
the first snapshot module is used for carrying out synchronous snapshot on files in the main catalog to obtain a target snapshot;
the confirming module is used for confirming the synchronous type according to the corresponding relation between the target snapshot and the reference snapshot; the reference snapshot is the data backup of the last synchronization success time point;
and the synchronization module is used for calling multithreading to synchronize the files according to the synchronization type.
In order to solve the above technical problems, the present application further provides another file synchronization device, including a memory, configured to store a computer program;
and a processor for implementing the steps of the file synchronization method as described above when executing the computer program.
To solve the above technical problem, the present application further provides a computer readable storage medium, where a computer program is stored, where the computer program implements the steps of the file synchronization method described above when executed by a processor.
According to the file synchronization method, the target snapshot is obtained by performing the synchronization snapshot on the files in the main catalog; confirming the synchronization type according to the corresponding relation between the target snapshot and the reference snapshot; the reference snapshot is the data backup of the last successful synchronization time point; and according to the synchronization type, calling multithreading to synchronize the files. Compared with the prior art, the method has the advantages that the synchronization efficiency caused by data synchronization by using a single thread is low, by adopting the technical scheme, the file in the main cluster is subjected to snapshot and is compared with the reference snapshot obtained by data backup at the last successful synchronization time point, so that the corresponding relation between the target snapshot and the reference snapshot is confirmed, the synchronization type is confirmed according to the corresponding relation between the target snapshot and the reference snapshot, and the multithreading is called according to the synchronization type to perform data synchronization, so that the synchronization efficiency is improved.
In addition, the file synchronization device and the medium provided by the application correspond to the file synchronization method, and have the same effects.
Drawings
For a clearer description of the embodiments of the present application, the drawings that are needed in the embodiments will be briefly described, it being apparent that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a file synchronization method according to an embodiment of the present application;
fig. 2 is a synchronization method of synchronization failure difference points provided in the embodiment of the present application;
FIG. 3 is a block diagram of a file synchronization device according to an embodiment of the present disclosure;
fig. 4 is a block diagram of another file synchronization device according to an embodiment of the present application.
Detailed Description
The following description of the technical solutions in the embodiments of the present application will be made clearly and completely with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments herein without making any inventive effort are intended to fall within the scope of the present application.
Along with the digitization of information, data gradually becomes the business core of a user, the stability requirement of the user on a storage system carrying data is gradually improved, and although most storage manufacturers can provide storage equipment with higher stability for the user, the storage equipment cannot prevent various natural disasters from damaging the production system in an unrecoverable manner, and various modes for protecting the data are realized, wherein most commonly, the data are copied and stored in a plurality of places, one place is failed to cause data loss, and the data can be recovered through the data at other places, so that the safety of the data is ensured. Remote copy technology has evolved to ensure data storage persistence, recoverability, high availability.
Remote copying requires that data be transferred as quickly as possible, improving the copying efficiency. Concurrent copying can remarkably improve copying efficiency, and the current mainstream mode is completed in a snapshot transmission mode. The definition of the snapshot is: with respect to a fully available copy of a given data set, the copy includes an image of the corresponding data at some point in time (the point in time at which the copy began).
Each remote copy essentially synchronizes a snapshot of the master cluster to the slave cluster, the difference between the master and slave clusters being in essence the point of difference between the two snapshots. Because the distributed file system service has strict time sequence, if a complete sequence mode is used when the master and slave clusters synchronize the difference points, only a single thread can be adopted, and the synchronization efficiency is lower. The multithread synchronization cannot achieve the complete sequency of the difference point synchronization because of the characteristic of simultaneous operation, and many difference points which must be operated sequentially fail synchronously.
It can be seen that how to improve the synchronization efficiency in data synchronization is a problem to be solved by those skilled in the art.
The core of the application is to provide a file synchronization method, a device and a medium, which are used for improving the synchronization efficiency during data synchronization.
In order to provide a better understanding of the present application, those skilled in the art will now make further details of the present application with reference to the drawings and detailed description.
Fig. 1 is a flowchart of a file synchronization method provided in an embodiment of the present application, as shown in fig. 1, where the method includes:
s10: performing synchronous snapshot on files in the main catalog to obtain a target snapshot;
s11: confirming the synchronization type according to the corresponding relation between the target snapshot and the reference snapshot; the reference snapshot is the data backup of the last successful synchronization time point;
s12: and according to the synchronization type, calling multithreading to synchronize the files.
The definition of the snapshot is: with respect to a fully available copy of a given data set, the copy includes an image of the corresponding data at some point in time (the point in time at which the copy began). In the application, when each file in the file system is changed, a difference point is recorded, the remote copy function performs remote synchronization according to the difference point, the synchronization sequence is not strictly sequential, when certain file operations which are required to be sequentially executed are synchronized, if failure occurs, the difference point of failure is recorded, and finally retry is performed. If the first retry fails, the failed difference points continue to be recorded for the second retry until all the difference points are successfully synchronized.
In the present application, when a file in the file system is changed, for example, when an operation such as adding, deleting, modifying, etc., an inode of the changed file is recorded as a difference point, and these difference points are logically sequential, for example, a folder is added and a file is added under the folder, and the difference point of creating the folder is necessarily before creating the file.
In this embodiment, the target snapshot is obtained by performing a synchronization snapshot on the files in the main directory, where the target snapshot is a modified file snapshot that needs to be synchronized to other clusters. When synchronization is started, multithreading concurrently reads the difference points, and files needing to be synchronized are selected to be synchronized to a far end according to the difference points. The difference point is the difference between the reference snapshot and the target snapshot, and the reference snapshot is the data backup at the time point when the last synchronization was successful. The latest state of the slave and the reference snapshot of the master are consistent. The purpose of synchronization is to calculate the type of synchronization from the points of difference between the snapshots, and then synchronize. And when the difference point does not exist on the reference snapshot and the target snapshot exists, the operation is described as a new operation. When the difference point exists on the reference snapshot and does not exist on the target snapshot, the deleting operation is described. A modification operation is described when a difference point exists on both the reference snapshot and the target snapshot.
In addition, in the implementation, after the synchronization is successful, the snapshot can be played on the slave end, the snapshot name is consistent with the target snapshot name of the master end, and the master and slave clusters achieve data consistency.
In a specific implementation, when the remote end performs synchronization concurrently, a strict sequence cannot be guaranteed, and a difference point may perform synchronization in advance. Such as synchronizing the files before creating the parent directory of the file. This is the case because the parent directory of the file does not exist and the synchronization file will fail. When the synchronization fails, the difference point of the synchronization failure should be recorded, and the difference point is retried after the first round of synchronization is completed. There may still be a difference point of synchronization failure after the first round of retry, at which time a second round of retry is recorded, if still failed, the next round is continued. Because the first point of the service sequence of each round of retry difference points must succeed synchronously, the number of difference points of each round of retry is necessarily reduced, and finally, the whole process is successful.
There are various situations of the difference point of the synchronization failure, and there are various operations besides the above, for example, when deleting the directory, if there are more files under the directory, deleting the failure and recording the difference point, which will not be described herein. In principle, the new operation must wait for the parent directory to complete synchronously, and the delete operation must wait for the child file to be deleted successfully.
According to the file synchronization method, index nodes of files recording service changes on a master cluster are different points, concurrent synchronization is carried out on slave clusters according to the different points, when the different points with synchronization failure are encountered, the files are recorded and wait for retry of next synchronization, synchronization can be carried out through multiple rounds, and synchronization is completed after all the different points are successfully synchronized. The method can remarkably improve the synchronization efficiency, thereby improving the efficiency of incremental copying. The inode with the changed file is recorded as a difference point and can be retried when the master-slave synchronization difference point fails. And finally, the master and slave data tend to be consistent after retrying.
According to the file synchronization method, the target snapshot is obtained by performing the synchronization snapshot on the files in the main catalog; confirming the synchronization type according to the corresponding relation between the target snapshot and the reference snapshot; the reference snapshot is the data backup of the last successful synchronization time point; and according to the synchronization type, calling multithreading to synchronize the files. Compared with the prior art, the method has the advantages that the synchronization efficiency caused by data synchronization by using a single thread is low, by adopting the technical scheme, the file in the main cluster is subjected to snapshot and is compared with the reference snapshot obtained by data backup at the last successful synchronization time point, so that the corresponding relation between the target snapshot and the reference snapshot is confirmed, the synchronization type is confirmed according to the corresponding relation between the target snapshot and the reference snapshot, and the multithreading is called according to the synchronization type to perform data synchronization, so that the synchronization efficiency is improved.
When the files in the file system are subjected to operations such as new addition, deletion, modification and the like, the inodes of the changed files are recorded as difference points, and the difference points are logically sequential, for example, a folder is added and a file is added under the folder, and the difference point of the created folder is necessarily before the file is created.
In the above embodiment, it is described that when a file in the file system is changed, the difference points are recorded, and the distribution situation of the difference points can be confirmed by comparing the reference snapshot with the target snapshot.
Therefore, in the present embodiment, confirming the synchronization type according to the correspondence between the target snapshot and the reference snapshot includes:
confirming a difference point between the target snapshot and the reference snapshot; wherein, the difference point is the index node of the changed file;
and confirming the synchronization type according to the distribution condition of the difference points in the target snapshot and the reference snapshot.
Specifically, confirming the synchronization type according to the distribution condition of the difference points between the target snapshot and the reference snapshot includes:
if the difference point does not exist on the reference snapshot but exists on the target snapshot, confirming that the synchronization type is a newly added operation;
if the difference point exists on the reference snapshot but does not exist on the target snapshot, confirming that the synchronization type is a deleting operation;
if the difference point exists on the reference snapshot and the target snapshot at the same time, confirming that the synchronization type is a modification operation.
By adopting the technical scheme, the distribution condition of the difference points is judged, and when the difference points do not exist on the reference snapshot and the target snapshot exists, the new operation is indicated. When the difference point exists on the reference snapshot and does not exist on the target snapshot, the deleting operation is described. The modification operation is described when the difference points exist on the reference snapshot and the target snapshot at the same time, so that the judgment of the operation is realized.
In implementations, each remote copy essentially synchronizes a snapshot of the master cluster to the slave cluster, and the difference between the master and slave clusters is essentially the point of difference between the two snapshots. Because the distributed file system service has strict time sequence, if a complete sequence mode is used when the master and slave clusters synchronize the difference points, only a single thread can be adopted, and the synchronization efficiency is far lower than that of a multi-thread mode. The multithread synchronization cannot achieve the complete sequency of the difference point synchronization because of the characteristic of simultaneous operation, and many difference points which must be operated sequentially fail synchronously.
Therefore, on the basis of the above embodiment, in the present embodiment, further includes:
if the synchronization of the difference points fails, recording the difference points of the synchronization failure;
and after the synchronization is completed, retrying the synchronization of the difference points with failed synchronization until all the difference points are completed.
For different file changing operations, the file changing operations are not the same when the file changing operations are performed, and when the file changing operations are performed concurrently, a strict sequence cannot be ensured, and the synchronization may be performed in advance by different points. Such as synchronizing the files before creating the parent directory of the file. This is the case because the parent directory of the file does not exist and the synchronization file will fail. Fig. 2 is a schematic diagram of a synchronization method of a synchronization failure difference point according to an embodiment of the present application, as shown in fig. 2, when the synchronization failure difference point is recorded, and the synchronization failure difference point is retried after the first round of synchronization is completed.
There may still be a difference point of synchronization failure after the first round of retry, at which time a second round of retry is recorded, and if there is a failure, the next round is continued. Because the first point of the service sequence of each round of retry difference points must succeed synchronously, the number of difference points of each round of retry is necessarily reduced, and finally, the whole process is successful.
There are various cases of the difference point of the synchronization failure, and there are various operations other than the above, such as deleting the failure and recording the difference point if there are files under the directory when deleting the directory. In principle, the new operation must wait for the parent directory to complete synchronously, and the delete operation must wait for the child file to be deleted successfully.
According to the file synchronization method provided by the embodiment, when the difference points of synchronization failure are encountered, the file synchronization method records the difference points and waits for the retry of the next round of synchronization, the synchronization can be completed after the synchronization of all the difference points is successful after multiple rounds of synchronization. The method can remarkably improve the synchronization efficiency, thereby improving the efficiency of incremental copying. The inode with the changed file is recorded as a difference point and can be retried when the master-slave synchronization difference point fails. And finally, the master and slave data tend to be consistent after retrying.
In a specific implementation, after the synchronization is completed, in order to ensure the completion degree of the synchronization and avoid data loss, the result of the synchronization can be verified. In this embodiment, after synchronization is completed at all the difference points, the method further includes:
snapshot the files in the master catalog of the slave cluster to obtain a slave catalog snapshot;
judging whether the name of the slave directory snapshot is consistent with the name of the target snapshot;
if the synchronization is consistent, the success of the synchronization is confirmed.
In addition, in the implementation, if the name of the slave directory snapshot and the name of the target snapshot are not consistent, the method further includes: recording the different points with inconsistent names, and sending alarm information. Further comprises: and storing the recorded difference points to a cloud server.
In the above embodiments, the file synchronization method is described in detail, and the present application further provides a corresponding embodiment of the file synchronization device. It should be noted that the present application describes an embodiment of the device portion from two angles, one based on the angle of the functional module and the other based on the angle of the hardware.
Fig. 3 is a block diagram of a file synchronization device according to an embodiment of the present application, as shown in fig. 3, where the device includes:
a first snapshot module 10, configured to perform a synchronous snapshot on a file in the main directory to obtain a target snapshot;
a confirmation module 11, configured to confirm the synchronization type according to the correspondence between the target snapshot and the reference snapshot; the reference snapshot is the data backup of the last successful synchronization time point;
and the synchronization module 12 is used for calling the multithreading to synchronize the files according to the synchronization type.
A snapshot is a complete available copy of a given data set that includes an image of the corresponding data at some point in time (the point in time when the copy began). When each file in the file system is changed, the difference points are recorded, the remote copy function carries out remote synchronization according to the difference points, the synchronization sequence is not strict, when certain file operations which are required to be executed sequentially are synchronized, if failure occurs, the failed difference points are recorded, and finally retry is carried out. If the first retry fails, the failed difference points continue to be recorded for the second retry until all the difference points are successfully synchronized.
When the files in the file system are subjected to operations such as new addition, deletion, modification and the like, the inodes of the changed files are recorded as difference points, and the difference points are logically sequential, for example, a folder is added and a file is added under the folder, and the difference point of the created folder is necessarily before the file is created.
When synchronization is started, multithreading concurrently reads the difference points, and files needing to be synchronized are selected to be synchronized to a far end according to the difference points. The difference point is the difference between the reference snapshot and the target snapshot, and the reference snapshot is the data backup at the time point when the last synchronization was successful. The latest state of the slave and the reference snapshot of the master are consistent. The purpose of synchronization is to calculate the type of synchronization from the points of difference between the snapshots, and then synchronize. And after the synchronization is successful, the snapshot is played on the slave end, the snapshot name is consistent with the target snapshot name of the master end, and the master and slave clusters achieve data consistency.
Since the embodiments of the apparatus portion and the embodiments of the method portion correspond to each other, the embodiments of the apparatus portion are referred to the description of the embodiments of the method portion, and are not repeated herein.
According to the file synchronization device provided by the embodiment of the application, the target snapshot is obtained by performing the synchronization snapshot on the files in the main catalog; confirming the synchronization type according to the corresponding relation between the target snapshot and the reference snapshot; the reference snapshot is the data backup of the last successful synchronization time point; and according to the synchronization type, calling multithreading to synchronize the files. Compared with the prior art, the method has the advantages that the synchronization efficiency caused by data synchronization by using a single thread is low, by adopting the technical scheme, the file in the main cluster is subjected to snapshot and is compared with the reference snapshot obtained by data backup at the last successful synchronization time point, so that the corresponding relation between the target snapshot and the reference snapshot is confirmed, the synchronization type is confirmed according to the corresponding relation between the target snapshot and the reference snapshot, and the multithreading is called according to the synchronization type to perform data synchronization, so that the synchronization efficiency is improved.
Fig. 4 is a block diagram of another file synchronization device according to an embodiment of the present application, as shown in fig. 4, where the device includes: a memory 20 for storing a computer program;
a processor 21 for implementing the steps of the file synchronization method according to the above embodiment when executing a computer program.
The file synchronization device provided in this embodiment may include, but is not limited to, a smart phone, a tablet computer, a notebook computer, a desktop computer, or the like.
Processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 may be implemented in hardware in at least one of a digital signal processor (Digital Signal Processor, DSP), a Field programmable gate array (Field-Programmable Gate Array, FPGA), a programmable logic array (Programmable Logic Array, PLA). The processor 21 may also comprise a main processor, which is a processor for processing data in an awake state, also called central processor (Central Processing Unit, CPU), and a coprocessor; a coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 21 may be integrated with an image processor (Graphics Processing Unit, GPU) for taking care of rendering and rendering of the content that the display screen is required to display. In some embodiments, the processor 21 may also include an artificial intelligence (Artificial Intelligence, AI) processor for processing computing operations related to machine learning.
Memory 20 may include one or more computer-readable storage media, which may be non-transitory. Memory 20 may also include high-speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used for storing a computer program 201, which, when loaded and executed by the processor 21, is capable of implementing the relevant steps of the file synchronization method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may further include an operating system 202, data 203, and the like, where the storage manner may be transient storage or permanent storage. The operating system 202 may include Windows, unix, linux, among others. The data 203 may include, but is not limited to, a target snapshot, a reference snapshot, and the like.
In some embodiments, the file synchronization device may further include a display 22, an input/output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.
Those skilled in the art will appreciate that the structure shown in fig. 4 is not limiting of the file synchronization device and may include more or fewer components than shown.
The file synchronization device provided by the embodiment of the application comprises a memory and a processor, wherein the processor can realize the following method when executing a program stored in the memory: performing synchronous snapshot on files in the main catalog to obtain a target snapshot; confirming the synchronization type according to the corresponding relation between the target snapshot and the reference snapshot; the reference snapshot is the data backup of the last successful synchronization time point; and according to the synchronization type, calling multithreading to synchronize the files.
According to the file synchronization device provided by the embodiment of the application, the target snapshot is obtained by performing the synchronization snapshot on the files in the main catalog; confirming the synchronization type according to the corresponding relation between the target snapshot and the reference snapshot; the reference snapshot is the data backup of the last successful synchronization time point; and according to the synchronization type, calling multithreading to synchronize the files. Compared with the prior art, the method has the advantages that the synchronization efficiency caused by data synchronization by using a single thread is low, by adopting the technical scheme, the file in the main cluster is subjected to snapshot and is compared with the reference snapshot obtained by data backup at the last successful synchronization time point, so that the corresponding relation between the target snapshot and the reference snapshot is confirmed, the synchronization type is confirmed according to the corresponding relation between the target snapshot and the reference snapshot, and the multithreading is called according to the synchronization type to perform data synchronization, so that the synchronization efficiency is improved.
Finally, the present application also provides a corresponding embodiment of the computer readable storage medium. The computer-readable storage medium has stored thereon a computer program which, when executed by a processor, performs the steps as described in the method embodiments above.
It will be appreciated that the methods of the above embodiments, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored on a computer readable storage medium. With such understanding, the technical solution of the present application, or a part contributing to the prior art or all or part of the technical solution, may be embodied in the form of a software product stored in a storage medium, performing all or part of the steps of the method described in the various embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The computer readable storage medium provided by the embodiment of the application obtains a target snapshot by performing synchronous snapshot on files in a main catalog; confirming the synchronization type according to the corresponding relation between the target snapshot and the reference snapshot; the reference snapshot is the data backup of the last successful synchronization time point; and according to the synchronization type, calling multithreading to synchronize the files. Compared with the prior art, the method has the advantages that the synchronization efficiency caused by data synchronization by using a single thread is low, by adopting the technical scheme, the file in the main cluster is subjected to snapshot and is compared with the reference snapshot obtained by data backup at the last successful synchronization time point, so that the corresponding relation between the target snapshot and the reference snapshot is confirmed, the synchronization type is confirmed according to the corresponding relation between the target snapshot and the reference snapshot, and the multithreading is called according to the synchronization type to perform data synchronization, so that the synchronization efficiency is improved.
The file synchronization method, device and medium provided by the application are described in detail above. In the description, each embodiment is described in a progressive manner, and each embodiment is mainly described by the differences from other embodiments, so that the same similar parts among the embodiments are mutually referred. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section. It should be noted that it would be obvious to those skilled in the art that various improvements and modifications can be made to the present application without departing from the principles of the present application, and such improvements and modifications fall within the scope of the claims of the present application.
It should also be noted that in this specification, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
Claims (10)
1. A method for synchronizing files, comprising:
performing synchronous snapshot on files in the main catalog to obtain a target snapshot;
confirming a synchronization type according to the corresponding relation between the target snapshot and the reference snapshot; the reference snapshot is the data backup of the last synchronization success time point;
and according to the synchronization type, calling multithreading to synchronize the files.
2. The file synchronization method according to claim 1, wherein said confirming the synchronization type according to the correspondence between the target snapshot and the reference snapshot comprises:
confirming a difference point between the target snapshot and the reference snapshot; wherein, the difference point is an index node of the changed file;
and confirming the synchronization type according to the distribution condition of the difference points in the target snapshot and the reference snapshot.
3. The file synchronization method of claim 2, further comprising:
if the difference point fails to synchronize, recording the difference point which fails to synchronize;
and after the synchronization is completed, retrying the synchronization of the difference points with failed synchronization until all the difference points are completed.
4. A file synchronization method in accordance with claim 3, wherein said validating said synchronization type based on a distribution of said difference points between said target snapshot and said reference snapshot comprises:
if the difference point does not exist on the reference snapshot but exists on the target snapshot, confirming that the synchronization type is a new operation;
if the difference point exists on the reference snapshot but does not exist on the target snapshot, confirming that the synchronization type is a deleting operation;
and if the difference point exists on the reference snapshot and the target snapshot at the same time, confirming that the synchronization type is a modification operation.
5. A method of synchronizing files according to claim 3, further comprising, after synchronization is completed at all of the points of difference:
snapshot the files in the master catalog of the slave cluster to obtain a slave catalog snapshot;
judging whether the name of the slave catalog snapshot is consistent with the name of the target snapshot;
if the synchronization is consistent, the success of the synchronization is confirmed.
6. The file synchronization method of claim 5, further comprising, if the name of the slave directory snapshot and the name of the target snapshot do not match:
recording the difference points with inconsistent names, and sending alarm information.
7. The file synchronization method of claim 6, further comprising:
and storing the recorded difference points to a cloud server.
8. A document synchronization apparatus, comprising:
the first snapshot module is used for carrying out synchronous snapshot on files in the main catalog to obtain a target snapshot;
the confirming module is used for confirming the synchronous type according to the corresponding relation between the target snapshot and the reference snapshot; the reference snapshot is the data backup of the last synchronization success time point;
and the synchronization module is used for calling multithreading to synchronize the files according to the synchronization type.
9. A file synchronization device comprising a memory for storing a computer program;
processor for implementing the steps of the file synchronization method according to any of claims 1 to 7 when executing said computer program.
10. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the steps of the file synchronization method according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310018869.8A CN116383161A (en) | 2023-01-06 | 2023-01-06 | File synchronization method, device and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310018869.8A CN116383161A (en) | 2023-01-06 | 2023-01-06 | File synchronization method, device and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116383161A true CN116383161A (en) | 2023-07-04 |
Family
ID=86977592
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310018869.8A Pending CN116383161A (en) | 2023-01-06 | 2023-01-06 | File synchronization method, device and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116383161A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117290328A (en) * | 2023-11-22 | 2023-12-26 | 苏州元脑智能科技有限公司 | Storage system file concurrency synchronization method, device, equipment and readable storage medium |
-
2023
- 2023-01-06 CN CN202310018869.8A patent/CN116383161A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117290328A (en) * | 2023-11-22 | 2023-12-26 | 苏州元脑智能科技有限公司 | Storage system file concurrency synchronization method, device, equipment and readable storage medium |
| CN117290328B (en) * | 2023-11-22 | 2024-02-27 | 苏州元脑智能科技有限公司 | Storage system file concurrency synchronization method, device, equipment and readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106325933B (en) | Batch data synchronous method and device | |
| US9946716B2 (en) | Distributed file system snapshot | |
| US20150213100A1 (en) | Data synchronization method and system | |
| US20190129976A1 (en) | Apparatus for controlling synchronization of metadata on network and method for the same | |
| US11119978B2 (en) | Snapshot version control | |
| CN110209735B (en) | Database backup method, database backup device, computing device, and storage medium | |
| US10102083B1 (en) | Method and system for managing metadata records of backups | |
| GB2565179A (en) | Component-based synchronization of digital assets | |
| CN107479881B (en) | Method for synchronizing difference codes, storage medium, electronic device and system | |
| US20150227605A1 (en) | Information processing terminal, synchronization control method, and computer-readable recording medium | |
| CN111104069A (en) | Multi-region data processing method and device of distributed storage system and electronic equipment | |
| EP2542976B1 (en) | Method and apparatus of backing-up subversion repository | |
| CN107357920B (en) | Incremental multi-copy data synchronization method and system | |
| US11762738B2 (en) | Reducing bandwidth during synthetic restores from a deduplication file system | |
| CN116383161A (en) | File synchronization method, device and medium | |
| CN114356844B (en) | Method and device for comparing file snapshots based on directory tree polling | |
| CN115098299A (en) | Backup method, disaster recovery method, device and equipment for virtual machine | |
| US10223206B1 (en) | Method and system to detect and delete uncommitted save sets of a backup | |
| CN112231288A (en) | Log storage method and device and medium | |
| US10445183B1 (en) | Method and system to reclaim disk space by deleting save sets of a backup | |
| JP6251965B2 (en) | Information system and database recovery method | |
| CN112860376B (en) | Snapshot chain manufacturing method and device, electronic equipment and storage medium | |
| CN115129676A (en) | Data synchronization method, device, equipment and medium | |
| US20050165862A1 (en) | Autonomic and fully recovering filesystem operations | |
| CN108874592B (en) | Data cold standby method and system for Log-structured storage engine |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |