CN107491565B

CN107491565B - Data synchronization method

Info

Publication number: CN107491565B
Application number: CN201710928744.3A
Authority: CN
Inventors: 冯泽康
Original assignee: Language Networking (wuhan) Information Technology Co Ltd
Current assignee: Language Networking (wuhan) Information Technology Co Ltd
Priority date: 2017-10-10
Filing date: 2017-10-10
Publication date: 2020-01-14
Anticipated expiration: 2037-10-10
Also published as: WO2019071801A1; CN107491565A

Abstract

The invention provides a data synchronization method applied to at least two data nodes, which considers the aspects of file I/O use frequency, system cpu occupation rate, time, space complexity and the like and provides a corresponding optimization mode, thereby realizing a universal and efficient file synchronization method; the calculation amount is saved from the algorithm flow, and the calculation speed of the file blocking strong hash (such as the md5 value) is optimized; when data is read, a self-adaptive block size adjusting mode is adopted, and a memory mapping method and a Hash value calculating and judging method are adopted, so that the file synchronization speed can be greatly increased, the user satisfaction degree is improved, and the resource overhead is saved.

Description

Data synchronization method

Technical Field

The invention belongs to the technical field of synchronization, and particularly relates to a data synchronization method among different data nodes.

Background

In the era of unprecedented fire and heat in cloud computing, with the increase of social information volume, information data which needs to be managed by computers in various industries is also larger and larger, the mass data cannot be stored by a single PC or a server, and the corresponding data needs to be stored in a facility with mass storage capacity. The cloud storage system integrates storage spaces of multiple machines in combination with cloud computing technology, some personal-oriented cloud storage services such as a hundred-degree cloud disk and enterprise-oriented cloud storage solutions such as seven-cow cloud storage have come and come, and the cloud storage systems provide data mass storage functions based on the Internet.

However, most of current network cloud disks do not support an incremental data synchronization function, a user modifies one byte in a large file locally and synchronizes all data of the whole file to the cloud, which is an aspect of consuming traffic for the user.

Cloud storage services supporting incremental synchronization are also available in the market, but all of the cloud storage services adopt a traditional sliding window data check mode algorithm of rsync to achieve file incremental synchronization. The method has certain limitation, and for a large amount of file data, especially under the condition that a plurality of same data blocks exist, the weak hash and the strong hash values of the data blocks are still inevitably calculated, so that cpu is consumed, and meanwhile, the time overhead of comparing the differentiation of all the data blocks of the file completely by calculating the hash is also large.

In addition, in the internet environment, when a plurality of users share a large file, a plurality of terminals and a file storage server form a star topology structure, and when one user changes certain data in the file and triggers file synchronization of the plurality of terminals, the file storage server at this time inevitably becomes a bottleneck of transmission, and network IO and cpu occupation are all violent.

In the implementation of a distributed file system, files stored by a master node and a slave node are required to be strictly consistent, so that file synchronization between the master node and the slave node is indispensable. In order to meet the requirement that a user can access a new file instead of a historical file after successfully uploading a data file to the distributed file system, the distributed file system considers that the user is allowed to access when the corresponding file is complete and completely consistent in data at a master node and a slave node. Based on the scenario, the data synchronization requirements of the master node and the slave node of the distributed file system must be efficient, and after a user uploads a file to one node of the distributed file system, another peer node of the distributed file system must quickly calculate and synchronize the difference data with the new file. In such a scenario, a higher requirement is put forward on a data synchronization algorithm, and network I/O and CPU resources must be saved as much as possible, so that the user experience is better under the condition that the strict consistency of the whole system is maintained.

Therefore, it is imperative to improve the performance of the data synchronization algorithm, and it should be considered to moderately share the computation pressure of the server to the file synchronization terminal to improve the processing performance of the server.

Disclosure of Invention

The invention aims to provide an effective data resource synchronization solution under the background of widely using cloud storage at present, so that the synchronization of cloud storage data files is accelerated, network I/O (input/output) resources are saved, and the flow overhead is saved. For individuals or units using the commercialized cloud storage scheme, an efficient and traffic-saving synchronization scheme may not only improve user satisfaction, but may also save costs of using a data file synchronization service. For a user who builds the cloud storage service, saving the synchronous flow of the data files means that the request throughput and the response speed of the system can be greatly improved under the same hardware investment, and a more favorable background guarantee is provided for a client.

Aiming at the current situation that a plurality of cloud storages do not have a file increment synchronization function or provide a primary file synchronization function but the calculation amount for data differentiation calculation is large, the invention provides a method for solving the problem in a targeted manner, and the method is comprehensively considered from the aspects of reducing the disk I/O frequency, saving the CPU calculation amount, reducing the data amount of network communication and the interaction times, so that the file synchronization performance is effectively improved.

The method is not only suitable for incremental synchronization of the cloud-stored files, but also suitable for the file synchronization of two sites which are mirror images of each other, and the differentiation comparison and incremental synchronization of duplex and non-duplex end-to-end files, so that the speed of file synchronization can be greatly increased, the satisfaction degree of users can be improved, and simultaneously the resource overhead can be saved.

Specifically, the invention is realized by the following technical scheme:

a data synchronization method applied to at least two data nodes comprises at least a first data node and a second data node, wherein the first data node and the second data node correspond to store at least one synchronized data file; when the data of the first data node changes, the data synchronization method is applied to enable the data of the second data node to keep synchronous; the data synchronization method comprises the following steps:

(1) sequentially reading at least one data block of a first data node according to the size of a preset value from the starting point of the first data node, and calculating a first sampling characteristic value of the data block of the at least one first data node;

(2) acquiring a second sampling characteristic value of a corresponding data block of the second data node;

(3) comparing whether the first sampling characteristic value is consistent with the second sampling characteristic value in the step (1) and the step (2);

(4) if the comparison result in the step (3) is inconsistent, determining that the current data block of the first data node is inconsistent with the corresponding data block of the second data node;

(5) repeating the step (1) to the step (4) until the data blocks which are inconsistent with all the corresponding data blocks in the first data node in the second data node are determined;

(6) automatically editing the data blocks which are determined in the step (5) and are inconsistent with all the corresponding data blocks in the first data nodes in the second data nodes, so that the data of the second data nodes and the data of the first data nodes are kept synchronous;

the method is characterized in that:

in the step (1), when reading the data block of the first data node, firstly, sequentially mapping the data block with a predetermined value size to the memory according to the size of the buffer area;

and, the method further comprises:

the first sampled characteristic value includes an MD5 value of a data block of the first data node of the predetermined value size and MD5 values of data blocks of the first data node other than the data block of the first data node of the predetermined value size;

the second sampled characteristic value includes an MD5 value of the data block of the corresponding one of the second data nodes read according to the size of the predetermined value and MD5 values of data blocks of the second data nodes other than the data block of the corresponding one of the second data nodes read according to the size of the predetermined value.

Those skilled in the art will appreciate that each data block of the first data node can have its own sampling characteristic value. However, in the present invention, it is not necessary to calculate the feature value for each data block, but it is determined which data blocks need to perform the above calculation steps through reasonable sampling point selection, so that the calculation speed is greatly increased compared with the algorithm in the prior art that each feature value (e.g. hash value) is calculated even if there are a large number of identical data blocks.

Preferably, the method further comprises that the predetermined value is different when different data blocks are read.

Further, different predetermined values may be set when reading different data blocks.

Specifically, if at a certain time node, the comparison results of the blocked weak hash value and the strong hash value of the data block of a first data node read according to the size of a first predetermined value and the data block of a corresponding second data node read according to the size of the first predetermined value are consistent,

then reading the data block of the next first data node according to the size of a second preset value; the second predetermined value is greater than the first predetermined value;

otherwise, reading the data block of the next first data node according to the size of the third preset value; the third predetermined value is less than the first predetermined value.

In addition, in the step (1) and the step (2), calculating a first sampling characteristic value of the data block of the at least one first data node and obtaining a second sampling characteristic value of the data block of the second data node, further includes:

the second data node calculates and stores block sampling hash values and non-sampling data hash values of a plurality of data blocks of the second data node in advance, and stores byte data with preset length values at the front ends of the data blocks;

after reading the data block of at least one first data node, comparing whether byte data of the preset length value at the front end of the corresponding data block of a second data node exists at the starting position of the data block of the first data node;

if not, the block sample hash value and the non-sample data hash value of the current data block of the at least one first data node do not need to be calculated.

Wherein the predetermined length value is much smaller than the first predetermined value, the second predetermined value and the third predetermined value.

The inventors have determined, by statistical analysis, that the predetermined length value should be at least less than one tenth of the first, second and third predetermined values.

Further, when the data blocks are read from the first data node, at least two data blocks are read each time; reading the at least two data blocks by adopting at least one first thread, and reading the at least two data blocks each time when reading the data blocks from the first data node; and after all the data blocks of the first data node are read, the difference data and the difference information are sent to the second data node and are combined by the second data node.

After the first thread reads the first data block, the second thread simultaneously carries out consistency judgment and simultaneously reads the second data block; and in the same way, the reading of the first thread and the judgment of the second thread are simultaneously and concurrently carried out.

Preferably, at least two data blocks are read each time, including a first data block and a second data block, the first thread reads the first data block into the cache region, and simultaneously notifies the second thread to perform consistency judgment;

during specific implementation, a data block information table may be maintained in the second data node in advance, where the information table stores an attribution relationship mapping table of block data and corresponding blocks, where the attribution relationship mapping table includes a weak hash value and a strong hash value of each block of data, an offset of each block of data at the second data node, and a size of each block of data; when the second data node receives the synchronous data for the first time, reading and writing the file by adopting a memory mapping technology; when the data is stored, the data is stored in blocks, and the attribution relation mapping table is calculated and stored.

When the synchronization method is applied to a cloud storage environment, the first data node may be a local data storage of a user side, and the second data node may be a cloud data server;

when applied to data synchronization of a distributed file system, the first data node and the second data node may be a master-slave node, one of master nodes, one of slave nodes of the distributed file system.

The method can be realized in a file synchronization software environment based on a C/S or B/S architecture, and can be realized by a person skilled in the art through a specified language code and then used as a core of file synchronization, and the client/server, the client/cloud and the master/slave nodes only need to realize corresponding logic processes according to an interaction flow specified in an algorithm

In addition, in the file synchronization environment of p2p, each node needs to implement the functions of a client and a server, so that data can be synchronized with each other, for example, in the application scenario of a distributed file system, efficient storage node file synchronization can be implemented by using the method.

Therefore, the method can be implemented by a corresponding file synchronization device, and can also be implemented by using related computer instructions stored in a computer-readable storage medium.

Drawings

FIG. 1 is a schematic diagram of a prior art rsync-based sliding window data check pattern algorithm

FIG. 2 is a flow chart of a prior art synchronization method

FIG. 3 is a schematic diagram of the proposed data synchronization method of the present invention

DETAILED DESCRIPTION OF EMBODIMENT (S) OF INVENTION

As described above, the traditional method for implementing file incremental synchronization by using the sliding window data check mode algorithm based on rsync has certain limitations, and for a large amount of file data, especially under the condition that many identical data blocks exist, it is still inevitable to calculate weak hash and strong hash values of the data blocks, cpu is consumed very much, and meanwhile, the time overhead for comparing the differences of all the data blocks of a file completely by calculating hash is also large.

The above-mentioned drawbacks can be more clearly understood in conjunction with fig. 1.

In the synchronization method, a sliding window (1) with a fixed size is set at first, a data stream (2) of a user end is read, and the data read each time is a byte; and calculating a weak hash value (such as an Alder32 check code) and a strong hash value (such as an md5 check code), comparing the weak hash value with a hash value of a corresponding data stream transmitted from a server, determining that the current block of the client is inconsistent with the block corresponding to the server if the weak hash value and the hash value are inconsistent, reading data of one byte, moving a sliding window backwards, removing the first byte, filling the read byte at the tail part, calculating the strong and weak check codes, comparing the strong and weak check codes with the server, and so on until the difference between all blocks of the file and the blocks of the server is compared.

Fig. 2 is a schematic flow chart of the prior art adopting the method.

In practical implementation, the method has at least the following defects: the method has the advantages that files need to be read frequently, the I/O of a disk is influenced obviously, the moving speed of a sliding window is low, only one byte is moved backwards each time, the strong and weak hash can be calculated repeatedly under the condition that the same blocks are more, the CPU consumption is high, the hash calculation times are more, and in the case of a large file and few changes, a lot of extra calculation is carried out, and the speed is still slow.

The synchronization method of the present invention, as shown in fig. 3, optimizes the above-mentioned drawbacks from several aspects.

(1) Optimizing disk I/O reads and writes

Considering that the sliding window moves backwards once to read one byte from the file, the reading of the I/O of the disk can be reduced by reading one data block at a time, but the speed of directly reading the file is still slow, and the data block can be directly mapped to the memory from the file by adopting the memory mapping technology, so that the speed of reading the file can be greatly improved.

Therefore, in fig. 3, first, starting from the start of the first data node (310), the data blocks of at least one first data node are read sequentially according to the size of the predetermined value (31,41, 51); when reading the data block of the first data node, firstly mapping the relevant data of the first data node into the memory, and then sequentially reading the data block of the first data node in the memory according to the size of the preset value; meanwhile, the predetermined values (31,41,51) are different in size when different data blocks are read.

Here, the file read is no longer a read byte, but a read data block; meanwhile, the size of the data block is read, namely the size of a traditional fixed sliding window is not larger, and the size of a variable preset value (31,41,51) is larger; in addition, a memory mapping technology is adopted, the data blocks are directly mapped to the memory from the file, and the file reading speed can be greatly improved.

Wherein the predetermined value (31,41,51) of variation employs the following transformation strategy:

if at a certain time node the comparison result of the data block of a first data node read according to the size of a first predetermined value (31,32) and the block weak hash value and the strong hash value (312,311) of the data block of a corresponding second data node read according to the size of the first predetermined value (31,32) is consistent,

then, reading the data block of the next first data node according to the size of a second preset value (41); the second predetermined value (41) is greater than the first predetermined value (31);

otherwise, reading the data block of the next first data node according to the size of a third preset value (51); the third predetermined value (51) is less than the first predetermined value (31).

In fig. 3, the reference numerals 32,42, and 52 are schematic data block size marks when reading data of a size (31,41,51) corresponding to a user terminal in the second data node (320). In general, the sizes of the two are consistent, and of course, the sizes can be set to other corresponding sizes according to actual situations.

It can be seen that, in the present invention, the predetermined value (31,41,51, 32,42, 52) is adaptively changed according to the comparison result of the two data nodes, so that the jumping speed of the unchanged data block can be increased, the missing of the changed data block can be avoided, and the comparison and synchronization rate can be increased as a whole.

(2) Taking advantage of multiple cores

The multithreading is adopted, more than 2 data blocks can be read in advance at one time to serve as a buffer pool, one thread is responsible for reading the data blocks, the other thread is responsible for synchronizing the data blocks, therefore, data synchronization and data reading can be carried out simultaneously, the memory mapping is combined, the higher speed can be improved, and the I/O overhead of a disk is reduced.

In the invention, when the data blocks are read from the first data node, at least two data blocks are read each time; reading the at least two data blocks by adopting at least one first thread, and reading the at least two data blocks each time when reading the data blocks from the first data node; and after all the data blocks of the first data node are read, the difference data and the difference information are sent to the second data node and are combined by the second data node.

(3) saving CPU computing resources

Aiming at the characteristic that the hash value calculation operation of the Rsync algorithm is too intensive, the technical scheme of the invention further adds a pre-judgment strategy; judging whether each data block is necessary before calculating the hash value, wherein the server not only stores the hash value of each data block, but also stores a small segment of bytes in front of the current block, and before actually carrying out comparison and verification, firstly, whether the bytes exist at the beginning of the current slider or not is judged, and if not, the hash value does not need to be calculated; otherwise, the sampled hash value is compared with the non-sampled partial hash value, the former weak hash is replaced by the random sampled data calculation md5, so that the calculation of the strong hash is replaced by directly calculating the hash value of the partial data except the sampled data in the data block, the whole data block is not required to calculate the hash value, the hash operation times are obviously saved, and the pressure of the CPU can be greatly relieved.

Specifically, the method for calculating the blocking weak hash value and the strong hash value of the data block further comprises the following steps:

the second data node calculates and stores blocking weak hash values and strong hash values of a plurality of data blocks of the second data node in advance, and stores byte data with preset length values at the front ends of the data blocks;

after reading the data blocks of at least one first data node, comparing whether bytes of the predetermined length value corresponding to the front end of the data block of a second data node exist in the data blocks of the first data node;

if not, the blocking weak hash value and the strong hash value of the data block of the at least one first data node do not need to be calculated.

Preferably, in the present invention, the first sampling characteristic value and the second sampling characteristic value may also be obtained by sampling according to the following method:

assume that the first data node A includes data chunks A1, A2, A3 … … An; the second data node B data are B1, B2, … … Bm.

Then for data block a1, some set of samples M for which it is possible may be selected as

M = {A1[0~N/6], A1[N/3~N/2],A1[5N/6,N-1]}

Where N represents the byte length of data block A1, A1[0 ~ N/6] represents a portion of data in the data block from the beginning of the A data block to length N/6, A1[ N/3 ~ N/2] represents data from length N/3 to length N/2, A1[5N/6, N-1] represents data from length 5N/6 to length N-1

TA1 = MD5(M)

TA2=MD5(C_AM)

Wherein, C_AM denotes the other part of the data block a excluding the sample set M.

When the consistency determination is performed, TA1 may be understood as a predicted value of the sampling characteristic value, and TA2 is a predicated value of the sampling characteristic value, that is, TA1 may further use the predicated value to confirm that the data blocks are completely consistent when the consistency of the data is predicted in the first step and the data blocks are consistent; conversely, if TA1 does not match, then the default value calculation need not be performed, and of course TA2 need not be calculated.

The sampling characteristic value algorithm and the consistency determination process of other data blocks are similar.

The calculation of the sampling characteristic value is given according to the formula above. If the consistency judgment result is yes, determining that a same data block is found; if not, the buffer area is moved backwards for a certain length, and the characteristic value of the new data block is calculated according to the size of the preset value and the same consistency judgment is carried out.

Preferably, the feature value may also be one of a block sample hash value, and/or a non-sample data hash value, and/or an MD5 value, and/or a strong hash value, and/or a weak hash value, or a combination of at least two of them, but the present invention focuses on speeding up the calculation of a file block strong hash (e.g., MD5 value).

(4) Reducing the computation load of the server

The server side can directly store the file blocks without storing complete files. The server side maintains a file block information table, the attribution relationship between the blocks and the file, the md5 value of each block and the like are stored on the surface and the inside of the server side, and in addition, the server side can also adopt a memory mapping technology when storing the file, so that the reading and writing speed of the file is greatly improved.

In the present invention, the above strategy is adopted, specifically: and maintaining a data block information table at the second data node, wherein the information table stores an attribution relation mapping table of the data blocks and corresponding blocks, and the attribution relation mapping table comprises a weak block hash value and a strong hash value of each data block, an offset of each data block at the second data node and the size of each data block.

When the second data node receives the synchronous data for the first time, reading and writing the file by adopting a memory mapping technology; when the data is stored, the data is stored in blocks, and the attribution relation mapping table is calculated and stored.

It can be seen that, compared with the conventional file synchronization algorithm, the invention at least obtains the following beneficial effects: the file I/O use frequency, the system CPU occupation rate, the time, the space complexity and the like are considered and a corresponding optimization mode is provided, so that a universal and efficient file synchronization method is realized; the calculation amount is saved from the algorithm flow, and the calculation speed of the file blocking strong hash (such as the md5 value) is optimized; if the synchronous client and the synchronous server exist on the server (such as a distributed file system storage service), considering that the hardware of the server is generally high-configuration and the GPU configuration is higher, the optimization of the invention greatly improves the high availability and high consistency of distributed storage.

Claims

1. A data synchronization method applied to at least two data nodes comprises at least a first data node and a second data node, wherein the first data node and the second data node correspond to store at least one synchronized data file; when the data of the first data node changes, the data synchronization method is applied to enable the data of the second data node to keep synchronous; the data synchronization method comprises the following steps:

(6) automatically editing the data blocks which are determined in the step (5) and are inconsistent with all the corresponding data blocks in the first data nodes in the second data nodes, so that the data of the second data nodes and the data of the first data nodes are kept synchronous; the method is characterized in that:

and, the method further comprises:

the second sampled characteristic value comprises an MD5 value of the data block of the corresponding one of the second data nodes read according to the size of the predetermined value and MD5 values of data blocks of the second data nodes other than the data block of the corresponding one of the second data nodes read according to the size of the predetermined value;

wherein the method further comprises: firstly, sequentially reading data blocks of at least one first data node according to the size of a preset value (31,41,51) from the starting point of the first data node (310); when reading the data block of the first data node, firstly mapping the relevant data of the first data node into the memory, and then sequentially reading the data block of the first data node in the memory according to the size of the preset value; when different data blocks are read, the preset values are different in size;

if the comparison result of the blocked weak hash value and the strong hash value of the data block of one first data node read according to the size of the first preset value and the data block of one corresponding second data node read according to the size of the first preset value is consistent at a certain time node,

then reading the data block of the next first data node according to the size of a second preset value; the second predetermined value is greater than the first predetermined value; otherwise, reading the data block of the next first data node according to the size of the third preset value; the third predetermined value is less than the first predetermined value.

2. The method of claim 1, wherein in the steps (1) and (2), calculating a first sampling characteristic value of the data block of the at least one first data node and obtaining a second sampling characteristic value of the data block of the second data node further comprises:

3. The method of claim 1, wherein when reading data blocks from a first data node, at least two data blocks are read at a time; and after all the data blocks of the first data node are read, the difference data and the difference information are sent to the second data node and are combined by the second data node.

4. The method of claim 2, wherein the predetermined length value is much smaller than the first predetermined value, the second predetermined value, and the third predetermined value, specifically: the predetermined length value should be at least less than one tenth of the first, second and third predetermined values.

5. The method of claim 3, wherein at least two data blocks are read at a time, including a first data block and a second data block, and the first data block is read into the cache using the first thread while the second thread is notified to make the coherency determination.

6. The method of any of claims 1 to 5, wherein: and maintaining a data block information table at the second data node, wherein the information table stores an attribution relation mapping table of the data blocks and corresponding blocks, and the attribution relation mapping table comprises a weak block hash value and a strong hash value of each data block, an offset of each data block at the second data node and the size of each data block.

7. The method of claim 6, wherein: and when the second data node receives the synchronous data for the first time, reading and writing the file by adopting a memory mapping technology.

8. A data synchronization apparatus for performing the data synchronization method of any one of claims 1 to 7.

9. A computer-readable storage medium having stored thereon computer-executable instructions; executing the instructions by a processor and a memory for implementing the data synchronization method according to any of claims 1-7.