CN114385747A - Mobile internet rapid data synchronization method - Google Patents
Mobile internet rapid data synchronization method Download PDFInfo
- Publication number
- CN114385747A CN114385747A CN202111106914.2A CN202111106914A CN114385747A CN 114385747 A CN114385747 A CN 114385747A CN 202111106914 A CN202111106914 A CN 202111106914A CN 114385747 A CN114385747 A CN 114385747A
- Authority
- CN
- China
- Prior art keywords
- check
- check code
- client terminal
- data
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 30
- 230000001360 synchronised effect Effects 0.000 claims abstract description 27
- 238000012795 verification Methods 0.000 claims abstract description 4
- 230000005540 biological transmission Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005265 energy consumption Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention provides a method for quickly synchronizing data of a mobile internet, which comprises a server terminal and a client terminal, and comprises the following steps: s1, a client terminal creates a verification cache table TA for a file to be synchronized, and the client terminal initiates a synchronization request of the file to be synchronized to a server terminal; s2, the server terminal receives the synchronization request, creates a check code table TB according to the file to be synchronized, and feeds the check code table TB back to the client terminal; s3, the client terminal receives the check code table TB, generates a difference check table TC according to the check cache table TA and the check code table TB, and sends the difference check table TC to the server terminal; s4, the server terminal receives the difference checking table TC, searches data corresponding to the difference checking table TC and sends the searched data to the client terminal; the client terminal and the server terminal can perform good data synchronization, the calculation amount of the terminal is reduced, and the flow in the synchronization process is reduced.
Description
Technical Field
The invention relates to a data synchronization method, in particular to a mobile internet rapid data synchronization method.
Background
In the era of mobile internet, the popularization of mobile terminals and the demand of people for information sharing have greatly promoted the development of technologies such as mobile cloud downloading, remote firmware/package updating, and the like. However, the mobile terminal is battery powered and limited by battery capacity, and the existing data synchronization algorithm often causes a large amount of bandwidth and traffic requirements, so that the energy consumption is too large.
In the prior art, a fast synchronization algorithm is mainly used for synchronizing files of a client terminal and a server terminal, and the fast data synchronization algorithm mainly comprises an rsync algorithm and an improved algorithm thereof. The algorithm was proposed by Andrew Tridgell in 1996 and has been integrated into a Linux system for updating files and directories of two computers simultaneously, with the idea of using differential encoding to reduce data transmission. Previous studies have shown that for files whose content similarity is substantially independent of the similarity of stored bytes (e.g., rar and exe), the rsync algorithm suffers from performance degradation.
Therefore, in order to solve the above technical problems, it is necessary to provide a new technical means.
Disclosure of Invention
In view of the above, an object of the present invention is to provide a method for fast data synchronization in a mobile internet, which can perform good data synchronization between a client terminal and a server terminal, and effectively reduce the amount of computation of the terminal in the synchronization process, thereby reducing the traffic in the synchronization process, and having higher synchronization efficiency.
The invention provides a method for quickly synchronizing data of a mobile internet, which comprises a server terminal and a client terminal, and comprises the following steps:
s1, a client terminal creates a verification cache table TA for a file to be synchronized, and the client terminal initiates a synchronization request of the file to be synchronized to a server terminal;
s2, the server terminal receives the synchronization request, creates a check code table TB according to the file to be synchronized, and feeds the check code table TB back to the client terminal;
s3, the client terminal receives the check code table TB, generates a difference check table TC according to the check cache table TA and the check code table TB, and sends the difference check table TC to the server terminal;
and S4, the server terminal receives the difference check table TC, searches data corresponding to the difference check table TC, and sends the searched data to the client terminal to complete synchronization.
Further, the synchronization request for the file to be synchronized must include the data block size Z.
Further, the check cache table TA is created as follows:
the client terminal divides the file to be synchronized into a plurality of data blocks according to the data block size Z;
and calculating the check code of each data block, judging whether the check code of each data block exists in the client or not, and if not, adding the check code to the check cache table TA.
Further, the check code table TB is created as follows:
the server terminal divides the file of the latest version stored by the server terminal into a plurality of data blocks according to the data block size Z;
the check code for each data block is calculated and added to the check code table TB.
Further, the check code includes a hash value of the data block, a check value, and an offset position of the data block in the file.
Further, the difference check table TC is generated as follows:
the client terminal acquires the hash value of each check code in the check code table TB;
the client terminal searches in the check cache table TA by taking the hash value of the check code in the check code table TB as a key word, if the corresponding hash value exists in the check cache table TA, the hash value is not processed, otherwise, the check code corresponding to the hash value is stored in the difference check table TC.
Further, the method also includes step S5:
the client terminal receives the synchronous data sent by the server terminal;
and the client terminal updates the synchronous data to a local database and updates the checking cache table TA.
The invention has the beneficial effects that: the invention can lead the client terminal and the server terminal to carry out good data synchronization, effectively reduces the calculated amount of the terminal in the synchronization process, thereby reducing the flow in the synchronization process and having higher synchronization efficiency.
Drawings
The invention is further described below with reference to the following figures and examples:
FIG. 1 is a flow chart of the present invention.
Detailed Description
The invention is described in further detail below with reference to the drawings of the specification:
the invention provides a method for quickly synchronizing data of a mobile internet, which comprises a server terminal and a client terminal, and comprises the following steps:
s1, a client terminal creates a verification cache table TA for a file to be synchronized, and the client terminal initiates a synchronization request of the file to be synchronized to a server terminal; the synchronous request must include a data block size Z;
s2, the server terminal receives the synchronization request, creates a check code table TB according to the file to be synchronized, and feeds the check code table TB back to the client terminal;
s3, the client terminal receives the check code table TB, generates a difference check table TC according to the check cache table TA and the check code table TB, and sends the difference check table TC to the server terminal;
and S4, the server terminal receives the difference check table TC, searches data corresponding to the difference check table TC, and sends the searched data to the client terminal to complete synchronization. The check cache table TA and the check code table TB are both composed of check codes, where the check codes include hash values of the data blocks, check values, and offset positions of the data blocks in the file, that is, the check code Cd is equal to<hashIDi,checki,offseti>Wherein i represents the ith data block, offsetiIs the offset position of the ith data block in the file, checkiFor the check value, hashID, of the ith data blockiThe hash value of the ith data block is obtained, wherein the check value is calculated by using a check function, such as an MD5 function, but other existing check functions are also available; the hash value is calculated by using a hash function, such as a linear probing hash function hash (x) ═ x +11 MOD199, although other existing hash functions may also be used in the present invention; by the method, the client terminal and the server terminal can be well synchronized, the calculated amount of the terminal is effectively reduced in the synchronization process, namely, the check code cannot be repeatedly calculated, the transmission frequency of the check code in the transmission process is greatly reduced, the transmission of repeated data is reduced, the flow in the synchronization process is reduced, high synchronization efficiency is achieved, and efficient and rapid data synchronization is achieved.
In this embodiment, the check cache table TA is created as follows:
the client terminal divides the file to be synchronized into a plurality of data blocks according to the data block size Z;
and calculating the check code of each data block, judging whether the check code of each data block exists in the client or not, and if not, adding the check code to the check cache table TA.
The check code table TB is created as follows:
the server terminal divides the file of the latest version stored by the server terminal into a plurality of data blocks according to the data block size Z;
the check code for each data block is calculated and added to the check code table TB.
The difference check table TC is generated as follows:
the client terminal acquires the hash value of each check code in the check code table TB;
the client terminal searches in the check cache table TA by taking the hash value of the check code in the check code table TB as a key word, if the corresponding hash value exists in the check cache table TA, the hash value is not processed, otherwise, the check code corresponding to the hash value is stored in the difference check table TC. By the method, data synchronization is facilitated, data retrieval is performed based on the hash value, retrieval speed and efficiency can be improved, unnecessary data transmission in the data synchronization process is reduced, and accordingly traffic usage of the synchronized data and bandwidth resources occupied by the data are reduced.
In this embodiment, the method further includes step S5:
the client terminal receives the synchronous data sent by the server terminal;
the client terminal updates the synchronous data to the local database and updates the checking cache table TA, and the method is favorable for the next synchronous operation.
The invention is described in further detail below using an example:
setting a file size of 10 on a client8And (6) bit. A new version of the file is set on the server, the size of the file on the client is the same, but there is a difference in the contents of one data block. Client press (Z10)6bit) to obtain 100 data blocks, and respectively calculating check codes, so that the local check cache table TA contains 100 entries. When the client requests the server for synchronization, the check code table TB contains 100 check codes. According to the assumption, the discrepancy check table TC contains only 1 check code. The server sends 1 corresponding data block to the client according to the difference check table TCAnd the client receives the data block, updates the data block into a local file, and updates a check cache table to prepare for next synchronization.
As can be seen from the above example, the communication overhead of the client and the server for transmitting the check code table TB and the difference check table TC is 101 × 128bit — 140,928 bit. The communication overhead of 1 data block sent by the server is 106And (6) bit. The communication overhead saved by the rapid data synchronization method is about 1- (140,928bit + 10)6bit)/108bit≈98.9％。
Finally, the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all of them should be covered in the claims of the present invention.
Claims (7)
1. A mobile internet rapid data synchronization method comprises a server terminal and a client terminal, and is characterized in that: the method comprises the following steps:
s1, a client terminal creates a verification cache table TA for a file to be synchronized, and the client terminal initiates a synchronization request of the file to be synchronized to a server terminal;
s2, the server terminal receives the synchronization request, creates a check code table TB according to the file to be synchronized, and feeds the check code table TB back to the client terminal;
s3, the client terminal receives the check code table TB, generates a difference check table TC according to the check cache table TA and the check code table TB, and sends the difference check table TC to the server terminal;
and S4, the server terminal receives the difference check table TC, searches data corresponding to the difference check table TC, and sends the searched data to the client terminal to complete synchronization.
2. The mobile internet fast data synchronization method as claimed in claim 1, wherein: the synchronization request for the file to be synchronized must include the data chunk size Z.
3. The mobile internet fast data synchronization method as claimed in claim 2, wherein: the check cache table TA is created as follows:
the client terminal divides the file to be synchronized into a plurality of data blocks according to the data block size Z;
and calculating the check code of each data block, judging whether the check code of each data block exists in the client or not, and if not, adding the check code to the check cache table TA.
4. The mobile internet fast data synchronization method as claimed in claim 3, wherein: the check code table TB is created as follows:
the server terminal divides the file of the latest version stored by the server terminal into a plurality of data blocks according to the data block size Z;
the check code for each data block is calculated and added to the check code table TB.
5. The mobile internet fast data synchronization method as claimed in claim 4, wherein: the check code includes a hash value of the data block, a check value, and an offset location of the data block in the file.
6. The mobile internet fast data synchronization method as claimed in claim 5, wherein: the difference check table TC is generated as follows:
the client terminal acquires the hash value of each check code in the check code table TB;
the client terminal searches in the check cache table TA by taking the hash value of the check code in the check code table TB as a key word, if the corresponding hash value exists in the check cache table TA, the hash value is not processed, otherwise, the check code corresponding to the hash value is stored in the difference check table TC.
7. The mobile internet fast data synchronization method as claimed in claim 1, wherein: further comprising step S5:
the client terminal receives the synchronous data sent by the server terminal;
and the client terminal updates the synchronous data to a local database and updates the checking cache table TA.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111106914.2A CN114385747A (en) | 2021-09-22 | 2021-09-22 | Mobile internet rapid data synchronization method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111106914.2A CN114385747A (en) | 2021-09-22 | 2021-09-22 | Mobile internet rapid data synchronization method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114385747A true CN114385747A (en) | 2022-04-22 |
Family
ID=81195013
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111106914.2A Pending CN114385747A (en) | 2021-09-22 | 2021-09-22 | Mobile internet rapid data synchronization method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114385747A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116055542A (en) * | 2022-12-29 | 2023-05-02 | 南京先进计算产业发展有限公司 | Network adapter and high-performance computing cluster server communication connection method |
-
2021
- 2021-09-22 CN CN202111106914.2A patent/CN114385747A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116055542A (en) * | 2022-12-29 | 2023-05-02 | 南京先进计算产业发展有限公司 | Network adapter and high-performance computing cluster server communication connection method |
| CN116055542B (en) * | 2022-12-29 | 2024-03-15 | 南京先进计算产业发展有限公司 | Network adapter and high-performance computing cluster server communication connection method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10552040B2 (en) | Fixed size extents for variable size deduplication segments | |
| RU2581551C2 (en) | Method for optimisation of data storage and transmission | |
| US8935487B2 (en) | Fast and low-RAM-footprint indexing for data deduplication | |
| US9053032B2 (en) | Fast and low-RAM-footprint indexing for data deduplication | |
| US9984093B2 (en) | Technique selection in a deduplication aware client environment | |
| US7359956B2 (en) | Data transfer scheme using caching and differential compression techniques for reducing network load | |
| US7640354B2 (en) | Scalable differential compression of network data | |
| US20070124415A1 (en) | Method and apparatus for reducing network traffic over low bandwidth links | |
| US20080297379A1 (en) | Determination of compression state information for use in interactive compression | |
| US20180357217A1 (en) | Chunk compression in a deduplication aware client environment | |
| US20050027731A1 (en) | Compression dictionaries | |
| US20100293141A1 (en) | Method and a System for Obtaining Differential Backup | |
| US9917894B2 (en) | Accelerating transfer protocols | |
| US10459886B2 (en) | Client-side deduplication with local chunk caching | |
| US8959155B1 (en) | Data compression through redundancy removal in an application acceleration environment | |
| WO2019237594A1 (en) | Session persistence method and apparatus, and computer device and storage medium | |
| US10339124B2 (en) | Data fingerprint strengthening | |
| EP2153609A1 (en) | Method and apparatus for communicating compression state information for interactive compression | |
| CN113885780A (en) | Data synchronization method, device, electronic equipment, system and storage medium | |
| CN114385747A (en) | Mobile internet rapid data synchronization method | |
| WO2014117729A1 (en) | Scalable data deduplication | |
| CN108900621B (en) | Fog computing mode-based difference cloud synchronization method | |
| US20210019231A1 (en) | Method, device and computer program product for backing up data | |
| CN111258623A (en) | Server and method for providing application and file, user terminal and computer readable storage medium | |
| CN109697136B (en) | Data recovery method, device and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |