CN113064909B - Data synchronization verification method and device - Google Patents
Data synchronization verification method and device Download PDFInfo
- Publication number
- CN113064909B CN113064909B CN202110616581.1A CN202110616581A CN113064909B CN 113064909 B CN113064909 B CN 113064909B CN 202110616581 A CN202110616581 A CN 202110616581A CN 113064909 B CN113064909 B CN 113064909B
- Authority
- CN
- China
- Prior art keywords
- data
- inconsistent
- time
- interval
- source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/273—Asynchronous replication or reconciliation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data synchronization verification method, which comprises the following steps: acquiring a source table and a target table; acquiring the historical time of successful synchronization of the previous data to obtain the time range of data updating; determining a data interval to be checked according to the time range, performing binary search on the source table and the target table, and judging whether the information of the source table and the information of the target table in the data interval are consistent; when the information is consistent, terminating the binary search; when the information is inconsistent, outputting a difference interval with inconsistent data; comparing the source data and the target data in the difference interval to obtain inconsistent data; and comparing the inconsistent data with inconsistent data in other data intervals, and correcting the data based on the comparison result.
Description
Technical Field
The invention belongs to the technical field of database data synchronization, and particularly relates to a data synchronization verification method and device.
Background
In order to ensure the normal operation of the service system, after the data synchronization/migration is completed, the data after the synchronization/migration needs to be checked. The verification after data synchronization/migration is also the check of synchronization/migration quality, and the result of data verification is also an important basis for guaranteeing the operation of the service system.
In the prior art, under the condition that data files at the bottom of a database are consistent, whether the data are consistent or not can be judged through md5, but under the condition of inconsistency (such as data of a general table and data of sub tables and sub databases), the data can only be judged through full data comparison, and then the data of inconsistent files cannot be quickly positioned.
Disclosure of Invention
The present invention is to solve the above technical problems, and provide a method and an apparatus for data synchronization verification.
In order to solve the problems, the invention is realized according to the following technical scheme:
in a first aspect, the present invention provides a data synchronization verification method, including the following steps:
acquiring a source table and a target table;
acquiring the historical time of successful synchronization of the previous data to obtain the time range of data updating;
determining a data interval to be checked according to the time range, performing binary search on the source table and the target table, and judging whether the information of the source table and the information of the target table in the data interval are consistent;
when the information is consistent, terminating the binary search; when the information is inconsistent, outputting a difference interval with inconsistent data;
comparing the source data and the target data in the difference interval based on the record ID and the updating time of the data to obtain inconsistent data;
and comparing the inconsistent data with inconsistent data in other data intervals, and correcting the inconsistent data based on the comparison result.
With reference to the first aspect, the present invention further provides a 1 st implementation manner of the first aspect, where determining a data interval to be checked according to a time range, and performing binary search on a source table and a target table, specifically including:
calculating the data quantity of the source table and the target table in the data interval;
when the data quantity of the source table and the data quantity of the target table are consistent, counting key values of the source table and the target table in a data interval, wherein the key values are the quantity, the maximum ID, the minimum ID, the ID average value and the updating time average value;
and when the key values of the source table and the target table are consistent, judging that the information of the source table and the target table in the data interval is consistent, and terminating binary search.
With reference to the first aspect, the present invention further provides a 1 st implementation manner of the first aspect, where determining a data interval to be checked according to a time range, and performing binary search on a source table and a target table, specifically including:
when the data quantity of the source table is inconsistent with that of the target table, indicating that the data in the data interval is inconsistent;
judging whether the data quantity is larger than a preset threshold value or not;
if yes, continuing binary search; if not, the inconsistent difference interval of the output data is searched based on the dichotomy.
With reference to the first aspect, the present invention further provides a 1 st implementation manner of the first aspect, where comparing, based on a record ID and an update time of data, source data and target data in a difference interval to obtain inconsistent data, specifically includes:
comparing the record ID and the update time of the source data and the target data in the difference interval based on the record ID and the update time of the data;
searching for inconsistent data in the source data, and performing increase, deletion and modification marking on the inconsistent data.
With reference to the first aspect, the present invention further provides a 1 st implementation manner of the first aspect, where the correcting the inconsistent data based on the comparison result specifically includes:
comparing the inconsistent data with inconsistent data in other data intervals to obtain an intersection of the inconsistent data;
and modifying the adding and deleting change marks of the inconsistent data in the intersection into change marks.
With reference to the first aspect, the present invention further provides an implementation manner of the first aspect in item 1, where the data synchronization checking method further includes a step of updating metadata, including:
setting a temporary synchronization success time;
acquiring the minimum update time of inconsistent data;
when the temporary synchronization success time is greater than the minimum updating time, setting the minimum updating time as the temporary synchronization success time;
when the key values of the source table and the target table are consistent and the data quantity is smaller than a preset threshold value, detecting whether the database supports deletion operation;
and when detecting that the deletion operation is not supported, setting the temporary synchronization success time as the synchronization success time, and persistently recording the difference interval and the inconsistent data into a database.
With reference to the first aspect, the present invention further provides a 1 st implementation manner of the first aspect, where the obtaining of the historical time of successful data synchronization before the previous time to obtain the time range of data update specifically includes:
acquiring the historical time of successful synchronization of previous data;
subtracting the buffering time to obtain a time range based on the time period from the historical time to the current time;
and if the historical time for successfully synchronizing the data before acquisition does not exist, giving an initial value to the historical time, and further calculating the time range.
In a second aspect, the present invention further provides a data synchronization verification apparatus, including:
an obtaining module for obtaining a source table and a target table;
the time range module is used for acquiring the historical time of successful data synchronization before the data synchronization, and obtaining the time range of data updating;
the searching module is used for determining a data interval to be checked according to the time range, performing binary searching on the source table and the target table, and judging whether the information of the source table and the information of the target table in the data interval are consistent; when the information is consistent, terminating the binary search; when the information is inconsistent, outputting a difference interval with inconsistent data;
the comparison module is used for comparing the source data and the target data in the difference interval based on the record ID and the updating time of the data to obtain inconsistent data;
and the correction module is used for comparing the inconsistent data with the inconsistent data in other data intervals and correcting the inconsistent data based on the comparison result.
With reference to the second aspect, the present invention further provides a 1 st implementation manner of the second aspect, where the searching module determines a data interval to be checked according to a time range, and performs binary search on the source table and the target table, specifically including:
calculating the data quantity of the source table and the target table in the data interval;
when the data quantity of the source table and the data quantity of the target table are consistent, counting key values of the source table and the target table in a data interval, wherein the key values are the quantity, the maximum ID, the minimum ID, the ID average value and the updating time average value;
and when the key values of the source table and the target table are consistent, judging that the information of the source table and the target table in the data interval is consistent, and terminating binary search.
With reference to the second aspect, the present invention further provides a second implementation manner of the second aspect, in which the lookup module determines a data interval to be checked according to a time range, and performs binary lookup on the source table and the target table, and the method further includes:
when the data quantity of the source table is inconsistent with that of the target table, indicating that the data in the data interval is inconsistent;
judging whether the data quantity is larger than a preset threshold value or not;
if yes, continuing binary search; if not, the inconsistent difference interval of the output data is searched based on the dichotomy.
Compared with the prior art, the invention has the beneficial effects that:
according to the data synchronization verification method and device, the data interval of the updating time range is subjected to binary search, the inconsistent difference interval is quickly located, the inconsistent data is searched based on the inconsistent difference interval, the contrast time is shortened, and efficient verification is achieved.
Drawings
Embodiments of the invention are described in further detail below with reference to the attached drawing figures, wherein:
FIG. 1 is a flow chart of a data synchronization verification method according to the present invention;
fig. 2 is a block diagram of a data synchronization verification apparatus according to the present invention.
Detailed Description
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. This application is capable of implementation in many different ways than those herein set forth and of similar import by those skilled in the art without departing from the spirit of this application and is therefore not limited to the specific implementations disclosed below.
In the application, a data synchronization verification method and device and a data migration system are provided. Details are described in the following examples one by one.
The core innovation points of the data verification method provided by the application comprise the following steps:
1. and performing binary search on the data according to the world range of the update time, and recording the data synchronization time (on the premise that the data is not deleted), which is determined after the verification, namely the starting time of performing secondary search in the next verification. The invention improves the verification efficiency through binary search, and can update the time of successful data synchronization after each verification, so long as the verification before the data belongs to the synchronous data, if no change, such as updating or adding, the updating time is not changed, and the repeated judgment is not needed in the next verification.
2. And comparing key values (number, maximum ID, minimum ID and average ID) of the current binary search interval, quickly comparing and positioning inconsistent intervals. In the prior art, the conventional means is to pull the full amount of data to perform the operation such as MD5 or other comparison, and the current method for comparing key values simplifies the amount and complexity of data acquisition.
3. By comparing the update time of the source data and the target data with the order ID, inconsistent data is quickly located. In the prior art, the conventional means is to compare all fields of the data record, but the program can ensure that if the update time is the same by ensuring that the data is updated, the update time is changed, the values of other fields of the data are the same, and vice versa.
Please refer to fig. 1, which is a flowchart illustrating an embodiment of a data synchronization verification method according to the present application. The method comprises the following steps:
s100: acquiring a source table and a target table;
s200: acquiring the historical time of successful synchronization of the previous data to obtain the time range of data updating;
s300: determining a data interval to be checked according to the time range, performing binary search on the source table and the target table, and judging whether the information of the source table and the information of the target table in the data interval are consistent;
when the information is consistent, terminating the binary search; when the information is inconsistent, outputting a difference interval with inconsistent data;
s400: comparing the source data and the target data in the difference interval based on the record ID and the updating time of the data to obtain inconsistent data;
s500: and comparing the inconsistent data with inconsistent data in other data intervals, and correcting the inconsistent data based on the comparison result.
Specifically, the following detailed analysis is performed for each step:
s100: a source table and a target table are obtained.
According to the data synchronization verification method provided by the embodiment of the application, a specific data verification task is executed through a universal data verification program. The general data verification program needs to acquire various information related to a specific data verification task in the operation process.
Information as described in embodiments of the present application includes, but is not limited to: source table and table information, target table and table information. Wherein, the source data can be obtained according to the source table information, namely: data before data synchronization/migration. The source table information includes but is not limited to: the name of a source data table storing the source data, the name of a source database to which the source data table belongs, and the name of a data identifier of the source data. The name of the data identifier of the original data refers to a field name of the unique identifier of the source data. Accordingly, target data can be acquired according to the target table information, that is: data synchronized/migrated data. The target table information includes but is not limited to: the name of a target data table storing target data, the name of a target database to which the target data table belongs, and the name of a data identifier of the target data corresponding to the name of the data identifier of the source data. The name of the data identifier of the target data is a field name of the unique identifier of the target data.
It should be noted that the field name of the unique identifier of the source data may be a name of one field (usually, a primary key ID) or may be names of a plurality of fields. In short, any field name or combination of field names that can uniquely identify source data can be used as the data identifier of the source data. Likewise, so is the data identification of the target data.
S200: and acquiring the historical time of successful data synchronization before the data is synchronized to obtain the time range of data updating.
In the present invention, the obtaining of the previous history time of successful data synchronization to obtain the time range of data update specifically includes:
s210: and acquiring the historical time of successful data synchronization before.
S220: and subtracting the buffering time to obtain a time range based on the time period from the historical time to the current time.
In one implementation, such as a history of 1 month 1 day 00 at 2021 year: 00, current time 2021 year 1 month 2 day 00: 00, it is 1 month, 1 day, 00: 00-1 month, 2 days 00: 00 for a 24 hour time frame. In one implementation, the buffering time may be 2-5 seconds.
S230: and if the historical time for successfully synchronizing the data before acquisition does not exist, giving an initial value to the historical time, and further calculating the time range.
In the present invention, if there is no history time of the last synchronization success, an initial value is given, and the initial value may be 0 or a reference time.
In steps S210 to S230, the present invention can update the history time of successful data synchronization after each verification, as long as the previous verification of the data belongs to the synchronization/migration data, and if the data has no change operation, such as updating or adding, the update time of the data does not change, so that it is not necessary to repeatedly determine the data at the next verification, and only the data in the time range corresponding to the update time is needed for verification.
S300: determining a data interval to be checked according to the time range, performing binary search on the source table and the target table, and judging whether the information of the source table and the information of the target table in the data interval are consistent:
s301: when the information is consistent, terminating the binary search;
s302: and when the information is inconsistent, outputting a difference interval with inconsistent data.
In the present invention, a data interval to be checked is determined according to a time range, and binary search is performed on a source table and a target table, specifically including:
s310: calculating the data quantity of the source table and the target table in the data interval;
s320: and when the data quantity of the source table and the data quantity of the target table are consistent, counting key values of the source table and the target table in a data interval, wherein the key values are the quantity, the maximum ID, the minimum ID, the ID average value and the updating time average value.
In one implementation of the present invention, the key values are: number, maximum ID, minimum ID, ID average, update time average. And searching and calculating corresponding key values, if the key values are consistent, indicating that the data in the interval are consistent, terminating binary search in the interval, and completing synchronous verification if the data in the interval are consistent.
And comparing key values (number, maximum ID, minimum ID and average ID) of the current binary search interval, quickly comparing and positioning inconsistent intervals. In the prior art, the conventional means is to pull the full amount of data to perform the operation such as MD5 or other comparison, and the current method for comparing key values simplifies the amount and complexity of data acquisition.
S330: and when the key values of the source table and the target table are consistent, judging that the information of the source table and the target table in the data interval is consistent, and terminating binary search.
S340: when the data quantity of the source table is inconsistent with that of the target table, indicating that the data in the data interval is inconsistent; judging whether the data quantity is larger than a preset threshold value or not; if yes, continuing binary search; if not, the inconsistent difference interval of the output data is searched based on the dichotomy.
In steps S310 to S330, the data interval corresponding to the binary search update time is obtained in step S200. The first step is comparing the number, if the number is consistent, the step S320 [ compare key value ]. If the data in the data interval is inconsistent, and whether the data quantity is larger than a certain threshold value is further judged (the efficiency is improved, individual data query is avoided), if so, binary comparison is continuously carried out, otherwise, the step S400 is carried out: find the data record that is inconsistent specifically, and terminate the interval binary search.
S400: and comparing the source data and the target data in the difference interval based on the record ID and the updating time of the data to obtain inconsistent data.
In the present invention, comparing the source data and the target data in the difference interval based on the record ID and the update time of the data to obtain inconsistent data specifically includes:
s410: comparing the record ID and the update time of the source data and the target data in the difference interval based on the record ID and the update time of the data;
s420: searching for inconsistent data in the source data, and performing increase, deletion and modification marking on the inconsistent data.
In the invention, the ID of the order and the updating time are compared, and the data of the current difference interval are compared, so that the data which are asynchronous with the source data are found out quickly through the ID and the updating time of the source data and the target data, such as (more data, missing data and inconsistent content). And correspondingly marking the data, wherein the marks belong to the types of the added and deleted data.
S500: and comparing the inconsistent data with inconsistent data in other data intervals, and correcting the inconsistent data based on the comparison result.
In the present invention, the correction of inconsistent data based on the comparison result specifically includes:
s510: comparing the inconsistent data with inconsistent data in other data intervals to obtain an intersection of the inconsistent data;
s520: and modifying the adding and deleting change marks of the inconsistent data in the intersection into change marks.
In the method, the data of the current section and other sections are corrected by comparing the unsynchronized data (excess, missing and inconsistent content) of the current binary search section and other sections. Since the binary search is performed according to the update time range of the data update, there may be data whose update is inconsistent (content is inconsistent), and in different binary, data which belongs to (extra, missing) is determined in this section, and actually belongs to data whose update is inconsistent (content is inconsistent). So the data in other intervals need to be compared again.
Preferably, the present invention further includes an S600 updating metadata step, configured to update the current data synchronization success time as a history time of the next verification, where the S600 updating metadata step includes:
s610: setting a temporary synchronization success time;
s620: acquiring the minimum update time of inconsistent data;
s630: when the temporary synchronization success time is greater than the minimum updating time, setting the minimum updating time as the temporary synchronization success time;
s640: when the key values of the source table and the target table are consistent and the data quantity is smaller than a preset threshold value, detecting whether the database supports deletion operation; and when detecting that the deletion operation is not supported, setting the temporary synchronization success time as the synchronization success time, and persistently recording the difference interval and the inconsistent data into a database.
The invention carries out binary search on the time range of the updating time, and carries out the following steps on the key values recorded in the search range: the number, the maximum ID, the minimum ID and the ID average value are compared, the inconsistent difference interval is quickly located, the inconsistent data is searched by comparing the ID with the updating time, and the verification is efficient, accurate and reliable. Furthermore, the next time data synchronization is checked, the comparison (i.e., historical data) may be started based on the earliest update time at which currently found data is consistent, and the comparison time may be further reduced.
The invention also provides a data synchronization verification device to realize the data synchronization verification method. The data synchronization checking device comprises an acquisition module, a time range module, a searching module, a comparison module, a correction module and an updating module.
The acquisition module is used for acquiring a source table and a target table;
the time range module is used for acquiring the historical time of successful synchronization of the previous data to obtain the time range of data updating;
the search module is used for determining a data interval to be checked according to the time range, performing binary search on the source table and the target table, and judging whether the information of the source table and the information of the target table in the data interval are consistent; when the information is consistent, terminating the binary search; when the information is inconsistent, outputting a difference interval with inconsistent data;
the comparison module is used for comparing the source data and the target data in the difference interval based on the record ID and the updating time of the data to obtain inconsistent data;
the correction module is used for comparing the inconsistent data with inconsistent data in other data intervals and correcting the data based on the comparison result;
the updating module is used for updating the metadata, and when the updating module updates the metadata, the following steps are executed, including:
setting a temporary synchronization success time;
acquiring the minimum update time of inconsistent data;
when the temporary synchronization success time is greater than the minimum updating time, setting the minimum updating time as the temporary synchronization success time;
when the key values of the source table and the target table are consistent and the data quantity is smaller than a preset threshold value, detecting whether the database supports deletion operation;
and when detecting that the deletion operation is not supported, setting the temporary synchronization success time as the synchronization success time, and persistently recording the difference interval and the inconsistent data into a database.
The search module determines a data interval to be checked according to a time range, and performs binary search on a source table and a target table, and specifically includes:
calculating the data quantity of the source table and the target table in the data interval;
when the data quantity of the source table and the data quantity of the target table are consistent, counting key values of the source table and the target table in a data interval, wherein the key values are the quantity, the maximum ID, the minimum ID, the ID average value and the updating time average value;
and when the key values of the source table and the target table are consistent, judging that the information of the source table and the target table in the data interval is consistent, and terminating binary search.
Specifically, the search module determines a data interval to be checked according to a time range, performs binary search on the source table and the target table, and further includes:
when the data quantity of the source table is inconsistent with that of the target table, indicating that the data in the data interval is inconsistent;
judging whether the data quantity is larger than a preset threshold value or not;
if yes, continuing binary search; if not, the inconsistent difference interval of the output data is searched based on two divisions
Specifically, when each function module implements each function described above, each step flow corresponding to the data synchronization verification method is executed.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the present invention in any way, so that any modification, equivalent change and modification made to the above embodiment according to the technical spirit of the present invention are within the scope of the technical solution of the present invention.
Claims (4)
1. A data synchronization verification method is characterized by comprising the following steps:
acquiring a source table and a target table;
acquiring the historical time of successful synchronization of the previous data to obtain the time range of data updating;
determining a data interval to be checked according to the time range, performing binary search on the source table and the target table, judging whether the information of the source table and the target table in the data interval is consistent, and terminating the binary search when the information is consistent; when the information is inconsistent, outputting a difference interval with inconsistent data, specifically comprising: calculating the data quantity of the source table and the target table in the data interval;
when the data quantity of the source table and the data quantity of the target table are consistent, counting key values of the source table and the target table in a data interval, wherein the key values are used for quickly locating a difference interval with inconsistent data, and the key values are the quantity, the maximum ID, the minimum ID, an ID average value and an update time average value;
when the key values of the source table and the target table are consistent, judging that the information of the source table and the information of the target table in the data interval are consistent, and terminating binary search;
when the data quantity of the source table is inconsistent with that of the target table, indicating that the data in the data interval is inconsistent;
judging whether the data quantity is greater than a preset threshold value, wherein the preset threshold value is used for improving the efficiency so as to avoid inquiring the full data one by one;
if yes, continuing binary search; if not, searching a difference interval with inconsistent output data based on dichotomy;
comparing the source data and the target data in the difference interval based on the record ID and the updating time of the data to obtain inconsistent data, which specifically comprises the following steps:
comparing the record ID and the update time of the source data and the target data in the difference interval based on the record ID and the update time of the data;
searching for inconsistent data in the obtained source data, and carrying out addition, deletion and modification marking on the inconsistent data;
the inconsistent data of inconsistent data and other data intervals is compared, and the inconsistent data is corrected based on the comparison result, which specifically comprises the following steps:
comparing the inconsistent data with inconsistent data in other data intervals to obtain an intersection of the inconsistent data;
and modifying the adding and deleting change marks of the inconsistent data in the intersection into change marks.
2. The data synchronization verification method of claim 1, further comprising the step of updating metadata, comprising:
setting a temporary synchronization success time;
acquiring the minimum update time of inconsistent data;
when the temporary synchronization success time is greater than the minimum updating time, setting the minimum updating time as the temporary synchronization success time;
when the key values of the source table and the target table are consistent and the data quantity is smaller than a preset threshold value, detecting whether the database supports deletion operation;
and when detecting that the deletion operation is not supported, setting the temporary synchronization success time as the synchronization success time, and persistently recording the difference interval and the inconsistent data into a database.
3. The data synchronization verification method according to claim 1, wherein obtaining a history time of previous data synchronization success to obtain a time range of data update specifically comprises:
acquiring the historical time of successful synchronization of previous data;
subtracting the buffering time to obtain a time range based on the time period from the historical time to the current time;
and if the historical time for successfully synchronizing the data before acquisition does not exist, giving an initial value to the historical time, and further calculating the time range.
4. A data synchronization verification apparatus, comprising:
an obtaining module for obtaining a source table and a target table;
the time range module is used for acquiring the historical time of successful data synchronization before the data synchronization, and obtaining the time range of data updating;
the searching module is used for determining a data interval to be checked according to the time range, performing binary searching on the source table and the target table, and judging whether the information of the source table and the information of the target table in the data interval are consistent; when the information is consistent, terminating the binary search; when the information is inconsistent, outputting a difference interval with inconsistent data;
the method includes the steps of determining a data interval to be checked according to a time range, and performing binary search on a source table and a target table, and specifically includes the following steps:
calculating the data quantity of the source table and the target table in the data interval;
when the data quantity of the source table and the data quantity of the target table are consistent, counting key values of the source table and the target table in a data interval, wherein the key values are used for quickly locating a difference interval with inconsistent data, and the key values are the quantity, the maximum ID, the minimum ID, an ID average value and an update time average value;
when the key values of the source table and the target table are consistent, judging that the information of the source table and the information of the target table in the data interval are consistent, and terminating binary search;
when the data quantity of the source table is inconsistent with that of the target table, indicating that the data in the data interval is inconsistent;
judging whether the data quantity is greater than a preset threshold value, wherein the preset threshold value is used for improving the efficiency so as to avoid inquiring the full data one by one;
if yes, continuing binary search; if not, searching a difference interval with inconsistent output data based on dichotomy;
the comparison module is used for comparing the source data and the target data in the difference interval based on the record ID and the updating time of the data to obtain inconsistent data, and specifically comprises the following steps:
comparing the record ID and the update time of the source data and the target data in the difference interval based on the record ID and the update time of the data;
searching for inconsistent data in the obtained source data, and carrying out addition, deletion and modification marking on the inconsistent data; the correction module is used for comparing the inconsistent data with the inconsistent data in other data intervals, and correcting the inconsistent data based on the comparison result, and specifically comprises the following steps:
comparing the inconsistent data with inconsistent data in other data intervals to obtain an intersection of the inconsistent data;
and modifying the adding and deleting change marks of the inconsistent data in the intersection into change marks.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110616581.1A CN113064909B (en) | 2021-06-03 | 2021-06-03 | Data synchronization verification method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110616581.1A CN113064909B (en) | 2021-06-03 | 2021-06-03 | Data synchronization verification method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113064909A CN113064909A (en) | 2021-07-02 |
| CN113064909B true CN113064909B (en) | 2021-10-22 |
Family
ID=76568554
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110616581.1A Active CN113064909B (en) | 2021-06-03 | 2021-06-03 | Data synchronization verification method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113064909B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114385631A (en) * | 2021-12-17 | 2022-04-22 | 北京达佳互联信息技术有限公司 | Verification method, verification device, server and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104021132A (en) * | 2013-12-08 | 2014-09-03 | 郑州正信科技发展股份有限公司 | Method and system for verification of consistency of backup data of host database and backup database |
| CN105989044A (en) * | 2015-02-04 | 2016-10-05 | 阿里巴巴集团控股有限公司 | Database verification method and system |
| CN106326398A (en) * | 2016-08-19 | 2017-01-11 | 东软集团股份有限公司 | Data consistency comparison method and device |
| CN106411530A (en) * | 2016-10-25 | 2017-02-15 | 国家电网公司 | Weak password rapid comparison and searching method |
| US9977807B1 (en) * | 2017-02-13 | 2018-05-22 | Sas Institute Inc. | Distributed data set indexing |
| CN108153784A (en) * | 2016-12-06 | 2018-06-12 | 北京京东尚科信息技术有限公司 | Synchrodata treating method and apparatus |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104036029B (en) * | 2014-06-27 | 2016-04-13 | 深圳市腾讯计算机***有限公司 | Large data consistency control methods and system |
| CN106033427A (en) * | 2015-03-11 | 2016-10-19 | 阿里巴巴集团控股有限公司 | A sampling data verification method and device |
| CN108763304B (en) * | 2018-04-20 | 2020-12-29 | 国家电网有限公司 | Blood-cause relationship-based power standing book data verification method and device |
| CN111159482A (en) * | 2019-12-19 | 2020-05-15 | 贝壳技术有限公司 | Data verification method and system |
| CN112631839A (en) * | 2020-12-31 | 2021-04-09 | 中国农业银行股份有限公司 | Method, device and equipment for realizing data backup |
-
2021
- 2021-06-03 CN CN202110616581.1A patent/CN113064909B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104021132A (en) * | 2013-12-08 | 2014-09-03 | 郑州正信科技发展股份有限公司 | Method and system for verification of consistency of backup data of host database and backup database |
| CN105989044A (en) * | 2015-02-04 | 2016-10-05 | 阿里巴巴集团控股有限公司 | Database verification method and system |
| CN106326398A (en) * | 2016-08-19 | 2017-01-11 | 东软集团股份有限公司 | Data consistency comparison method and device |
| CN106411530A (en) * | 2016-10-25 | 2017-02-15 | 国家电网公司 | Weak password rapid comparison and searching method |
| CN108153784A (en) * | 2016-12-06 | 2018-06-12 | 北京京东尚科信息技术有限公司 | Synchrodata treating method and apparatus |
| US9977807B1 (en) * | 2017-02-13 | 2018-05-22 | Sas Institute Inc. | Distributed data set indexing |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113064909A (en) | 2021-07-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8249801B2 (en) | Navigation device and map data updating method thereof | |
| CN105138635B (en) | A method of it is compared using cryptographic Hash and carries out data increment duplication | |
| US20080209310A1 (en) | Annotation validity using partial checksums | |
| CN113064909B (en) | Data synchronization verification method and device | |
| CN111143368A (en) | Relational database data comparison method and system | |
| CN111008241A (en) | Data synchronization method and device, storage medium and computer equipment | |
| CN115577694B (en) | Intelligent recommendation method for standard writing | |
| CN111241293A (en) | Knowledge graph algorithm constructed based on academic literature | |
| CN114840497B (en) | Line migration preprocessing method, system and device for database and storage medium | |
| CN110502524B (en) | Phoenix index data asynchronous updating method and device | |
| JP3260587B2 (en) | Time series data management method | |
| CN117272953B (en) | Automatic document information filling method, system and storage medium | |
| CN115544048B (en) | Method and terminal for monitoring data change | |
| CN109446022B (en) | Method and device for detecting abnormal overflow page of database and storage medium | |
| CN116719824A (en) | Data management method and related device | |
| CN117971279A (en) | Rollback method and device of system version and server | |
| CN117454858A (en) | Document management information adjustment and update method, system and storage medium | |
| CN116842104A (en) | Data synchronization method, device and medium based on java microservice | |
| CN113220670A (en) | Method and device for correcting address data | |
| JPH05108437A (en) | Program file alteration history management method | |
| CN117725059A (en) | Database data comparison method, medium and device | |
| CN117493464A (en) | Database data synchronization method, device, electronic equipment and readable storage medium | |
| CN115540847A (en) | Map updating and displaying method, device, medium and map data center station | |
| CN116166720A (en) | Data sorting method and device and electronic equipment | |
| CN115291932A (en) | Method for acquiring similarity threshold, data processing method and product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |