CN110765212A - Data synchronization system and method based on database as platform - Google Patents
Data synchronization system and method based on database as platform Download PDFInfo
- Publication number
- CN110765212A CN110765212A CN201911079511.6A CN201911079511A CN110765212A CN 110765212 A CN110765212 A CN 110765212A CN 201911079511 A CN201911079511 A CN 201911079511A CN 110765212 A CN110765212 A CN 110765212A
- Authority
- CN
- China
- Prior art keywords
- data
- synchronous
- file
- receiving
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 19
- 230000001360 synchronised effect Effects 0.000 claims abstract description 104
- 238000004519 manufacturing process Methods 0.000 claims abstract description 41
- 238000012545 processing Methods 0.000 claims description 11
- 238000004458 analytical method Methods 0.000 claims description 4
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 238000012795 verification Methods 0.000 description 7
- 230000005540 biological transmission Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000004075 alteration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data synchronization system and a data synchronization method based on a database as a platform, wherein the data synchronization system comprises a sub-server, a synchronous file production system, a synchronous file receiving system and a target server; the sub-server comprises a source database, and the source database is provided with a plurality of groups; the synchronous file production system is used for acquiring synchronous data in the sub-server and generating a Structured Query Language (SQL) data file according to the synchronous data; the invention sets a sub-server, a synchronous file production system, a synchronous file receiving system and a target server, and utilizes the synchronous file production system and the synchronous file receiving system to transmit the data generated or changed in the sub-server to the target server in time so as to update the content in the target server at any time.
Description
Technical Field
The invention relates to the technical field of data transmission, in particular to a data synchronization system and a data synchronization method based on a database as a platform.
Background
With the advent of the cloud era, big data has attracted more and more attention, and the big data needs special technology to effectively process a large amount of data within the tolerance of the elapsed time. Technologies applicable to big data include: the system comprises a large-scale parallel processing database, a data mining power grid, a distributed file system, a distributed database, a cloud computing platform, the Internet, an extensible storage system and the like.
With the rapid development of science and technology, more than one database platform exists in the existing data synchronization system, data in each database platform is continuously updated and increased, if data synchronization is not performed in time, information errors are generated, and when the data synchronization is performed in the existing data synchronization system, data transmission errors are easily generated, so that information on the database platforms is inconsistent.
Disclosure of Invention
The invention aims to provide a data synchronization system and a data synchronization method based on a database as a platform.
In order to achieve the purpose, the invention provides the following technical scheme: the data synchronization system and method based on the database as the platform, including sub-server, synchronous file production system, synchronous file receiving system and target server;
the sub-server comprises a source database, and the source database is provided with a plurality of groups;
the synchronous file production system is used for acquiring synchronous data in the sub-server and generating a Structured Query Language (SQL) data file according to the synchronous data;
the synchronous file receiving system is used for receiving synchronous data generated by the synchronous file production system and transmitting the synchronous data to the target server, and comprises a receiving unit, an judging unit and a processing and sending unit;
and the target server is used for receiving the processed synchronous data in the synchronous file receiving system and performing storage and updating.
Preferably, the synchronous file receiving system is further configured to perform consistency judgment on the received synchronous data.
Preferably, the receiving unit of the synchronous file receiving system is configured to receive synchronous data transmitted by the synchronous file production system.
Preferably, the judging unit of the synchronous file receiving system is configured to judge whether the SQL data file is consistent with the SQL data file received by the receiving unit.
Preferably, the processing and sending unit is configured to send the SQL data file to the target server.
Preferably, a warning module is arranged in the judging unit.
The method of the data synchronization system based on the database as the platform comprises the following steps:
s1, when new data is generated or the data changes in one of the multiple source databases in the sub-server, the synchronous file production system synchronously copies the new data;
s2, the synchronous file production system analyzes the synchronously copied data pair, and filters, maps and converts the synchronously copied data after the analysis is finished to generate an executable SQL data file;
s3, sending the SQL data file generated by conversion to a synchronous file receiving system;
s4, the receiving unit in the synchronous file receiving system receives the SQL data file of the synchronous file production system;
s5, after the synchronous file receiving system completes the file receiving, the judging unit judges the SQL data file received by the receiving unit and judges whether the SQL data file is consistent with the SQL data file received by the receiving unit;
s6, when the results are consistent, sending the group of SQL data files to the target server through the processing and sending unit;
s7, when the results are not consistent, the warning module of the judge unit generates warning information and transmits the warning information back to the synchronous file production system, and the steps S1, S2, S3, S4 and S5 are continuously executed until the results are consistent.
The invention provides a data synchronization system and a data synchronization method based on a database as a platform, which have the following beneficial effects:
(1) the invention utilizes the synchronous file production system and the synchronous file receiving system to transmit the data generated or changed in the sub-server to the target server in time so as to update the content in the target server at any time by arranging the sub-server, the synchronous file production system, the synchronous file receiving system and the target server.
(2) The method of the data synchronization system of the invention checks and compares the received synchronization data, and after the comparison and verification are passed, if the verification is unqualified, the transmission step is repeatedly executed until the comparison and verification result is passed, thereby ensuring the real-time update of the data and the accuracy of the updated data.
Drawings
FIG. 1 is a block diagram of the system architecture of the present invention.
Detailed Description
The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.
Example 1:
as shown in fig. 1, the present invention provides a technical solution: the data synchronization system and method based on the database as the platform, including sub-server, synchronous file production system, synchronous file receiving system and target server;
the sub-server comprises a source database, and the source database is provided with a plurality of groups;
the synchronous file production system is used for acquiring synchronous data in the sub-server and generating a Structured Query Language (SQL) data file according to the synchronous data;
the synchronous file receiving system is used for receiving synchronous data generated by the synchronous file production system and transmitting the synchronous data to the target server, and comprises a receiving unit, an judging unit and a processing and sending unit;
and the target server is used for receiving the processed synchronous data in the synchronous file receiving system and performing storage and updating.
Preferably, the synchronous file receiving system is further configured to perform consistency judgment on the received synchronous data.
Preferably, the receiving unit of the synchronous file receiving system is configured to receive synchronous data transmitted by the synchronous file production system.
Preferably, the judging unit of the synchronous file receiving system is configured to judge whether the SQL data file is consistent with the SQL data file received by the receiving unit.
Preferably, the processing and sending unit is configured to send the SQL data file to the target server.
Preferably, a warning module is arranged in the judging unit.
The invention uses the synchronous file production system and the synchronous file receiving system to transmit the data generated or changed in the sub-server to the target server in time by arranging the sub-server, the synchronous file production system, the synchronous file receiving system and the target server so as to update the content in the target server at any time
Example 2:
s1, when the synchronous file production system detects that a certain source database in the sub-server stores new data, synchronously copying the new data;
s2, after the new data synchronously copied enter the synchronous file production system, the synchronous file production system analyzes the synchronously copied data, and after the analysis is finished, the synchronously copied synchronous data is filtered, mapped and converted to finally generate an executable SQL data file;
s3, sending the SQL data file generated by conversion to a synchronous file receiving system;
s4, the receiving unit in the synchronous file receiving system receives the SQL data file of the synchronous file production system;
s5, after the synchronous file receiving system completes the file receiving, the judging unit immediately judges the SQL data file received by the receiving unit and judges whether the verification information of the received SQL data file is consistent with that of the SQL data file in the synchronous file production system;
s6, when the results are consistent, sending the group of SQL data files to the target server through the processing and sending unit;
s7, when the results are not consistent, the warning module of the judging unit generates warning information and transmits the warning information back to the synchronous file production system;
returning to step S1, steps S2, S3, S4 and S5 are continued until the verification result is consistent.
Example 3:
s1, when the synchronous file production system detects that data updating occurs in a source database in the sub-server, the synchronous file production system synchronously copies the new data;
s2, after the new data synchronously copied enter the synchronous file production system, the synchronous file production system analyzes the synchronously copied data, and after the analysis is finished, the synchronously copied synchronous data is filtered, mapped and converted to finally generate an executable SQL data file;
s3, sending the SQL data file generated by conversion to a synchronous file receiving system;
s4, the receiving unit in the synchronous file receiving system receives the SQL data file of the synchronous file production system;
s5, after the synchronous file receiving system completes the file receiving, the judging unit immediately judges the SQL data file received by the receiving unit and judges whether the verification information of the received SQL data file is consistent with that of the SQL data file in the synchronous file production system;
s6, when the results are consistent, sending the group of SQL data files to the target server through the processing and sending unit;
s7, when the results are not consistent, the warning module of the judging unit generates warning information and transmits the warning information back to the synchronous file production system;
returning to step S1, steps S2, S3, S4 and S5 are continued until the verification result is consistent.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (7)
1. The data synchronization system and method based on the database as the platform are characterized by comprising a sub-server, a synchronous file production system, a synchronous file receiving system and a target server;
the sub-server comprises a source database, and the source database is provided with a plurality of groups;
the synchronous file production system is used for acquiring synchronous data in the sub-server and generating a Structured Query Language (SQL) data file according to the synchronous data;
the synchronous file receiving system is used for receiving synchronous data generated by the synchronous file production system and transmitting the synchronous data to the target server, and comprises a receiving unit, an judging unit and a processing and sending unit;
and the target server is used for receiving the processed synchronous data in the synchronous file receiving system and performing storage and updating.
2. The database-based platform data synchronization system and method according to claim 1, wherein: the synchronous file receiving system is also used for judging the consistency of the received synchronous data.
3. The database-based platform data synchronization system and method according to claim 1, wherein: and the receiving unit of the synchronous file receiving system is used for receiving the synchronous data transmitted by the synchronous file production system.
4. The database-based platform data synchronization system and method according to claim 1, wherein: and the judging unit of the synchronous file receiving system is used for judging whether the SQL data file is consistent with the SQL data file received by the receiving unit.
5. The database-based platform data synchronization system and method according to claim 1, wherein: and the processing and sending unit is used for sending the SQL data file of the synchronous data to the target server.
6. The database-based platform data synchronization system and method according to claim 1, wherein: and a warning module is arranged in the judging unit.
7. The method of database-based platform data synchronization system according to any of claims 1-6, wherein: the method comprises the following steps:
s1, when new data is generated or the data changes in one of the multiple source databases in the sub-server, the synchronous file production system synchronously copies the new data;
s2, the synchronous file production system analyzes the synchronously copied data pair, and filters, maps and converts the synchronously copied data after the analysis is finished to generate an executable SQL data file;
s3, sending the SQL data file generated by conversion to a synchronous file receiving system;
s4, the receiving unit in the synchronous file receiving system receives the SQL data file of the synchronous file production system;
s5, after the synchronous file receiving system completes the file receiving, the judging unit judges the SQL data file received by the receiving unit and judges whether the SQL data file is consistent with the SQL data file received by the receiving unit;
s6, when the results are consistent, sending the group of SQL data files to the target server through the processing and sending unit;
s7, when the results are not consistent, the warning module of the judge unit generates warning information and transmits the warning information back to the synchronous file production system, and the steps S1, S2, S3, S4 and S5 are continuously executed until the results are consistent.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911079511.6A CN110765212A (en) | 2019-11-07 | 2019-11-07 | Data synchronization system and method based on database as platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911079511.6A CN110765212A (en) | 2019-11-07 | 2019-11-07 | Data synchronization system and method based on database as platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110765212A true CN110765212A (en) | 2020-02-07 |
Family
ID=69336616
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911079511.6A Pending CN110765212A (en) | 2019-11-07 | 2019-11-07 | Data synchronization system and method based on database as platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110765212A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112732838A (en) * | 2021-01-21 | 2021-04-30 | 上海海事大学 | Database data synchronous updating system and method based on software engineering |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106227822A (en) * | 2016-07-22 | 2016-12-14 | 深圳市中润四方信息技术有限公司 | A kind of data distributing method and data synchronous system |
| CN109101627A (en) * | 2018-08-14 | 2018-12-28 | 交通银行股份有限公司 | heterogeneous database synchronization method and device |
-
2019
- 2019-11-07 CN CN201911079511.6A patent/CN110765212A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106227822A (en) * | 2016-07-22 | 2016-12-14 | 深圳市中润四方信息技术有限公司 | A kind of data distributing method and data synchronous system |
| CN109101627A (en) * | 2018-08-14 | 2018-12-28 | 交通银行股份有限公司 | heterogeneous database synchronization method and device |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112732838A (en) * | 2021-01-21 | 2021-04-30 | 上海海事大学 | Database data synchronous updating system and method based on software engineering |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110908997B (en) | Data blood relationship construction method and device, server and readable storage medium | |
| JP2019501448A (en) | Automated anomaly detection service in heterogeneous log streams | |
| CN103699638A (en) | Method for realizing cross-database type synchronous data based on configuration parameters | |
| CN103064933A (en) | Data query method and system | |
| CN111709527A (en) | Operation and maintenance knowledge map library establishing method, device, equipment and storage medium | |
| US11003635B2 (en) | Database scheme for storing generic data | |
| CN106547781A (en) | A kind of method for realizing distributed transaction, device and database server | |
| CN111737355A (en) | MongoDB metadata management-based heterogeneous data source synchronization method and system | |
| CN114691704A (en) | Metadata synchronization method based on MySQL binlog | |
| CN111625540A (en) | Method and device for verifying data synchronization integrity of relational database | |
| CN110765212A (en) | Data synchronization system and method based on database as platform | |
| CN111538573A (en) | Asynchronous task processing method and device and computer readable storage medium | |
| CN110895547A (en) | Multi-source heterogeneous database data synchronization system and method based on DB2 federal characteristics | |
| CN114817177A (en) | Industrial internet data storage method, system, storage medium and electronic equipment | |
| EP2904520A1 (en) | Reference data segmentation from single to multiple tables | |
| CN107562943B (en) | Data calculation method and system | |
| CN111444536A (en) | Database design method and system suitable for mimicry environment | |
| US9762645B2 (en) | Modifying data collection systems responsive to changes to data providing systems | |
| CN115658815A (en) | CDC (control data center) -based data synchronization method | |
| CN109829016B (en) | Data synchronization method and device | |
| CN116016554A (en) | Configurable cloud edge real-time database transmission system and method | |
| CN115470355A (en) | Rail transit information query method and device, electronic equipment and storage medium | |
| CN113127036A (en) | Software development system, method, apparatus and medium for continuous integration of code | |
| CN102609419B (en) | Similar data de-duplication method | |
| EP4325357A3 (en) | Clustering processes using traffic data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200207 |
|
| RJ01 | Rejection of invention patent application after publication |