CN112131312A - Data synchronization system, method, device, equipment and storage medium - Google Patents
Data synchronization system, method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN112131312A CN112131312A CN202010973201.5A CN202010973201A CN112131312A CN 112131312 A CN112131312 A CN 112131312A CN 202010973201 A CN202010973201 A CN 202010973201A CN 112131312 A CN112131312 A CN 112131312A
- Authority
- CN
- China
- Prior art keywords
- data source
- data
- target
- field
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data synchronization system, a method, a device, equipment and a storage medium, wherein the data synchronization system comprises a configuration subsystem and a working node subsystem; the configuration subsystem is used for configuring initial data source information, target data source information and associated information of at least two data synchronization tasks; the working node subsystem is used for loading the starting data source information, the target data source information and the target associated information of the target data synchronization task, establishing a first communication link between the first data source appointed in the target associated information and a second communication link between the first communication link and a second data source appointed in the target associated information according to the starting data source information and the target data source information, and synchronizing the target data from the first data source to the second data source through the first communication link and the second communication link. The embodiment of the invention can realize synchronous transmission of data of different data sources and has high transmission speed.
Description
Technical Field
The embodiment of the invention relates to the technical field of computer application, in particular to a data synchronization system, a data synchronization method, a data synchronization device, data synchronization equipment and a storage medium.
Background
Nowadays, internet technology develops rapidly, and data in the network shows explosive growth, and the data processing and statistical analysis of diversified structure bring not little trouble, and heterogeneous data synchronous analysis receives attention gradually in order to handle the data of diversified structure.
In the prior art, when data of a heterogeneous data source is synchronized, multiple modules such as a scheduling module, a monitoring module, an alarm module, a management node and the like are generally adopted, the management node is adopted to determine an optimal working node according to node information of a working node, data source configuration information and the like, synchronous tasks of the same data source are distributed to the same working node, synchronous tasks executed by the working node are monitored according to synchronous information fed back from the working node, and a task alarm event is generated when the synchronous tasks are monitored to be abnormal. The heterogeneous data synchronization system is high in complexity, the failure rate is increased, and the operation and maintenance cost of the whole system is increased. And the scene of data synchronization is single, and only the initial data source and the target data source are the same type of memory, the data synchronization can be realized mutually, and the data synchronization of the cross-data type can not be realized. Therefore, there is an urgent need for a system that can be configured simply and can achieve data synchronization across data types.
Disclosure of Invention
The invention provides a data synchronization system, a data synchronization method, a data synchronization device, data synchronization equipment and a storage medium, which are used for realizing data synchronization transmission of different data sources (including heterogeneous data sources).
In a first aspect, an embodiment of the present invention provides a data synchronization system, where the system includes:
a configuration subsystem and a working node subsystem;
the configuration subsystem is used for configuring initial data source information, target data source information and associated information of at least two data synchronization tasks;
the working node subsystem is used for loading the starting data source information, the target data source information and the target associated information of the target data synchronization task, establishing a first communication link between the first data source appointed in the target associated information and a second communication link between the first communication link and a second data source appointed in the target associated information according to the starting data source information and the target data source information, and synchronizing the target data from the first data source to the second data source through the first communication link and the second communication link.
In a second aspect, an embodiment of the present invention further provides a data synchronization method, where the method includes:
when a data synchronization instruction is received, loading target associated information of a target data synchronization task matched with the data synchronization instruction, and configuring initial data source information and target data source information configured in a subsystem;
establishing a first communication link with a first data source specified in the target association information according to the starting data source information and the target data source information, and configuring a second communication link between a second data source specified in the target association information and configured in a subsystem;
synchronizing target data from the first data source to the second data source over the first communication link and the second communication link.
In a third aspect, an embodiment of the present invention provides a data synchronization apparatus, where the apparatus includes:
the loading module is used for loading target associated information of a target data synchronization task matched with the data synchronization instruction and configuring initial data source information and target data source information configured in the subsystem when the data synchronization instruction is received;
the establishing module is used for establishing a first communication link with a first data source appointed in the target association information and a second communication link with a second data source appointed in the target association information according to the starting data source information and the target data source information;
a synchronization module to synchronize target data from the first data source to the second data source over the first communication link and the second communication link.
In a fourth aspect, an embodiment of the present invention further provides a data synchronization apparatus, where the apparatus includes:
one or more processors;
a storage device for storing one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors implement the data synchronization method according to any of the embodiments of the present invention.
In a fifth aspect, the present invention further provides a storage medium containing computer-executable instructions, which when executed by a computer processor, are configured to perform the data synchronization method according to any one of the embodiments of the present invention.
According to the technical scheme of the embodiment of the invention, the incidence relation among the initial data source, the target data source and at least two data synchronization tasks is configured, so that when a data synchronization task exists, the working node subsystem can realize synchronous data transmission between any appointed initial data source and any appointed target data source by a certain appointed working node. The data synchronization system of the embodiment of the invention can realize the synchronous transmission of the data of different data sources, and the data synchronization system only comprises the initial data source, the target data source and the working node subsystem to realize the synchronous transmission of the data among the different data sources, and has the advantages of high flexibility of synchronous transmission among the data, simple system structure and high transmission speed.
Drawings
In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, a brief description is given below of the drawings used in describing the embodiments. It should be clear that the described figures are only views of some of the embodiments of the invention to be described, not all, and that for a person skilled in the art, other figures can be derived from these figures without inventive effort.
Fig. 1 is a schematic structural diagram of a data synchronization system according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a data synchronization system according to a second embodiment of the present invention;
fig. 3 is a schematic flowchart of a data synchronization method according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of a data synchronization apparatus according to a fourth embodiment of the present invention;
fig. 5 is a schematic structural diagram of a data synchronization device according to a fifth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 1 is a schematic structural diagram of a data synchronization system according to an embodiment of the present invention, where the embodiment is applicable to data synchronization in heterogeneous data sources, and the method may be executed by the data synchronization system, and the system may be implemented in a form of software and/or hardware.
As shown in fig. 1, the system of the present embodiment includes: a configuration subsystem 110 and a working node subsystem 120;
the configuration subsystem 110 is configured to configure initial data source information, target data source information, and associated information of at least two data synchronization tasks. The configuration of the initial data source information, the target data source information and the associated information of at least two data synchronization tasks in the configuration subsystem is configured in real time when the data synchronization system receives the data synchronization tasks, when the data synchronization tasks are received, the data synchronization system performs real-time configuration according to the associated relation of an initial data source, a target data source, a working node and the data synchronization tasks contained in one data synchronization task, and connects the initial data source, the working node and the target data source required in the data synchronization tasks so as to facilitate subsequent data synchronization transmission. The incidence relation comprises a certain specified initial data source, a certain target data source, a data conversion relation between the initial data source and the target data source and the like. It should be noted that the data synchronization system performs static configuration work on each piece of initial data source information and each piece of target data source information in advance, including but not limited to storing each piece of initial data source information and each piece of target data source information in a respective source file, and when the data synchronization system receives a data synchronization task, acquires the initial data source information and the target data source information according to a certain specified initial data source and a certain target data source to perform real-time configuration and establish communication connection.
Optionally, the initial data source information and the target data source information of the data synchronization system may be preset by the system before synchronous transmission of data. The multiple databases in the initial data source and the multiple databases in the target data source are connected in a mesh manner through the working node subsystem, for example, the initial data source is HBase, MongoDB, MySQL, and the target data source can be the three databases. When the data synchronization system receives a data synchronization task, the working node subsystem loads an incidence relation in the data synchronization task, acquires a certain specified initial data source, a certain target data source and a certain working node from the incidence relation, then establishes links between the initial data source and the working node and between the working node and the target data source, and performs a data synchronization transmission task.
The working node subsystem 120 is configured to load the start data source information, the target data source information, and the target association information of the target data synchronization task, to establish a first communication link with a first data source specified in the target association information and a second communication link with a second data source specified in the target association information according to the start data source information and the target data source information, and to synchronize target data from the first data source to the second data source through the first communication link and the second communication link.
The target associated information of the target data synchronization task refers to a certain specified initial data source, a certain target data source, a data conversion relation between the initial data source and the target data source, and the like contained in the target data synchronization task when the data synchronization system receives the target data synchronization task. The first communication link is a link established between the first data source and a designated working node in the working node subsystem, and is used for data transmission between the first data source and the working node. Similarly, the second communication link is a link established between the second data source and a designated working node in the working node subsystem, and is used for data transmission between the second data source and the working node. And an additional management node and the like are not needed, communication connection is established between the first data source and the second data source only through the working node, and the system is simple in structure.
Specifically, the target data synchronization task designates a certain initial data source as a database of a data source of the target data synchronization task, designates a certain target data source as a database where data of the target data synchronization task is finally placed, the designated certain initial data source is the first data source, the first data source is linked with the working node, the designated certain target data source is the second data source, and the second data source is linked with the working node. The target data refers to data which needs to be synchronously transmitted in the data synchronization task, and comprises pictures, audio files, video files, text data, log files or stream data and the like. For example, the target data is the identity information of a person, including text data of name, age, sex, and identification number. When the data synchronization system receives a data synchronization task, a designated working node in the working node subsystem loads pre-configured initial data source information, target data source information and associated information of the data synchronization task, so as to establish a first communication link with a first data source designated in the associated information according to the initial data source information and the target data source information, and call the target data from the first data source. And establishing a second communication link between the working node and a second data source specified in the target association information, and synchronizing the target data called from the first data source to the second data source through the second communication link.
Illustratively, the starting data source information includes: the storage type of the initial data source, the IP address of the initial data source, and the connection account and the password; the target data source information includes: the storage type of the target data source, the IP address of the target data source, and the connection account and the password.
The storage type of the initial data source refers to the type of a database originally placed in data synchronization transmission, and the storage type of the target data source refers to the type of a database finally stored after data synchronization processing. The storage type of the starting data source or the target data source comprises a relational database, a non-relational database and the like. Relational databases include MySQL, Server, SQL, Oracle, etc. The non-relational database includes: HDFS, MongoDB, HBase, etc. The IP address of the originating/destination data source refers to the address of the TCP/IP network protocol port. The connection account and password of the starting/target data source are specific to each starting/target data source and are used for connecting with the working node. It should be understood that, in the present application, the storage types of the starting data source and the target data source may be the same or different.
The establishing a first communication link with a first data source specified in the target association information and a second communication link with a second data source specified in the target association information according to the starting data source information and the target data source information comprises: determining the first data source and the second data source specified in the target association information; determining the IP address, the connection account and the password of the first data source according to the initial data source information, and determining the IP address, the connection account and the password of the second data source according to the target data source information; establishing a first communication link with the first data source according to the IP address of the first data source, the connection account and the password; and establishing a second communication link with the second data source according to the IP address, the connection account and the password of the second data source.
Illustratively, respective clients of different starting/target data sources are established in the working node subsystem, one client corresponding to each database of the starting data source and one client corresponding to each database of the target data source. When a data synchronization task is received, acquiring an association relation to determine a first data source and a second data source, and loading an IP address, a connection account and a password in the first data source through a client to enable the first data source and a working node to establish a first communication link. Similarly, the IP address, the connection account and the password in the second data source are loaded through the client, so that the second communication link is established between the second data source and the working node.
For example, the connection between each initial data source and the target data source through the working node may be preconfigured, and the form of the client of each initial/target data source is established in the working node subsystem, so that the IP address, the connection account, and the password of the initial/target data source may be loaded at the client to perform communication connection on the initial/target data source. During synchronous data transmission, the pre-established client can directly call the associated information of the starting data source and the target data source to establish the first communication link and the second communication link.
According to the technical scheme of the embodiment of the invention, the incidence relation among the initial data source, the target data source and at least two data synchronization tasks is configured, so that when a data synchronization task exists, the working node subsystem can realize synchronous data transmission between any appointed initial data source and any appointed target data source by a certain appointed working node. The data synchronization system of the embodiment of the invention can realize synchronous transmission of data of different data sources, and the data synchronization system only comprises the initial data source, the target data source and the working node subsystem to realize synchronous transmission of data among different data sources, thereby having the advantages of flexible synchronization among data, simple system structure and high transmission speed.
Example two
The data synchronization system provided in the embodiment of the present invention is optimized based on the above embodiments, wherein technical terms the same as or similar to those in the above embodiments will not be described again, and the system of the embodiment includes: a configuration subsystem and a working node subsystem;
the configuration subsystem is used for configuring initial data source information, target data source information and associated information of at least two data synchronization tasks;
the working node subsystem is used for loading the initial data source information, the target data source information and the target associated information of the target data synchronization task so as to establish a first communication link with a first data source appointed in the target associated information and a second communication link with a second data source appointed in the target associated information according to the initial data source information and the target data source information.
Wherein the associated information of each data synchronization task comprises: the data synchronization method comprises a first data source corresponding to a current data synchronization task, a data structure type in the first data source, a field list to be synchronized, a second data source, a data structure type in the second data source, and a mapping relation between a field to be synchronized and a field to be written.
The first data structure type refers to a data structure type in the first data source, and includes but is not limited to an array, a queue, a linked list, a hash table, and the like. Similarly, the second data structure type refers to a type of a data structure stored in the second data source after the target data of the synchronous transmission is converted, and it should be understood that the data structure type of the first data source and the data structure type of the second data source are set according to types of databases of the respective data sources. The field list to be synchronized refers to a field list in a specified first data source, for example, the first data source includes a data table, the data table includes a field list, and the field list to be synchronized may refer to a field list in the data table in the first data source. The mapping relation between the fields to be synchronized and the fields to be written refers to the mapping relation between the fields to be synchronized in the first data source and the fields to be written in the second data source.
Exemplarily, a target field list to be synchronized corresponding to the target data synchronization task is determined according to the target association information; acquiring data of a target field to be synchronized in the target field to be synchronized list from a first data source corresponding to the target data synchronization task; determining a target field to be written corresponding to the target field to be synchronized according to the mapping relation between the field to be synchronized and the field to be written; and writing the data matched with the target field to be written into the target field to be written.
Specifically, when the data synchronization system receives a target data synchronization task, a first data source, a second data source and a working node are determined according to initial data source information, target data source information and target association information of the target data synchronization task in the target data synchronization task, and a communication link between the first data source and the working node and a communication link between the second data source and the working node are established. Determining data of a target to-be-synchronized field in a to-be-synchronized field list in the first data source, wherein the data is target data, and the structure type of the target data is the data structure type in the first data source. And writing the data in the target field to be written corresponding to the target field to be synchronized into the target field to be written in the second data source according to the mapping relation between the field to be synchronized in the first data source and the field to be written in the second data source.
Exemplarily, a target field to be written corresponding to the target field to be synchronized is determined according to a mapping relation between the field to be synchronized and the field to be written; writing the data matched with the target field to be written into the target field to be written comprises: and converting the data of the target field to be synchronized into the data matched with the target field to be written according to the conversion logic between the field to be synchronized and the field to be written.
The mapping relationship between the fields to be synchronized and the fields to be written comprises the following steps: the first field in the first data source corresponds to the second field in the target data source, and the conversion logic is arranged between the first field and the second field. The conversion logic includes: conversion of data types, merging of two fields into one field, mathematical operation of the fields, and the like. Specifically, according to the conversion logic between the field to be synchronized and the field to be written, the data of the target field to be synchronized is converted into the data matched with the target field to be written, so as to form a one-to-one correspondence relationship, for example, the target data includes personal identity information including name, identification card number and age, the identity information of the target field to be synchronized in the data table of the first data source is converted into the identity information matched with the target field to be written, and the identity information can be stored in the second data source.
As shown in fig. 2, a schematic structural diagram of a system for synchronous data transmission according to an embodiment of the present invention is provided, where three data sources are respectively a source data source and a target data source, which are MySQL, HBase, and mongoDB. Each database has respective IP address, connection account and password to facilitate connection with the working node subsystem. In the data synchronization system shown in fig. 2, the information of each source/destination data source, including the IP address, the connection account number, and the password, is pre-configured, including but not limited to static configuration in the respective source file.
When a data synchronization task is received, determining a first data source, a second data source, a field mapping relation between the first data source and the second data source and the like according to the incidence relation in the data synchronization task, and establishing a communication link between the first data source and a working node and a communication link between the working node and the second data source. Determining a field list and target data in a first data source, reading data from the first data source through a first communication link, obtaining data of a field to be written according to a mapping relation between the field to be synchronized and the field to be written, and transmitting the data to a second data source through a second communication link to complete a data synchronous transmission task. Optionally, when some data is stored, in order to ensure the security of the data, the data is encrypted, so that, in the working node subsystem, before the working node performs data mapping, the working node performs parsing, including decryption processing and the like, on the read data, and then performs data mapping processing. The system can realize synchronous data transmission among different data sources, can synchronize from any one data source to another data source, and realizes synchronous transmission among loosely-coupled and high-efficiency heterogeneous data.
The technical scheme provided by the embodiment of the invention determines the first data source and the second data source through the incidence relation in the data processing task, establishes the first data source and the working node as well as the communication link between the working node and the target data source, and converts and transmits the data in the field to be synchronized in the first data source to the data in the field to be written in the second data source through the mapping relation between the field to be synchronized and the field to be written so as to complete the synchronous transmission of the data, so that one data synchronization task can realize the data synchronization between different data sources through only one working node.
EXAMPLE III
Fig. 3 is a schematic flow chart of a data synchronization method provided in an embodiment of the present invention, where the method is applied to a work node subsystem in a data synchronization system, and the method includes the following steps:
s310, when a data synchronization instruction is received, loading target associated information of a target data synchronization task matched with the data synchronization instruction, and configuring initial data source information and target data source information configured in the subsystem.
The target associated information includes a certain specified initial data source, a certain target data source, and a data conversion relationship between the initial data source and the target data source. When a data synchronization task instruction is received, connecting a starting data source, a working node and a target data source which are required in a data synchronization task so as to facilitate subsequent data synchronization transmission. It should be noted that the data synchronization system performs static configuration work on the source data information and the destination data source information in advance, including but not limited to storing the source data information and the destination data source information in respective source files. When the data synchronization system receives a data synchronization task, acquiring initial data source information and target data source information according to a certain specified initial data source and a certain target data source, and configuring in real time to establish communication connection.
S320, establishing a first communication link with a first data source specified in the target association information and a second communication link with a second data source specified in the target association information according to the starting data source information and the target data source information.
The initial data source information refers to information of a certain initial data source specified in the data synchronization task, and similarly, the target data source information refers to information of a certain target data source specified in the data synchronization task. The first data source refers to a starting data source designated in the data synchronization task, and the second data source refers to a target data source designated in the data synchronization task. And establishing a first communication link between a first data source and one working node in the working node subsystem through the acquired initial data source information, and establishing a second communication link between a second data source and the working node according to the acquired target data source information. It should be understood that the communication connection between the origin/target data source and the worker node is a real-time configuration connection between the first data source and the worker node and the second data source.
S330, synchronizing target data from the first data source to the second data source through the first communication link and the second communication link.
The target data refers to data that needs to be synchronized from the first data source to the second data source, and the format of the target data can be determined according to actual situations. After the first communication link and the second communication link are constructed, the first data source and the second data source are connected through the working node, and the target data of the first data source can be transmitted to the working node through the first communication link and synchronously transmitted from the working node to the second data source through the second communication link.
Illustratively, the starting data source information includes: the storage type of the initial data source, the IP address of the initial data source, and the connection account and the password; the target data source information includes: the storage type of the target data source, the IP address of the target data source, and the connection account and the password.
The storage type of the starting/target data source comprises a relational database, a non-relational database and the like. The IP address of the originating/destination data source refers to the address of the TCP/IP network protocol port. The connection account and password of the starting/target data source are specific to each starting/target data source and are used for connecting with the working node.
The establishing a first communication link with a first data source specified in the target association information and a second communication link with a second data source specified in the target association information according to the starting data source information and the target data source information comprises: determining the first data source and the second data source specified in the target association information; determining the IP address, the connection account and the password of the first data source according to the initial data source information, and determining the IP address, the connection account and the password of the second data source according to the target data source information; establishing a first communication link with the first data source according to the IP address of the first data source, the connection account and the password; and establishing a second communication link with the second data source according to the IP address, the connection account and the password of the second data source.
The method includes the steps of determining a first data source and a second data source according to target associated information in a data synchronization task, establishing a first communication link between the first data source and a working node according to an IP address, a connection account and a password of the first data source, and establishing a second communication link between the second data source and the working node according to an IP address, a connection account and a password of the second data source to achieve connection between the first data source and the second data source through the working node And connecting the account and the password.
Illustratively, the association information of each data synchronization task includes: the data synchronization method comprises a first data source corresponding to a current data synchronization task, a data structure type in the first data source, a field list to be synchronized, a second data source, a data structure type in the second data source, and a mapping relation between a field to be synchronized and a field to be written.
The first data structure type refers to a data structure type in the first data source, and includes but is not limited to an array, a queue, a linked list, a hash table, and the like. Similarly, the second data structure type refers to a type of a data structure stored in the second data source after the target data of the synchronous transmission is converted, and it should be understood that the data structure type of the first data source and the data structure type of the second data source are set according to types of databases of the respective data sources. The list of fields to be synchronized refers to a list of fields in the specified first data source. The mapping relation between the fields to be synchronized and the fields to be written refers to the mapping relation between the fields to be synchronized in the first data source and the fields to be written in the second data source.
The synchronizing target data from the first data source to the second data source over the first communication link and the second communication link comprises: determining a target field list to be synchronized corresponding to the target data synchronization task according to the target associated information; acquiring data of a target field to be synchronized in the target field to be synchronized list from a first data source corresponding to the target data synchronization task; and writing the data matched with the target field to be written into the target field to be written according to the mapping relation between the field to be synchronized and the field to be written.
The target incidence relation refers to the incidence relation in the target data task. When the data synchronization system receives a target data synchronization task, determining a first data source, a second data source and a working node according to initial data source information, target data source information and target association information of the target data synchronization task in the target data synchronization task, and establishing a communication link between the first data source and the working node and a communication link between the second data source and the working node. Determining data of a target to-be-synchronized field in a to-be-synchronized field list in the first data source, wherein the data is target data, and the structure type of the target data is the data structure type in the first data source. And writing the data in the target field to be written corresponding to the target field to be synchronized into the target field to be written in the second data source according to the mapping relation between the field to be synchronized in the first data source and the field to be written in the second data source.
Illustratively, the writing the data matched with the target field to be written into according to the mapping relationship between the field to be synchronized and the field to be written into includes: determining a target field to be written corresponding to the target field to be synchronized according to the mapping relation between the field to be synchronized and the field to be written; and converting the data of the target field to be synchronized into the data matched with the target field to be written according to the conversion logic between the field to be synchronized and the field to be written.
Wherein, the mapping relationship between the fields to be synchronized and the fields to be written comprises: the conversion logic between a first field in a data table of the first data source and a second field in a data table of the target data source; the conversion logic includes: conversion of data types, merging of two fields into one field, mathematical operation of the fields, and the like. Specifically, according to the conversion logic between the field to be synchronized and the field to be written, the data of the target field to be synchronized is converted into the data matched with the target field to be written, so as to form a one-to-one correspondence relationship, for example, the target data includes personal identity information including name, identification card number and age, the identity information of the target field to be synchronized in the data table of the first data source is converted into the identity information matched with the target field to be written, and the identity information can be stored in the second data source.
According to the technical scheme of the embodiment of the invention, the incidence relation among the initial data source, the target data source and at least two data synchronization tasks is configured, so that when a data synchronization task exists, the working node subsystem can realize synchronous data transmission between any appointed initial data source and any appointed target data source by a certain appointed working node. The data synchronization system of the embodiment of the invention can realize synchronous transmission of data of different data sources, and the data synchronization system only comprises the initial data source, the target data source and the working node subsystem to realize synchronous transmission of data among different data sources, thereby having the advantages of flexible synchronization among data, simple system structure and high transmission speed.
Example four
Fig. 4 is a schematic structural diagram of a data synchronization apparatus provided in an embodiment of the present invention, which is capable of executing a data synchronization method provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method. The data synchronization device of the embodiment of the invention is integrated in a working node subsystem of a data synchronization system, and comprises:
a loading module 410, configured to, when a data synchronization instruction is received, load target association information of a target data synchronization task that matches the data synchronization instruction, and configure initial data source information and target data source information configured in a subsystem;
an establishing module 420, configured to establish a first communication link with a first data source specified in the target association information and a second communication link with a second data source specified in the target association information according to the starting data source information and the target data source information;
a synchronization module 430 to synchronize target data from the first data source to the second data source over the first communication link and the second communication link.
Further, the establishing module 420 includes:
a data source determination submodule configured to determine the first data source and the second data source specified in the target association information;
the information determining submodule is used for determining the IP address, the connection account and the password of the first data source according to the initial data source information, and determining the IP address, the connection account and the password of the second data source according to the target data source information;
the first communication link establishing submodule is used for establishing a first communication link with the first data source according to the IP address of the first data source, the connection account and the password;
and the second communication link establishing submodule is used for establishing a second communication link with the second data source according to the IP address of the second data source, the connection account and the password.
Further, the synchronization module 430 includes:
a field list determining submodule, configured to determine, according to the target association information, a target to-be-synchronized field list corresponding to the target data synchronization task;
the data acquisition submodule is used for acquiring data of a target field to be synchronized in the target field to be synchronized list from a first data source corresponding to the target data synchronization task;
and the data writing module is used for writing the data matched with the target field to be written into the target field to be written according to the mapping relation between the field to be synchronized and the field to be written.
Further, the mapping relationship between the field to be synchronized and the field to be written includes: the conversion logic between a first field in a data table of the first data source and a second field in a data table of the target data source; the data writing submodule includes:
the data writing unit is used for determining a target field to be written corresponding to the target field to be synchronized according to the mapping relation between the field to be synchronized and the field to be written; and the data conversion module is used for converting the data of the target field to be synchronized into the data matched with the target field to be written according to the conversion logic between the field to be synchronized and the field to be written.
Further, the mapping relationship between the field to be synchronized and the field to be written includes: the conversion logic between a first field in a data table of the first data source and a second field in a data table of the target data source; the data writing unit includes:
the data writing subunit is used for determining a target field to be written corresponding to the target field to be synchronized according to the mapping relation between the field to be synchronized and the field to be written; and converting the data of the target field to be synchronized into the data matched with the target field to be written according to the conversion logic between the field to be synchronized and the field to be written.
According to the technical scheme of the embodiment of the invention, the incidence relation among the initial data source, the target data source and at least two data synchronization tasks is configured, so that when a data synchronization task exists, the working node subsystem can realize synchronous data transmission between any appointed initial data source and any appointed target data source by a certain appointed working node. The data synchronization system of the embodiment of the invention can realize synchronous transmission of data of different data sources, and the data synchronization system only comprises the initial data source, the target data source and the working node subsystem to realize synchronous transmission of data among different data sources, thereby having the advantages of flexible synchronization among data, simple system structure and high transmission speed.
It should be noted that, the units and modules included in the system are merely divided according to functional logic, but are not limited to the above division as long as the corresponding functions can be realized; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the embodiment of the invention.
EXAMPLE five
Fig. 5 is a schematic structural diagram of a data synchronization device according to an embodiment of the present invention. FIG. 5 illustrates a block diagram of an exemplary device 50 suitable for use in implementing embodiments of the present invention. The device 50 shown in fig. 5 is only an example and should not bring any limitation to the function and scope of use of the embodiments of the present invention.
As shown in FIG. 5, device 50 is embodied in a general purpose computing device. The components of the device 50 may include, but are not limited to: one or more processors or processing units 501, a system memory 502, and a bus 503 that couples the various system components (including the system memory 502 and the processing unit 501).
The system memory 502 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)504 and/or cache memory 505. The device 50 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 506 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 5, commonly referred to as a "hard drive"). Although not shown in FIG. 5, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 503 by one or more data media interfaces. Memory 502 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
A program/utility 508 having a set (at least one) of program modules 507 may be stored, for instance, in memory 502, such program modules 507 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 507 generally perform the functions and/or methodologies of embodiments of the invention as described herein.
The processing unit 501 executes various functional applications and data processing, for example, implementing a data synchronization method provided by an embodiment of the present invention, by executing a program stored in the system memory 502.
EXAMPLE six
Embodiments of the present invention also provide a storage medium containing computer-executable instructions for performing a method of data synchronization when executed by a computer processor.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for embodiments of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (15)
1. A data synchronization system, comprising:
a configuration subsystem and a working node subsystem;
the configuration subsystem is used for configuring initial data source information, target data source information and associated information of at least two data synchronization tasks;
the working node subsystem is used for loading the starting data source information, the target data source information and the target associated information of the target data synchronization task, establishing a first communication link between the first data source appointed in the target associated information and a second communication link between the first communication link and a second data source appointed in the target associated information according to the starting data source information and the target data source information, and synchronizing the target data from the first data source to the second data source through the first communication link and the second communication link.
2. The system of claim 1, wherein the starting data source information comprises: the storage type of the initial data source, the IP address of the initial data source, and the connection account and the password;
the target data source information includes: the storage type of the target data source, the IP address of the target data source, and the connection account and the password.
3. The system of claim 2, wherein establishing a first communication link with a first data source specified in the target association information and a second communication link with a second data source specified in the target association information according to the starting data source information, the target data source information comprises:
determining the first data source and the second data source specified in the target association information;
determining the IP address, the connection account and the password of the first data source according to the initial data source information, and determining the IP address, the connection account and the password of the second data source according to the target data source information;
establishing a first communication link with the first data source according to the IP address of the first data source, the connection account and the password;
and establishing a second communication link with the second data source according to the IP address, the connection account and the password of the second data source.
4. The system of claim 1, wherein the association information of each of the data synchronization tasks comprises:
the data synchronization method comprises a first data source corresponding to a current data synchronization task, a data structure type in the first data source, a field list to be synchronized, a second data source, a data structure type in the second data source, and a mapping relation between a field to be synchronized and a field to be written.
5. The system of claim 4, wherein the synchronizing target data from the first data source to the second data source via the first communication link and the second communication link comprises:
determining a target field list to be synchronized corresponding to the target data synchronization task according to the target associated information;
acquiring data of a target field to be synchronized in the target field to be synchronized list from a first data source corresponding to the target data synchronization task;
and writing the data matched with the target field to be written into the target field to be written according to the mapping relation between the field to be synchronized and the field to be written.
6. The system according to claim 5, wherein the mapping relationship between the field to be synchronized and the field to be written comprises:
the conversion logic between the first field and the second field of the target data source;
the writing the data matched with the target field to be written into according to the mapping relation between the field to be synchronized and the field to be written into comprises:
determining a target field to be written corresponding to the target field to be synchronized according to the mapping relation between the field to be synchronized and the field to be written;
and converting the data of the target field to be synchronized into the data matched with the target field to be written according to the conversion logic between the field to be synchronized and the field to be written.
7. A data synchronization method is applied to a working node subsystem of a data synchronization system, and is characterized by comprising the following steps:
when a data synchronization instruction is received, loading target associated information of a target data synchronization task matched with the data synchronization instruction, and configuring initial data source information and target data source information configured in a subsystem;
establishing a first communication link with a first data source specified in the target association information and a second communication link with a second data source specified in the target association information according to the starting data source information and the target data source information;
synchronizing target data from the first data source to the second data source over the first communication link and the second communication link.
8. The method of claim 7, wherein the starting data source information comprises: the storage type of the initial data source, the IP address of the initial data source, and the connection account and the password;
the target data source information includes: the storage type of the target data source, the IP address of the target data source, and the connection account and the password.
9. The method of claim 8, wherein establishing a first communication link with a first data source specified in the target association information and a second communication link with a second data source specified in the target association information according to the starting data source information, the target data source information comprises:
determining the first data source and the second data source specified in the target association information;
determining the IP address, the connection account and the password of the first data source according to the initial data source information, and determining the IP address, the connection account and the password of the second data source according to the target data source information;
establishing a first communication link with the first data source according to the IP address of the first data source, the connection account and the password;
and establishing a second communication link with the second data source according to the IP address, the connection account and the password of the second data source.
10. The method of claim 7, wherein the association information of each of the data synchronization tasks comprises:
the data synchronization method comprises a first data source corresponding to a current data synchronization task, a data structure type in the first data source, a field list to be synchronized, a second data source, a data structure type in the second data source, and a mapping relation between a field to be synchronized and a field to be written.
11. The method of claim 10, wherein the synchronizing target data from the first data source to the second data source over the first communication link and the second communication link comprises:
determining a target field list to be synchronized corresponding to the target data synchronization task according to the target associated information;
acquiring data of a target field to be synchronized in the target field to be synchronized list from a first data source corresponding to the target data synchronization task;
and writing the data matched with the target field to be written into the target field to be written according to the mapping relation between the field to be synchronized and the field to be written.
12. The method according to claim 11, wherein the mapping relationship between the field to be synchronized and the field to be written comprises:
the conversion logic between a first field in a data table of the first data source and a second field in a data table of the target data source;
the writing the data matched with the target field to be written into according to the mapping relation between the field to be synchronized and the field to be written into comprises:
determining a target field to be written corresponding to the target field to be synchronized according to the mapping relation between the field to be synchronized and the field to be written;
and converting the data of the target field to be synchronized into the data matched with the target field to be written according to the conversion logic between the field to be synchronized and the field to be written.
13. A data synchronization apparatus integrated with a work node subsystem of a data synchronization system, comprising:
the loading module is used for loading target associated information of a target data synchronization task matched with the data synchronization instruction and configuring initial data source information and target data source information configured in the subsystem when the data synchronization instruction is received;
the establishing module is used for establishing a first communication link with a first data source appointed in the target association information and a second communication link with a second data source appointed in the target association information according to the starting data source information and the target data source information;
a synchronization module to synchronize target data from the first data source to the second data source over the first communication link and the second communication link.
14. A data synchronization apparatus, characterized in that the apparatus comprises:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement a data synchronization method as recited in any of claims 7-12.
15. A storage medium containing computer-executable instructions, characterized in that the program, when executed by a processor, implements the data synchronization method according to any one of claims 7-12.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010973201.5A CN112131312A (en) | 2020-09-16 | 2020-09-16 | Data synchronization system, method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010973201.5A CN112131312A (en) | 2020-09-16 | 2020-09-16 | Data synchronization system, method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112131312A true CN112131312A (en) | 2020-12-25 |
Family
ID=73845858
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010973201.5A Pending CN112131312A (en) | 2020-09-16 | 2020-09-16 | Data synchronization system, method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112131312A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113076208A (en) * | 2021-06-04 | 2021-07-06 | 上海燧原科技有限公司 | Multi-event synchronization circuit, method and computing chip |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150142740A1 (en) * | 2013-11-18 | 2015-05-21 | International Business Machines Corporation | Automatically managing mapping and transform rules when synchronizing systems |
| CN105005618A (en) * | 2015-07-21 | 2015-10-28 | 杭州合众数据技术有限公司 | Data synchronization method and system among heterogeneous databases |
| CN108133007A (en) * | 2017-12-22 | 2018-06-08 | 北京明朝万达科技股份有限公司 | A kind of method of data synchronization and system |
-
2020
- 2020-09-16 CN CN202010973201.5A patent/CN112131312A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150142740A1 (en) * | 2013-11-18 | 2015-05-21 | International Business Machines Corporation | Automatically managing mapping and transform rules when synchronizing systems |
| CN105005618A (en) * | 2015-07-21 | 2015-10-28 | 杭州合众数据技术有限公司 | Data synchronization method and system among heterogeneous databases |
| CN108133007A (en) * | 2017-12-22 | 2018-06-08 | 北京明朝万达科技股份有限公司 | A kind of method of data synchronization and system |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113076208A (en) * | 2021-06-04 | 2021-07-06 | 上海燧原科技有限公司 | Multi-event synchronization circuit, method and computing chip |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109274731B (en) | Method and device for deploying and calling web service based on multi-tenant technology | |
| WO2017041398A1 (en) | Data transmission method and device | |
| CN112667414A (en) | Message queue-based message consumption method and device, computer equipment and medium | |
| US10135763B2 (en) | System and method for secure and efficient communication within an organization | |
| CN112765152B (en) | Method and apparatus for merging data tables | |
| CN103677844A (en) | Lightweight class data processing frame based on message chain | |
| CN111338834B (en) | Data storage method and device | |
| US10331484B2 (en) | Distributed data platform resource allocator | |
| CN112995261A (en) | Configuration method and device of service table, network equipment and storage medium | |
| CN114124916A (en) | Data transmission method and device, electronic equipment and storage medium | |
| CN112445860B (en) | Method and device for processing distributed transaction | |
| CN112181307A (en) | Block chain based distributed data redundancy storage method and electronic equipment | |
| CN112131312A (en) | Data synchronization system, method, device, equipment and storage medium | |
| CN113051086B (en) | Data processing method and device, electronic equipment and storage medium | |
| CN109151033B (en) | Communication method and device based on distributed system, electronic equipment and storage medium | |
| CN107861821B (en) | Mining method and device for module call relation and computer readable medium | |
| CN113656502A (en) | Data synchronization method, system, electronic device and storage medium | |
| CN111679892A (en) | Distributed transaction processing method, device, equipment and medium | |
| CN110719233B (en) | Method and device for transmitting information | |
| CN109617789B (en) | Method, system, electronic device and storage medium for processing session message | |
| CN112511636B (en) | Data transmission system, method, device, computer equipment and storage medium | |
| CN112860447B (en) | Interaction method and system between different applications | |
| CN113760842A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN113849449A (en) | Communication system and information interaction method, device and medium | |
| CN114265966A (en) | Data processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |