CN116881368A - Data synchronization method and device, electronic equipment and storage medium - Google Patents
Data synchronization method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN116881368A CN116881368A CN202310987386.9A CN202310987386A CN116881368A CN 116881368 A CN116881368 A CN 116881368A CN 202310987386 A CN202310987386 A CN 202310987386A CN 116881368 A CN116881368 A CN 116881368A
- Authority
- CN
- China
- Prior art keywords
- rule
- data
- data synchronization
- synchronized
- extraction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 64
- 230000001360 synchronised effect Effects 0.000 claims abstract description 114
- 238000000605 extraction Methods 0.000 claims abstract description 100
- 238000013507 mapping Methods 0.000 claims description 44
- 238000004590 computer program Methods 0.000 claims description 16
- 238000012545 processing Methods 0.000 abstract description 14
- 230000008569 process Effects 0.000 description 13
- 238000004891 communication Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 238000007726 management method Methods 0.000 description 7
- 230000003287 optical effect Effects 0.000 description 3
- 238000013075 data extraction Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- FLDSMVTWEZKONL-AWEZNQCLSA-N 5,5-dimethyl-N-[(3S)-5-methyl-4-oxo-2,3-dihydro-1,5-benzoxazepin-3-yl]-1,4,7,8-tetrahydrooxepino[4,5-c]pyrazole-3-carboxamide Chemical compound CC1(CC2=C(NN=C2C(=O)N[C@@H]2C(N(C3=C(OC2)C=CC=C3)C)=O)CCO1)C FLDSMVTWEZKONL-AWEZNQCLSA-N 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data synchronization method, a data synchronization device, electronic equipment and a storage medium. The data synchronization method comprises the following steps: acquiring a data synchronization rule applied in a merging rule of a source database; acquiring extraction rules of cluster nodes corresponding to the source database; determining data to be synchronized at the cluster node according to the data synchronization rule and the extraction rule; and determining a target end database according to the data synchronization rule, and synchronizing the corresponding data to be synchronized to the target end database. According to the embodiment of the invention, the data synchronization rule is stored as the merging rule, so that the data synchronization rule is convenient to check and manage, the data to be synchronized is obtained by simultaneously analyzing and processing each storage node, the data to be synchronized is synchronized to the target database according to the data synchronization rule, the resource consumption of data synchronization is reduced, and the use experience of a user is improved.
Description
Technical Field
The present invention relates to the field of data synchronization technologies, and in particular, to a data synchronization method, a data synchronization device, an electronic device, and a storage medium.
Background
In the process of synchronization of the database, the synchronization needs to be managed, namely, a data synchronization rule is established, a target position of the data synchronization is determined, and some parameters are set during the synchronization. When synchronizing from the source database to the target database, a plurality of different data synchronization rules can be established according to different requirements.
If the data synchronization rules are not combined for the same source database, the same database content needs to be analyzed for multiple times, so that resources are wasted to a certain extent. The database cluster node can comprise computing nodes and storage nodes, data are distributed on a plurality of storage nodes according to a certain strategy, the data structures of the storage nodes are identical, the data content is relatively independent, the storage nodes are required to be analyzed and processed independently in the data synchronization process, the resource waste can be caused, and when the storage nodes are changed, the rule is required to be re-established, so that on the premise of normal data synchronization, the problem that the resource consumption is required to be solved is solved urgently.
Disclosure of Invention
The invention provides a data synchronization method, a data synchronization device, electronic equipment and a storage medium, which are used for realizing unified management of data synchronization rules in the same database and reducing resource consumption of data synchronization.
According to an aspect of the present invention, there is provided a data synchronization method, wherein the method includes:
acquiring a data synchronization rule applied in a merging rule of a source database;
acquiring extraction rules of cluster nodes corresponding to the source database;
determining data to be synchronized at the cluster nodes according to the data synchronization rule and the extraction rule;
and determining a target end database according to the data synchronization rule, and synchronizing corresponding data to be synchronized to the target end database.
According to another aspect of the present invention, there is provided a data synchronization apparatus, wherein the apparatus includes:
the synchronization rule acquisition module is used for acquiring the data synchronization rule applied in the merging rule of the source database;
the extraction rule acquisition module is used for acquiring the extraction rule of the cluster node corresponding to the source end database;
the synchronous data determining module is used for determining data to be synchronized at the cluster nodes according to the data synchronizing rule and the extraction rule;
and the synchronous data synchronization module is used for determining a target end database according to the data synchronization rule and synchronizing the corresponding data to be synchronized to the target end database.
According to another aspect of the present invention, there is provided an electronic apparatus including:
at least one processor;
and a memory communicatively coupled to the at least one processor;
wherein the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the data synchronization method of any one of the embodiments of the present invention.
According to another aspect of the present invention, there is provided a computer readable storage medium storing computer instructions for causing a processor to perform the data synchronization method of any one of the embodiments of the present invention.
According to the technical scheme, the extraction rule of the cluster node corresponding to the source end database is obtained by obtaining the data synchronization rule applied in the merging rule of the source end database, the data to be synchronized is determined at the cluster node according to the data synchronization rule and the extraction rule, the target end database is determined according to the data synchronization rule, and the corresponding data to be synchronized is synchronized to the target end database, so that the data synchronization rule merging management of the same database is realized, all storage nodes are processed uniformly, the data to be synchronized is obtained by simultaneously analyzing and processing all the storage nodes, the resource consumption of data synchronization is reduced, and the use experience of a user is improved.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the invention or to delineate the scope of the invention. Other features of the present invention will become apparent from the description that follows.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a data synchronization method according to a first embodiment of the present invention;
FIG. 2 is a flow chart of a data synchronization method according to a second embodiment of the present invention;
FIG. 3 is a flow chart of a data synchronization method according to a third embodiment of the present invention;
fig. 4 is a mapping relation structure diagram provided in accordance with a third embodiment of the present invention;
FIG. 5 is a flow chart of a rule management method according to a third embodiment of the present invention;
FIG. 6 is a flow chart of a data synchronization method according to a third embodiment of the present invention;
fig. 7 is a schematic structural diagram of a data synchronization device according to a fourth embodiment of the present invention;
fig. 8 is a schematic structural diagram of an electronic device implementing a data synchronization method according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example 1
Fig. 1 is a flowchart of a data synchronization method according to a first embodiment of the present invention, where the method may be performed by a data synchronization device, which may be implemented in hardware and/or software, and the data synchronization device may be configured in an electronic device, where the data synchronization method is applicable to a case of synchronizing data of a source database to a target database. As shown in fig. 1, the method includes:
s110, acquiring a data synchronization rule applied in the merging rule of the source database.
The source database may refer to a database that needs to be synchronized. The merging rule may refer to a merging rule generated by merging all data synchronization rules in the source database. In one embodiment, the merging rule may include at least one data synchronization rule. The data synchronization rule may be a rule established according to data synchronization requirements for data synchronization. In the actual operation process, the data synchronization rule may be created according to the user requirement, and the data synchronization rule may include the to-be-synchronized data identifier and the target end database identifier, so as to perform data synchronization according to the to-be-synchronized data identifier and the target end database identifier.
In the embodiment of the invention, the merging rule of the source database can be acquired, and the data synchronization rule applied in the merging rule can be determined. In the actual operation process, a corresponding relation can exist between the merging rule and the data synchronization rule, and the data synchronization rule in the merging rule can be obtained according to the corresponding relation. For example, the correspondence may include a mapping relationship, and the data synchronization rule may be identified as being included in a certain merge rule. In one embodiment, each data synchronization rule may be identified by a universally unique identification code (Universally Unique Identifier, UUID).
After the data synchronization rule in the merge rule is acquired, the data synchronization rule applied in the merge rule can be determined. In the actual operation process, the rule state of each data synchronization rule can be obtained, the rule state of each data synchronization rule is determined to be starting or stopping, and the data synchronization rule with the rule state being starting is taken as the applied data synchronization rule.
S120, acquiring extraction rules of cluster nodes corresponding to the source database.
The cluster nodes may refer to storage nodes (Date Node, DN) in the cluster that are responsible for storing data, and an extraction rule may be established for each cluster Node, so as to facilitate extraction of data stored by the cluster nodes. The extraction rule may be a rule for extracting data stored in the cluster nodes, and the extraction rule may be a rule established according to data extraction requirements, where each cluster node has a corresponding extraction rule.
In the embodiment of the invention, the cluster nodes of the cluster where the source database is located can be determined, and the extraction rule corresponding to each cluster node is obtained. In an embodiment, there may be situations where cluster nodes increase or decrease in the cluster. When the extraction rule of each cluster node is obtained, whether each cluster node has a corresponding extraction rule or not and whether each extraction rule has a corresponding cluster node or not can be determined. When any cluster node does not have a corresponding extraction rule, the extraction rule can be established for the cluster node; when any extraction rule does not have a corresponding cluster node, the extraction rule can be deleted to update the extraction rule of the cluster node, so that each cluster node has a corresponding extraction rule.
S130, determining data to be synchronized at the cluster nodes according to the data synchronization rule and the extraction rule.
In the embodiment of the invention, after determining the applied data synchronization rule and acquiring the extraction rule of the cluster node, the data can be extracted according to the applied data synchronization rule and the extraction rule of the cluster node, so as to determine the data to be synchronized. In the actual operation process, the extraction rule can be called according to the applied data synchronization rule, and the data corresponding to each data synchronization rule is extracted from the cluster nodes to serve as the data to be synchronized.
In an embodiment, since the number of the data synchronization rules may be multiple, the data to be synchronized extracted in each data synchronization rule may be repeated, and the data extracted according to each data synchronization rule may be used as initial synchronization data, and the same initial synchronization data may be combined according to the identifier of the initial synchronization data, to generate the data to be synchronized. In an embodiment, the identification of the initial synchronization data may include, but is not limited to, a name of the initial synchronization data, field information, and the like.
And S140, determining a target end database according to the data synchronization rule, and synchronizing the corresponding data to be synchronized to the target end database.
The target database may refer to a database that may refer to destination synchronization.
In the embodiment of the invention, corresponding data to be synchronized can be matched according to each data synchronization rule, a target end database is determined, and the matched data to be synchronized is synchronized to the corresponding target end database. In the actual operation process, the data to be synchronized and the target end database identification in the data synchronization rule can be extracted, the data to be synchronized and the target end database corresponding to the data synchronization rule are determined, and the data to be synchronized are synchronized according to each data synchronization rule.
In an embodiment, data to be synchronized may be extracted and analyzed in a Write Ahead Log (WAL), the data to be synchronized corresponding to the data synchronization rule is matched according to each data synchronization rule, and the data to be synchronized is synchronized according to a target database corresponding to the data synchronization rule.
According to the embodiment of the invention, the extraction rule of the cluster node corresponding to the source database is obtained by obtaining the data synchronization rule applied in the merging rule of the source database, the data to be synchronized is determined at the cluster node according to the data synchronization rule and the extraction rule, the target database is determined according to the data synchronization rule, and the corresponding data to be synchronized is synchronized to the target database, so that the data synchronization rule merging management of the data synchronization rules belonging to the same database is realized, and all the data synchronization rules can be performed simultaneously. And uniformly processing each storage node, and simultaneously analyzing and processing each storage node to acquire data to be synchronized, so that the resource consumption of data synchronization is reduced, and the use experience of a user is improved.
In an embodiment, before acquiring the data synchronization rule applied in the merging rule of the source database, the method further includes:
creating at least two data synchronization rules according to the data synchronization requirement; the data synchronization rule includes: a data identifier to be synchronized and a target end database identifier;
and merging and storing at least two data synchronization rules as merging rules, and adding the mapping relation between the merging rules and the data synchronization rules in the mapping relation set.
The data to be synchronized identification may be information for identifying the data to be synchronized. In an embodiment, the identifier of the data to be synchronized may include location information of the data to be synchronized, and the data to be synchronized may be extracted according to the identifier of the data to be synchronized. The target database identifier may be information for identifying a target synchronization position of the data to be synchronized, and the target synchronization position of the data to be synchronized may be determined according to the target database identifier. The mapping relation set may be a set storing mapping relations between the merging rules and the data synchronization rules, and the mapping relation between the merging rules and the data synchronization rules may be determined in the mapping relation set.
In the embodiment of the invention, the data synchronization rule can be created according to the data synchronization requirement, and the number of the data synchronization rules is not limited. In the actual operation process, the data identification to be synchronized and the target end database identification can be determined according to the data synchronization requirement, and the data synchronization rule is created according to the data identification to be synchronized and the target end database identification. After the data synchronization rule is created, at least two data synchronization rules can be merged and stored as a merging rule, the mapping relation between each data synchronization rule and the merging rule is determined, and the mapping relation is stored in a mapping relation set.
In an embodiment, each data synchronization rule may be identified by a UUID, and the merging rules may store all data synchronization rules belonging to the source database, and may query whether a corresponding merging rule exists in the mapping relation set according to the UUID, if so, establish a mapping relation between the data synchronization rule and the merging rule, and store the mapping relation in the mapping relation set; if not, establishing a merging rule.
Example two
Fig. 2 is a flowchart of a data synchronization method according to a second embodiment of the present invention, and this embodiment is a further description of a data synchronization method based on the foregoing embodiment. As shown in fig. 2, the method includes:
s210, searching at least one data synchronization rule according to the mapping relation of the merging rule.
In the embodiment of the invention, the mapping relation of the merging rule can be determined, and the data synchronization rule associated with the merging rule is searched according to the mapping relation. In the actual operation process, all the data synchronization rules associated with the merging rules can be extracted through the mapping relation, and the number of the data synchronization rules can be not limited.
S220, taking the data synchronization rule with the rule state of each data synchronization rule being the starting data synchronization rule as the applied data synchronization rule.
The rule state may be an identifier for indicating whether the data synchronization rule is applied, and the rule state may include start and stop, and when the rule state of the data synchronization rule is start, the data synchronization rule may be operated to perform data synchronization; when the rule state of the data synchronization rule is stopped, the data synchronization rule may not be run. In one embodiment, the rule states may be set according to user requirements.
In the embodiment of the invention, after the data synchronization rule is acquired, the rule state of each data synchronization rule can be extracted, wherein the rule state is the started data synchronization rule as the applied data synchronization rule.
S230, extracting cluster nodes corresponding to the source database.
In the embodiment of the invention, the data of the source database can be stored in a plurality of cluster nodes, the cluster nodes of the cluster where the source database is located can be obtained, and the cluster nodes for storing the information of the source database are determined.
S240, determining extraction rules matched with all cluster nodes.
In the embodiment of the invention, the extraction rule can be extracted, and the extraction rule matched with each cluster node can be determined. The cluster nodes may be in one-to-one correspondence with the extraction rules. In an embodiment, a mapping relationship between each cluster node and the extraction rule may be pre-established, and the extraction rule matched with each cluster node is extracted through the mapping relationship.
In an embodiment, determining the extraction rule matching each cluster node further comprises:
determining whether each cluster node has a corresponding unique extraction rule;
if not, the extraction rule is added or deleted so that each cluster node has a corresponding unique extraction rule.
In the embodiment of the invention, after the extraction rule is obtained, the cluster nodes can be matched with the extraction rule, and whether the corresponding unique extraction rule exists in each cluster node is determined. When the cluster node does not have the corresponding extraction rule, the extraction rule can be added or deleted. For example, when any cluster node does not have a corresponding unique extraction rule, the extraction rule corresponding to the cluster node may be newly added; when any extraction rule does not exist corresponding cluster nodes, the extraction rule can be deleted, so that each cluster node has a corresponding unique extraction rule.
S250, extracting initial synchronous data from the cluster nodes according to the data synchronous rule, and merging the initial synchronous data with the same data name.
The initial synchronization data may be synchronization data extracted according to each data synchronization rule.
In the embodiment of the invention, initial synchronous data can be extracted from the cluster nodes according to each data synchronous rule. Since the initial synchronization data extracted from each data synchronization rule has a repetition, the data name of the initial synchronization data can be obtained, and the initial synchronization data with the same data name can be combined.
S260, taking the initial synchronous data as data to be synchronized.
S270, extracting target synchronous data from the corresponding data to be synchronized according to the data synchronous rule.
In the embodiment of the invention, corresponding data to be synchronized can be matched as target synchronization data according to each data synchronization rule. In the actual operation process, the data identification to be synchronized corresponding to the data synchronization rule can be extracted, the target synchronization data corresponding to each data synchronization rule is determined, and the target synchronization data is extracted from the data to be synchronized.
S280, synchronizing the target synchronous data to the target database according to the target database identification of the data synchronization rule.
In the embodiment of the invention, the target position of each target data synchronization can be identified and determined in the target end database, and the target synchronization data is synchronized to the target end database according to the target position.
According to the embodiment of the invention, at least one data synchronization rule is searched according to the mapping relation of the merging rule, the data synchronization rule with the rule state of starting in each data synchronization rule is used as the data synchronization rule applied, the cluster node corresponding to the source end database is extracted, the extraction rule matched with each cluster node is determined, the initial synchronization data is extracted at the cluster node according to the data synchronization rule, the initial synchronization data with the same data name are merged, the initial synchronization data is used as the data to be synchronized, the target synchronization data is extracted in the corresponding data to be synchronized according to the data synchronization rule, the target synchronization data is synchronized to the target end database according to the target end database identification of the data synchronization rule, unified processing of each storage node is realized, simultaneous analysis processing is carried out on each storage node to obtain the data to be synchronized, the resource consumption of data synchronization is reduced, and the use experience of a user is improved.
Example III
Fig. 3 is a flowchart of a data synchronization method according to a third embodiment of the present invention, in which a database is a postgres database, two data synchronization rules are exemplified by rule 1 and rule 2, respectively, a rule state of rule 1 is started, and a start state of rule 2 is stopped, based on the above embodiment, to further describe a data synchronization method.
S310, creating a data synchronization rule 1 and a data synchronization rule 2 of the source database.
In one embodiment, rule 1 and rule 2 may be identified by uuid. Illustratively, rule 1 is uuid1 and rule 2 is uuid2.
S320, inquiring whether a corresponding merging rule uuid_combination exists according to the source database identification in the rule 1.
S330, if the corresponding merging rule is not queried, using the uuid1 of the current rule 1 as a merging rule uuid_combination, and establishing a source database mapping relation combination.
Exemplary, mapping relationship of source database: uuid_combination; the mapping relationship of the merging rule and the data synchronization rules (rule 1 and rule 2) is origin. The mapping relation between the merging rule and the rule 1 is uuid_combination, namely uuid1.
And S340, if the corresponding merging rule is inquired, adding a mapping relation between the new data synchronization rule and the merging rule to the origin.
For example, the added mapping relationship may be uuid_combination, uuid1, uuid2.
S350, cluster nodes (dn 1, dn2 and dn 3) corresponding to the extraction database are obtained, and extraction rules corresponding to each cluster node are established.
S360, inquiring the mapping relation child.map of the extraction rules, and if the cluster node does not find the corresponding extraction rule, creating a new mapping relation child.map.
Illustratively, the child. Map mapping relationship may include:
dn1:uuid_child1；
dn2:uuid_child2
dn3:uuid_child3
and S370, matching whether the cluster nodes are consistent with the extraction rules, if not, adding or deleting the corresponding extraction rules, and updating the child_map.
S380, acquiring rule states of the rule 1 and the rule 2, wherein the rule states in the data synchronization rules are the started data synchronization rules as the applied data synchronization rules, and acquiring data to be synchronized.
During actual operation, the rule states of rule 1 and rule 2 may be pre-set and saved. The rule states of the rule 1 and the rule 2 can be extracted, and the started data synchronization rule is used as the applied data synchronization rule to acquire the data to be synchronized.
Illustratively, rule 1 may be started, saving rule state to origin_rule. Namely uuid1: enable; stopping rule 2, and storing the rule state to origin_rule.status; namely uuid2: disable. If all rules have stopped, the extraction rule is terminated.
In one embodiment, data to be synchronized having the same data name may be merged.
S390, determining a target end database according to the data synchronization rule, and synchronizing the corresponding data to be synchronized to the target end database.
In actual operation, the wal log may be analyzed, the analyzed data to be synchronized is matched according to the original rule (rule 1 and rule 2), and if the data to be synchronized is matched, the result is written into the target end database corresponding to the rule.
In one embodiment, fig. 4 is a mapping relation structure diagram according to a third embodiment of the present invention. As shown in fig. 4, the source database mapping (map) may include source data: uuid_combination. The mapping relationship (origin. Map) of the merge rule and the data synchronization rule (rule 1 and rule 2) includes: uuid_combination, uuid1, uuid2. The extraction rule mapping relation (child. Map) may include dn1: uuid_child1; dn2, uuid_child2; dn 3. Uuid_child3. Rule states (rule_status) may be uuid1: enabled and uuid2: disabled.
In one embodiment, fig. 5 is a flowchart of a rule management method according to a third embodiment of the present invention. As shown in fig. 5, the method includes:
creating a data synchronization rule 1 (identified as uuid 1) and a data synchronization rule 2 (identified as uuid 2) of the source database, and inquiring whether a corresponding merging rule (uuid_combination) exists according to the source database identification in the rule 1 and the rule 2. If the merging rule does not have the rule 1 and the rule 2, adding the mapping relation between the rule 1 and the merging rule and between the rule 2 and the merging rule to the mapping relation combination. If the merging rule has the rule 1 and the rule 2, inquiring the extraction rule corresponding to the cluster node, and determining whether each cluster node has the corresponding extraction rule; if the extraction rule does not exist, the extraction rule is added or deleted so that each cluster node has the corresponding extraction rule, and the Wal log extraction is performed when each cluster node has the corresponding extraction rule.
In an embodiment, fig. 6 is a flowchart of a data synchronization method according to a third embodiment of the present invention. In this embodiment, taking the example of starting rule 1 and stopping rule 2 as shown in fig. 6, the method includes:
and starting the rule 1 and stopping the rule 2 according to the data synchronization requirement, and saving the rule states of the rule 1 and the rule 2 to origin_rule. Illustratively, rule states uuid1: enabled and uuid2: disabled may be saved. Determining whether all data synchronization rules (rule 1 and rule 2) are all stopped, and if so, terminating the extraction rule; if all the cluster nodes stop, determining the extraction rules corresponding to the cluster nodes, acquiring initial synchronous data, merging the initial synchronous data according to the data name of the initial synchronous data to serve as data to be synchronized, and refreshing the extraction rules corresponding to the cluster nodes. Analyzing the data to be synchronized in the Wal log, after analyzing to record, respectively determining target synchronous data corresponding to the rule 1 and data to be synchronized corresponding to the rule 2, and respectively transmitting the data to be synchronized corresponding to the rule 1 and the rule 2 to a target database.
Example IV
Fig. 7 is a schematic structural diagram of a data synchronization device according to a fourth embodiment of the present invention. As shown in fig. 7, the apparatus includes: a synchronization rule acquisition module 41, an extraction rule acquisition module 42, a synchronization data determination module 43 and a synchronization data synchronization module 44.
The synchronization rule obtaining module 41 is configured to obtain a data synchronization rule applied in a merge rule of the source database.
The extraction rule obtaining module 42 is configured to obtain an extraction rule of the cluster node corresponding to the source database.
The synchronization data determining module 43 is configured to determine data to be synchronized at the cluster node according to the data synchronization rule and the extraction rule.
The synchronization data synchronization module 44 is configured to determine a target database according to a data synchronization rule, and synchronize corresponding data to be synchronized to the target database.
According to the embodiment of the invention, the data synchronization rule applied in the merging rule of the source database is acquired through the synchronization rule acquisition module, the extraction rule acquisition module acquires the extraction rule of the cluster node corresponding to the source database, the synchronous data determination module determines the data to be synchronized at the cluster node according to the data synchronization rule and the extraction rule, the synchronous data synchronization module determines the target database according to the data synchronization rule and synchronizes the corresponding data to be synchronized to the target database, so that the data synchronization rule merging management of the data synchronization rule belonging to the same database is realized, and all the data synchronization rules can be performed simultaneously. And uniformly processing each storage node, and simultaneously analyzing and processing each storage node to acquire data to be synchronized, so that the resource consumption of data synchronization is reduced, and the use experience of a user is improved.
In an embodiment, the data synchronization device further includes:
the rule creation module is used for creating at least two data synchronization rules according to the data synchronization requirement; the data synchronization rule includes: a data identifier to be synchronized and a target end database identifier;
and the mapping relation determining module is used for merging and storing at least two data synchronization rules as merging rules and adding the mapping relation between the merging rules and the data synchronization rules into the mapping relation set.
In one embodiment, the synchronization rule acquisition module 41 includes:
the synchronization rule searching unit is used for searching at least one data synchronization rule according to the mapping relation of the merging rule;
and the synchronization rule determining unit is used for taking the data synchronization rule with the rule state of each data synchronization rule being started as the applied data synchronization rule.
In one embodiment, the extraction rule acquisition module 42 includes:
the cluster node extraction unit is used for extracting cluster nodes corresponding to the source end database;
and the extraction rule determining unit is used for determining the extraction rule matched with each cluster node.
In an embodiment, the extraction rule determining unit includes:
the extraction rule matching unit is used for determining whether each cluster node has a corresponding unique extraction rule;
and the extraction rule updating unit is used for adding or deleting the extraction rule if the extraction rule does not exist, so that each cluster node has a corresponding unique extraction rule.
In one embodiment, the synchronization data determination module 43 includes:
the initial data merging unit is used for extracting initial synchronous data from the cluster nodes according to the data synchronization rule and merging the initial synchronous data with the same data name;
and the synchronous data determining unit is used for taking the initial synchronous data as the data to be synchronized.
In one embodiment, the synchronization data synchronization module 44 includes:
the data extraction unit is used for extracting target synchronous data from the corresponding data to be synchronized according to the data synchronization rule;
and the data synchronization unit is used for synchronizing the target synchronous data to the target database according to the target database identification of the data synchronization rule.
The data synchronization device provided by the embodiment of the invention can execute the data synchronization method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
Example five
Fig. 8 is a schematic structural diagram of an electronic device 10 implementing a data synchronization method according to an embodiment of the present invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Electronic equipment may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 8, the electronic device 10 includes at least one processor 11, and a memory, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, etc., communicatively connected to the at least one processor 11, in which the memory stores a computer program executable by the at least one processor, and the processor 11 may perform various appropriate actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from the storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 may also be stored. The processor 11, the ROM 12 and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
Various components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, etc.; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as the data synchronization method.
In some embodiments, the data synchronization method may be implemented as a computer program tangibly embodied on a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more of the steps of data synchronization method X described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data synchronization method in any other suitable way (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
A computer program for carrying out methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be implemented. The computer program may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service are overcome.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present invention may be performed in parallel, sequentially, or in a different order, so long as the desired results of the technical solution of the present invention are achieved, and the present invention is not limited herein.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (10)
1. A method of data synchronization, comprising:
acquiring a data synchronization rule applied in a merging rule of a source database;
acquiring extraction rules of cluster nodes corresponding to the source database;
determining data to be synchronized at the cluster node according to the data synchronization rule and the extraction rule;
and determining a target end database according to the data synchronization rule, and synchronizing the corresponding data to be synchronized to the target end database.
2. The method of claim 1, further comprising, prior to the obtaining the data synchronization rule applied in the merge rule of the source database:
creating at least two data synchronization rules according to the data synchronization requirement; wherein the data synchronization rule includes: a data identifier to be synchronized and a target end database identifier;
and merging and storing at least two data synchronization rules as the merging rules, and adding the mapping relation between the merging rules and the data synchronization rules in a mapping relation set.
3. The method according to claim 2, wherein the obtaining the data synchronization rule applied in the merging rule of the source database includes:
searching at least one data synchronization rule according to the mapping relation of the merging rule;
and taking the data synchronization rule with the rule state of the data synchronization rule as the starting data synchronization rule as the applied data synchronization rule.
4. The method according to claim 1, wherein the obtaining the extraction rule of the cluster node corresponding to the source database includes:
extracting cluster nodes corresponding to the source database;
and determining extraction rules matched with the cluster nodes.
5. The method of claim 4, wherein said determining the extraction rules that match each of the cluster nodes further comprises:
determining whether each cluster node has a corresponding unique extraction rule;
if not, the extraction rule is added or deleted so that each cluster node has the corresponding unique extraction rule.
6. The method according to claim 1, wherein said determining data to be synchronized at the cluster node according to the data synchronization rule and the extraction rule comprises:
extracting initial synchronous data from the cluster nodes according to the data synchronous rule, and merging the initial synchronous data with the same data name;
and taking the initial synchronous data as the data to be synchronized.
7. The method according to claim 1, wherein determining a target database according to the data synchronization rule and synchronizing the corresponding data to be synchronized to the target database includes:
extracting target synchronous data from the corresponding data to be synchronized according to the data synchronization rule;
and synchronizing the target synchronous data to the target database according to the target database identification of the data synchronization rule.
8. A data synchronization device, comprising:
the synchronization rule acquisition module is used for acquiring the data synchronization rule applied in the merging rule of the source database;
the extraction rule acquisition module is used for acquiring the extraction rule of the cluster node corresponding to the source end database;
the synchronous data determining module is used for determining data to be synchronized at the cluster node according to the data synchronizing rule and the extraction rule;
and the synchronous data synchronization module is used for determining a target end database according to the data synchronization rule and synchronizing the corresponding data to be synchronized to the target end database.
9. An electronic device, the electronic device comprising:
at least one processor;
and a memory communicatively coupled to the at least one processor;
wherein the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the data synchronization method of any one of claims 1-7.
10. A computer readable storage medium storing computer instructions for causing a processor to perform the data synchronization method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310987386.9A CN116881368A (en) | 2023-08-07 | 2023-08-07 | Data synchronization method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310987386.9A CN116881368A (en) | 2023-08-07 | 2023-08-07 | Data synchronization method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116881368A true CN116881368A (en) | 2023-10-13 |
Family
ID=88262171
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310987386.9A Pending CN116881368A (en) | 2023-08-07 | 2023-08-07 | Data synchronization method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116881368A (en) |
-
2023
- 2023-08-07 CN CN202310987386.9A patent/CN116881368A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN116108094A (en) | Data integration method and device, electronic equipment and storage medium | |
| CN112925811B (en) | Method, apparatus, device, storage medium and program product for data processing | |
| CN116860751A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN116383207A (en) | Data tag management method and device, electronic equipment and storage medium | |
| CN116578646A (en) | Time sequence data synchronization method, device, equipment and storage medium | |
| CN113691403B (en) | Topology node configuration method, related device and computer program product | |
| CN116028517A (en) | Fusion database system and electronic equipment | |
| CN115905322A (en) | Service processing method and device, electronic equipment and storage medium | |
| CN112860811B (en) | Method and device for determining data blood relationship, electronic equipment and storage medium | |
| CN116881368A (en) | Data synchronization method and device, electronic equipment and storage medium | |
| CN114691781A (en) | Data synchronization method, system, device, equipment and medium | |
| CN116431698B (en) | Data extraction method, device, equipment and storage medium | |
| CN115174447B (en) | Network communication method, device, system, equipment and storage medium | |
| CN112783507B (en) | Data stream guiding playback method and device, electronic equipment and readable storage medium | |
| CN117093274A (en) | Application running stack analysis method, device, equipment and storage medium | |
| CN118012936A (en) | Data extraction method, device, equipment and storage medium | |
| CN115730000A (en) | Medical data integration method, device, equipment and medium based on data lake | |
| CN117609171A (en) | Authority data synchronization method, device, equipment and medium | |
| CN118093211A (en) | Distributed lock processing method, device, equipment and medium | |
| CN115237968A (en) | Node management method, device, equipment and storage medium in database system | |
| CN117076048A (en) | Container application starting method and device, storage medium and electronic equipment | |
| CN117271667A (en) | Data reading method and device, electronic equipment and storage medium | |
| CN116881229A (en) | Database exercise environment construction method, device, equipment and storage medium | |
| CN117171214A (en) | Database information comparison method and device, electronic equipment and storage medium | |
| CN117573677A (en) | Data updating method, device, equipment and medium based on transverse table database structure |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |