CN111182014B - Data synchronization method and device - Google Patents
Data synchronization method and device Download PDFInfo
- Publication number
- CN111182014B CN111182014B CN201811334308.4A CN201811334308A CN111182014B CN 111182014 B CN111182014 B CN 111182014B CN 201811334308 A CN201811334308 A CN 201811334308A CN 111182014 B CN111182014 B CN 111182014B
- Authority
- CN
- China
- Prior art keywords
- data
- key information
- version number
- target key
- service node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application discloses a data synchronization method and a related device, when a synchronization request of a service node is obtained, a snapshot copy of a data set to be synchronized corresponding to an identifier is obtained from a key value database based on the identifier in the synchronization request. In the process of carrying out data synchronization on the service node according to the snapshot copy, if updated data aiming at the data set to be synchronized is received, the version number of target key information corresponding to the updated data is changed in an increasing mode, then the updated data is synchronized to the service node as the data corresponding to the target key information, the read version number of the target key information is updated to the version number after the target key information is changed, and the condition that the newer data corresponding to the key information is covered by the older data is avoided.
Description
Technical Field
The present application relates to the field of data processing, and in particular, to a data synchronization method and apparatus.
Background
In the distributed system, when a service node is online, in order to ensure that the online of a new service does not affect the normal operation of the existing service, the distributed system synchronously subscribes a data set corresponding to the relationship for the service node according to the subscription relationship of the service node to data, and the synchronized data set generally comprises two parts, wherein one part is historical data of the data set before data synchronization, and the other part is real-time data updated to the data set in the data synchronization process.
In one possible implementation, when a service node is online, the distributed system uses a Message Queue (MQ) and a Key-Value (KV) database to synchronize data of the service node. The KV database is used for storing historical data, and the MQ is used for caching updated real-time data in the data synchronization process. Since real-time data is generally classified into two types: one is an update to the historical data; the other is newly generated service data. In order to avoid the update of the historical data from being covered by the original data, the adopted data synchronization mode is that the historical data in the KV database is synchronized for the newly online service, in the synchronization process, the newly generated real-time data is cached in the MQ for waiting, and the real-time data cached in the MQ is synchronized for the service after the synchronization is completed.
For the two real-time data, if the first data cannot be pushed in time, the service node makes a mistake in processing the existing service because the historical data is usually wrong. The failure of the second data to be pushed in time may result in the failure of the service data to be processed by the service node in time. In either case, the new online service node is adversely affected.
Disclosure of Invention
The embodiment of the application provides a data synchronization method and device, so that acquired updating data can be directly synchronized to a service node in the data synchronization process, the historical data does not need to be waited for being synchronized, the updating data serving as real-time data is pushed to the service node in time, and the data processing problem caused by the fact that the service node appears is avoided.
In a first aspect, an embodiment of the present application provides a data synchronization method, where the method includes:
acquiring a synchronization request of a service node, wherein the synchronization request comprises an identifier of a data set to be synchronized;
acquiring a snapshot copy generated according to the data set to be synchronized from a key value database through the identification, wherein the key information corresponding to the data in the snapshot copy is provided with a version number, and the version number of one key information is used for identifying the updating information of the data corresponding to the key information;
in the process of carrying out data synchronization on the service node according to the snapshot copy, if the update data aiming at the data set to be synchronized is obtained, the version number of the target key information corresponding to the update data is changed in an increasing mode;
synchronizing the updated data serving as the data corresponding to the target key information to the service node, and updating the read version number of the target key information to the modified version number of the target key information;
when the data of the target key information is synchronized with the service node again, the read version number is used for avoiding that the data corresponding to the target key information and having the version number lower than the read version number is synchronized with the service node in the data synchronization process.
Optionally, before synchronizing the update data to the service node as the data corresponding to the target key information, the method further includes:
judging whether the target key information has a read version number;
and if not, determining that the data corresponding to the target key information is not synchronized to the service node, and executing the step of synchronizing the updated data serving as the data corresponding to the target key information to the service node.
Optionally, the method further includes:
and if the target key information is judged not to read the version number, the data corresponding to the target key information in the snapshot copy is not synchronized to the service node.
Optionally, if it is determined that the target key information has the read version number, the method further includes:
judging whether the modified version number of the target key information is larger than the read version number of the target key information;
and if so, executing the step of synchronizing the updated data to the service node as the data corresponding to the target key information.
Optionally, after completing data synchronization to the service node according to the snapshot copy, the method further includes:
and deleting the version number of the key information corresponding to the data in the data set to be synchronized.
In a second aspect, an embodiment of the present application provides a data synchronization apparatus, where the apparatus includes a first obtaining unit, a second obtaining unit, a changing unit, and a synchronization unit:
the first obtaining unit is configured to obtain a synchronization request of a service node, where the synchronization request includes an identifier of a data set to be synchronized;
the second obtaining unit is configured to obtain, from the key value database, a snapshot copy generated according to the data set to be synchronized through the identifier, where key information corresponding to data in the snapshot copy is provided with a version number, and the version number of one piece of key information is used to identify update information of the data corresponding to the key information;
the change unit is configured to, in a process of performing data synchronization on the service node according to the snapshot copy, change, in an incremental manner, a version number of target key information corresponding to the update data if the update data for the set of data to be synchronized is acquired;
the synchronization unit is configured to synchronize the update data to the service node as data corresponding to the target key information, and update the read version number of the target key information to the version number of the modified target key information;
when the data of the target key information is synchronized with the service node again, the read version number is used for avoiding that the data corresponding to the target key information and having the version number lower than the read version number is synchronized with the service node in the data synchronization process.
Optionally, the apparatus further includes a determining unit:
the judging unit is used for judging whether the target key information has a read version number or not;
if the judging unit judges that the target key information does not read out the version number, the synchronizing unit is used for determining that the data corresponding to the target key information is not synchronized to the service node, and executing the step of synchronizing the updated data serving as the data corresponding to the target key information to the service node.
Optionally, if the determining unit determines that the target key information does not read the version number, the synchronizing unit is configured to not synchronize the data corresponding to the target key information in the snapshot copy with the service node.
Optionally, if the determining unit determines that the target key information has a read version number, the determining unit is further configured to determine whether the modified version number of the target key information is greater than the read version number of the target key information;
and if the judging unit judges that the modified version number of the target key information is greater than the read version number of the target key information, triggering the synchronizing unit to execute the step of synchronizing the updated data serving as the data corresponding to the target key information to the service node.
Optionally, the apparatus further includes a deleting unit:
and the deleting unit is used for deleting the version number of the key information corresponding to the data in the data set to be synchronized.
In a third aspect, an apparatus for data synchronization is provided in embodiments herein and includes a memory and one or more programs, where the one or more programs are stored in the memory and configured for execution by the one or more processors to include instructions for:
acquiring a synchronization request of a service node, wherein the synchronization request comprises an identifier of a data set to be synchronized;
acquiring a snapshot copy generated according to the data set to be synchronized from a key value database through the identification, wherein the key information corresponding to the data in the snapshot copy is provided with a version number, and the version number of one key information is used for identifying the updating information of the data corresponding to the key information;
in the process of carrying out data synchronization on the service node according to the snapshot copy, if the update data aiming at the data set to be synchronized is obtained, the version number of the target key information corresponding to the update data is changed in an increasing mode;
synchronizing the updated data serving as the data corresponding to the target key information to the service node, and updating the read version number of the target key information to the modified version number of the target key information;
when the data of the target key information is synchronized with the service node again, the read version number is used for avoiding that the data corresponding to the target key information and having the version number lower than the read version number is synchronized with the service node in the data synchronization process.
In a fourth aspect, embodiments of the present application provide a computer-readable storage medium, which includes instructions that, when executed on a computer, cause the computer to perform the data synchronization method according to the first aspect.
According to the technical scheme, the embodiment of the application has the following advantages:
when a synchronization request of a service node is obtained, a snapshot copy of a data set to be synchronized corresponding to an identifier can be obtained from a key value database based on the identifier in the synchronization request, and key information corresponding to the data in the snapshot copy is provided with a version number of update information used for identifying the corresponding data. In the process of data synchronization to the service node according to the snapshot copy, if updated data aiming at the data set to be synchronized is received, the version number of the target key information corresponding to the updated data is changed in an increasing mode, then the updated data can be synchronized to the service node as the data corresponding to the target key information, and the read version number of the target key information is updated to the version number after the target key information is changed, because the version number of the key information is gradually increased along with the update of the data, the read version number of the key information can identify the latest updated information of the data corresponding to the key information synchronized to the service node, therefore, in the process of data synchronization, the read version number can avoid the data which is lower than the read version number and corresponds to the target key information, namely, the data with the older updated information is prevented from being synchronized to the service node, therefore, the condition that the newer data corresponding to the key information is covered by the older data is avoided, the acquired updated data can be directly synchronized to the service node in the data synchronization process, the historical data does not need to be synchronized, the updated data serving as the real-time data is pushed to the service node in time, and the data processing problem caused by the fact that the service node appears is avoided.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a schematic system architecture diagram of a data synchronization method according to an embodiment of the present application;
fig. 2 is a flowchart of a data synchronization method according to an embodiment of the present application;
fig. 3 is a schematic system architecture diagram of a data synchronization method according to an embodiment of the present application;
fig. 4 is a structural diagram of a data synchronization apparatus according to an embodiment of the present application;
fig. 5 is a schematic hardware structure diagram of an apparatus for data synchronization according to an embodiment of the present disclosure.
Detailed Description
Embodiments of the present application are described below with reference to the accompanying drawings.
The conventional data synchronization method is performed according to a timing sequence that all historical data are synchronized to a service node first and then updated data are synchronized to the service node, otherwise, the historical data may cover the updated data.
For example, a data set to be synchronized of a service node includes historical data X1The corresponding key information is K1The Message Queue distributor (MQ BROKER) acquires data in the process of synchronizing data to the service nodeK is1Update data X of2If MQ BROKER is acquiring the updated data X2Data X will be updated immediately afterwards2Pushed to the service node, and the historical data X at the moment1If the data is not pushed to the service node, then MQ BROKER will send the historical data X again through data synchronization1As K1Is synchronized to the service node, resulting in K at the service node1Is composed of update data X2Is replaced with history data X1. Thus, after data synchronization, the service node uses K if processing service needs to use K1Then only the historical data X is obtained1Causing service nodes to handle the traffic with errors. Therefore, the traditional data method can only synchronize the historical data and then push the updated data to the service node.
It can be understood that, since it is difficult for the MQ BROKER to distinguish which data is the most recent data, in order to avoid the case that the history data covers the updated data, only the conventional method can be used for data synchronization. In order to solve the technical problem caused by the conventional method, an embodiment of the present application provides a data synchronization method, which may be applied to the system architecture shown in fig. 1. The system architecture comprises a service node 101, an MQ BROKER 102, a KV database 103 and an existing service node 104. Service node 101 and existing service node 104 are both service nodes in a distributed system. The service node may be, for example, a router, a mobile terminal accessing a serving cell, a client, or the like; preferably, the mobile terminal may be a smartphone, a Personal Digital Assistant (PDA), a tablet computer, or the like; preferably, the client may be an Application (APP) configured on the terminal device.
The existing service node 104 is a service node already connected to the system, and is configured to produce various types of service data and publish the produced service data to the system through the MQ BROKER 102, where the MQ BROKER 102 serves as a control unit of the MQ and provides publishing, subscribing and storing functions of the data through the MQ. The KV database 103 is used for storing historical data corresponding to data required by the service node 101, and the historical data may be produced by the service node 104 or acquired by other devices; the KV database 103 stores historical data in KV, where in a KV pair, K is key information for identifying a storage location of V, and V is a specific historical data content.
The service node 101 may send a synchronization request to the MQ BROKER 102 according to a data processing requirement to obtain historical data corresponding to the data processing requirement, where the data processing requirement may be clear through an identifier carried in the synchronization request, the MQ BROKER 102 may determine a data set to be synchronized through the identifier in the synchronization request, and obtain a snapshot copy corresponding to the data set to be synchronized through the KV database 103, and the MQ BROKER 102 synchronizes content in the snapshot copy to the service node 101, which is a process of data synchronization in this embodiment.
In this embodiment, the service node 101 that sends the synchronization request may include multiple types of service nodes. For example, it may refer to a newly online service node, and the newly online service node may include two cases. In one case, service nodes belonging to the distributed system but temporarily not on-line are on-line, and according to their own data processing requirements, these service nodes need to obtain data, which is generated in the distributed system before on-line and corresponds to the data processing requirements, through a synchronization request, and such service nodes may be used as service nodes 101.
In another case, service nodes that originally do not belong to the distributed system are added to the system and brought online, and according to the data processing requirements of the service nodes, the service nodes need to obtain data that is generated in the distributed system before the access of the service nodes and corresponds to the data processing requirements through a synchronization request, and such service nodes can be used as the service nodes 101.
The service node 101 sending the synchronization request in the embodiment of the present application may be not only the service node 101 that is newly online, but also an online service node in the distributed system, for example, some data processing functions are newly added to some online service nodes, and in order to implement these new functions, these online service nodes may obtain data required for implementing the new functions in the distributed system through the synchronization request, so as to provide corresponding functions.
The synchronization request comprises an identifier of a data set to be synchronized, and the data set to be synchronized is a set of key information corresponding to data required to be acquired by the service node determined according to the identifier. The identifier is used to identify the type of data that the service node needs to obtain, for example, the mobile terminal accesses a serving cell as the service node 101, and provides a function of forwarding data, so that the mobile terminal needs to obtain routing information needed for forwarding data, a synchronization request sent to the MQ BROKER 102 carries an identifier for identifying the needed routing information, and the MQ may obtain data of the routing information needed by the mobile terminal from the KV database 103.
The MQ obtains a snapshot copy of the data set to be synchronized corresponding to the identifier from the KV database 103 according to the identifier, in the embodiment of the application, the key information corresponding to the data in the snapshot copy is additionally provided with a version number by the MQ BROKER 102, the version number is used for identifying the update information of the corresponding data, the specific content of the data does not need to be identified, and the update information can reflect that the corresponding data is updated for several times in the data synchronization process.
The version number may be represented by the numbers 0, 1 … …, but may also be represented by other forms of letters plus numbers V0, V1 … …, etc. Generally, the version number set by MQ BROKER 102 for the key information corresponding to the data in the snapshot copy may be 0 or V0, etc.
During the process of data synchronization from the MQ BROKER 102 to the service node according to the snapshot copy, if the MQ receives the update data for the data set to be synchronized, the MQ BROKER 102 changes the version number of the target key information corresponding to the update data in an incremental manner. For example, the version number of the target key information is increased by 1 every time data corresponding to the target key information is updated.
And the MQ BROKER 102 synchronizes the updated data serving as the data corresponding to the target key information to the service node, and updates the read version number of the target key information into the modified version number of the target key information. Because the version number of the key information is gradually increased along with the updating of the data in the key information, the read version number of the key information can identify the latest updated information of the data corresponding to the key information which is synchronized to the service node, if the data to be determined of the target key information needs to be synchronized to the service node again, the MQ BROKER 102 judges whether the version number of the target key information is higher than the read version number, if not, the data to be determined is definitely old data relative to the data of the target key information which is pushed to the service node 101 before, and the MQ BROKER 102 does not synchronize the data to be determined to the service node any more, so that the data corresponding to the target key information and having the version number lower than the read version number is prevented from being synchronized to the service node, and the situation that the newer data corresponding to the key information is covered by the older data is avoided.
Example one
Next, a data synchronization method provided by an embodiment of the present application will be described in detail with reference to the accompanying drawings.
Referring to fig. 2, fig. 2 shows a flow chart of a data synchronization method, the method comprising:
s201, acquiring a synchronization request of the service node.
The service node needs to acquire the latest data required for realizing the functions in the system according to the data processing requirements of the service node, so that the service node can send a synchronization request to the MQ BROKER 102 so as to acquire the required synchronization data from the MQ BROKER 102. At this point, MQ BROKER 102 may be considered to enter the data recovery state.
S202, acquiring a snapshot copy generated according to the data set to be synchronized from a key value database through the identification.
In one implementation, the identification in the synchronization request may be the subject (Topic) of the data needed by the service node, by which the MQ BROKER 102 may be made aware of what type of data is needed by the service node.
For example, the KV database 103 determines that key information K included in the data set to be synchronized is 1 to N according to the identifier, and the KV database 103 may read corresponding data from the storage location identified by K being 1 to N, where the data constitute a snapshot copy of the data set to be synchronized. The snapshot copy may be constructed as shown at 105 in fig. 1, if K: v corresponding to 1 is data X1… …, K: v corresponding to N is data Z1If the snapshot copy includes data X1… …, data Z1。
The MQ obtains data corresponding to the data set to be synchronized from the KV database 103 in a snapshot copy mode, the snapshot copy means that the KV database 103 backs up the data corresponding to the data set to be synchronized, and the data in the snapshot copy remains unchanged, so that when the MQ obtains the updated data, the MQ BROKER 102 can still update the data in the KV database without affecting the data in the snapshot copy.
Assume that the snapshot copy is shown as 105 in FIG. 1, and includes data X1… …, data Z1Wherein, the data X1Corresponding key information K is 1, data Z1And if the corresponding key information K is N, the version numbers set by the MQ BROKER 102 for the key information 1 to N corresponding to the data in the snapshot copy are all 0.
It should be noted that, since the data In the snapshot copy is the data read into the MQ, the version number of the key information setting corresponding to the data In the snapshot copy may be referred to as a read-In version number, and is denoted by In Ver, and at this time, the value of In Ver is 0.
S203, in the process of carrying out data synchronization on the service node according to the snapshot copy, if the updated data aiming at the data set to be synchronized is obtained, the version number of the target key information corresponding to the updated data is changed in an incremental mode.
The MQ BROKER 102 synchronizes the data in the snapshot copy to the service node, and each time the data is synchronized to the service node, the MQ BROKER 102 may assign a read version number to the key information K corresponding to the data, and the read version number may be represented by Out Ver. And if the updated data aiming at the data set to be synchronized is not obtained In the process of carrying Out data synchronization on the service node according to the snapshot copy, directly synchronizing the data In the snapshot copy to the service node, wherein at the moment, the Out Ver value of the key information K corresponding to each data is equal to the In Ver value of the key information K corresponding to the data In the snapshot copy.
For example, as shown in FIG. 3, data X in the snapshot copy in FIG. 31Is synchronized to the service node, then data X1Corresponding key information K: out Ver of 1 is the data X in the snapshot copy1Corresponding key information K: 1, In Ver, i.e., bond information K: out Ver of 1 is 0.
If, In the process of performing data synchronization on the service node by the MQ BROKER 102 according to the snapshot copy, the MQ obtains the updated data for the data set to be synchronized, the version number of the target key information corresponding to the updated data is changed In an incremental manner, where the version number may also be referred to as a read-In version number and may be represented by In Ver, except that the data corresponding to the target key information is data newly generated by the existing service node 104, and is not data from the KV database. For the version number, the value incremented each time may be the same or different, and for convenience of description, in this embodiment, the value incremented each time is the same and is 1.
It should be noted that the update data for the data set to be synchronized includes data updated for existing data in the snapshot copy, and data that does not belong to the snapshot copy but corresponds to K in the data set to be synchronized.
For example, MQ BROKER 102 determines that the data set to be synchronized includes according to the identificationK of (1) to (100), however, corresponding data are not stored in the positions where K is 51 to (100) in the KV database 103, and the snapshot copy obtained by the MQ only includes the data X corresponding to K is 1 to (50)1、Y1… …. When MQ BROKER 102 synchronizes data in the snapshot copy corresponding to the data set to be synchronized to the service node, if the existing service node 104 generates new data as K: 1. v: x2，K：60、V：Y2，K：120、V：M2Since the data set to be synchronized comprises K from 1 to 100, K: 1 and K: 120 are in the data set to be synchronized, then the new data V: x2And V: y is2Is the updated data for the set of data to be synchronized. Wherein, the data V: x2The corresponding K is 1, and K is already included in the snapshot copy: 1, data V: x2Is the data that updates the existing data in the snapshot copy. Data V: y is2The corresponding K is 60, although K is not included in the snapshot copy: 60, but K: 60 should the data corresponding to the snapshot copy be read to the data synchronized to the service node, then data V: x2Is data that does not belong to the snapshot copy but corresponds to K in the data set to be synchronized.
The version number of the target key information corresponding to the updated data may be a version number set by MQ BROKER 102 for the target key information corresponding to the data when the MQ obtains the snapshot copy, or may be a version number after the target key information is changed. For example, the MQ obtains the update data as K: 1. v: x2If MQ BROKER 102 is based on V: x2Corresponding target key information K: if 1 determines that Out Ver is 0, it indicates that the update data is the data updated for the first time according to the data corresponding to the target key information, and at this time, MQ BROKER 102 will update the update data in the target key information K: 1 is increased to change the version number of the target key information on the basis that the version number corresponding to the 1 is 0, and the changed version number can be InVer: 1. if MQ BROKER 102 is based on V: x2Corresponding target key information K: if 1 determines that Out Ver is not 0, for example, 1, it indicates that the target key information K: 1 has been modified in an incremental manner, and for the target key information K corresponding to the updated data: 1, canWith the target key information K obtained in the previous change: version number of 1 In Ver: 1, and the modified version number can be InVer: 2.
s204, synchronizing the updated data to the service node as the data corresponding to the target key information, and updating the read version number of the target key information to the modified version number of the target key information.
Since the update data is the latest data corresponding to the target key information, the MQ BROKER 102 needs to synchronize the update data as the data corresponding to the target key information to the service node, and update the read version number of the target key information to the version number after the target key information is changed.
Referring to fig. 3, for target key information K: 1, for update data V: x2And the target key information corresponding to the updating data is K: the version number after 1 change may be In Ver: 1, MQ BROKER 102 needs to update data V: x1And synchronizing the data corresponding to the target key information to the service node to cover the old data V: x1And the target key information K: the read version number Out Ver of 1 is updated to the target key information K: 1 modified version number 1, i.e., target key information K: the read version number Out Ver of 1 is 1.
It can be seen from the above technical solutions that, when a synchronization request of a service node is obtained, a snapshot copy of a data set to be synchronized corresponding to an identifier can be obtained from a key value database based on the identifier in the synchronization request, and key information corresponding to data in the snapshot copy is set with a version number of update information used for identifying the corresponding data. In the process of data synchronization to the service node according to the snapshot copy, if updated data aiming at the data set to be synchronized is received, the version number of the target key information corresponding to the updated data is changed in an increasing mode, then the updated data can be synchronized to the service node as the data corresponding to the target key information, and the read version number of the target key information is updated to the version number after the target key information is changed, because the version number of the key information is gradually increased along with the update of the data, the read version number of the key information can identify the latest updated information of the data corresponding to the key information synchronized to the service node, therefore, in the process of data synchronization, the read version number can avoid the data which is lower than the read version number and corresponds to the target key information, namely, the data with the older updated information is prevented from being synchronized to the service node, therefore, the possibility that the newer data corresponding to the key information is covered by the older data is reduced, the acquired updated data can be directly synchronized to the service node in the data synchronization process, the historical data does not need to be synchronized, the updated data serving as real-time data is pushed to the service node in time, and the data processing problem caused by the fact that the service node appears is avoided.
The foregoing embodiment describes a data synchronization method, the core of which is to read the version number, however, because the MQ may obtain the updated data at different times in the data synchronization process, the read version number may have different situations, and further, the specific process of data synchronization may be different. Therefore, before S204 is executed, it is necessary to determine the case of reading the version number, so as to determine an appropriate data synchronization process according to the difference of the case of reading the version number.
Specifically, before executing S204, MQ BROKER 102 may first determine whether the target key information has the read version number, and if MQ BROKER 102 determines that the target key information does not have the read version number, MQ BROKER 102 determines that the data corresponding to the target key information is not synchronized to the service node, and executes the step of S204.
Under the condition that MQ BROKER 102 judges that the target key information does not have the read version number, MQ BROKER 102 synchronizes the updated data to the service node as the data corresponding to the target key information, and updates the read version number of the target key information to the modified version number of the target key information, MQ BROKER 102 can compare the version number of the target key information in the snapshot copy with the read version number, if the version number of the target key information in the snapshot copy is smaller than the read version number, the data corresponding to the target key information in the snapshot copy is not synchronized to the service node, and therefore the updated data synchronized to the service node is prevented from being covered by the data corresponding to the target key information in the snapshot copy. However, when the MQ acquires the update data for the data set to be synchronized again, if the MQ BROKER 102 determines through comparison that the version number of the target key information corresponding to the update data is greater than the read version number, the MQ BROKER 102 pushes the update data acquired again to the service node.
For example, the snapshot copy includes data V: z1And its corresponding key information K: and N, the version number InVer of the key information is 0, and the MQ acquires the update data V: z2And updating target key information K corresponding to the data: n, then MQ BROKE 102 changes the version number InVer of the target key information to 1. If MQ BROKER 102 judges the target key information K: n does not have a read version number, MQ BROKE 102 will update data V: z2Synchronizing to the service node, and transmitting the target key information K: the read version number Out Ver of N is set to 1. To avoid the target key information K in the snapshot copy: n corresponding to data V: z1The update data V: z2Covering, since MQ BROKER 102 judges data V in the snapshot copy: z1Corresponding target key information K: n has a version number 0 less than the read version number 1, and therefore, MQ BROKER 102 will not convert the target key information K: n corresponding to data V: z1The synchronization is performed to the service node. If the MQ obtains the update data V again: z3And updating target key information K corresponding to the data: n, then MQ BROKE 102 changes the version number In Ver of the target key information to 2, since MQ BROKE 102 judges that the data V is updated: z3Corresponding target key information K: n is greater than the read version number 1, and therefore, MQ BROKER 102 will update data V: z3Synchronizing to the service node.
It should be noted that, the manner in which the MQ BROKER 102 does not synchronize the data corresponding to the target key information in the snapshot copy to the service node may include multiple manners, and this embodiment will be described in two manners.
The first way may be not to process the data corresponding to the target key information in the snapshot copy. The specific implementation manner may be that MQ BROKER 102 marks data corresponding to the target key information that does not have the read version number in the snapshot copy, and MQ BROKER 102 directly skips the marked data in the process of performing data synchronization to the service node according to the snapshot copy, and does not process the marked data. Or, the MQ BROKER 102 deletes the data corresponding to the target key information without the read version number in the snapshot copy, so that the data corresponding to the target key information without the read version number is not processed in the process of performing data synchronization to the service node according to the snapshot copy.
The second way may be to process the data corresponding to the target key information in the snapshot copy, when the data corresponding to the target key information in the snapshot copy is processed, the MQ BROKER 102 compares the version number of the target key information in the snapshot copy with the modified version number of the target key information, and if the version number of the target key information in the snapshot copy is smaller than the modified version number of the target key information, the MQ BROKER 102 does not synchronize the data corresponding to the target key information in the snapshot copy to the service node.
If MQ BROKER 102 determines that the target key information has the read version number, MQ BROKER 102 determines that the data corresponding to the target key information is synchronized to the service node, so that when MQ BROKER 102 acquires the updated data, MQ BROKER 102 can further determine whether the modified version number of the target key information is greater than the read version number of the target key information; if yes, it indicates that the update data is the latest update information, and MQ BROKER 102 executes step S204.
It is understood that, in a normal case, the updated data generated by the existing service node 104 is the latest data, the modified version number of the target key information is greater than the read version number of the target key information, and there is rarely a case where the modified version number of the target key information is not greater than the read version number of the target key information.
After the MQ BROKER 102 completes data synchronization to the service node according to the snapshot copy, the version number of the key information corresponding to the data in the data set to be synchronized does not need to be used, and the version number of the key information corresponding to the data in the data set to be synchronized has no value. If the version number of the key information corresponding to the data in the data set to be synchronized is still stored, a certain storage space is occupied, so that in order to save the storage space, after the MQ BROKER 102 completes data synchronization to the service node according to the snapshot copy, the version number of the key information corresponding to the data in the data set to be synchronized can be deleted. And, service node 101 leaves the data recovery state.
Example two
Based on the data synchronization method provided in the first embodiment, the present embodiment provides a data synchronization apparatus. Fig. 4 is a structural diagram of a data synchronization apparatus provided in this embodiment, where the data synchronization apparatus 400 includes a first obtaining unit 401, a second obtaining unit 402, a changing unit 403, and a synchronization unit 404:
the first obtaining unit 401 is configured to obtain a synchronization request of a service node, where the synchronization request includes an identifier of a data set to be synchronized;
the second obtaining unit 402 is configured to obtain, from a key value database, a snapshot copy generated according to the data set to be synchronized through the identifier, where key information corresponding to data in the snapshot copy is provided with a version number, and the version number of one piece of key information is used to identify update information of the data corresponding to the key information;
the changing unit 403 is configured to, in the process of performing data synchronization on the service node according to the snapshot copy, change, in an incremental manner, a version number of the target key information corresponding to the update data if the update data for the data set to be synchronized is acquired;
the synchronizing unit 404 is configured to synchronize the update data to the service node as data corresponding to the target key information, and update the read version number of the target key information to the version number of the modified target key information;
when the data of the target key information is synchronized with the service node again, the read version number is used for avoiding that the data corresponding to the target key information and having the version number lower than the read version number is synchronized with the service node in the data synchronization process.
Optionally, the apparatus further includes a determining unit:
the judging unit is used for judging whether the target key information has a read version number or not;
if the judging unit judges that the target key information does not read out the version number, the synchronizing unit is used for determining that the data corresponding to the target key information is not synchronized to the service node, and executing the step of synchronizing the updated data serving as the data corresponding to the target key information to the service node.
Optionally, if the determining unit determines that the target key information does not read the version number, the synchronizing unit is configured to not synchronize the data corresponding to the target key information in the snapshot copy with the service node.
Optionally, if the determining unit determines that the target key information has a read version number, the determining unit is further configured to determine whether the modified version number of the target key information is greater than the read version number of the target key information;
and if the judging unit judges that the modified version number of the target key information is greater than the read version number of the target key information, triggering the synchronizing unit to execute the step of synchronizing the updated data serving as the data corresponding to the target key information to the service node.
Optionally, the apparatus further includes a deleting unit:
and the deleting unit is used for deleting the version number of the key information corresponding to the data in the data set to be synchronized.
It can be seen from the above technical solutions that, when a synchronization request of a service node is obtained, a snapshot copy of a data set to be synchronized corresponding to an identifier can be obtained from a key value database based on the identifier in the synchronization request, and key information corresponding to data in the snapshot copy is set with a version number of update information used for identifying the corresponding data. In the process of data synchronization to the service node according to the snapshot copy, if updated data aiming at the data set to be synchronized is received, the version number of the target key information corresponding to the updated data is changed in an increasing mode, then the updated data can be synchronized to the service node as the data corresponding to the target key information, and the read version number of the target key information is updated to the version number after the target key information is changed, because the version number of the key information is gradually increased along with the update of the data, the read version number of the key information can identify the latest updated information of the data corresponding to the key information synchronized to the service node, therefore, in the process of data synchronization, the read version number can avoid the data which is lower than the read version number and corresponds to the target key information, namely, the data with the older updated information is prevented from being synchronized to the service node, therefore, the condition that the newer data corresponding to the key information is covered by the older data is avoided, the acquired updated data can be directly synchronized to the service node in the data synchronization process, the historical data does not need to be synchronized, the updated data serving as the real-time data is pushed to the service node in time, and the data processing problem caused by the fact that the service node appears is avoided.
EXAMPLE III
Referring to fig. 5, fig. 5 is a schematic diagram of a hardware structure of an apparatus for data synchronization according to an embodiment of the present application, where the apparatus 500 includes a memory 501, a receiver 502, and a transmitter 503, and a processor 504 respectively connected to the memory 501, the receiver 502, and the transmitter 503, where the memory 501 is configured to store a set of program instructions, and the processor 504 is configured to call the program instructions stored in the memory 501 to perform the following operations:
triggering the receiver 502 to obtain a synchronization request of a service node, where the synchronization request includes an identifier of a data set to be synchronized;
acquiring a snapshot copy generated according to the data set to be synchronized from a key value database through the identification, wherein the key information corresponding to the data in the snapshot copy is provided with a version number, and the version number of one key information is used for identifying the updating information of the data corresponding to the key information;
in the process of performing data synchronization on the service node according to the snapshot copy, if the receiver 502 obtains the update data for the data set to be synchronized, the version number of the target key information corresponding to the update data is changed in an incremental manner;
triggering the sender 503 to synchronize the updated data to the service node as the data corresponding to the target key information, and updating the read version number of the target key information to the version number after the target key information is changed;
when the data of the target key information is synchronized with the service node again, the read version number is used for avoiding that the data corresponding to the target key information and having the version number lower than the read version number is synchronized with the service node in the data synchronization process.
Alternatively, the processor 504 may be a Central Processing Unit (CPU), the Memory 501 may be an internal Memory of a Random Access Memory (RAM) type, and the receiver 502 and the transmitter 503 may include a common physical interface, which may be an Ethernet (Ethernet) interface or an Asynchronous Transfer Mode (ATM) interface. The processor 504, transmitter 503, receiver 502, and memory 501 may be integrated into one or more separate circuits or hardware, such as: application Specific Integrated Circuit (ASIC).
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product.
The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store or a data storage device, such as a server, a data center, etc., that is integrated with one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.
Claims (12)
1. A method for synchronizing data, the method comprising:
acquiring a synchronization request of a service node, wherein the synchronization request comprises an identifier of a data set to be synchronized;
acquiring a snapshot copy generated according to the data set to be synchronized from a key value database through the identification, wherein the key information corresponding to the data in the snapshot copy is provided with a version number, and the version number of one key information is used for identifying the updating information of the data corresponding to the key information;
in the process of carrying out data synchronization on the service node according to the snapshot copy, if the update data aiming at the data set to be synchronized is obtained, the version number of the target key information corresponding to the update data is changed in an increasing mode;
synchronizing the updated data serving as the data corresponding to the target key information to the service node, and updating the read version number of the target key information to the modified version number of the target key information;
when the data of the target key information is synchronized with the service node again, the read version number is used for avoiding that the data corresponding to the target key information and having the version number lower than the read version number is synchronized with the service node in the data synchronization process.
2. The method of claim 1, wherein before synchronizing the update data to the service node as the data corresponding to the target key information, the method further comprises:
judging whether the target key information has a read version number;
and if not, determining that the data corresponding to the target key information is not synchronized to the service node, and executing the step of synchronizing the updated data serving as the data corresponding to the target key information to the service node.
3. The method of claim 2, further comprising:
and if the target key information is judged not to read the version number, the data corresponding to the target key information in the snapshot copy is not synchronized to the service node.
4. The method of claim 2, wherein if it is determined that the target key information has a read version number, the method further comprises:
judging whether the modified version number of the target key information is larger than the read version number of the target key information;
and if so, executing the step of synchronizing the updated data to the service node as the data corresponding to the target key information.
5. The method of any of claims 1-4, wherein after completing data synchronization to the serving node from the snapshot copy, the method further comprises:
and deleting the version number of the key information corresponding to the data in the data set to be synchronized.
6. A data synchronization apparatus, characterized in that the apparatus comprises a first acquisition unit, a second acquisition unit, a change unit, and a synchronization unit:
the first obtaining unit is configured to obtain a synchronization request of a service node, where the synchronization request includes an identifier of a data set to be synchronized;
the second obtaining unit is configured to obtain, from the key value database, a snapshot copy generated according to the data set to be synchronized through the identifier, where key information corresponding to data in the snapshot copy is provided with a version number, and the version number of one piece of key information is used to identify update information of the data corresponding to the key information;
the change unit is configured to, in a process of performing data synchronization on the service node according to the snapshot copy, change, in an incremental manner, a version number of target key information corresponding to the update data if the update data for the set of data to be synchronized is acquired;
the synchronization unit is configured to synchronize the update data to the service node as data corresponding to the target key information, and update the read version number of the target key information to the version number of the modified target key information;
when the data of the target key information is synchronized with the service node again, the read version number is used for avoiding that the data corresponding to the target key information and having the version number lower than the read version number is synchronized with the service node in the data synchronization process.
7. The apparatus according to claim 6, further comprising a judging unit:
the judging unit is used for judging whether the target key information has a read version number or not;
if the judging unit judges that the target key information does not read out the version number, the synchronizing unit is used for determining that the data corresponding to the target key information is not synchronized to the service node, and executing the step of synchronizing the updated data serving as the data corresponding to the target key information to the service node.
8. The apparatus according to claim 7, wherein if the determining unit determines that the target key information does not read the version number, the synchronizing unit is configured to not synchronize the data corresponding to the target key information in the snapshot copy with the service node.
9. The apparatus according to claim 7, wherein if said judging unit judges that said target key information has a read version number, said judging unit is further configured to judge whether a modified version number of said target key information is greater than said read version number of said target key information;
and if the judging unit judges that the modified version number of the target key information is greater than the read version number of the target key information, triggering the synchronizing unit to execute the step of synchronizing the updated data serving as the data corresponding to the target key information to the service node.
10. The apparatus according to any one of claims 6 to 9, wherein the apparatus further comprises a deletion unit:
and the deleting unit is used for deleting the version number of the key information corresponding to the data in the data set to be synchronized.
11. An apparatus for data synchronization comprising a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured for execution by the one or more processors to perform the one or more programs include instructions for:
acquiring a synchronization request of a service node, wherein the synchronization request comprises an identifier of a data set to be synchronized;
acquiring a snapshot copy generated according to the data set to be synchronized from a key value database through the identification, wherein the key information corresponding to the data in the snapshot copy is provided with a version number, and the version number of one key information is used for identifying the updating information of the data corresponding to the key information;
in the process of carrying out data synchronization on the service node according to the snapshot copy, if the update data aiming at the data set to be synchronized is obtained, the version number of the target key information corresponding to the update data is changed in an increasing mode;
synchronizing the updated data serving as the data corresponding to the target key information to the service node, and updating the read version number of the target key information to the modified version number of the target key information;
when the data of the target key information is synchronized with the service node again, the read version number is used for avoiding that the data corresponding to the target key information and having the version number lower than the read version number is synchronized with the service node in the data synchronization process.
12. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the data synchronization method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811334308.4A CN111182014B (en) | 2018-11-09 | 2018-11-09 | Data synchronization method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811334308.4A CN111182014B (en) | 2018-11-09 | 2018-11-09 | Data synchronization method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111182014A CN111182014A (en) | 2020-05-19 |
| CN111182014B true CN111182014B (en) | 2022-04-26 |
Family
ID=70647932
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811334308.4A Active CN111182014B (en) | 2018-11-09 | 2018-11-09 | Data synchronization method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111182014B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103294675A (en) * | 2012-02-23 | 2013-09-11 | 上海盛霄云计算技术有限公司 | Method and device for updating data in distributed storage system |
| WO2014199568A1 (en) * | 2013-06-12 | 2014-12-18 | 日本電気株式会社 | Data writing control method for persistent storage device |
| CN105404679A (en) * | 2015-11-24 | 2016-03-16 | 华为技术有限公司 | Data processing method and apparatus |
| WO2016050112A1 (en) * | 2014-09-30 | 2016-04-07 | 华为技术有限公司 | Data storage method, storage apparatus and storage system |
| CN106662983A (en) * | 2015-12-31 | 2017-05-10 | 华为技术有限公司 | Method, apparatus and system for data reconstruction in distributed storage system |
| CN107733957A (en) * | 2016-08-12 | 2018-02-23 | 北京融聚世界网络科技有限公司 | Distributed Services configure system and version number's distribution method |
| CN108121804A (en) * | 2017-12-22 | 2018-06-05 | 百度在线网络技术（北京）有限公司 | Method, apparatus, terminal and the storage medium of cross-region distributed storage data |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10235404B2 (en) * | 2014-06-25 | 2019-03-19 | Cohesity, Inc. | Distributed key-value store |
| US10353920B2 (en) * | 2015-10-23 | 2019-07-16 | Oracle International Corporation | Efficient mirror data re-sync |
| US10261913B2 (en) * | 2017-04-20 | 2019-04-16 | Alibaba Group Holding Limited | Persistent memory for key-value storage |
-
2018
- 2018-11-09 CN CN201811334308.4A patent/CN111182014B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103294675A (en) * | 2012-02-23 | 2013-09-11 | 上海盛霄云计算技术有限公司 | Method and device for updating data in distributed storage system |
| WO2014199568A1 (en) * | 2013-06-12 | 2014-12-18 | 日本電気株式会社 | Data writing control method for persistent storage device |
| WO2016050112A1 (en) * | 2014-09-30 | 2016-04-07 | 华为技术有限公司 | Data storage method, storage apparatus and storage system |
| CN105404679A (en) * | 2015-11-24 | 2016-03-16 | 华为技术有限公司 | Data processing method and apparatus |
| CN106662983A (en) * | 2015-12-31 | 2017-05-10 | 华为技术有限公司 | Method, apparatus and system for data reconstruction in distributed storage system |
| CN107733957A (en) * | 2016-08-12 | 2018-02-23 | 北京融聚世界网络科技有限公司 | Distributed Services configure system and version number's distribution method |
| CN108121804A (en) * | 2017-12-22 | 2018-06-05 | 百度在线网络技术（北京）有限公司 | Method, apparatus, terminal and the storage medium of cross-region distributed storage data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111182014A (en) | 2020-05-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109842694B (en) | Method for synchronizing MAC addresses, network equipment and computer readable storage medium | |
| US9411869B2 (en) | Replication between sites using keys associated with modified data | |
| CN107315825B (en) | Index updating system, method and device | |
| CN111049928B (en) | Data synchronization method, system, electronic device and computer readable storage medium | |
| CN111259072A (en) | Data synchronization method and device, electronic equipment and computer readable storage medium | |
| CN112632029B (en) | Data management method, device and equipment of distributed storage system | |
| CN110535776B (en) | Gateway current limiting method, device, gateway, system and storage medium | |
| CN112839076B (en) | Data storage method, data reading method, gateway, electronic equipment and storage medium | |
| CN109391548B (en) | Table entry migration method and device and network communication system | |
| CN104580425A (en) | Client data synchronization method and system | |
| CN113886743A (en) | Method, device and system for refreshing cache resources | |
| CN117112508B (en) | File synchronization method and device based on serial numbers, computer equipment and storage medium | |
| US9767023B2 (en) | Method of controlling data writing to persistent storage device | |
| EP3624462B1 (en) | Data management system | |
| CN111355765A (en) | Network request processing and sending method and device | |
| CN111182014B (en) | Data synchronization method and device | |
| CN112749172A (en) | Data synchronization method and system between cache and database | |
| CN110955460A (en) | Service process starting method and device, electronic equipment and storage medium | |
| CN111629054B (en) | Message processing method, device and system, electronic equipment and readable storage medium | |
| CN114077587A (en) | Rule engine based business processing method, rule engine, medium and device | |
| CN111708780A (en) | Distributed table system and fragment master selection method, device, server and medium | |
| CN113037797A (en) | Data processing method and device | |
| JP2020123875A5 (en) | ||
| CN112395263B (en) | OSD data recovery method and device | |
| CN116388921B (en) | Link point time synchronization method, device and storage medium based on precompiled contracts |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |