CN111726375A - Data synchronization method and device and computer equipment - Google Patents

Data synchronization method and device and computer equipment Download PDF

Info

Publication number
CN111726375A
CN111726375A CN201910209353.5A CN201910209353A CN111726375A CN 111726375 A CN111726375 A CN 111726375A CN 201910209353 A CN201910209353 A CN 201910209353A CN 111726375 A CN111726375 A CN 111726375A
Authority
CN
China
Prior art keywords
data
version
local
latest version
registry
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201910209353.5A
Other languages
Chinese (zh)
Inventor
肖士良
丁硕青
赵鹏
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Shuju Xinyun Information Technology Co ltd
Original Assignee
Beijing Shuju Xinyun Information Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Shuju Xinyun Information Technology Co ltd filed Critical Beijing Shuju Xinyun Information Technology Co ltd
Priority to CN201910209353.5A priority Critical patent/CN111726375A/en
Publication of CN111726375A publication Critical patent/CN111726375A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • H04L67/1095Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/50Network services
    • H04L67/55Push-based network services

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The invention provides a data synchronization method, a data synchronization device and computer equipment. The method relates to the computer internet technology and solves the problem of poor reliability of data updating. The method comprises the following steps: receiving version information of registry data pushed by a registry; checking the version of the local data; and when the version of the local data is inconsistent with the version information of the registry data, acquiring the data of the latest version from the management module. The technical scheme provided by the invention is suitable for data synchronization in a distributed node network, and realizes reliable and efficient data synchronization.

Description

Data synchronization method and device and computer equipment
Technical Field
The present invention relates to computer internet technologies, and in particular, to a data synchronization method, apparatus, and computer device.
Background
At present, internet users grow rapidly, the pressure of network service performance grows rapidly, and caching and distributed deployment become an essential link in back-end design. Data synchronization is a relatively complex problem in use of distributed multiple nodes, and data inconsistency and service instability can be caused by improper processing.
The existing data synchronization is simple to realize, and the basic principle is as follows:
1. and passive synchronization, namely setting a data validity period and acquiring the data again after the data validity period is invalid.
The disadvantages are as follows: the lock preemption problem needs to be processed, and the large-area invalidation of the data cache is easily triggered, so that the overlarge rear-end pressure and the service avalanche are caused; data before cache failure is old data, which may result in dirty data.
2. And actively synchronizing, actively pushing the rear-end updating data to each node, and executing the node to update the current data.
The disadvantages are as follows: the data has a single point problem, and the synchronization mode is easily affected by the network problem.
Disclosure of Invention
The present invention is directed to solving the problems described above.
According to a first aspect of the present invention, there is provided a data synchronization method, comprising:
receiving version information of registry data pushed by a registry;
checking the version of the local data;
and when the version of the local data is inconsistent with the version information of the registry data, acquiring the data of the latest version from the management module.
Preferably, when the version of the local data is inconsistent with the version information of the registry data, after the step of obtaining the latest version of the data from the management module, the method further includes:
acquiring a timestamp carried in the data of the latest version;
comparing the local current system time with the timestamp;
enabling the latest version of data if a local current system time is later than the timestamp.
Preferably, after the step of comparing the local current system time with the timestamp, the method further includes:
in the case that the local current system time is earlier than the timestamp, delaying the current local data;
and after the local system time reaches the timestamp, enabling the latest version of data.
According to another aspect of the present invention, there is also provided a data synchronization method, including:
the method comprises the steps that a registry receives version information of data of the latest version pushed by a management module, wherein the version information carries a timestamp of the data of the latest version;
and the registration center pushes the data information of the latest version to each cache node to indicate the cache node with the older data version to update the local data to the latest version.
Preferably, after the step of pushing the data information of the latest version to each cache node by the registration center to instruct the cache node with an older data version to update the local data to the latest version, the method further includes:
and the registration center receives the notification information of the update completion sent by the cache node and records the latest version information of the local data of the cache node.
Preferably, after the step of pushing the data information of the latest version to each cache node by the registration center to instruct the cache node with an older data version to update the local data to the latest version, the method further includes:
the registry provides the latest version information of the local data of each cache node recorded to the load balancing platform so that the load balancing platform sends a user request to the cache node which completes the local data updating.
According to another aspect of the present invention, there is also provided a data synchronization apparatus, including:
the version information receiving module is used for receiving the version information of the registry data pushed by the registry;
the version checking module is used for checking the version of the local data;
and the active updating module is used for acquiring the data of the latest version from the management module when the version of the local data is inconsistent with the version information of the data of the registration center.
Preferably, the apparatus further comprises:
and the data management module is used for acquiring the timestamp carried in the latest version of data, comparing the local current system time with the timestamp, and enabling the latest version of data under the condition that the local current system time is later than the timestamp.
Preferably, the data management module is further configured to, in a case that a local current system time is earlier than the timestamp, delay the current local data, and enable the latest version of data after the local system time reaches the timestamp.
According to another aspect of the present invention, there is also provided a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the program:
receiving version information of registry data pushed by a registry;
checking the version of the local data;
and when the version of the local data is inconsistent with the version information of the registry data, acquiring the data of the latest version from the management module.
The invention provides a data synchronization method, a data synchronization device and computer equipment.A registration center receives version information of data of the latest version pushed by a management module and pushes the data information of the latest version to each cache node so as to indicate the cache node with the older data version to update local data to the latest version; each cache node receives the version information of the registry data pushed by the registry, then checks the version of the local data, and acquires the data of the latest version from the management module when the version of the local data is inconsistent with the version information of the registry data. The data can be updated in time, and the problem of poor reliability of data updating is solved.
Other characteristic features and advantages of the invention will become apparent from the following description of exemplary embodiments, which is to be read in connection with the accompanying drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and together with the description, serve to explain the principles of the invention. In the drawings, like reference numerals are used to indicate like elements. The drawings in the following description are directed to some, but not all embodiments of the invention. For a person skilled in the art, other figures can be derived from these figures without inventive effort.
Fig. 1 schematically shows a flow of a data synchronization method provided by an embodiment of the present invention;
FIG. 2 schematically shows a detailed flow of step 107 in FIG. 1;
fig. 3 schematically shows a structure of a data synchronization apparatus according to an embodiment of the present invention;
fig. 4 exemplarily shows a structure of a computer device provided by an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention. It should be noted that the embodiments and features of the embodiments in the present application may be arbitrarily combined with each other without conflict.
The existing data synchronization is simple to realize, and the basic principle is as follows:
1. and passive synchronization, namely setting a data validity period and acquiring the data again after the data validity period is invalid.
The disadvantages are as follows: the lock preemption problem needs to be processed, and the large-area invalidation of the data cache is easily triggered, so that the overlarge rear-end pressure and the service avalanche are caused; data before cache failure is old data, which may result in dirty data.
2. And actively synchronizing, actively pushing the rear-end updating data to each node, and executing the node to update the current data.
The disadvantages are as follows: the data has a single point problem, and the synchronization mode is easily affected by the network problem.
In view of the above problems, embodiments of the present invention provide a data synchronization method, apparatus, and computer device, where version information is pushed to all cache nodes by a registry, and each cache node is triggered to autonomously acquire updated data, so as to solve the problems that data is not updated timely and errors are easily generated in the existing synchronization method, and update data timely and increase reliability.
An embodiment of the present invention provides a data synchronization method, which takes updating API data as an example for description.
The application environment of the embodiment of the invention is as follows:
the cloud integration platform comprises an API management module and an API execution module, and the API execution module responds to the user request according to the setting of the API management module. The API execution module needs to acquire API information from the management module. The API information can be cached on the server where the API execution module is located in order to quickly respond to the user request. The API execution module needs to be horizontally expanded and deployed in multiple shares, and preferably, the API execution module may be disposed on each cache node server.
The specific process of the embodiment of the invention is shown in fig. 1, and comprises the following steps:
step 101, a registry receives version information of the latest version of data pushed by a management module, wherein the version information carries a timestamp of the latest version of data.
In the embodiment of the present invention, the API execution module of the cache node is started and then connected to the registry, and maintains long connection, and at the same time, obtains API information data, that is, current version information version (preferably, the specific content of the version is an API information start timestamp) from the management module, and constructs a version: the map structure of API-info is shown in Table 1.
TABLE 1
version1 API-info
And after the management module updates the API information, pushing the information to the registration center, and informing the registration center of the version information of the API information of the latest version.
And 102, the registration center pushes the data information of the latest version to each cache node to indicate the cache node with the older data version to update the local data to the latest version.
In this step, the registry pushes version information of the API information to all API execution modules connected to the registry (preferably, the version is a time stamp greater than the time for updating the API information).
Step 103, after receiving the version information of the registry data pushed by the registry, the cache node checks the version of the local data.
In this step, the API execution module determines whether the data currently using the cache is the latest version according to the version received from the registry.
And step 104, when the version of the local data is inconsistent with the version information of the registry data, acquiring the data of the latest version from the management module.
In this step, if the API information currently cached locally is not the latest version, the latest API information is acquired from the management module and added to the version that is constructed locally: in the API-info map structure, it can be arranged in ascending version order (the order of version timestamps from early to late), as shown in Table 2.
TABLE 2
version1 API-info
version2 API-info
Therefore, in the running process of the cloud integrated platform, data such as API information and the like can be continuously updated, timely updating of application data is guaranteed, and response to user requests is optimized.
And 105, the registry receives the update completion notification information sent by the cache node and records the latest version information of the local data of the cache node.
And 106, providing the latest version information of the local data of each cache node recorded by the registry to the load balancing platform so that the load balancing platform sends a user request to the cache node which completes the local data updating.
The load balancing platform judges the currently available API execution module (namely the API execution module updated to the cache node of the API information of the latest version) according to the registration center, distributes the user request to the available API execution module and avoids the user request from being sent to the non-updated API execution module.
Step 107, determining the used data version according to the current system time, and processing the user request.
In the operation process of the cloud integrated platform, for newly acquired data, start management needs to be performed to avoid the problem of dirty data, which is specifically shown in fig. 2 and includes:
step 1071, obtaining the timestamp carried in the data of the latest version.
Step 1072, comparing the local current system time with the timestamp.
Step 1073, if the local current system time is earlier than said timestamp, the current local data is delayed.
Step 1074, after the local system time reaches the timestamp, the latest version of data is enabled.
Step 1075, enabling the latest version of data if the local current system time is later than the timestamp.
Taking the API information version updating situation shown in table 2 as an example, the API execution module processes the user request, and if the current system timestamp is greater than version2, the API-info corresponding to version2 in table 2 is used, otherwise, the API-info corresponding to previous version1 is used.
When the system time is larger than version2, using the API-info corresponding to version2, deleting the API-info information corresponding to version1 in the map, and updating the new version: the API-info map structure is shown in Table 3.
TABLE 3
version2 API-info
According to the technical scheme provided by the embodiment of the invention, after the management platform modifies the API information, the execution module can acquire the API data in time and start new data at the same time to continue processing the user request.
An embodiment of the present invention further provides a data synchronization apparatus, whose structure is shown in fig. 3, including:
the version information receiving module 301 is configured to receive version information of registry data pushed by a registry;
a version check module 302 for checking a version of the local data;
and the active updating module 303 is configured to obtain the data of the latest version from the management module when the version of the local data is inconsistent with the version information of the registry data.
Preferably, the apparatus further comprises:
the data management module 304 is configured to obtain a timestamp carried in the latest version of data, compare a local current system time with the timestamp, and enable the latest version of data when the local current system time is later than the timestamp.
Preferably, the data management module 304 is further configured to delay the current local data if the local current system time is earlier than the timestamp, and enable the latest version of data after the local system time reaches the timestamp.
An embodiment of the present invention further provides a computer device, whose structure is shown in fig. 4, and which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor executes the computer program to implement the following steps:
receiving version information of registry data pushed by a registry;
checking the version of the local data;
and when the version of the local data is inconsistent with the version information of the registry data, acquiring the data of the latest version from the management module.
The embodiment of the invention provides a data synchronization method, a data synchronization device and computer equipment.A registration center receives version information of data of a latest version pushed by a management module and pushes the data information of the latest version to each cache node so as to indicate the cache node with an older data version to update local data to the latest version; each cache node receives the version information of the registry data pushed by the registry, then checks the version of the local data, and acquires the data of the latest version from the management module when the version of the local data is inconsistent with the version information of the registry data. The data can be updated in time, and the problem of poor reliability of data updating is solved.
The cache can be updated in time by the back-end data change, and whether the current cache data is the latest or not is judged by using version information such as the version number of the data. The inconsistent calling data is avoided, dirty data is caused, or user experience is reduced.
The method avoids using cache locks, removes the performance loss caused by lock preemption, uses cache services and adds monitoring to avoid avalanche caused by cache invalidation, and improves the reliability of the system.
The above-described aspects may be implemented individually or in various combinations, and such variations are within the scope of the present invention.
Finally, it should be noted that: the above examples are only for illustrating the technical solutions of the present invention, and are not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims (10)

1. A method of data synchronization, comprising:
receiving version information of registry data pushed by a registry;
checking the version of the local data;
and when the version of the local data is inconsistent with the version information of the registry data, acquiring the data of the latest version from the management module.
2. The data synchronization method according to claim 1, wherein after the step of obtaining the latest version of data from the management module when the version of the local data is not consistent with the version information of the registry data, the method further comprises:
acquiring a timestamp carried in the data of the latest version;
comparing the local current system time with the timestamp;
enabling the latest version of data if a local current system time is later than the timestamp.
3. The data synchronization method of claim 2, wherein the step of comparing the local current system time to the time stamp is followed by:
in the case that the local current system time is earlier than the timestamp, delaying the current local data;
and after the local system time reaches the timestamp, enabling the latest version of data.
4. A method of data synchronization, comprising:
the method comprises the steps that a registry receives version information of data of the latest version pushed by a management module, wherein the version information carries a timestamp of the data of the latest version;
and the registration center pushes the data information of the latest version to each cache node to indicate the cache node with the older data version to update the local data to the latest version.
5. The data synchronization method according to claim 4, wherein after the step of the registration center pushing the latest version of data information to each cache node to instruct cache nodes with older data versions to update local data to the latest version, the method further comprises:
and the registration center receives the notification information of the update completion sent by the cache node and records the latest version information of the local data of the cache node.
6. The data synchronization method according to claim 5, wherein after the step of the registration center pushing the latest version of data information to each cache node to instruct cache nodes with older data versions to update local data to the latest version, the method further comprises:
the registry provides the latest version information of the local data of each cache node recorded to the load balancing platform so that the load balancing platform sends a user request to the cache node which completes the local data updating.
7. A data synchronization apparatus, comprising:
the version information receiving module is used for receiving the version information of the registry data pushed by the registry;
the version checking module is used for checking the version of the local data;
and the active updating module is used for acquiring the data of the latest version from the management module when the version of the local data is inconsistent with the version information of the data of the registration center.
8. The data synchronization apparatus of claim 7, further comprising:
and the data management module is used for acquiring the timestamp carried in the latest version of data, comparing the local current system time with the timestamp, and enabling the latest version of data under the condition that the local current system time is later than the timestamp.
9. The data synchronization apparatus according to claim 7,
the data management module is further configured to delay using the current local data when the local current system time is earlier than the timestamp, and enable the data of the latest version after the local system time reaches the timestamp.
10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps when executing the program of:
receiving version information of registry data pushed by a registry;
checking the version of the local data;
and when the version of the local data is inconsistent with the version information of the registry data, acquiring the data of the latest version from the management module.
CN201910209353.5A 2019-03-19 2019-03-19 Data synchronization method and device and computer equipment Pending CN111726375A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910209353.5A CN111726375A (en) 2019-03-19 2019-03-19 Data synchronization method and device and computer equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910209353.5A CN111726375A (en) 2019-03-19 2019-03-19 Data synchronization method and device and computer equipment

Publications (1)

Publication Number Publication Date
CN111726375A true CN111726375A (en) 2020-09-29

Family

ID=72563311

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910209353.5A Pending CN111726375A (en) 2019-03-19 2019-03-19 Data synchronization method and device and computer equipment

Country Status (1)

Country Link
CN (1) CN111726375A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112818060A (en) * 2021-01-29 2021-05-18 北京百度网讯科技有限公司 Data synchronization method and device, electronic equipment and readable storage medium
CN112860708A (en) * 2021-03-16 2021-05-28 深圳前海微众银行股份有限公司 Data consistency restoration method, system, device and equipment

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103220342A (en) * 2013-03-29 2013-07-24 厦门雅迅网络股份有限公司 Grouped remote and synchronous program updating method for terminals
CN104881485A (en) * 2015-06-04 2015-09-02 中国银行股份有限公司 Data synchronizing method and system
CN106599195A (en) * 2016-12-14 2017-04-26 北京邮电大学 Method and system for synchronizing metadata under mass network data environment
CN106973099A (en) * 2017-03-28 2017-07-21 广东欧珀移动通信有限公司 A kind of data-updating method, apparatus and system
CN107172169A (en) * 2017-05-27 2017-09-15 广东欧珀移动通信有限公司 Method of data synchronization, device, server and storage medium
CN107426309A (en) * 2017-07-17 2017-12-01 北京深思数盾科技股份有限公司 A kind of information synchronization method, apparatus and system
CN108769172A (en) * 2018-05-21 2018-11-06 杭州有赞科技有限公司 A kind of method of data synchronization and system
CN109413127A (en) * 2017-08-18 2019-03-01 北京京东尚科信息技术有限公司 A kind of method of data synchronization and device

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103220342A (en) * 2013-03-29 2013-07-24 厦门雅迅网络股份有限公司 Grouped remote and synchronous program updating method for terminals
CN104881485A (en) * 2015-06-04 2015-09-02 中国银行股份有限公司 Data synchronizing method and system
CN106599195A (en) * 2016-12-14 2017-04-26 北京邮电大学 Method and system for synchronizing metadata under mass network data environment
CN106973099A (en) * 2017-03-28 2017-07-21 广东欧珀移动通信有限公司 A kind of data-updating method, apparatus and system
CN107172169A (en) * 2017-05-27 2017-09-15 广东欧珀移动通信有限公司 Method of data synchronization, device, server and storage medium
CN107426309A (en) * 2017-07-17 2017-12-01 北京深思数盾科技股份有限公司 A kind of information synchronization method, apparatus and system
CN109413127A (en) * 2017-08-18 2019-03-01 北京京东尚科信息技术有限公司 A kind of method of data synchronization and device
CN108769172A (en) * 2018-05-21 2018-11-06 杭州有赞科技有限公司 A kind of method of data synchronization and system

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112818060A (en) * 2021-01-29 2021-05-18 北京百度网讯科技有限公司 Data synchronization method and device, electronic equipment and readable storage medium
CN112818060B (en) * 2021-01-29 2024-04-19 北京百度网讯科技有限公司 Data synchronization method, device, electronic equipment and readable storage medium
CN112860708A (en) * 2021-03-16 2021-05-28 深圳前海微众银行股份有限公司 Data consistency restoration method, system, device and equipment
CN112860708B (en) * 2021-03-16 2024-05-24 深圳前海微众银行股份有限公司 Data consistency restoration method, system, device and equipment

Similar Documents

Publication Publication Date Title
US7500020B1 (en) Coherency of replicas for a distributed file sharing system
US20180150501A1 (en) Database system, server device, computer program product, and information processing method
US10069941B2 (en) Scalable event-based notifications
CN105592139A (en) HA implementation method and device of distributed file system management platform
US9529772B1 (en) Distributed caching cluster configuration
CN111726375A (en) Data synchronization method and device and computer equipment
CN110213331B (en) Service request processing method, terminal device, electronic device and storage medium
CN113094430B (en) Data processing method, device, equipment and storage medium
CN112632029B (en) Data management method, device and equipment of distributed storage system
CN111694865A (en) Four-layer structure data acquisition method and device based on distributed system
WO2017097181A1 (en) Data pushing method and apparatus
CN111159233A (en) Distributed caching method, system, computer device and storage medium
CN107025257B (en) Transaction processing method and device
CN106533751B (en) SDN controller cluster merging method and device
CN111752577B (en) Upgrading method and equipment for system version
CN113190619B (en) Data read-write method, system, equipment and medium for distributed KV database
CN113596187B (en) Domain name configuration management system
CN113268540B (en) Data synchronization method and device
CN111382132A (en) Medical image data cloud storage system
US20240061754A1 (en) Management of logs and cache for a graph database
CN110851527B (en) Data synchronization method for main and standby servers
JP4224289B2 (en) Data replication management method
CN113391759B (en) Communication method and equipment
CN107819858B (en) Method and device for managing cloud service during dynamic expansion and contraction of cloud service
CN113094074A (en) Service cluster updating method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20200929

RJ01 Rejection of invention patent application after publication