CN106528574A - Data synchronization method and device - Google Patents
Data synchronization method and device Download PDFInfo
- Publication number
- CN106528574A CN106528574A CN201510584004.3A CN201510584004A CN106528574A CN 106528574 A CN106528574 A CN 106528574A CN 201510584004 A CN201510584004 A CN 201510584004A CN 106528574 A CN106528574 A CN 106528574A
- Authority
- CN
- China
- Prior art keywords
- data
- node
- task
- synchronized
- data acquisition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/273—Asynchronous replication or reconciliation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
- G06F16/1824—Distributed file systems implemented using Network-attached Storage [NAS] architecture
- G06F16/1827—Management specifically adapted to NAS
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data synchronization method. When a cluster node receives a data obtaining message, a data obtaining task corresponding to the data obtaining message is generated, and to-be-synchronized data is obtained from a database according to the data obtaining task. The data obtaining message is sent to all cluster nodes in a cluster after the to-be-synchronized data is processed and written in the database by a node which receives the to-be-synchronized data, so that on the basis of ensuring configuration data of the nodes in the cluster to be kept consistent, the data synchronization efficiency is improved and the resources are saved.
Description
Technical field
The application is related to communication technical field, more particularly to a kind of method of data synchronization.The application also simultaneously
It is related to a kind of server.
Background technology
With the development of distributed system, the continuous expansion of Network and the network user of rapid growth
Quantity, a distributed operation system generally require tens, up to a hundred or thousands of equipment to prop up
Support existing extensive business.After some client connects certain equipment, need for user is subjective
Ask or objective factor, client is possible to change the state of this equipment or the equipment run
A certain attribute configuration in program, due to current operation system be distributed system, therefore cluster in its
His node needs to know occurred change and the configuration before and after change.Send out in distributed business system
During raw consecutive variations, it is ensured that can be known between clustered node at any time and there occurs change, and with it is newest
Configuration is synchronous.
Current technology typically takes the synchronization that following three kinds of modes are carried out between cluster, its respective characteristic
And shortcoming is as follows：
(1) synchronization based on broadcast
The existing reproduction process based on broadcast is as shown in figure 1, when certain server is connected to client request
When, the server changes this node state, and subsequently changing content is directly synchronized to by this node based on broadcast
The brotgher of node, so that the brotgher of node keeps content consistent.
However, the simultaneous techniquess based on broadcast are, based on udp broadcast, to it requires all clustered node positions
In the same network segment, and must be that IP broadcast (UDP) is accessibility.In addition, the synchronous skill based on broadcast
Art be a node will to all of node synchrodata, when synchrodata amount than it is larger when, to Netowrk tape
Wide and performance impact is larger, and increasing with clustered node number, and the nodes of cluster are more, needs
Synchronous machine is more, so as to the ability of linear extendible is subject to severely restricts.
(2) leaving concentratedly based on DB Backup
It is existing to be sent out as shown in Fig. 2 being connected to certain node in client based on centralized database storage configuration
Rise change request after, the node directly change write into Databasce, other nodes have request needs obtain
Then directly obtain from data base when taking this newest configuration item.It is not direct between each node in the program
Synchrodata, but centralized stores are configured by data base's persistence, so as to accomplish that each nodes sharing is matched somebody with somebody
Put.
As the essence of leaving concentratedly based on DB Backup is that the shared data of cluster are directly stored in data
Storehouse, when the node in cluster needs constantly to obtain this configuration, then can carry out no small pressure to data library tape
Power, so that bring impact on stability.
(3) peer synchronization mode
As shown in figure 3, in existing Peer, every server will arbitrarily select a server standby
Session information in part its internal memory, so every server have a peer server of oneself, and not
It is other all of servers, this mode is eliminated if add in the cluster excessive server instance to be affected
The problem of retractility.Correspondingly, more complexity are brought to load equalizer based on peer synchronization mode
Property.After a service failure, it is this own failed server that load equalizer must be known by that service
Peer backup server.This will reduce the range of choice of load equalizer, while some hardware are not yet
This requirement can be met.And in addition to processing normal request, server also will be responsible for replicating for task.
As the task of backup session data is also required to take the cycle of CPU, so at the request of every server
Reason ability is also reduced
As can be seen here, how the node configuration variation in distributed business system frequently in the case of, protect
Card clustered node can real-time synchronization update newest configuration data, become those skilled in the art and urgently solve
Technical problem certainly.
The content of the invention
This application provides a kind of data-updating method.To improve notice efficiency and save equipment money
On the premise of source, the configuration and attribute data for enabling the node in cluster is obtained timely updates.The party
Method is applied in the cluster for include multiple clustered nodes and data base, and the clustered node includes receiving
The data write node of data to be synchronized and data pull node, comprise the following steps：
When clustered node receives data acquisition message, the clustered node is generated and the data acquisition
The corresponding data acquisition task of message；Wherein, the data acquisition message is that data write node exists
The data processing to be synchronized is finished and backward itself and the data pull of the data base is write
What node sent；
The clustered node obtains the number to be synchronized from the data base according to the data acquisition task
According to.
Preferably, before the clustered node receives data acquisition message, also include：
If the clustered node is the data writing node, the data write node receives the client
The data described to be synchronized that end sends, and according to the type of the data to be synchronized to the data to be synchronized
Processed；
Data write node generates mark corresponding with the data to be synchronized, and by the mark with
And the data correspondence to be synchronized writes the data base；
The data write node is arranged and trigger data synchronous event；
When data write node listens to the data synchronization event, the data write node to
All clustered nodes in the cluster send the data acquisition message for carrying the mark.
Preferably, write node and send to itself and the data pull node in the data and carry institute
After stating the data acquisition message of mark, also include：
The data write node judges whether the data pull node is properly received the data acquisition and disappears
Breath；
If the data pull node is not successfully received the data acquisition message, the data write node
Rear in default time threshold resends the data acquisition message to the data pull node.
Preferably, the node is treated according to the data acquisition task is obtained from the data base together
Step data, specially：
During the data acquisition task to be added the clustered node task queue of itself；
The clustered node is monitored the task queue and currently whether there is data acquisition task；
If existing, the clustered node is according to the corresponding mark of the data acquisition task from the data base
Middle to obtain the data to be synchronized, the mark is carried in the data acquisition message, and the mark exists
Before the data to be synchronized are written into the data base, correspondence is generated.
Preferably, the clustered node adds the data acquisition task in the task queue of itself, tool
Body is：
The clustered node whether there is configuration and the data acquisition task one in judging the task queue
The duplicate data of cause obtains task；
If existing, the duplicate data is obtained task replacement and is appointed for the data acquisition by the clustered node
Business；
If not existing, the clustered node adds the data acquisition task into the task queue.
Preferably, institute is obtained from the data base according to the data acquisition task in the clustered node
After stating data to be synchronized, also include：
If the clustered node is restarted, the clustered node by current local all internal storage datas with
And data cached deletion；
The clustered node obtains corresponding with itself internal storage data and data cached from the data base,
And initialized according to acquired data.
Preferably, also include：
If the clustered node is obtained from the data base according to the corresponding mark of the data acquisition task
The data acquisition task is updated weight after timestamp by the data failure to be synchronized, the clustered node
The task queue is added newly.
Correspondingly, the application also proposed a kind of server, and the server is applied to bag as clustered node
Include in the cluster of multiple clustered nodes and data base, the clustered node includes receiving data to be synchronized
Data write node and data pull node, the server includes：
Generation module, when data acquisition message is received, generates corresponding with the data acquisition message
Data acquisition task；Wherein, the data acquisition message is that data write node is treating same by described
Step data is disposed and writes rear all clustered nodes transmissions in the cluster of the data base
's；
Acquisition module, obtains the data to be synchronized from the data base according to the data acquisition task.
Preferably, also include：
Processing module, receives the data described to be synchronized that client sends, and according to the data to be synchronized
Type the data to be synchronized are processed；
Writing module, generates mark corresponding with the data to be synchronized, and by the mark and described
Data correspondence to be synchronized writes the data base；
Setup module, is arranged and trigger data synchronous event；
Sending module, the Servers-all when the data synchronization event is listened to, in the cluster
Send the data acquisition message for carrying the mark.
Preferably, also include：
Judge module, judges whether the data pull node is properly received the data acquisition message, and
Indicate the sending module pre- when the data pull node is not successfully received the data acquisition message
If the backward data pull node of time threshold resend the data acquisition message.
Preferably, the acquisition module is specifically included：
Addition submodule, the data acquisition task is added in the task queue of itself；
Submodule is monitored, the task queue is monitored and currently be whether there is data acquisition task；
Acquisition submodule, monitors the task queue in the monitoring submodule and there is currently data acquisition times
The number to be synchronized is obtained from the data base according to the corresponding mark of the data acquisition task during business
According to the mark is carried in the data acquisition message, and the mark is write in the data to be synchronized
Before entering the data base, correspondence is generated.
Preferably, it is described addition submodule specifically for：
Judge whether that configuring the duplicate data consistent with the data acquisition task obtains task；
If existing, it is the data acquisition task that the duplicate data is obtained task replacement；
If not existing, the data acquisition task is added into the task queue.
Preferably, also include：
Removing module, when the server is restarted, by current local all internal storage datas and slow
Deposit data is deleted；
Initialization module, obtains corresponding with itself internal storage data and data cached from the data base,
And initialized according to acquired data.
Preferably, also include：
Weight processing module, when the server according to the corresponding mark of the data acquisition task from the number
According to the data failure to be synchronized is obtained in storehouse, the data acquisition task is updated after timestamp again
Add the task queue.
As can be seen here, by the technical scheme using the application, disappear when clustered node receives data acquisition
During breath, data acquisition task corresponding with the data acquisition message is generated, subsequently appointed according to data acquisition
Business obtains data to be synchronized from data base, as data acquisition message is the number that receives data to be synchronized
According to write node in rear all clusters in cluster that data processing to be synchronized is finished simultaneously write into Databasce
Node sends, on the basis of being consistent in the configuration data for ensureing each clustered node in cluster,
Improve.
Description of the drawings
Fig. 1 is the existing reproduction process schematic diagram based on broadcast；
Fig. 2 deposits configuration schematic diagram for existing based on centralized database；
Fig. 3 is existing peer synchronization schematic diagram；
Fig. 4 is a kind of schematic flow sheet of method of data synchronization that the application is proposed；
Fig. 5 is the schematic flow sheet of cluster synchronization in the application specific embodiment；
Fig. 6 be the application specific embodiment in load schematic flow sheet；
Fig. 7 is the cluster synchronization mode flow chart of lightweight in the application specific embodiment；
Fig. 8 is a kind of server module composition schematic diagram that the application is proposed.
Specific embodiment
In view of the problem in background technology, present applicant proposes a kind of method of data synchronization, to realize
The cluster synchronization mode of lightweight, so as to effectively ensure in cluster on the basis of performance is taken into account each
Clustered node content is consistent.
For ease of distinguishing the clustered node of different role and function, for certain writing events once, this
Application will receive data to be synchronized and the node of the data write into Databasce to be synchronized be referred to as data
Node is write, now other clustered nodes in cluster are then referred to as counted for data write node
According to pulling node.
As indicated at 4, be the application propose a kind of method of data synchronization schematic flow sheet, the method should
Treat same for, in the cluster including multiple clustered nodes and data base, the clustered node includes receiving
The data write node of step data and data pull node, comprise the following steps：
S401, when clustered node receives data acquisition message, the clustered node is generated and the number
According to the corresponding data acquisition task of acquisition message；Wherein, the data acquisition message is write for the data
The data processing to be synchronized is being finished and is writing the rear institute in the cluster of the data base by node
Have what clustered node sent.
As stated in the Background Art, from the clustered node for receiving synchrodata or from data base simultaneously to all
Other clustered nodes write data to be synchronized and will increase the burden of clustered node and data base, and expend
Resource will also become many.In order to solve the technical problem, the application ought have client to connect
When change request is initiated to certain clustered node, indicate that the clustered node will first change write into Databasce first,
And while trigger the event of a configuration change, rather than to each clustered node broadcast data synchronization.Phase
Ying Di, all clustered nodes (including the node and other nodes) in the cluster listen to configuration change
After event, data syn-chronization is not directly carried out as in the prior art, but with the advice method of lightweight
Tell that other clustered nodes go data base to obtain the configuration of change,.
In the application preferred embodiment, if the clustered node is the data writes node, then should
Clustered node is to receive the notice flow process before data acquisition message as follows：
Step a) the data write nodes receive the data described to be synchronized that the client sends, and root
The data to be synchronized are processed according to the type of the data to be synchronized；
Step b) data write node generates mark corresponding with the data to be synchronized, and will be described
Mark and the data correspondence write to be synchronized data base；
Step c) the data write nodes are arranged and trigger data synchronous event；
When data write node listens to the data synchronization event, the data write step d)
Node sends the data acquisition message for carrying the mark to all clustered nodes in the cluster.
By the advice method triggered using above event, synchronous cost is significantly reduced, and is improve
The linear extendible ability of cluster.And the allocating cache for receiving is in this node memory or disk, so after
It is continuous to need to obtain this configuration when client needs to obtain this configuration or the node, then directly from local
Internal memory or the corresponding data of local disk direct access, it is to avoid directly access the database.
It should be noted which is only a kind of preferred implementation that the application is provided, section is being realized
The data processing to be synchronized of received client can be finished by point and to write the data base backward
On the basis of all clustered nodes in cluster send data acquisition message, those skilled in the art can also
Other similar implementation methods are taken, these belong to the protection domain of the application.
Additionally, for the data acquisition message for having sent, whether data write node also will be sent to for which
Judged to other clustered nodes.In the application preferred embodiment, data write node judges number
According to node is pulled, whether successful data reception obtains message, if data pull node is not successfully received data obtaining
Message is taken, data write node resends data in the rear of default time threshold to data pull node
Obtain message.
S402, the clustered node are treated according to the data acquisition task is obtained from the data base
Synchrodata.
Due to the application by lightweight information come reduce the communication resource burden, therefore with prior art in take
Band data except that, only comprising mark corresponding with data to be synchronized in the data acquisition message of the application
Know, the mark was generated before by data write into Databasce to be synchronized by data write node, subsequently other
Clustered node is directly inquired about corresponding data to be synchronized in data base by the mark and synchronizes place
Reason.
In the application preferred embodiment, in order that each clustered node can in an orderly manner from data base
Data to be synchronized are obtained, during data acquisition task is added the task queue of itself by clustered node first, so
Task queue is monitored afterwards and currently whether there is data acquisition task, and obtained according to data in case of presence
Take the corresponding mark of task and obtain data to be synchronized from data base, the mark is carried on data acquisition message
In and before data to be synchronized are written into data base correspondence generate.
It should be noted that data to be synchronized involved in above scheme can be related to clustered node
Configuration data, or client needs the data storage being stored in clustered node, or
Node processing is write by client upload to data and needs to preserve the data of result.Additionally, as visitor
After the request of family end changes the configuration of the corresponding server of certain clustered node, if the configuration needs simultaneously
When applying other clustered nodes in other clusters, can also be by the scheme of the application by client institute
The setting of change is synchronized to the corresponding server of other clustered nodes, specifically, technical scheme
In " data to be synchronized " at least include following several types：
(1) data to be synchronized are that user passes through client upload to the data of data write node, the data
After data base is written into, need other clustered nodes to be synchronized to itself.The upload data can be in the number
According to the related configuration data of the system run in write node, user's answering in using system or system
With the related configuration data for producing, the configuration number being configured for clustered node in itself during program
According to this and common needs storage data (such as document, picture, multimedia file etc.) etc.,
So user can make collection only by client to after clustered node upload configuration data in cluster
Other clustered nodes in group are quickly taken and the clustered node identical configuration data, and such user is follow-up
System in using other clustered nodes or before can adopting when running application program therein it is identical
Using configuration.
(2) data to be synchronized are the data that user is modified to clustered node by client, work as modification
After end, the result (the processed data to be synchronized for finishing) of modification needs which after being written into data base
He is synchronized to itself by clustered node.
(3) (request may for the request that data to be synchronized are sent to clustered node by client for user
Directly to change data) in entrained critical data or the critical data generated according to the request.
For example, clustered node is needed in the logical process of program after the request of the client that is disposed
Certain the ID value (such as ID) in the request is preserved, in being written into data base and notifying cluster
Other clustered node synchronizations；Or the root in the logical process of program is needed after the request that is disposed
Another critical data (ID) is generated according to the value (such as ID) in the request, the critical data that will be generated
Write into Databasce other clustered node synchronizations in notifying cluster.
If additionally, configuration item continuously change causes clustered node to continuously receive multiple data acquisitions to disappear
(for example there are a large amount of changes) during breath, the application can ensure data base by the ability of Database lock itself
Middle storage is newest configuration.Meanwhile, clustered node is generating data acquisition according to data acquisition message
Whether there is similarly configuring for task during task queue can be judged after task, and appoint new task replacement old
Business.So as to ensure to avoid repetition from obtaining out-of-date data, in the application preferred embodiment, cluster section
Put when the data acquisition task to be added the task queue of itself, in first determining whether the task queue
Task is obtained with the presence or absence of the duplicate data consistent with the data acquisition task is configured；By institute if existing
It is the data acquisition task to state duplicate data and obtain task replacement；By the data acquisition if not existing
Task is added into the task queue.Additionally, corresponding according to data acquisition task clustered node is worked as
When mark obtains data failure to be synchronized from data base, clustered node also can be by the data acquisition task more
Task queue is rejoined after new timestamp.
During daily use, the clustered node in cluster is probably due to itself or other objective originals
Thus result in and restart, after having clustered node to restart, the clustered node needs the full dose from data base
Loading latest data.In the preferred embodiment of the application, if clustered node is restarted, the cluster section
Current local all internal storage datas and data cached deletion are obtained and itself by point from the data base
Corresponding internal storage data and data cached, and initialized according to acquired data.
By adopting above scheme, after certain clustered node in cluster is by configuration change write into Databasce,
It is provided for indicating the trigger event that other clustered nodes obtain change configuration from data base, clustered node exists
Listen to the rear of the trigger event and data acquisition message is sent to other clustered nodes, if certain cluster section
Point obtains the data acquisition message failure, then data acquisition message is thrown by clustered node again that write data
It is handed to the clustered node for obtaining message failure, it is ensured that all of clustered node can receive loading messages.
So as on the premise of save resources, it is ensured that the final consistency of clustered node local disk.In the party
In case, the data write for writing configuration can be notified by way of directly notifying or sending a message to bus
Other clustered nodes are synchronized, and in former mode, pass through Http between each clustered node of cluster
Call notification directly notify other side's synchronization；And in latter approach, data write node and will configure and become
A message is issued after more write into Databasce to messaging bus, and other all clustered nodes disappear as subscribing to
Breath, starts synchronously after having listened to such message, and these belong to the protection domain of the application.
In order to the technological thought of the application is expanded on further, in conjunction with specific application scenarios, to the application
Technical scheme illustrate.
As shown in figure 5, for the schematic flow sheet of cluster synchronization in the application specific embodiment, concrete interaction
Process is as follows：
Step 1.1, client initiate the request of modification；
Step 2.1, after certain node in cluster is connected to request, will directly change write into Databasce；
Step 2.2, after the node is connected to the request that write is completed, goes to step 2.3；
Step 2.3, the node initiate the event of a configuration item change；
Step 2.4, the node listen to configuration item change, go to step 3.1；
Step 3.1, notifies this node and other brotgher of node loading latest configuration changes；
After step 3.2, the brotgher of node and this node have notice, data acquisition task is generated, and is added
Task queue node monitors task queue, if any task, then carries out pulling configuration by task
Step 3.3, after pulling configuration change, is cached to internal memory or is saved in local disk.
In the above process, when client has continuous change request, protected by the lock of data base itself
Card is concurrent, so as to ensure that last renewal can be persisted in data base, when loading from data base,
It is newest configuration forever so as to ensure to load.
When certain Server is restarted, need the configuration for being buffered in local node to need from data base entirely
Amount loads out, so as to ensure the newest of local disk caching, in flow chart as shown in Figure 6, plus
The flow process of load is as follows：
Step 1. manager restarts Server
Step 2.Server deletes the allocating cache on all of disk in restarting process, from disk.
Step 3.Server loads all of configuration item from database full-volume, and is saved in disk
Step 4. other initialization.
The startup of step 5.Server is completed, and externally provides service
When step 6. client or this node need configuration information, directly obtain from disk buffering.
If in Server A start-up courses, other nodes there occurs configuration change, but Server A add
Before load terminates, this configuration is not also put in storage, or when another node notifies ServerA because net
Network reason, causes to notify that failure, the cluster synchronization mode of lightweight can constantly be led to by other nodes
Know trial to ensure that this renewal is synchronized to Server A, the flow process is as shown in fig. 7, comprises following steps：
1.ServerB notifies that Server A nodes are configured from database synchronization
2., because network or other reasonses, failure is notified
After 3.Server B are connected to the result of failure, at regular intervals, initiate again to notify
4.ServerB notifies that ServerA nodes are configured from database synchronization
5.Server A have notice after request, generate loading tasks and are put into load queue.
6.Server B have notice.
7.ServerA monitors load queue, and discovery has loading tasks, starts from data base's loading is corresponding to match somebody with somebody
Put.
8.ServerA has loaded configuration, and has updated local disk caching.
For reaching above technical purpose, the application also proposed a kind of server, and the server is used as section
Point is applied in the cluster for include multiple nodes and data base, as shown in figure 8, the server includes：
Generation module 810, when data acquisition message is received, generates corresponding with the data acquisition message
Data acquisition task；Wherein, the data acquisition message is that data write node is being treated described
Synchrodata is disposed and writes rear all clustered nodes transmissions in the cluster of the data base
's；
Acquisition module 820, obtains the number to be synchronized from the data base according to the data acquisition task
According to.
In specific application scenarios, also include：
Processing module, receives the data described to be synchronized that client sends, and according to the data to be synchronized
Type the data to be synchronized are processed；
Writing module, generates mark corresponding with the data to be synchronized, and by the mark and described
Data correspondence to be synchronized writes the data base；
Setup module, is arranged and trigger data synchronous event；
Sending module, the Servers-all when the data synchronization event is listened to, in the cluster
Send the data acquisition message for carrying the mark.
In specific application scenarios, also include：
Judge module, judges whether the data pull node is properly received the data acquisition message, and
Indicate the sending module pre- when the data pull node is not successfully received the data acquisition message
If the backward data pull node of time threshold resend the data acquisition message.
In specific application scenarios, the acquisition module is specifically included：
Addition submodule, the data acquisition task is added in the task queue of itself；
Submodule is monitored, the task queue is monitored and currently be whether there is data acquisition task；
Acquisition submodule, monitors the task queue in the monitoring submodule and there is currently data acquisition times
The number to be synchronized is obtained from the data base according to the corresponding mark of the data acquisition task during business
According to the mark is carried in the data acquisition message, and the mark is write in the data to be synchronized
Before entering the data base, correspondence is generated.
In specific application scenarios, it is described addition submodule specifically for：
Judge whether that configuring the duplicate data consistent with the data acquisition task obtains task；
If existing, it is the data acquisition task that the duplicate data is obtained task replacement；
If not existing, the data acquisition task is added into the task queue.
In specific application scenarios, also include：
Removing module, when the server is restarted, by current local all internal storage datas and slow
Deposit data is deleted；
Initialization module, obtains corresponding with itself internal storage data and data cached from the data base,
And initialized according to acquired data.
In specific application scenarios, also include：
Weight processing module, when the server according to the corresponding mark of the data acquisition task from the number
According to the data failure to be synchronized is obtained in storehouse, the data acquisition task is updated after timestamp again
Add the task queue.
Through the above description of the embodiments, those skilled in the art can be understood that this Shen
Please be realized by hardware, it is also possible to by software plus necessary general hardware platform mode realizing.
Based on such understanding, the technical scheme of the application can be embodied in the form of software product, and this is soft
It (can be CD-ROM, USB flash disk, movement are hard that part product can be stored in a non-volatile memory medium
Disk etc.) in, use including some instructions so that a computer equipment (can be personal computer, take
Business device, or the network equipment etc.) perform method described in the application each implement scene.
It will be appreciated by those skilled in the art that accompanying drawing is a schematic diagram for being preferable to carry out scene, in accompanying drawing
Module or flow process not necessarily implement necessary to the application.
It will be appreciated by those skilled in the art that the module in device in implement scene can be according to implement scene
Description carries out being distributed in the device of implement scene, it is also possible to carries out respective change and is disposed other than this enforcement
In one or more devices of scene.The module of above-mentioned implement scene can merge into a module, also may be used
To be further split into multiple submodule.
Above-mentioned the application sequence number is for illustration only, does not represent the quality of implement scene.
Disclosed above is only that the several of the application are embodied as scene, but, the application is not limited to
This, the changes that any person skilled in the art can think of should all fall into the protection domain of the application.
Claims (14)
1. a kind of method of data synchronization, it is characterised in that methods described is applied to include multiple clustered nodes
And in the cluster of data base, the clustered node includes the data write node for receiving data to be synchronized
With data pull node, the method includes：
When clustered node receives data acquisition message, the clustered node is generated and the data acquisition
The corresponding data acquisition task of message；Wherein, the data acquisition message is that data write node exists
The data processing to be synchronized is finished and all clusters in the backward cluster of the data base are write
What node sent；
The clustered node obtains the number to be synchronized from the data base according to the data acquisition task
According to.
2. the method for claim 1, it is characterised in that receive data in the clustered node
Before obtaining message, also include：
If the clustered node is the data writing node, the data write node receives the client
The data described to be synchronized that end sends, and according to the type of the data to be synchronized to the data to be synchronized
Processed；
Data write node generates mark corresponding with the data to be synchronized, and by the mark with
And the data correspondence to be synchronized writes the data base；
The data write node is arranged and trigger data synchronous event；
When data write node listens to the data synchronization event, the data write node to
All clustered nodes in the cluster send the data acquisition message for carrying the mark.
3. method as claimed in claim 2, it is characterised in that write node to itself in the data
And after the data pull node sends the data acquisition message for carrying the mark, also include：
The data write node judges whether the data pull node is properly received the data acquisition and disappears
Breath；
If the data pull node is not successfully received the data acquisition message, the data write node
Rear in default time threshold resends the data acquisition message to the data pull node.
4. the method for claim 1, it is characterised in that the clustered node is according to the data
Acquisition task obtains the data to be synchronized from the data base, specially：
During the data acquisition task to be added the clustered node task queue of itself；
The clustered node is monitored the task queue and currently whether there is data acquisition task；
If existing, the clustered node is according to the corresponding mark of the data acquisition task from the data base
Middle to obtain the data to be synchronized, the mark is carried in the data acquisition message, and the mark exists
Before the data to be synchronized are written into the data base, correspondence is generated.
5. method as claimed in claim 4, it is characterised in that the data are obtained by the clustered node
During the task of taking adds the task queue of itself, specially：
The clustered node whether there is configuration and the data acquisition task one in judging the task queue
The duplicate data of cause obtains task；
If existing, the duplicate data is obtained task replacement and is appointed for the data acquisition by the clustered node
Business；
If not existing, the clustered node adds the data acquisition task into the task queue.
6. the method for claim 1, it is characterised in that in the clustered node according to the number
After the data to be synchronized are obtained from the data base according to acquisition task, also include：
If the clustered node is restarted, the clustered node by current local all internal storage datas with
And data cached deletion；
The clustered node obtains corresponding with itself internal storage data and data cached from the data base,
And initialized according to acquired data.
7. method as claimed in claim 6, it is characterised in that also include：
If the clustered node is obtained from the data base according to the corresponding mark of the data acquisition task
The data acquisition task is updated weight after timestamp by the data failure to be synchronized, the clustered node
The task queue is added newly.
8. a kind of server, it is characterised in that the server is applied to include multiple as clustered node
In the cluster of clustered node and data base, the clustered node includes that the data for receiving data to be synchronized are write
Ingress and data pull node, the server include：
Generation module, when data acquisition message is received, generates corresponding with the data acquisition message
Data acquisition task；Wherein, the data acquisition message is that data write node is treating same by described
Step data is disposed and writes rear all clustered nodes transmissions in the cluster of the data base
's；
Acquisition module, obtains the data to be synchronized from the data base according to the data acquisition task.
9. server as claimed in claim 8, also includes：
Processing module, receives the data described to be synchronized that client sends, and according to the data to be synchronized
Type the data to be synchronized are processed；
Writing module, generates mark corresponding with the data to be synchronized, and by the mark and described
Data correspondence to be synchronized writes the data base；
Setup module, is arranged and trigger data synchronous event；
Sending module, the Servers-all when the data synchronization event is listened to, in the cluster
Send the data acquisition message for carrying the mark.
10. server as claimed in claim 9, it is characterised in that also include：
Judge module, judges whether the data pull node is properly received the data acquisition message, and
Indicate the sending module pre- when the data pull node is not successfully received the data acquisition message
If the backward data pull node of time threshold resend the data acquisition message.
11. servers as claimed in claim 8, it is characterised in that the acquisition module is specifically included：
Addition submodule, the data acquisition task is added in the task queue of itself；
Submodule is monitored, the task queue is monitored and currently be whether there is data acquisition task；
Acquisition submodule, monitors the task queue in the monitoring submodule and there is currently data acquisition times
The number to be synchronized is obtained from the data base according to the corresponding mark of the data acquisition task during business
According to the mark is carried in the data acquisition message, and the mark is write in the data to be synchronized
Before entering the data base, correspondence is generated.
12. servers as claimed in claim 11, it is characterised in that the addition submodule is specifically used
In：
Judge whether that configuring the duplicate data consistent with the data acquisition task obtains task；
If existing, it is the data acquisition task that the duplicate data is obtained task replacement；
If not existing, the data acquisition task is added into the task queue.
13. servers as claimed in claim 8, it is characterised in that also include：
Removing module, when the server is restarted, by current local all internal storage datas and slow
Deposit data is deleted；
Initialization module, obtains corresponding with itself internal storage data and data cached from the data base,
And initialized according to acquired data.
14. servers as claimed in claim 6, it is characterised in that also include：
Weight processing module, when the server according to the corresponding mark of the data acquisition task from the number
According to the data failure to be synchronized is obtained in storehouse, the data acquisition task is updated after timestamp again
Add the task queue.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510584004.3A CN106528574A (en) | 2015-09-14 | 2015-09-14 | Data synchronization method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510584004.3A CN106528574A (en) | 2015-09-14 | 2015-09-14 | Data synchronization method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106528574A true CN106528574A (en) | 2017-03-22 |
Family
ID=58348461
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510584004.3A Pending CN106528574A (en) | 2015-09-14 | 2015-09-14 | Data synchronization method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106528574A (en) |
Cited By (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107070730A (en) * | 2017-06-16 | 2017-08-18 | 郑州云海信息技术有限公司 | A kind of startup method and apparatus of CTDB |
| CN107087038A (en) * | 2017-06-29 | 2017-08-22 | 珠海市魅族科技有限公司 | A kind of method of data syn-chronization, synchronizer, device and storage medium |
| CN108696595A (en) * | 2018-05-28 | 2018-10-23 | 郑州云海信息技术有限公司 | Distributed type assemblies method of data synchronization, master node, slave node, system and medium |
| CN108833569A (en) * | 2018-06-27 | 2018-11-16 | 郑州云海信息技术有限公司 | A kind of method of data synchronization and relevant apparatus |
| CN108875016A (en) * | 2018-06-20 | 2018-11-23 | 上海百林通信网络科技服务股份有限公司 | A kind of sample technology of sharing and evaluation method based on face recognition application |
| CN109167810A (en) * | 2018-07-27 | 2019-01-08 | 阿里巴巴集团控股有限公司 | Monitoring, notice, method for refreshing and device calculate equipment and storage medium |
| CN109165261A (en) * | 2018-10-10 | 2019-01-08 | 深圳市欧瑞博科技有限公司 | Method of data synchronization, device, server and computer storage medium |
| CN109639777A (en) * | 2018-11-28 | 2019-04-16 | 优刻得科技股份有限公司 | Data synchronous method, apparatus, system and non-volatile memory medium |
| CN109766347A (en) * | 2017-07-21 | 2019-05-17 | 腾讯科技（深圳）有限公司 | A kind of data-updating method, device, system, computer equipment and storage medium |
| CN110011855A (en) * | 2019-04-12 | 2019-07-12 | 苏州浪潮智能科技有限公司 | Distributed type assemblies roll up QoS adjusting method, device, equipment and storage medium |
| CN110021339A (en) * | 2017-12-27 | 2019-07-16 | 北京大学 | Cluster parallel computing accelerated method based on protein folding measuring and calculating protein structure |
| CN110471780A (en) * | 2019-08-21 | 2019-11-19 | 北京百佑科技有限公司 | Distributed event processing unit, terminal and computer storage medium |
| CN110580258A (en) * | 2019-09-20 | 2019-12-17 | 中国银行股份有限公司 | big data free query method and device |
| CN110795499A (en) * | 2019-09-17 | 2020-02-14 | 中国平安人寿保险股份有限公司 | Cluster data synchronization method, device and equipment based on big data and storage medium |
| CN110866062A (en) * | 2018-08-09 | 2020-03-06 | 菜鸟智能物流控股有限公司 | Data synchronization method and device based on distributed cluster |
| CN111142789A (en) * | 2019-11-29 | 2020-05-12 | 浪潮(北京)电子信息产业有限公司 | Method, device, equipment and storage medium for synchronizing configuration information between nodes |
| CN111628895A (en) * | 2020-05-28 | 2020-09-04 | 苏州浪潮智能科技有限公司 | Configuration data synchronization method, device, equipment and readable storage medium |
| CN111625511A (en) * | 2020-05-22 | 2020-09-04 | 广东浪潮大数据研究有限公司 | Configuration file management method and device, electronic equipment and storage medium |
| CN112612771A (en) * | 2020-11-24 | 2021-04-06 | 深圳市和讯华谷信息技术有限公司 | Data writing method and system |
| CN113064950A (en) * | 2021-03-18 | 2021-07-02 | 北京沃东天骏信息技术有限公司 | Data synchronization method, device, equipment and storage medium |
| CN113992669A (en) * | 2021-10-25 | 2022-01-28 | 哈尔滨理工大学 | Distributed data distribution method for trusted messages in industry |
| WO2022037173A1 (en) * | 2020-08-21 | 2022-02-24 | 苏州浪潮智能科技有限公司 | Method and apparatus for data persistence in storage cluster runtime, and device and medium |
| CN114697346A (en) * | 2022-04-22 | 2022-07-01 | 厦门飞鱼科技有限公司 | Method for data synchronization between server nodes |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040010510A1 (en) * | 2002-07-10 | 2004-01-15 | Timo Hotti | Method and system for database synchronization |
| US20120084286A1 (en) * | 2010-09-30 | 2012-04-05 | Verizon Patent And Licensing Inc. | Method and apparatus for group coordination of calendar events |
| CN102693324A (en) * | 2012-01-09 | 2012-09-26 | 西安电子科技大学 | Distributed database synchronization system, synchronization method and node management method |
| CN102722579A (en) * | 2012-06-07 | 2012-10-10 | 浪潮电子信息产业股份有限公司 | Bisynchronous method for internal database of cluster memory system |
| CN102902746A (en) * | 2012-09-18 | 2013-01-30 | 杭州勒卡斯广告策划有限公司 | Method, device and system for processing mass data |
| CN103248504A (en) * | 2012-02-06 | 2013-08-14 | 上海软智信息科技有限公司 | Cluster node matching method, cluster communicating module, equipment and system |
| CN104679611A (en) * | 2015-03-05 | 2015-06-03 | 浙江宇视科技有限公司 | Data resource copying method and device |
| CN104753994A (en) * | 2013-12-27 | 2015-07-01 | 杭州海康威视***技术有限公司 | Method and device for data synchronization based on cluster server system |
-
2015
- 2015-09-14 CN CN201510584004.3A patent/CN106528574A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040010510A1 (en) * | 2002-07-10 | 2004-01-15 | Timo Hotti | Method and system for database synchronization |
| US20120084286A1 (en) * | 2010-09-30 | 2012-04-05 | Verizon Patent And Licensing Inc. | Method and apparatus for group coordination of calendar events |
| CN102693324A (en) * | 2012-01-09 | 2012-09-26 | 西安电子科技大学 | Distributed database synchronization system, synchronization method and node management method |
| CN103248504A (en) * | 2012-02-06 | 2013-08-14 | 上海软智信息科技有限公司 | Cluster node matching method, cluster communicating module, equipment and system |
| CN102722579A (en) * | 2012-06-07 | 2012-10-10 | 浪潮电子信息产业股份有限公司 | Bisynchronous method for internal database of cluster memory system |
| CN102902746A (en) * | 2012-09-18 | 2013-01-30 | 杭州勒卡斯广告策划有限公司 | Method, device and system for processing mass data |
| CN104753994A (en) * | 2013-12-27 | 2015-07-01 | 杭州海康威视***技术有限公司 | Method and device for data synchronization based on cluster server system |
| CN104679611A (en) * | 2015-03-05 | 2015-06-03 | 浙江宇视科技有限公司 | Data resource copying method and device |
Cited By (33)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107070730A (en) * | 2017-06-16 | 2017-08-18 | 郑州云海信息技术有限公司 | A kind of startup method and apparatus of CTDB |
| CN107070730B (en) * | 2017-06-16 | 2020-03-31 | 郑州云海信息技术有限公司 | CTDB starting method and device |
| CN107087038A (en) * | 2017-06-29 | 2017-08-22 | 珠海市魅族科技有限公司 | A kind of method of data syn-chronization, synchronizer, device and storage medium |
| CN109766347A (en) * | 2017-07-21 | 2019-05-17 | 腾讯科技（深圳）有限公司 | A kind of data-updating method, device, system, computer equipment and storage medium |
| CN110021339B (en) * | 2017-12-27 | 2021-04-30 | 北京大学 | Cluster parallel computing acceleration method based on protein folding calculation protein structure |
| CN110021339A (en) * | 2017-12-27 | 2019-07-16 | 北京大学 | Cluster parallel computing accelerated method based on protein folding measuring and calculating protein structure |
| CN108696595A (en) * | 2018-05-28 | 2018-10-23 | 郑州云海信息技术有限公司 | Distributed type assemblies method of data synchronization, master node, slave node, system and medium |
| CN108875016A (en) * | 2018-06-20 | 2018-11-23 | 上海百林通信网络科技服务股份有限公司 | A kind of sample technology of sharing and evaluation method based on face recognition application |
| CN108833569A (en) * | 2018-06-27 | 2018-11-16 | 郑州云海信息技术有限公司 | A kind of method of data synchronization and relevant apparatus |
| CN109167810A (en) * | 2018-07-27 | 2019-01-08 | 阿里巴巴集团控股有限公司 | Monitoring, notice, method for refreshing and device calculate equipment and storage medium |
| CN110866062A (en) * | 2018-08-09 | 2020-03-06 | 菜鸟智能物流控股有限公司 | Data synchronization method and device based on distributed cluster |
| CN110866062B (en) * | 2018-08-09 | 2023-11-24 | 菜鸟智能物流控股有限公司 | Data synchronization method and device based on distributed cluster |
| CN109165261A (en) * | 2018-10-10 | 2019-01-08 | 深圳市欧瑞博科技有限公司 | Method of data synchronization, device, server and computer storage medium |
| CN109639777A (en) * | 2018-11-28 | 2019-04-16 | 优刻得科技股份有限公司 | Data synchronous method, apparatus, system and non-volatile memory medium |
| CN110011855A (en) * | 2019-04-12 | 2019-07-12 | 苏州浪潮智能科技有限公司 | Distributed type assemblies roll up QoS adjusting method, device, equipment and storage medium |
| CN110011855B (en) * | 2019-04-12 | 2023-01-10 | 苏州浪潮智能科技有限公司 | Distributed cluster volume QoS (quality of service) adjusting method, device, equipment and storage medium |
| CN110471780A (en) * | 2019-08-21 | 2019-11-19 | 北京百佑科技有限公司 | Distributed event processing unit, terminal and computer storage medium |
| CN110795499B (en) * | 2019-09-17 | 2024-04-16 | 中国平安人寿保险股份有限公司 | Cluster data synchronization method, device, equipment and storage medium based on big data |
| CN110795499A (en) * | 2019-09-17 | 2020-02-14 | 中国平安人寿保险股份有限公司 | Cluster data synchronization method, device and equipment based on big data and storage medium |
| CN110580258B (en) * | 2019-09-20 | 2022-05-17 | 中国银行股份有限公司 | Big data free query method and device |
| CN110580258A (en) * | 2019-09-20 | 2019-12-17 | 中国银行股份有限公司 | big data free query method and device |
| CN111142789A (en) * | 2019-11-29 | 2020-05-12 | 浪潮(北京)电子信息产业有限公司 | Method, device, equipment and storage medium for synchronizing configuration information between nodes |
| CN111625511B (en) * | 2020-05-22 | 2023-11-03 | 广东浪潮大数据研究有限公司 | Configuration file management method and device, electronic equipment and storage medium |
| CN111625511A (en) * | 2020-05-22 | 2020-09-04 | 广东浪潮大数据研究有限公司 | Configuration file management method and device, electronic equipment and storage medium |
| CN111628895A (en) * | 2020-05-28 | 2020-09-04 | 苏州浪潮智能科技有限公司 | Configuration data synchronization method, device, equipment and readable storage medium |
| CN111628895B (en) * | 2020-05-28 | 2022-06-17 | 苏州浪潮智能科技有限公司 | Configuration data synchronization method, device, equipment and readable storage medium |
| WO2022037173A1 (en) * | 2020-08-21 | 2022-02-24 | 苏州浪潮智能科技有限公司 | Method and apparatus for data persistence in storage cluster runtime, and device and medium |
| CN112612771A (en) * | 2020-11-24 | 2021-04-06 | 深圳市和讯华谷信息技术有限公司 | Data writing method and system |
| CN113064950A (en) * | 2021-03-18 | 2021-07-02 | 北京沃东天骏信息技术有限公司 | Data synchronization method, device, equipment and storage medium |
| CN113064950B (en) * | 2021-03-18 | 2024-04-16 | 北京沃东天骏信息技术有限公司 | Data synchronization method, device, equipment and storage medium |
| CN113992669A (en) * | 2021-10-25 | 2022-01-28 | 哈尔滨理工大学 | Distributed data distribution method for trusted messages in industry |
| CN114697346A (en) * | 2022-04-22 | 2022-07-01 | 厦门飞鱼科技有限公司 | Method for data synchronization between server nodes |
| CN114697346B (en) * | 2022-04-22 | 2024-06-11 | 厦门飞鱼科技有限公司 | Method for synchronizing data between server nodes |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106528574A (en) | Data synchronization method and device | |
| CN106331098B (en) | Server cluster system | |
| CN107465767B (en) | Data synchronization method and system | |
| US9495252B2 (en) | Continuous data backup using real time delta storage | |
| US8843581B2 (en) | Live object pattern for use with a distributed cache | |
| US10229181B2 (en) | System and method for synchronizing data between communication devices in a networked environment without a central server | |
| CN104796445B (en) | Server node carries out the method, apparatus of source synchronous | |
| CN106888245B (en) | Data processing method, device and system | |
| CN106570007A (en) | Method and equipment for data synchronization of distributed caching system | |
| CN113568624A (en) | Mirror image pulling method and system, computer equipment and readable storage medium | |
| CN105610947A (en) | Method, device and system for realizing high-available distributed queue service | |
| CN107231400B (en) | Data synchronization method and device | |
| CN106452836B (en) | main node setting method and device | |
| CN105956032B (en) | Data cached synchronous method, system and device | |
| CN105208058A (en) | Information exchange system based on web session sharing | |
| JP6405255B2 (en) | COMMUNICATION SYSTEM, QUEUE MANAGEMENT SERVER, AND COMMUNICATION METHOD | |
| CN112199192B (en) | Method and system for deploying Kubernetes cluster refined management quota based on server | |
| CN111538710A (en) | Method and system for real-time data synchronization based on rsync | |
| CN111726388A (en) | Cross-cluster high-availability implementation method, device, system and equipment | |
| CN107786650B (en) | Storage medium and management strategy synchronization method, device and system | |
| CN110324564B (en) | Video conference data synchronization method and device | |
| CN116185697B (en) | Container cluster management method, device and system, electronic equipment and storage medium | |
| CN111404978B (en) | Data storage method and cloud storage system | |
| CN110688228A (en) | Method, system and electronic equipment for guaranteeing distributed system message time sequence | |
| CN107846476B (en) | Information synchronization method, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170322 |
|
| RJ01 | Rejection of invention patent application after publication |