CN111049928B - Data synchronization method, system, electronic device and computer readable storage medium - Google Patents
Data synchronization method, system, electronic device and computer readable storage medium Download PDFInfo
- Publication number
- CN111049928B CN111049928B CN201911351527.8A CN201911351527A CN111049928B CN 111049928 B CN111049928 B CN 111049928B CN 201911351527 A CN201911351527 A CN 201911351527A CN 111049928 B CN111049928 B CN 111049928B
- Authority
- CN
- China
- Prior art keywords
- node
- master
- nodes
- slave
- master node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
- H04L67/1044—Group management mechanisms
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Theoretical Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a data synchronization method, a data synchronization system, electronic equipment and a computer readable storage medium, and belongs to the technical field of computers. In the method, for any master node, the master node performs data synchronization with other master nodes in a master node cluster under the condition that stored data are changed, if the master node comprises at least one slave node belonging to the master node, the master node performs data synchronization with the slave nodes belonging to the master node, and for any slave node, when a master node switching instruction is received, the other master nodes in the master node cluster are determined as the master nodes to which the slave nodes belong. Because the master node synchronizes data between the master nodes before synchronizing the data with the slave nodes, the master node to which the slave node is replaced can provide the latest data for the slave node even if the slave node does not successfully synchronize the data with the previous master node, thereby ensuring the effect of data synchronization.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data synchronization method, system, electronic device, and computer-readable storage medium.
Background
In order to ensure the consistency of the stored data, data synchronization is often performed between the master node and the slave node. The master node is a node currently used for processing a transaction submitted by a client, and the slave nodes are nodes subordinate to the master node and used for backing up data in the master node.
In the prior art, a master node often executes a transaction submitted by a client, returns an execution result to the client after correspondingly modifying stored data, and then performs data synchronization on slave nodes subordinate to the master node. However, due to the influence of external factors, sometimes the master node to which the slave node belongs needs to be replaced, and at this time, one slave node is often designated as a new master node. However, the former master node may not perform data synchronization with the slave node in time after the latest transaction is executed, and thus, the data in the newly designated new master node is missing, and the data synchronization effect is poor.
Disclosure of Invention
The invention provides a data synchronization method, a data synchronization system, electronic equipment and a computer readable storage medium, which are used for solving the problems of data loss and poor data synchronization effect in a newly specified main node.
In a first aspect of the present invention, there is provided a data synchronization method, applied to a system including a master node cluster and at least one slave node, where the master node cluster includes at least two master nodes, the method including:
for any main node, under the condition that the data stored in the main node changes, performing data synchronization with other main nodes in the main node cluster;
if the main node comprises at least one slave node belonging to the main node, the main node and the slave node belonging to the main node perform data synchronization;
and for any slave node, when receiving a master node switching instruction, determining other master nodes in the master node cluster as master nodes to which the slave nodes belong.
In a second aspect of the present invention, there is also provided a data synchronization system, including:
the master node cluster comprises at least two master nodes;
for any master node, the master node is configured to perform data synchronization with other master nodes in the master node cluster when the stored data changes;
the master node is used for carrying out data synchronization with the slave node belonging to the master node if the master node comprises at least one slave node belonging to the master node;
for any slave node, the slave node is configured to, when receiving a master node switching instruction, determine other master nodes in the master node cluster as master nodes to which the slave node belongs.
In yet another aspect of the present invention, there is also provided a computer-readable storage medium having stored therein instructions, which when run on a computer, cause the computer to execute any of the above-described data synchronization methods.
In yet another aspect of the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform any of the above described data synchronization methods.
In the data synchronization method provided in the embodiment of the present invention, for any master node, the master node performs data synchronization with other master nodes in a master node cluster when stored data changes, and if the master node includes at least one slave node belonging to the master node, the master node performs data synchronization with the slave node belonging to the master node, and for any slave node, when a master node switching instruction is received, the other master nodes in the master node cluster are determined as the master node to which the slave node belongs. Since the master node synchronizes data between the master nodes before synchronizing the data with the slave nodes, that is, the data in each master node is up-to-date and consistent, the master node is replaced again, and thus, when the slave node fails to synchronize data with the previous master node, the master node to which the slave node is replaced again can provide up-to-date data for the slave node, thereby ensuring the effect of data synchronization.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
FIG. 1 is a flow chart illustrating steps of a data synchronization method according to an embodiment of the present invention;
FIG. 2 is a flow chart illustrating steps of another data synchronization method according to an embodiment of the present invention;
FIG. 3 is a flow chart of steps of a further data synchronization method provided by an embodiment of the present invention;
FIG. 4 is a system diagram illustrating an embodiment of the present invention;
FIG. 5 is a block diagram of a data synchronization system provided by an embodiment of the present invention;
fig. 6 is a block diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
Fig. 1 is a flowchart of steps of a data synchronization method provided in an embodiment of the present invention, which is applied to a system including a master node cluster and at least one slave node, where the master node cluster includes at least two master nodes, as shown in fig. 1, the method may include:
In the embodiment of the present invention, the nodes in the system may be devices such as a server and the like having data processing capability and storage capability, and each node in the system may be preset and built. Changes to the data stored by the master node may occur due to commit of the executing client to the transaction, or due to manual modification of the data in the master node, etc. For example, the committed transaction may cause the master node to store new data for the stored data, or to delete portions of the stored data, or to modify the contents of portions of the stored data, and so on.
Further, the specific number of the master nodes included in the master node cluster may be set according to actual requirements, which is not limited in the embodiment of the present invention. In this step, after the master node generates new data, the data are synchronized to all other master nodes in the master node cluster, so that the strong consistency of the data among the master nodes can be ensured.
And 102, if the main node comprises at least one slave node belonging to the main node, performing data synchronization between the main node and the slave node belonging to the main node.
In this step, each slave node may determine the number of slave nodes belonging to the master node, specifically, read and record connection configuration information in the relevant information of the master node, and then determine the number of slave nodes recorded in the connection configuration information as the number of slave nodes currently belonging to the master node. Then, for a master node whose number is not 0, the master node may perform synchronization of data to slave nodes belonging to the master node. Since there may be some master nodes including slave nodes belonging to the master node and some master nodes not including slave nodes belonging to the master node among the plurality of master nodes, the number of slave nodes belonging to the master node included in the master node may also vary. Therefore, in this step, before performing synchronization each time, the number of slave nodes belonging to the master node currently included by each master node is determined, and then based on the determination result, only the master node whose number of slave nodes belonging to the master node is not 0 performs the operation of synchronization with the slave node, so that the master node not connected with the slave node can be accurately avoided, the invalid operation of synchronization with the slave node is performed, and further the resources of the master node are saved to a certain extent.
And 103, for any slave node, when receiving a master node switching instruction, determining other master nodes in the master node cluster as master nodes to which the slave nodes belong.
In the embodiment of the present invention, the master node switching instruction may be sent to the slave node when the master node to which the slave node belongs has a failure, or may be sent in other cases. The master node switching instruction may be sent by a user through a terminal used by the user, or may be sent directly through the slave node, which is not limited in the embodiment of the present invention. Since the master node synchronizes data between the master nodes before synchronizing the data with the slave nodes, that is, the data in each master node is up-to-date and consistent, in this way, when the slave node does not successfully synchronize the data with the previous master node, the master node to which the slave node reselects can also provide the up-to-date data for the slave node, thereby ensuring the effect of data synchronization. Correspondingly, after the slave node reselects the master node, data comparison can be performed with the master node, and if the data of the slave node and the data of the master node are different, data synchronization operation can be performed, so that data consistency is ensured. It should be noted that, a slave node may also directly replace the master node when detecting that the master node has failed, which is not limited in the embodiment of the present invention. Further, the execution order of the steps is not exclusive, and the steps may be executed before the step 102. Of course, if the slave node fails, the slave node reports the fault to remind maintenance personnel to perform maintenance in time.
In summary, in the data synchronization method provided in the embodiments of the present invention, for any master node, when the stored data changes, the master node performs data synchronization with other master nodes in the master node cluster, and if the master node includes at least one slave node belonging to the master node, the master node performs data synchronization with the slave node belonging to the master node, and for any slave node, when receiving a master node switching instruction, when the slave node fails in the master node to which the slave node belongs, the other master nodes in the master node cluster are determined as the master node to which the slave node belongs. Since the master node synchronizes data between the master nodes before synchronizing the data with the slave nodes, that is, the data in each master node is up-to-date and consistent, the master node is replaced again, and thus, when the slave node fails to synchronize data with the previous master node, the master node to which the slave node is replaced again can provide up-to-date data for the slave node, thereby ensuring the effect of data synchronization. .
Fig. 2 is a flowchart of another data synchronization method provided in an embodiment of the present invention, which is applied to a system including a master node cluster and at least one slave node, where the master node cluster includes at least two master nodes, as shown in fig. 2, the method may include:
In this step, the maximum distance between the set positions of the master nodes in the master node cluster is smaller than a preset distance threshold. The preset distance threshold value can be preset according to actual conditions, and can be inversely proportional to the longest time required for data synchronization under the condition that the normal operation of each main node in the main node cluster is not influenced. That is, the longer the required maximum time is, the smaller the preset distance threshold can be set. Since the larger the distance between the setting positions of the master nodes is, the more network resources are required for data synchronization, and accordingly, the longer the required time is, the maximum distance between the setting positions of the master nodes in the master node cluster can be set to be smaller than the preset distance threshold when the master node cluster is set. And further, the data synchronization of each main node in the main node cluster is ensured, and meanwhile, the main nodes can normally operate. Meanwhile, the main nodes are arranged as close as possible, and the efficiency of data synchronization between the main nodes can be improved to a certain extent.
Further, when the master node performs data synchronization with other master nodes in the master node cluster, the master node may send the stored data to other master nodes, and accordingly, the other master nodes may store the data, thereby implementing data synchronization. Specifically, the master node may package the stored data and then send the packaged data to another master node, so as to reduce the amount of data as much as possible.
And/or the master node may also send the target transaction log to other master nodes, and accordingly, the other master nodes may execute the target transaction according to the target transaction log. The target transaction is a transaction that causes a change in the data stored in the master node. The master node causes the stored data to change because the target transaction is executed. Therefore, in this step, the target transaction log is sent to other master nodes, and the other master nodes can make the stored data change similarly by executing the target transaction, thereby realizing data synchronization. Therefore, the data volume of the target transaction log is usually small, and the data synchronization can be realized only by sending the target transaction log, so that the efficiency of the data synchronization can be improved to a certain extent. The transaction log may be a binary log, which is a structured Query Statement (SQL) that records changes to data, and may be used to indicate which changes are specifically made to data. It should be noted that the master node may implement the sending operation by way of atomic broadcast.
In this step, when the master node performs data synchronization with the slave node, the master node may send the stored data to the slave node, and accordingly, the slave node may store the data, thereby implementing fast data synchronization. Alternatively, the master node may also send the target transaction log to the slave node, and accordingly, the slave node may execute the target transaction according to the target transaction log. The target transaction is a transaction that causes changes to data stored in other master nodes. Because the data volume of the target transaction log is usually small, the data synchronization can be realized only by sending the target transaction log, and the efficiency of the data synchronization can be improved to a certain extent. Specifically, the implementation manner of each operation is similar to that described in the foregoing step 201, and reference may be made to the foregoing step 201, which is not limited by the embodiment of the present invention.
Meanwhile, because the data in each master node is up-to-date and consistent, even if the master node fails during data synchronization with the slave nodes, the slave nodes can acquire the up-to-date data, and therefore, a larger time delay limit value can be set for master-slave synchronization, wherein the time delay limit value refers to the maximum time length occupied by the master node during data synchronization with the slave nodes. If the data synchronization is not completed before the delay tolerance value, the synchronization is considered to be failed, and retry is performed. Thus, by setting a larger latency width limit, a longer time can be provided for master-slave synchronization. Therefore, more slave nodes can be arranged in the system, and multi-data center deployment is realized.
It should be noted that, in order to facilitate management of the slave nodes, in the embodiment of the present invention, all the slave nodes may be controlled to connect to the same master node, so that only one master node and one slave node in the master node cluster need to perform data synchronization, thereby reducing the workload of data synchronization to a certain extent.
Since the master node synchronizes data with the slave nodes before synchronizing the data with the master nodes, that is, the data in each master node is up-to-date and consistent, the slave node may select one other master node as a new master node, determine the new master node as the master node to which the slave node belongs, and specifically, establish a network connection between the master node and the slave node. Because other information does not need to be determined in advance in optional operation, the required processing resource is less, and the selection efficiency can be improved to a certain extent.
Each slave node can select the same other master nodes as a new master node, so that the master node and the slave nodes can be synchronized conveniently and management is facilitated.
In summary, in the data synchronization method provided in the embodiments of the present invention, for any master node, when the stored data changes, the master node performs data synchronization with other master nodes in the master node cluster, and if the master node includes at least one slave node belonging to the master node, the master node performs data synchronization with the slave nodes belonging to the master node, and for any slave node, when the slave node receives a master node switching instruction, any one of the slave nodes is selected as a new master node from the other master nodes, and the new master node is determined as the master node to which the slave node belongs. Since the master node synchronizes data between the master nodes before synchronizing the data with the slave nodes, that is, the data in each master node is up-to-date and consistent, the master node is replaced again, and thus, when the slave node fails to synchronize data with the previous master node, the master node to which the slave node is replaced again can provide up-to-date data for the slave node, thereby ensuring the effect of data synchronization. .
Fig. 3 is a flowchart of steps of another data synchronization method provided in an embodiment of the present invention, which is applied to a system including a master node cluster, slave nodes, and a management node, where the master node cluster includes at least two master nodes, as shown in fig. 3, the method may include:
Specifically, the implementation manner of this step may refer to step 201 described above, and details of the embodiment of the present invention are not described herein.
Specifically, the implementation manner of this step may refer to step 202, which is not described herein again in this embodiment of the present invention.
In this step, the preset load threshold may be set according to a requirement. By selecting other main nodes with loads smaller than the preset load threshold, the problem that the other main nodes have overlarge loads and then cause the nodes to have problems after the connection with the other main nodes is subsequently established can be avoided.
Specifically, the slave node may send a master node switching request to the management node, and then the management node may determine a load of each of the other master nodes in the other master nodes when receiving the master node switching request, and finally send the node identifier of the other master node whose load is smaller than a preset load threshold value to the slave node. The node identifier may be a flag capable of uniquely indicating a node, and the management node may record current load amounts of all the master nodes, where the current load amounts may be sent by each master node to the management node in real time. Because the slave node usually belongs to a single master node, if the slave node directly determines the load capacity of other master nodes, the slave node needs to establish connection with each other master node first to obtain the load of each other master node, and then the connection needs to be disconnected, so that the operation process is complicated. Therefore, in this step, the slave node may determine, through the management node, other master nodes whose loads are smaller than the preset load threshold, and may further improve the operation efficiency to a certain extent. Of course, in the embodiment of the present invention, one slave node may also be set to belong to multiple master nodes, and accordingly, the slave node may directly obtain the load of the master node, which is not limited in the embodiment of the present invention.
And step 304, the slave node determines other master nodes with the load smaller than a preset load threshold value as the master nodes to which the slave node belongs.
In this step, there may be one or more other master nodes whose load is smaller than the preset load threshold, and when there is one master node, the slave node may directly determine the other master node as the master node to which the slave node belongs, and when there are a plurality of slave nodes, the slave node may optionally select one other master node whose load is smaller than the preset load threshold, and determine the other master node as the master node to which the slave node belongs. In this way, by determining other master nodes with loads smaller than the preset load threshold as the master node to which the slave node belongs, it can be ensured that the loads of the master nodes do not affect the normal operation of the master node after the other master nodes are determined as new master nodes.
In summary, in the data synchronization method provided in the embodiments of the present invention, for any master node, when the stored data changes, the master node performs data synchronization with other master nodes in the master node cluster, and if the master node includes at least one slave node belonging to the master node, the master node performs data synchronization with the slave node belonging to the master node, and for any slave node, when the slave node receives a master node switching instruction, the slave node determines another master node whose load is smaller than a preset load threshold value among the other master nodes, and determines the other master node whose load is smaller than the preset load threshold value as the master node to which the slave node belongs, so as to avoid a problem that the load of the other master node is too large after the other master node is determined as the master node to which the slave node belongs. Meanwhile, data synchronization is carried out between the master nodes before the master nodes synchronize data with the slave nodes, namely, the data in each master node are up-to-date and consistent, so that the master nodes are replaced again, and the master nodes replaced again can provide up-to-date data for the slave nodes under the condition that the slave nodes do not successfully synchronize data with the former master nodes, and further the data synchronization effect is ensured.
The following describes a data synchronization method provided in an embodiment of the present invention with a specific example. Fig. 4 is a schematic diagram of a system according to a specific example provided in an embodiment of the present invention, and as shown in fig. 4, the system includes a master node cluster and a slave node cluster, where the master node cluster includes a plurality of master nodes (denoted by nodes denoted by M in the figure), and the slave node cluster includes a plurality of slave nodes (denoted by nodes denoted by S in the figure). In the system, the plurality of main nodes form a Group (Group), each main node can independently execute transactions, and the read-write transactions executed by the main nodes are committed after the other main nodes in the Group coordinate. When a host node finishes processing a certain transaction and prepares to commit, it will automatically perform atomicity broadcast in group to inform other host nodes what content has been changed/what transaction has been executed. Therefore, by means of atomic broadcasting, the transaction keeps the same sequence on each master node, that is, each master node receives the same transaction log in the same sequence, and then each master node can replay the transaction logs in the same sequence, that is, corresponding transactions are executed according to the transaction logs, so that the write service is performed, the data stored in each master node in the whole group keeps a completely consistent state, and each node keeps a complete data copy. For any master node, the master node may perform data synchronization with other master nodes in the master node cluster under the condition that stored data changes, and when performing the data synchronization, the data synchronization may be implemented based on a distributed consistency algorithm (a variant of Paxos protocol), so that as long as most nodes still survive and communication therebetween is not problematic, the group still can provide services to the outside, that is, data synchronization can be provided for slave nodes and transactions submitted by processing clients are provided, and further, a part of the master nodes in the group can be tolerated to have faults, thereby improving the fault tolerance rate of the system.
In this system, all master nodes are deployed in the same available Area (AZ), that is, the maximum distance between the set positions of the master nodes in the master node cluster is smaller than a preset distance threshold. By way of example, it can be seen that each master node is located in Beijing. Further, a host node in the host node cluster may be used as a source to perform a read service. In particular, the synchronization of data from the master node may be accomplished using a native asynchronous replication protocol, or a semi-synchronous replication protocol. The system has higher fault-tolerant rate, so that data synchronization between a master node and a slave node does not need to be completed as soon as possible to avoid the problem of data loss caused by sudden failure of the master node, and therefore, the master node can be connected with a plurality of slave nodes, multi-data center deployment is realized, and the node deployment effect is improved. By way of example, it can be seen that different slave nodes may be located in different cities, e.g., some slave nodes are located in shanghai and some slave nodes are located in wuhan. Further, when the master node connected fails, the slave node may select another master node to connect, and perform a read service using the reconnected master node as a source. Since the master node synchronizes data between the master nodes before synchronizing the data with the slave nodes, that is, the data in each master node is up-to-date and consistent, in this way, in the case that the slave node does not successfully synchronize the data with the previous master node, the master node to which the slave node is reconnected can also provide the up-to-date data for the slave node, thereby ensuring the effect of data synchronization.
Fig. 5 is a block diagram of a data synchronization system according to an embodiment of the present invention, and as shown in fig. 5, the system 50 may include: the system comprises a main node cluster and at least one slave node 501, wherein the main node cluster comprises at least two main nodes 502;
for any master node, the master node 502 is configured to perform data synchronization with other master nodes in the master node cluster when stored data changes;
the master node 502 is configured to perform data synchronization with a slave node belonging to the master node if the master node includes at least one slave node belonging to the master node;
for any slave node, the slave node 501 is configured to, when receiving a master node switching instruction, determine other master nodes in the master node cluster as master nodes to which the slave nodes belong.
Optionally, the slave node 501 is specifically configured to select one of the other master nodes as a new master node, and establish a connection with the new master node.
Optionally, the slave node 501 is specifically configured to determine other master nodes whose loads are smaller than a preset load threshold among the other master nodes;
the slave node 501 is specifically configured to determine, as a master node to which the slave node belongs, another master node whose load is smaller than a preset load threshold.
Optionally, the system 50 further includes a management node, where the management node is connected to all the master nodes, current load amounts of all the master nodes are recorded in the management node, and the current load amounts are sent to the management node by each master node in real time; the slave node 501 is specifically configured to send a master node switching request to the management node;
the management node is specifically configured to, when the master node switching request is received, determine a load of each of the other master nodes in the other master nodes according to the recorded current load amount;
the management node is specifically configured to send the node identifier of the other master node whose load is smaller than the preset load threshold value to the slave node.
Optionally, the maximum distance between the setting positions of the master nodes in the master node cluster is smaller than a preset distance threshold.
Optionally, all slave nodes are connected to the same master node.
Optionally, the host node 502 is specifically configured to send the stored data to other host nodes; the other host node stores the data;
and/or the master node 502 is specifically configured to send the target transaction log to another master node; the other host nodes are specifically configured to execute the target transaction according to the target transaction log; the target transaction is a transaction that causes a change in data stored in the master node.
In summary, in the data synchronization system provided in the embodiments of the present invention, for any master node, when the stored data changes, the master node performs data synchronization with other master nodes in the master node cluster, if the master node includes at least one slave node belonging to the master node, the master node performs data synchronization with the slave node belonging to the master node, and for any slave node, when the slave node receives a master node switching instruction, the slave node determines that the other master nodes in the master node cluster are the master node to which the slave node belongs. Since the master node synchronizes data between the master nodes before synchronizing the data with the slave nodes, that is, the data in each master node is up-to-date and consistent, the master node is replaced again, and thus, when the slave node fails to synchronize data with the previous master node, the master node to which the slave node is replaced again can provide up-to-date data for the slave node, thereby ensuring the effect of data synchronization.
For the system embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
An embodiment of the present invention further provides an electronic device, as shown in fig. 6, including a processor 601, a communication interface 602, a memory 603, and a communication bus 604, where the processor 601, the communication interface 602, and the memory 603 complete mutual communication through the communication bus 604,
a memory 603 for storing a computer program;
the processor 601 is configured to implement the following steps when executing the program stored in the memory 603:
for any main node, under the condition that the data stored in the main node is changed, performing data synchronization with other main nodes in the main node cluster;
if the main node comprises at least one slave node belonging to the main node, the main node and the slave node belonging to the main node perform data synchronization;
and for any slave node, when receiving a master node switching instruction, determining other master nodes in the master node cluster as master nodes to which the slave nodes belong. The communication bus mentioned in the above terminal may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the terminal and other equipment.
The Memory may include a Random Access Memory (RAM) or a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the Integrated Circuit may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component.
In yet another embodiment of the present invention, a computer-readable storage medium is further provided, which has instructions stored therein, and when the computer-readable storage medium runs on a computer, the computer is caused to execute the data synchronization method described in any of the above embodiments.
In a further embodiment of the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the data synchronization method of any of the above embodiments.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (10)
1. A data synchronization method applied to a system including a master node cluster and at least one slave node, wherein the master node cluster includes at least two master nodes, the method comprising:
for any main node, under the condition that the data stored in the main node is changed, performing data synchronization with other main nodes in the main node cluster;
if the main node comprises at least one slave node belonging to the main node, the main node and the slave node belonging to the main node perform data synchronization;
and for any slave node, when receiving a master node switching instruction, determining other master nodes in the master node cluster as master nodes to which the slave nodes belong.
2. The method of claim 1, wherein the determining other master nodes in the master node cluster as master nodes to which the slave nodes belong comprises:
and the slave node selects one of the other master nodes as a new master node, and determines the new master node as the master node to which the slave node belongs.
3. The method of claim 1, wherein the determining other master nodes in the master node cluster as master nodes to which the slave nodes belong comprises:
the slave node determines other main nodes of which the loads are smaller than a preset load threshold value;
and the slave node determines other master nodes with the load smaller than a preset load threshold value as the master nodes to which the slave nodes belong.
4. The method according to claim 3, wherein the system further comprises a management node, the management node is connected with all the master nodes, current load amounts of all the master nodes are recorded in the management node, and the current load amounts are sent to the management node by each master node in real time; the determining, by the slave node, another master node whose load is smaller than a preset load threshold in the other master nodes includes:
the slave node sends a master node switching request to the management node;
the management node determines the load of each other main node in other main nodes according to the recorded current load capacity under the condition that the main node switching request is received;
and the management node sends the node identification of other main nodes with the load smaller than the preset load threshold value to the slave node.
5. The method of claim 1, wherein the synchronizing data with other master nodes in the master node cluster in the event of a change in data stored at the master node comprises:
the main node sends the stored data to other main nodes; the other host node stores the data;
and/or the master node sends the target transaction log to other master nodes; the other main nodes execute the target transaction according to the target transaction log; the target transaction is a transaction that causes a change in data stored in the master node.
6. A data synchronization system, the system comprising: the master node cluster comprises at least two master nodes;
for any master node, the master node is configured to perform data synchronization with other master nodes in the master node cluster when the stored data changes;
the master node is used for carrying out data synchronization with the slave node belonging to the master node if the master node comprises at least one slave node belonging to the master node;
for any slave node, the slave node is configured to, when receiving a master node switching instruction, determine other master nodes in the master node cluster as master nodes to which the slave node belongs.
7. The system according to claim 6, wherein the slave node is specifically configured to select one of the other master nodes as a new master node and establish a connection with the new master node.
8. The system according to claim 6, wherein the slave node is specifically configured to determine other master nodes whose loads are smaller than a preset load threshold among the other master nodes;
the slave node is specifically configured to determine, as the master node to which the slave node belongs, another master node whose load is smaller than a preset load threshold.
9. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any one of claims 1 to 5 when executing a program stored in the memory.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911351527.8A CN111049928B (en) | 2019-12-24 | 2019-12-24 | Data synchronization method, system, electronic device and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911351527.8A CN111049928B (en) | 2019-12-24 | 2019-12-24 | Data synchronization method, system, electronic device and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111049928A CN111049928A (en) | 2020-04-21 |
| CN111049928B true CN111049928B (en) | 2022-03-29 |
Family
ID=70240011
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911351527.8A Active CN111049928B (en) | 2019-12-24 | 2019-12-24 | Data synchronization method, system, electronic device and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111049928B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111641522A (en) * | 2020-05-25 | 2020-09-08 | 杭州安恒信息技术股份有限公司 | Method, system and computer equipment for node switching |
| CN112134887B (en) * | 2020-09-23 | 2022-08-09 | 哈尔滨海能达科技有限公司 | Data synchronization method and device for nodes in distributed cluster |
| CN114138825A (en) * | 2021-11-24 | 2022-03-04 | 聚好看科技股份有限公司 | Server and method for providing data query service for application program |
| CN113923222B (en) * | 2021-12-13 | 2022-05-31 | 云和恩墨(北京)信息技术有限公司 | Data processing method and device |
| CN114328604B (en) * | 2021-12-29 | 2024-01-12 | 苏州浪潮智能科技有限公司 | Method, device and medium for improving cluster data acquisition capacity |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108964948A (en) * | 2017-05-19 | 2018-12-07 | 北京金山云网络技术有限公司 | Principal and subordinate's service system, host node fault recovery method and device |
| CN110263093A (en) * | 2019-05-27 | 2019-09-20 | 东软集团股份有限公司 | Method of data synchronization, device, node, cluster and storage medium |
| CN110336707A (en) * | 2019-08-07 | 2019-10-15 | 卓尔智联(武汉)研究院有限公司 | Block chain common recognition device, method and computer readable storage medium |
| CN110502373A (en) * | 2019-07-26 | 2019-11-26 | 苏州浪潮智能科技有限公司 | A kind of method, equipment and readable medium that main and subordinate node data are synchronous |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR101335101B1 (en) * | 2009-03-19 | 2013-12-03 | 가부시키가이샤 무라쿠모 | Method and system for data replication management |
| CN104243527B (en) * | 2013-06-20 | 2018-08-21 | 华为技术有限公司 | Method of data synchronization, data synchronization unit and distributed system |
| CN109951331B (en) * | 2019-03-15 | 2021-08-20 | 北京百度网讯科技有限公司 | Method, device and computing cluster for sending information |
-
2019
- 2019-12-24 CN CN201911351527.8A patent/CN111049928B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108964948A (en) * | 2017-05-19 | 2018-12-07 | 北京金山云网络技术有限公司 | Principal and subordinate's service system, host node fault recovery method and device |
| CN110263093A (en) * | 2019-05-27 | 2019-09-20 | 东软集团股份有限公司 | Method of data synchronization, device, node, cluster and storage medium |
| CN110502373A (en) * | 2019-07-26 | 2019-11-26 | 苏州浪潮智能科技有限公司 | A kind of method, equipment and readable medium that main and subordinate node data are synchronous |
| CN110336707A (en) * | 2019-08-07 | 2019-10-15 | 卓尔智联(武汉)研究院有限公司 | Block chain common recognition device, method and computer readable storage medium |
Non-Patent Citations (1)
| Title |
|---|
| "基于PTP的多节点微震数据同步采集技术研究";谢静等;《计算机测量与控制》;20150930(第9期);第3195-3198页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111049928A (en) | 2020-04-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111049928B (en) | Data synchronization method, system, electronic device and computer readable storage medium | |
| US11379461B2 (en) | Multi-master architectures for distributed databases | |
| US11320991B2 (en) | Identifying sub-health object storage devices in a data storage system | |
| CN109683826B (en) | Capacity expansion method and device for distributed storage system | |
| US11907561B2 (en) | Data backup method and apparatus | |
| WO2018058949A1 (en) | Data storage method, device and system | |
| JP5686034B2 (en) | Cluster system, synchronization control method, server device, and synchronization control program | |
| WO2020063763A1 (en) | Data storage method, apparatus and system, and server, control node and medium | |
| CN108228102B (en) | Method and device for data migration between nodes, computing equipment and computer storage medium | |
| CN105069152B (en) | data processing method and device | |
| CN113268472B (en) | Distributed data storage system and method | |
| CN115599747B (en) | Metadata synchronization method, system and equipment of distributed storage system | |
| CN110351313B (en) | Data caching method, device, equipment and storage medium | |
| CN114385561A (en) | File management method and device and HDFS system | |
| CN112749178A (en) | Method for ensuring data consistency and related equipment | |
| CN111552701A (en) | Method for determining data consistency in distributed cluster and distributed data system | |
| CN112148206A (en) | Data reading and writing method and device, electronic equipment and medium | |
| CN105323271B (en) | Cloud computing system and processing method and device thereof | |
| CN111459913B (en) | Capacity expansion method and device of distributed database and electronic equipment | |
| CN107547605B (en) | message reading and writing method based on node queue and node equipment | |
| CN112000850A (en) | Method, device, system and equipment for data processing | |
| CN109992447B (en) | Data copying method, device and storage medium | |
| CN110569231B (en) | Data migration method, device, equipment and medium | |
| CN111405313A (en) | Method and system for storing streaming media data | |
| CN111797062B (en) | Data processing method, device and distributed database system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |