CN116633946A - Cluster state synchronous processing method and system based on distributed protocol - Google Patents
Cluster state synchronous processing method and system based on distributed protocol Download PDFInfo
- Publication number
- CN116633946A CN116633946A CN202310624235.7A CN202310624235A CN116633946A CN 116633946 A CN116633946 A CN 116633946A CN 202310624235 A CN202310624235 A CN 202310624235A CN 116633946 A CN116633946 A CN 116633946A
- Authority
- CN
- China
- Prior art keywords
- data
- state
- state change
- master node
- unique
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000001360 synchronised effect Effects 0.000 title claims abstract description 97
- 238000003672 processing method Methods 0.000 title claims abstract description 17
- 230000008859 change Effects 0.000 claims abstract description 312
- 238000012545 processing Methods 0.000 claims abstract description 112
- 238000000034 method Methods 0.000 claims abstract description 47
- 238000012790 confirmation Methods 0.000 claims abstract description 39
- 230000005540 biological transmission Effects 0.000 claims description 79
- 238000009825 accumulation Methods 0.000 claims description 25
- 230000011218 segmentation Effects 0.000 claims description 21
- 238000005538 encapsulation Methods 0.000 claims description 14
- 230000000903 blocking effect Effects 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 12
- 230000000694 effects Effects 0.000 abstract description 2
- 238000012544 monitoring process Methods 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000013144 data compression Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 230000008521 reorganization Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/04—Network management architectures or arrangements
- H04L41/042—Network management architectures or arrangements comprising distributed management centres cooperatively managing the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/43—Assembling or disassembling of packets, e.g. segmentation and reassembly [SAR]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/55—Push-based network services
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a cluster state synchronous processing method and a system thereof based on a distributed protocol, which are applied to a computer cluster, wherein the computer cluster comprises a unique master node and a plurality of slave nodes linked with the unique master node, and the method comprises the following steps: acquiring state change information of a unique master node in real time, synchronously pushing the state change information to all the slave nodes, receiving a receipt confirmation signal fed back by each slave node, when the unique master node receives the receipt confirmation signals exceeding a preset quantity threshold, carrying out real-time update processing on the data synchronization state of the state change information to obtain state change confirmation data, submitting the state change confirmation data to a state change queue of the unique master node to carry out superposition sorting processing, and obtaining state change synchronization data synchronized with the actual state change sequence. The application has the effect of improving the state synchronization efficiency of cluster data processing.
Description
Technical Field
The application relates to the technical field of data state synchronization, in particular to a cluster state synchronization processing method and system based on a distributed protocol.
Background
At present, when facing to massive stock early warning demands, the performance of a single machine is insufficient to meet the massive data processing demands, so that a plurality of machines are required to respectively process massive stock early warning data, when the running pressure of a cluster is high, the capacity of the machines is required to be increased in time to meet the increase of the stock early warning demands, and as the initial state of a newly-increased machine is empty, the machines are required to participate in data running through the cluster state quickly.
The existing cluster running state exists in a state change record form, so that machines in the cluster can be synchronized to the latest state in time, but after the cluster runs for a certain time, corresponding state change data are often far larger than actual state data, so that the state change data are required to be processed into state snapshots periodically to be subjected to mirror backup, and the latest state snapshot data are pulled to obtain the state change data of the corresponding state snapshots when a newly added machine is started, however, during the period of periodical snapshot processing, if the newly added state change record is more, the system overhead of snapshot processing of the state change record is increased, the cluster data accumulation easily causes higher system load, and the state change of the newly added machine is slow.
With respect to the related art described above, the inventors consider that there is a drawback in that the efficiency of machine state synchronization of a newly added cluster is slow.
Disclosure of Invention
In order to improve the state synchronization efficiency of cluster data processing, the application provides a cluster state synchronization processing method, a system, computer equipment and a storage medium based on a distributed protocol.
The first object of the present application is achieved by the following technical solutions:
a cluster state synchronization processing method based on a distributed protocol, which is applied to a computer cluster, wherein the computer cluster comprises a unique master node and a plurality of slave nodes linked with the unique master node, and the method comprises the following steps:
acquiring state change information of a unique master node in real time;
synchronously pushing the state change information to all the slave nodes, and receiving a receiving confirmation signal fed back by each slave node;
when the unique master node receives a receiving confirmation signal exceeding a preset quantity threshold, the data synchronization state of the state change information is updated in real time to obtain state change confirmation data;
and submitting the state change confirmation data to a state change queue of the unique master node for superposition sorting processing to obtain state change synchronous data synchronous with the actual state change sequence.
By adopting the technical scheme, because the newly added cluster machine needs to perform data state synchronization when accessing the system, when the state change records are more, the cluster performs regular snapshot processing on the change record data to easily increase system overhead, so that the load of system data transmission is higher, and the efficiency of system data transmission is easily affected by data accumulation.
The present application may be further configured in a preferred example to: after submitting the state change confirmation data to the state change queue of the unique master node to perform superposition sorting processing to obtain state change synchronous data synchronous with the actual state change sequence, the method further comprises:
performing state snapshot processing on the state change queue to obtain state snapshot data synchronized with the state change synchronization data;
when the unique master node is accessed to a newly added slave node, acquiring the access time of the newly added slave node;
synchronously pushing incremental state snapshot data in the state snapshot data at the access time and after the access time to the newly added slave node;
and carrying out data segmentation processing on the incremental state snapshot data, and respectively carrying out data encapsulation processing on the split incremental state snapshot data segments to obtain state change segmented data consisting of unique data splitting structures.
By adopting the technical scheme, the state snapshot processing is carried out on the changed state data through monitoring the change condition of the state change queue, the state snapshot data of the unique master node is kept in data synchronization with the state change synchronous data, the data updating instantaneity of the state snapshot is improved, the access time of the newly added slave node is acquired to directionally update the state data when the unique master node is accessed to the newly added slave node, the incremental state change data which is greater than or equal to the access time, namely from the access time is synchronously pushed, the segmentation processing of the incremental state change data which causes data accumulation is carried out independently, the overall data processing workload of the whole state data queue is reduced, the data segmentation processing is carried out on the incremental state change data according to the preset data splitting rule, the split incremental state change data segments are respectively subjected to data encapsulation, the data transmission convenience is improved through a data segmentation and independent data encapsulation mode, the data compression rate is reduced, and the state change segmented data carrying the unique state label, the data splitting start key value and the data splitting end key value are obtained, and the unique state change segmented data carrying the unique state label and the data splitting end key value are convenient to split and the data splitting end key value is convenient to restore the data.
The present application may be further configured in a preferred example to: after the incremental state snapshot data are subjected to data segmentation processing and the separated incremental state snapshot data segments are respectively subjected to data encapsulation processing to obtain state change segmented data composed of unique data separation structures, the method further comprises the steps of:
performing parallel execution task allocation processing on the state change segment data to obtain a synchronous task execution signal for controlling the synchronization of the execution threads of the state change segment data;
when each slave node receives the synchronous task execution signal, acquiring main state snapshot data of the unique master node according to the state change segmentation data;
and carrying out data updating processing on repeated change records which are the same as the main state snapshot data in the original change queue of each slave node.
By adopting the technical scheme, the incremental state change data of the unique master node are subjected to parallel execution task allocation processing, so that the execution threads of the data synchronization threads of the plurality of slave nodes are kept to synchronously execute, thereby obtaining the state snapshot segment synchronization task between the unique master node and each slave node, being beneficial to synchronizing the data through the state snapshots in the plurality of slave node synchronization threads, improving the data synchronization rate of the plurality of slave nodes, being beneficial to reducing the occupation of the data repeated storage on the memory of the slave node and improving the utilization rate of the memory resources of the slave node by carrying out unique state tag identification in the incremental state change data segment when the slave node receives a synchronization task execution signal, thereby obtaining the master state snapshot data corresponding to the access time of the slave node, improving the data transmission accuracy, removing the repeated change records which are identical to the master state snapshot data in the node original change queue of the slave node, and only keeping the latest state change data under the same unique state tag.
The present application may be further configured in a preferred example to: after performing data update processing on the repeated change records identical to the master state snapshot data in the original change queue of each slave node, the method further includes:
acquiring task execution progress data corresponding to each synchronous task execution signal;
when all the task execution progress data are in a completion state, performing incremental snapshot processing on the original change queue updated by each slave node to obtain slave node incremental snapshot data synchronized with the incremental state change data;
according to the slave node increment snapshot data, carrying out synchronous data updating processing on the historical state snapshot data of each slave node to obtain complete state snapshot data of each slave node;
and according to the complete state snapshot data, carrying out synchronous change processing on the slave node state of each slave node and the current master node state of the unique master node.
By adopting the technical scheme, the data synchronization progress of each slave node is monitored in real time through the task execution progress data of each state snapshot segment synchronization task, when all task execution progress data are in a finished state, namely the state change data of a unique master node are synchronously transmitted to the corresponding slave node, incremental snapshot processing is carried out on the synchronization state change queue data in the slave node, namely the synchronization state change data in the slave node are synchronously snapshot processing, so that the incremental state change data of the slave node are synchronously kept with the snapshot data of the slave node, the data consistency of the snapshot data and the data pushed by the unique master node is improved, the data update processing is carried out on the historical state snapshot data of the slave node according to the incremental snapshot data of the slave node, the complete state snapshot data of the slave node are kept consistent, the repeated state snapshot data in the complete state snapshot data are subjected to the data update processing, only one group of state change data is reserved under the same unique state label, and then the node state of the slave node is synchronously updated with the current state of the unique master node, and the state change data of the slave node and the unique state change nodes are kept, and the state of the slave node is synchronously updated.
The present application may be further configured in a preferred example to: before the obtaining, in real time, the state change information of the unique master node, the method further includes:
acquiring state meta information of a unique master node, wherein the state meta information comprises state memory data and a unique state label;
when the unique master node receives a state change application, judging whether the state meta information is registered according to the unique state label to obtain a registration judgment result;
and when the registration judgment result is successful registration, synchronously updating the state meta information into a state change queue of the unique master node.
By adopting the technical scheme, the unique main node comprises the state memory data and the state meta information of the unique state tag, the current storage condition of the unique main node is monitored in time through the state memory data, when the unique main node receives the state change application, the state meta information is registered and judged according to the unique state tag carried in the state meta information, so that whether the state meta information exists in the unique main node or not is evaluated according to the registration judgment result, the searching convenience of the state memory data of the unique main node is improved, when the registration judgment result is successful registration, the state meta information corresponding to the unique main node is indicated, the state meta information is updated to the state change queue preset by the unique main node, the state change queue consistent with the current state change condition of the unique main node is obtained, and the synchronism of the state change data of the unique main node and the actual state change condition is improved.
The present application may be further configured in a preferred example to: when the unique master node receives a state change application, judging whether the state meta information is registered according to the unique state label, and after obtaining a registration judgment result, the method further comprises the steps of:
when the registration judgment result is unregistered, searching whether the loaded state memory data has a unique state tag of the state meta information or not;
if the state change data exists, the state memory data corresponding to the unique state label is subjected to state meta information registration processing, and state change data synchronous with the state of the unique master node is generated.
By adopting the technical scheme, when the registration judgment result is unregistered, the unique state label of the state meta information is searched in the loaded state memory data, the data global searching speed is improved, when the unique state label exists, the fact that the corresponding state memory data exists in the unique master node is indicated, the unregistered state meta information is subjected to re-registration processing, a frame is formed according to a preset state meta information structure, the state memory data corresponding to the unique state label is subjected to registration processing, and the state change data which is synchronous with the current state of the unique master node is obtained, so that the state synchronism of the state change record of the unique master node and the current state of the unique master node is improved.
The present application may be further configured in a preferred example to: after synchronously pushing the state change information to all the slave nodes and receiving a receipt acknowledgement signal fed back by each slave node, and when the unique master node receives the receipt acknowledgement signal exceeding a preset number threshold, performing real-time update processing on the data synchronization state of the state change information, and before obtaining state change acknowledgement data, the method further comprises:
acquiring a data transmission load value of a data transmission channel between the unique master node and each slave node;
judging whether the data accumulation of each data transmission channel reaches a preset blocking threshold value or not according to the data transmission load value;
if yes, calculating state change waiting time of the corresponding slave node, and calculating a bandwidth transmission resource value required by state change according to the data accumulation;
and adding a supplementary slave node for sharing the state synchronous transmission pressure for the unique master node according to the state change waiting time and the bandwidth transmission resource value.
By adopting the technical scheme, the data transmission load value of the data transmission channel between the unique master node and each slave node is used for carrying out data transmission dynamic monitoring on each data transmission channel, carrying out blocking judgment on the data accumulation of each data transmission channel according to the data transmission load value, calculating the state change waiting time of the corresponding blocking slave node when the data accumulation of the data transmission channel reaches the preset blocking threshold value, evaluating the current data transmission performance according to the state change waiting time, calculating the bandwidth transmission resource value required by the state change according to the data accumulation condition, adjusting the bandwidth resource in the current distributed data architecture according to the bandwidth transmission resource value actually required, optimizing the number of the slave nodes of the unique master node according to the state change waiting time and the bandwidth transmission resource value, keeping the data accumulation condition of the data transmission channel of each slave node within the range of the preset blocking threshold value, reducing the state change waiting time, and sharing the state synchronous transmission pressure of the unique master node by optimizing the nodes of the plurality of slave nodes, thereby improving the data transmission rate before the unique master node and the plurality of slave nodes.
The second object of the present application is achieved by the following technical solutions:
a distributed protocol based cluster state synchronization system for a computer cluster, the computer cluster comprising a unique master node and a plurality of slave nodes linked to the unique master node, the system comprising:
the data acquisition module is used for acquiring the state change information of the unique master node in real time;
the data pushing module is used for synchronously pushing the state change information to all the slave nodes and receiving a receiving confirmation signal fed back by each slave node;
the data updating module is used for carrying out real-time updating processing on the data synchronization state of the state change information when the unique master node receives the receiving confirmation signals exceeding a preset quantity threshold value to obtain state change confirmation data;
and the data synchronization module is used for submitting the state change confirmation data to the state change queue of the unique master node for superposition sorting processing to obtain state change synchronization data synchronized with the actual state change sequence.
By adopting the technical scheme, because the newly added cluster machine needs to perform data state synchronization when accessing the system, when the state change records are more, the cluster performs regular snapshot processing on the change record data to easily increase system overhead, so that the load of system data transmission is higher, and the efficiency of system data transmission is easily affected by data accumulation.
The third object of the present application is achieved by the following technical solutions:
a computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the steps of the above-described distributed protocol based state synchronization method when the computer program is executed.
The fourth object of the present application is achieved by the following technical solutions:
a computer readable storage medium storing a computer program which, when executed by a processor, implements the steps of the distributed protocol based state synchronization method described above.
In summary, the present application includes at least one of the following beneficial technical effects:
1. the method comprises the steps of carrying out distributed management on data state synchronization through a distributed protocol, setting a unique master node for generating a state change record, setting a plurality of slave nodes for receiving the state change record pushed by the master node, so that synchronous processing of the data state is realized, synchronously pushing state change information to all slave nodes according to the association relation between the unique master node and the plurality of slave nodes, receiving acknowledgement signals fed back by all slave nodes, judging whether the information is pushed successfully, updating the data synchronous state of the state change information in the unique master node in real time when the receiving quantity of the information receiving acknowledgement information exceeds a preset threshold value, enabling the data synchronous state to be synchronous with the current data pushing progress, enabling the state change synchronous data of the unique master node to be synchronous with the actual state change sequence through superposition sequencing update of the state change queue of the unique master node, and improving the state synchronous efficiency of cluster data processing;
2. The method comprises the steps of monitoring the change condition of a state change queue, carrying out state snapshot processing on the changed state data, enabling state snapshot data of a unique master node to be synchronous with state change synchronous data, improving data updating instantaneity of the state snapshot, obtaining access time of a newly-added slave node to carry out directional updating on the state data when the unique master node is accessed to the newly-added slave node, synchronously pushing the increment state change data which is greater than or equal to the access time and is from the access time, independently carrying out segmentation processing on the increment state change data which causes data accumulation, reducing the overall data processing workload of the whole state data queue, carrying out data segmentation processing on the increment state change data according to a preset data splitting rule, respectively carrying out data encapsulation on the split increment state change data segments, improving data transmission convenience in a data segmentation and independently carrying unique state labels, data splitting start key values and data splitting end key values, helping to uniquely identify the state data through the unique state labels, and conveniently linking the plurality of state change segment data again through the data splitting start key values and the data splitting end key values;
3. The incremental state change data of the unique master node are subjected to parallel execution task allocation processing, so that the execution threads of the data synchronization threads of the plurality of slave nodes are kept to synchronously execute, the state snapshot segment synchronization task between the unique master node and each slave node is obtained, the data synchronization of the plurality of slave nodes is facilitated to be performed through the state snapshots in the plurality of slave node synchronization threads, the data synchronization rate of the plurality of slave nodes is improved, when the slave nodes receive a synchronization task execution signal, the unique state label identification can be performed in the incremental state change data segment, the master state snapshot data corresponding to the access time of the slave nodes is obtained, the data transmission accuracy is improved, the repeated change records identical to the master state snapshot data in the node original change queues of the slave nodes are removed, only the latest state change data is reserved under the same unique state label, the synchronous state change queue data of the unique change records of each slave node is obtained, the invalid occupation of the memory of the slave nodes is reduced, and the utilization rate of the memory resources of the slave nodes is improved.
Drawings
Fig. 1 is a flowchart of an implementation of a cluster state synchronization processing method based on a distributed protocol according to an embodiment of the present application.
Fig. 2 is a flowchart showing an implementation of step S10 of the state synchronization processing method according to an embodiment of the present application.
Fig. 3 is a flowchart showing an implementation of step S20 of the state synchronization processing method according to an embodiment of the present application.
Fig. 4 is a flowchart showing an implementation of step S40 of the state synchronization processing method according to an embodiment of the present application.
Fig. 5 is a flowchart showing an implementation of step S404 of the state synchronization processing method according to an embodiment of the present application.
Fig. 6 is a flowchart showing an implementation of step S407 of the state synchronization processing method according to an embodiment of the present application.
FIG. 7 is a block diagram illustrating a state synchronization processing system based on a distributed protocol according to an embodiment of the present application.
Fig. 8 is a schematic diagram of an internal structure of a computer device for implementing a distributed protocol-based cluster state synchronization processing method.
Detailed Description
The present application will be described in further detail with reference to the accompanying drawings.
In this embodiment, the system state snapshot data is synchronized through a distributed cluster provided with a unique master node and a plurality of slave nodes, where state change data is only generated by the unique master node of the cluster, and the unique master node submits a local change record and then pushes the state change record to the slave node, and the execution subject is a computer cluster.
In one embodiment, as shown in fig. 1, the application discloses a cluster state synchronization processing method based on a distributed protocol, which is applied to a computer cluster, wherein the computer cluster comprises a unique master node and a plurality of slave nodes linked with the unique master node, and specifically comprises the following steps:
S10: and acquiring the state change information of the unique master node in real time.
Specifically, when the unique master node generates a state change, the state change information of the unique master node is obtained according to the data synchronous push request after the state change, for example, a data search instruction is generated according to the unique state label during the state change, and the corresponding state change information is searched in the state change queue of the unique master node according to the data search instruction.
It should be noted that periodic information acquisition may be performed when the state change information is stored in a fixed amount, and the method is not limited to the one of the information acquisition method in real time in the present embodiment.
In one embodiment, in order to accurately evaluate the registration condition of the state meta information in the unique master node, before acquiring the state change information of the unique master node in real time, as shown in fig. 2, the method further includes:
s101: and acquiring the state meta information of the unique master node, wherein the state meta information comprises state memory data and a unique state label.
Specifically, the memory data of the unique master node is searched through the unique state label to obtain the state memory data, and the state meta-information data which accords with the preset state meta-information structure in the unique master node is subjected to information extraction according to the preset state meta-information structure frame to obtain the state meta-information structure composition data, so that the state meta-information of the unique master node is obtained.
S102: when the unique master node receives the state change application, judging whether the state meta information is registered according to the unique state label, and obtaining a registration judgment result.
Specifically, when the unique master node receives the state change application, a corresponding unique state label is searched according to the state change application, the unique state label judges whether the state meta-information is registered in the unique master node, if the unique state label corresponding to the unique master node exists, the state change registration information corresponding to the unique master node is indicated, if the unique state label corresponding to the unique master node does not exist, the state change registration information corresponding to the unique master node is indicated, and whether the state meta-information exists in the unique master node is evaluated according to a registration judgment result.
S103: and when the registration judgment result is successful registration, synchronously updating the state meta information into a state change queue of the unique master node.
Specifically, when the registration judgment result is successful registration, it is indicated that corresponding state change registration information exists in the unique master node, and state meta information corresponding to the registration state change application is updated to a state change queue preset by the unique master node, for example, the state change queue is sequentially numbered according to the unique state label of the state meta information, so as to obtain a state change queue which is continuously and incrementally numbered according to the unique state label.
In an embodiment, in order to improve the integrity of registration judgment of the state meta information, and perform orderly information registration processing on unregistered state meta information, after judging whether the state meta information is registered according to the unique state label when the unique master node receives the state change application, a registration judgment result is obtained, as shown in fig. 2, the method further includes:
s104: and when the registration judgment result is unregistered, searching whether the unique state tag of the state meta information exists in the loaded state memory data.
Specifically, when the registration judgment result is unregistered, it is indicated that no corresponding state change registration information exists in the unique master node, and then data searching is performed in the loaded state memory data of the unique master node, according to the unique state tag of the unregistered state meta information, state tag comparison is performed with the state memory data, so that the convenience of searching the state memory data is improved, and whether the corresponding state change data exists in the stored state memory data of the unique master node is judged according to the comparison result.
S105: if the state information exists, the state memory data corresponding to the unique state label is subjected to state meta information registration processing, and state change data synchronous with the state of the unique master node is generated.
Specifically, when the corresponding unique state label exists in the state memory data of the unique master node, a frame is formed according to a preset state meta information structure, such as the unique state label, a data splitting start key value and a data splitting end key value, so that a unique state meta information structure forming frame which accords with a data format preset by the state memory data is formed, state meta information registration processing is performed on the state memory data corresponding to the unique state label, and data forming structure reorganization processing is performed on the state memory data according to a state meta information structure forming rule, so that state change data synchronous with the data structure forming of the state change queue is obtained.
S20: and synchronously pushing the state change information to all the slave nodes, and receiving a receiving confirmation signal fed back by each slave node.
Specifically, according to a data transmission channel pre-constructed between the unique master node and all the slave nodes, for example, the node ID of each slave node is sent to the unique master node to be linked, and data pushing permission is allocated to each slave node according to the access time, an independent data transmission channel between the unique master node and each slave node is constructed according to the node ID and the data pushing permission, and state change information is synchronously pushed to all the slave nodes according to the access time and the node IDs of the slave nodes, and a receiving confirmation signal sent to the unique master node by each slave node is monitored through a preset monitoring mechanism, such as a heartbeat packet, wherein the receiving confirmation signal is the receiving confirmation signal generated according to the receiving state and the receiving data integrity condition, and the receiving confirmation signal is the state change information which is successfully received by the slave node and is not damaged.
In an embodiment, in order to more effectively mobilize the slave node resources to reduce data accumulation caused by congestion of the data transmission channel, after synchronously pushing the state change information to all the slave nodes and receiving the receipt acknowledgement signal fed back by each slave node, and when the unique master node receives the receipt acknowledgement signal exceeding the preset number threshold, the method further includes, before performing real-time update processing on the data synchronization state of the state change information to obtain state change acknowledgement data, as shown in fig. 3:
s201: and acquiring a data transmission load value of a data transmission channel between the unique master node and each slave node.
Specifically, a capacity difference between the total capacity of the data to be transmitted of the unique master node and the total capacity of the data received by each slave node is calculated, and the capacity difference of the data transmission is used as a data transmission load value of each data transmission channel.
S202: and judging whether the data accumulation of each data transmission channel reaches a preset blocking threshold value or not according to the data transmission load value.
Specifically, whether the data accumulation of each data transmission channel reaches a preset blocking threshold value is judged according to the data transmission load value, if the data transmission load value reaches the preset blocking threshold value, the data transmission channel is judged to generate a data accumulation phenomenon, the data synchronous pushing rate of the unique master node is influenced, when the data transmission load value is lower than the preset blocking threshold value, the data transmission channel is in a normal transmission working condition, and the data synchronous pushing rate meets the data transmission requirement between the unique master node and the slave node.
S203: if yes, calculating state change waiting time of the corresponding slave node, and calculating a bandwidth transmission resource value required by state change according to data accumulation.
Specifically, when the data accumulation of the data transmission channel reaches the preset blocking threshold value, calculating the state change waiting time of the slave node, for example, obtaining the single data receiving time of each state change data pushed by the unique master node, calculating the state change waiting time of the slave node according to the difference value between adjacent single data receiving times, calculating the data accumulation condition according to the capacity difference between the total capacity of the data to be pushed of the unique master node and the total capacity of the data receiving of the slave node, and taking the capacity difference value as the bandwidth transmission resource value required by the state change.
S204: and adding a supplementary slave node for sharing the state synchronous transmission pressure for the unique master node according to the state change waiting time and the bandwidth transmission resource value.
Specifically, the number of the slave nodes of the unique master node is optimized according to the state change waiting time and the bandwidth transmission resource value, if the state change waiting time exceeds a preset waiting threshold value, an idle state is searched according to the bandwidth transmission resource value or a slave node meeting the bandwidth transmission resource value is newly added, data sharing is carried out on state change data in a data accumulation state, and the state synchronous transmission pressure of the unique master node is shared through optimizing the number of the slave nodes, so that the data accumulation probability when the unique master node carries out data synchronous pushing is reduced.
S30: when the unique master node receives the receiving confirmation signals exceeding the preset quantity threshold, the data synchronization state of the state change information is updated in real time, and state change confirmation data are obtained.
Specifically, after the data synchronization pushing is performed on all the slave nodes, the unique master node starts to perform the statistics of the reception acknowledgement signals when receiving the reception acknowledgement signals, when the unique master node receives the reception acknowledgement signals exceeding a preset number threshold, for example, the preset threshold is set to be half of the number of all the slave nodes, namely, when the unique master node receives the reception acknowledgement signals fed back by more than half of the slave nodes, the data synchronization state of the state change information of the unique master node is updated in real time, for example, when the data synchronization pushing is performed on the unique master node, the data synchronization state of the state change information is marked as a 'submit' state, and when the unique master node receives the reception acknowledgement signals exceeding the preset threshold, the data synchronization state of the state change information is marked as a 'confirm' state, so that the state change acknowledgement data after the data synchronization pushing is completed is obtained.
S40: and submitting the state change confirmation data to a state change queue of the unique master node for superposition sorting processing to obtain state change synchronous data synchronous with the actual state change sequence.
Specifically, the state change confirmation data in the state of being synchronously pushed or marked as 'confirmed' is submitted to a state change queue, a unique state change record serial number is generated according to the confirmation time of the state change confirmation data and the unique change state label of the data, and sequential data update is carried out in the state change queue according to each state change record serial number, so that state change synchronous data synchronous with the actual state change sequence is obtained.
In an embodiment, in order to reduce error interference caused to an original data transmission sequence after the newly added slave node accesses the distributed cluster, improve data transmission sequence of the newly added slave node, and after submitting the state change confirmation data to a state change queue of the unique master node to perform superposition ordering processing, obtain state change synchronization data synchronized with an actual state change sequence, as shown in fig. 4, the method further includes:
s401: and carrying out state snapshot processing on the state change queue to obtain state snapshot data synchronous with the state change synchronous data.
Specifically, when the state change queue changes, a snapshot instruction for controlling the start of the state snapshot processing is generated, the state snapshot processing is performed on the state change queue, the state change queue is mapped to the globally consistent data mirror image according to the sequential number of continuous increment, so that state snapshot data is obtained, the data compression duty ratio of the unique master node is reduced through the state snapshot data synchronized with the state change synchronous data, and the space utilization rate of the unique master node is improved.
S402: and when the unique master node accesses the newly added slave node, acquiring the access time of the newly added slave node.
Specifically, when the unique master node accesses the newly added slave node, the newly added slave node is registered according to the node ID of the newly added slave node, and when the newly added slave node is successfully registered in the unique master node, the access time of the newly added slave node accessing the unique master node is acquired.
S403: and synchronously pushing incremental state snapshot data in the state snapshot data at the access time and after the access time to the newly added slave node.
Specifically, according to the access time of the newly added slave node, incremental state snapshot data which is greater than or equal to the access time is searched in the state snapshot data, the searched state snapshot data is marked as state snapshot data, and the state snapshot data is pushed to the newly added slave node through a preset data transmission channel.
S404: and carrying out data segmentation processing on the incremental state snapshot data, and respectively carrying out data encapsulation processing on the split incremental state snapshot data segments to obtain state change segmented data consisting of unique data splitting structures.
Specifically, according to a preset data composition architecture, data segmentation processing is performed on the incremental state snapshot data, for example, a unique change label in the incremental state snapshot data is used as a mark, a corresponding primary key setting is performed on a data start section and a data end section of each incremental state snapshot data respectively, so that a split incremental state snapshot data section is obtained, and according to a preset data encapsulation rule, data encapsulation processing is performed on each incremental state snapshot data section respectively, so that state change segmentation data which is more convenient for data transmission is obtained, wherein each state change segmentation data carries a unique state label, a data split start key value and a data split end key value.
In an embodiment, after adding a new slave node, in order to perform concurrent thread synchronous execution processing on all slave nodes more orderly, after performing data segmentation processing on incremental state snapshot data and performing data encapsulation processing on the split obtained incremental state snapshot data segments respectively to obtain state change segmented data composed of unique data splitting structures, as shown in fig. 5, the method further includes:
s405: and performing parallel execution task allocation processing on the state change segmented data to obtain a synchronous task execution signal for controlling the execution thread synchronization of the state change segmented data.
Specifically, the task allocation processing is performed on each increment state change data in parallel, and the task allocation processing is performed on each increment state change data according to the transmission bandwidth requirement of each increment state change data, the data encapsulation completion time and the idle degree of a data transmission channel between the unique master node and the slave node, so that the data synchronous push threads of all the slave nodes linked with the unique master node are kept synchronous, and the state snapshot segment synchronous task of the unique master node is obtained.
S406: when each slave node receives the synchronous task execution signal, the master state snapshot data of the unique master node is obtained according to the state change segment data.
Specifically, when each slave node receives the synchronous task execution signal pushed by the unique master node from the data transmission channel, the synchronous task execution signal is subjected to data analysis, the master state snapshot data of the unique master node is searched from the received state change segmentation data, and the master state snapshot data of the unique master node is mapped to the slave nodes according to the searching result, so that each slave node can synchronously receive the corresponding master state snapshot data.
S407: and carrying out data updating processing on repeated change records which are the same as the main state snapshot data in the original change queue of each slave node.
Specifically, according to the master state snapshot data, the repeated change records identical to the master state snapshot data are searched in the slave node original change queue, for example, data searching is performed according to the unique state label in the master state snapshot data, and data removal is performed on the repeated change records in the slave node original change queue according to the searching result, so that only the unique change state data is reserved under each unique state label in the slave node original change queue, and the synchronous state change queue data of the unique change records of the slave node is obtained.
In one embodiment, in order to improve synchronization between the snapshot data of the slave node and the state change progress of the slave node, after performing data update processing on the repeated change records in the original change queue of each slave node, which are the same as the main state snapshot data, as shown in fig. 6, the method further includes:
s501: and acquiring task execution progress data corresponding to each synchronous task execution signal.
Specifically, according to the data transmission progress in the data transmission channel between the unique master node and each slave node, the total capacity of data to be pushed and the residual capacity of data to be pushed of the unique master node are included, and the data capacity received by the slave node is combined, so as to comprehensively calculate the task execution progress of the state snapshot segment synchronous task, if the data capacity received by the slave node is the same as the total capacity of data to be pushed, the task execution progress is finished, and if the data capacity received by the slave node is smaller than the total capacity of data to be pushed, the task execution progress is in progress, and the task execution progress data is consistent with the residual capacity of data to be pushed.
S502: when all task execution progress data are in a completion state, incremental snapshot processing is carried out on the original change queue updated by each slave node, and slave node incremental snapshot data synchronous with the incremental state change data are obtained.
Specifically, when all task execution progress data are in a completion state, that is, when the only master node pushes all state change data to be pushed to all slave nodes, incremental snapshot processing is performed on synchronous state change queue data of the slave nodes, for example, snapshot mapping processing is performed on data in the synchronous state change queue in sequence according to the only state label and corresponding receiving time of the state change data, so that slave node incremental snapshot data synchronous with the incremental state change data are obtained.
S503: and according to the incremental snapshot data of the slave nodes, carrying out synchronous data updating processing on the historical state snapshot data of each slave node to obtain the complete state snapshot data of each slave node.
Specifically, according to the incremental snapshot data of the slave node, synchronous data updating processing is performed on the historical state snapshot data of the slave node, for example, according to the unique state label of the incremental snapshot data of the slave node, state snapshot data matched with the unique change state of the incremental snapshot data of the slave node is searched in the historical state snapshot data, so that the incremental snapshot data of the slave node is subjected to data coverage processing on the historical state snapshot data, the historical state snapshot data is removed, and only one group of latest state change data is reserved under the same unique state label, so that complete state snapshot data is obtained.
S504: and according to the complete state snapshot data, carrying out synchronous change processing on the slave node state of each slave node and the current master node state of the unique master node.
Specifically, according to the complete state snapshot data, the node state of the slave node and the current node state of the unique master node are synchronously changed, for example, the node state of the slave node is updated according to the current node state of the unique master node, so that the node state of the slave node and the current node state of the unique master node are kept synchronous, and similarly, the node state of each slave node is changed, so that the unique master node and the plurality of slave nodes with synchronous state change are obtained.
It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic, and should not limit the implementation process of the embodiment of the present application.
In an embodiment, a distributed protocol-based cluster state synchronization system is provided, and is applied to a computer cluster, where the computer cluster includes a unique master node and a plurality of slave nodes linked with the unique master node, and the distributed protocol-based cluster state synchronization system is in one-to-one correspondence with the distributed protocol-based state synchronization method in the above embodiment. As shown in fig. 7, the cluster state synchronization system based on the distributed protocol includes a data acquisition module, a data pushing module, a data updating module and a data synchronization module. The functional modules are described in detail as follows:
And the data acquisition module is used for acquiring the state change information of the unique master node in real time.
And the data pushing module is used for synchronously pushing the state change information to all the slave nodes and receiving a receiving confirmation signal fed back by each slave node.
And the data updating module is used for updating the data synchronization state of the state change information in real time when the unique master node receives the receiving confirmation signals exceeding the preset quantity threshold value to obtain state change confirmation data.
And the data synchronization module is used for submitting the state change confirmation data to a state change queue of the unique master node for superposition sorting processing to obtain state change synchronization data synchronized with the actual state change sequence.
Preferably, after submitting the state change confirmation data to the state change queue of the unique master node to perform superposition sorting processing to obtain state change synchronization data synchronized with the actual state change sequence, the method further includes:
and the state snapshot processing module is used for carrying out state snapshot processing on the state change queue to obtain state snapshot data synchronous with the state change synchronous data.
And the access time acquisition module is used for acquiring the access time of the newly added slave node when the unique master node accesses the newly added slave node.
And the incremental data pushing module is used for synchronously pushing the incremental state snapshot data in the state snapshot data at the access time and after the access time to the newly added slave node.
The data segmentation processing module is used for carrying out data segmentation processing on the incremental state snapshot data, and respectively carrying out data encapsulation processing on the obtained incremental state snapshot data segments after splitting to obtain state change segmented data consisting of unique data splitting structures.
Preferably, after performing data segmentation processing on the incremental state snapshot data and respectively performing data encapsulation processing on the incremental state snapshot data segments obtained after splitting to obtain state change segmented data composed of unique data splitting structures, the method further comprises the steps of:
and the thread synchronization module is used for carrying out parallel execution task allocation processing on the state change segment data to obtain a synchronous task execution signal for controlling the execution thread synchronization of the state change segment data.
And the master snapshot data acquisition module is used for acquiring the master state snapshot data of the unique master node according to the state change segment data when each slave node receives the synchronous task execution signal.
And the data removing module is used for carrying out data updating processing on repeated change records which are the same as the main state snapshot data in the original change queue of each slave node.
Preferably, after the data update processing is performed on the repeated change records identical to the master state snapshot data in the original change queue of each slave node, the method further includes:
the progress data acquisition module is used for acquiring task execution progress data corresponding to each synchronous task execution signal.
And the incremental snapshot processing module is used for performing incremental snapshot processing on the original change queue updated by each slave node when all the task execution progress data are in a completion state, so as to obtain slave node incremental snapshot data synchronous with the incremental state change data.
And the state snapshot updating module is used for carrying out synchronous data updating processing on the historical state snapshot data of each slave node according to the slave node increment snapshot data to obtain the complete state snapshot data of each slave node.
And the node synchronization updating module is used for carrying out synchronization change processing on the slave node state of each slave node and the current master node state of the unique master node according to the complete state snapshot data.
Preferably, before acquiring the state change information of the unique master node in real time, the method further comprises:
and the meta information acquisition module is used for acquiring the state meta information of the unique master node, wherein the state meta information comprises state memory data and a unique state label.
And the registration judging module is used for judging whether the state meta information is registered or not according to the unique state label when the unique master node receives the state change application, so as to obtain a registration judging result.
And the meta information updating module is used for synchronously updating the state meta information into the state change queue of the unique master node when the registration judgment result is successful registration.
Preferably, when the unique master node receives the state change application, determining whether the state meta information is registered according to the unique state label, and after obtaining the registration determination result, further including:
and the tag searching module is used for searching whether the unique state tag of the state meta information exists in the loaded state memory data or not when the registration judging result is unregistered.
And the meta information registration module is used for carrying out state meta information registration processing on the state memory data corresponding to the unique state label if the state memory data exists, and generating state change data synchronous with the state of the unique master node.
Preferably, after synchronously pushing the state change information to all the slave nodes and receiving the receipt acknowledgement signal fed back by each slave node, and when the unique master node receives the receipt acknowledgement signal exceeding the preset number threshold, the method further comprises the steps of:
And the load data acquisition module is used for acquiring the data transmission load value of the data transmission channel between the unique master node and each slave node.
And the data accumulation judging module is used for judging whether the data accumulation of each data transmission channel reaches a preset blocking threshold value or not according to the data transmission load value.
And the waiting data calculation module is used for calculating the state change waiting time of the corresponding slave node if yes, and calculating the bandwidth transmission resource value required by state change according to data accumulation.
And the node quantity optimizing module is used for adding a supplementary slave node for sharing the state synchronous transmission pressure for the unique master node according to the state change waiting time and the bandwidth transmission resource value.
For specific limitations on the distributed protocol based cluster state synchronization system, reference may be made to the above limitation on the distributed protocol based state synchronization method, and no further description is given here. The modules in the cluster state synchronization system based on the distributed protocol can be implemented in whole or in part by software, hardware and a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a computer cluster, and the internal structure of which may be as shown in fig. 8. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used for storing the state data synchronous push data of the distributed unique master node and the plurality of slave nodes. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program, when executed by a processor, implements a state synchronization method based on a distributed protocol.
In one embodiment, a computer readable storage medium is provided having a computer program stored thereon which, when executed by a processor, implements the steps of a distributed protocol based state synchronization method.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions.
The above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and are intended to be included in the scope of the present application.
Claims (10)
1. A method for processing cluster state synchronization based on a distributed protocol, which is applied to a computer cluster, wherein the computer cluster comprises a unique master node and a plurality of slave nodes linked with the unique master node, and the method comprises the following steps:
Acquiring state change information of a unique master node in real time;
synchronously pushing the state change information to all the slave nodes, and receiving a receiving confirmation signal fed back by each slave node;
when the unique master node receives a receiving confirmation signal exceeding a preset quantity threshold, the data synchronization state of the state change information is updated in real time to obtain state change confirmation data;
and submitting the state change confirmation data to a state change queue of the unique master node for superposition sorting processing to obtain state change synchronous data synchronous with the actual state change sequence.
2. The method for processing cluster state synchronization based on a distributed protocol according to claim 1, wherein after the submitting the state change confirmation data to the state change queue of the unique master node for superposition ordering processing, obtaining state change synchronization data synchronized with an actual state change order, the method further comprises:
performing state snapshot processing on the state change queue to obtain state snapshot data synchronized with the state change synchronization data;
when the unique master node is accessed to a newly added slave node, acquiring the access time of the newly added slave node;
Synchronously pushing incremental state snapshot data in the state snapshot data at the access time and after the access time to the newly added slave node;
and carrying out data segmentation processing on the incremental state snapshot data, and respectively carrying out data encapsulation processing on the split incremental state snapshot data segments to obtain state change segmented data consisting of unique data splitting structures.
3. The method for synchronously processing cluster states based on a distributed protocol according to claim 2, wherein after performing data segmentation processing on the incremental state snapshot data and performing data encapsulation processing on the divided incremental state snapshot data segments respectively to obtain state change segmented data composed of unique data splitting structures, the method further comprises:
performing parallel execution task allocation processing on the state change segment data to obtain a synchronous task execution signal for controlling the synchronization of the execution threads of the state change segment data;
when each slave node receives the synchronous task execution signal, acquiring main state snapshot data of the unique master node according to the state change segmentation data;
And carrying out data updating processing on repeated change records which are the same as the main state snapshot data in the original change queue of each slave node.
4. The distributed protocol-based cluster state synchronization processing method according to claim 3, wherein after performing data update processing on the repeated change records identical to the master state snapshot data in the original change queue of each of the slave nodes, the method further comprises:
acquiring task execution progress data corresponding to each synchronous task execution signal;
when all the task execution progress data are in a completion state, performing incremental snapshot processing on the original change queue updated by each slave node to obtain slave node incremental snapshot data synchronized with the incremental state change data;
according to the slave node increment snapshot data, carrying out synchronous data updating processing on the historical state snapshot data of each slave node to obtain complete state snapshot data of each slave node;
and according to the complete state snapshot data, carrying out synchronous change processing on the slave node state of each slave node and the current master node state of the unique master node.
5. The method for processing cluster state synchronization based on a distributed protocol according to claim 1, wherein before the obtaining, in real time, state change information of a unique master node, the method further comprises:
acquiring state meta information of a unique master node, wherein the state meta information comprises state memory data and a unique state label;
when the unique master node receives a state change application, judging whether the state meta information is registered according to the unique state label to obtain a registration judgment result;
and when the registration judgment result is successful registration, synchronously updating the state meta information into a state change queue of the unique master node.
6. The method for processing the cluster state synchronization based on the distributed protocol according to claim 5, wherein when the unique master node receives the state change application, it determines whether the state meta information is registered according to the unique state label, and after obtaining a registration determination result, the method further comprises:
when the registration judgment result is unregistered, searching whether the loaded state memory data has a unique state tag of the state meta information or not;
If the state change data exists, the state memory data corresponding to the unique state label is subjected to state meta information registration processing, and state change data synchronous with the state of the unique master node is generated.
7. The method according to claim 1, wherein after pushing the state change information to all the slave nodes synchronously and receiving a reception acknowledgement signal fed back by each of the slave nodes, and when the unique master node receives a reception acknowledgement signal exceeding a preset number threshold, the method further comprises, before obtaining state change acknowledgement data, updating the data synchronization state of the state change information in real time:
acquiring a data transmission load value of a data transmission channel between the unique master node and each slave node;
judging whether the data accumulation of each data transmission channel reaches a preset blocking threshold value or not according to the data transmission load value;
if yes, calculating state change waiting time of the corresponding slave node, and calculating a bandwidth transmission resource value required by state change according to the data accumulation;
and adding a supplementary slave node for sharing the state synchronous transmission pressure for the unique master node according to the state change waiting time and the bandwidth transmission resource value.
8. A distributed protocol based cluster state synchronization system for a computer cluster, the computer cluster including a unique master node and a plurality of slave nodes linked to the unique master node, the system comprising:
the data acquisition module is used for acquiring the state change information of the unique master node in real time;
the data pushing module is used for synchronously pushing the state change information to all the slave nodes and receiving a receiving confirmation signal fed back by each slave node;
the data updating module is used for carrying out real-time updating processing on the data synchronization state of the state change information when the unique master node receives the receiving confirmation signals exceeding a preset quantity threshold value to obtain state change confirmation data;
and the data synchronization module is used for submitting the state change confirmation data to the state change queue of the unique master node for superposition sorting processing to obtain state change synchronization data synchronized with the actual state change sequence.
9. Computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the distributed protocol based cluster state synchronization processing method according to any of claims 1 to 7 when the computer program is executed.
10. A computer readable storage medium storing a computer program, wherein the computer program when executed by a processor implements the steps of the distributed protocol based cluster state synchronization processing method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310624235.7A CN116633946B (en) | 2023-05-29 | 2023-05-29 | Cluster state synchronous processing method and system based on distributed protocol |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310624235.7A CN116633946B (en) | 2023-05-29 | 2023-05-29 | Cluster state synchronous processing method and system based on distributed protocol |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116633946A true CN116633946A (en) | 2023-08-22 |
| CN116633946B CN116633946B (en) | 2023-11-21 |
Family
ID=87602280
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310624235.7A Active CN116633946B (en) | 2023-05-29 | 2023-05-29 | Cluster state synchronous processing method and system based on distributed protocol |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116633946B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20100014177A (en) * | 2008-08-01 | 2010-02-10 | 한국전자통신연구원 | System and method for metadata management |
| CN108280080A (en) * | 2017-01-06 | 2018-07-13 | 阿里巴巴集团控股有限公司 | A kind of method of data synchronization, device and electronic equipment |
| CN108696595A (en) * | 2018-05-28 | 2018-10-23 | 郑州云海信息技术有限公司 | Distributed type assemblies method of data synchronization, master node, slave node, system and medium |
| CN111274317A (en) * | 2020-01-07 | 2020-06-12 | 书生星际(北京)科技有限公司 | Method and device for synchronizing multi-node data and computer equipment |
| CN113407527A (en) * | 2020-11-20 | 2021-09-17 | 广东美云智数科技有限公司 | Permission data acquisition method, management device and storage medium |
| CN115599747A (en) * | 2022-04-22 | 2023-01-13 | 北京志凌海纳科技有限公司（Cn） | Metadata synchronization method, system and equipment of distributed storage system |
-
2023
- 2023-05-29 CN CN202310624235.7A patent/CN116633946B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20100014177A (en) * | 2008-08-01 | 2010-02-10 | 한국전자통신연구원 | System and method for metadata management |
| CN108280080A (en) * | 2017-01-06 | 2018-07-13 | 阿里巴巴集团控股有限公司 | A kind of method of data synchronization, device and electronic equipment |
| CN108696595A (en) * | 2018-05-28 | 2018-10-23 | 郑州云海信息技术有限公司 | Distributed type assemblies method of data synchronization, master node, slave node, system and medium |
| CN111274317A (en) * | 2020-01-07 | 2020-06-12 | 书生星际(北京)科技有限公司 | Method and device for synchronizing multi-node data and computer equipment |
| CN113407527A (en) * | 2020-11-20 | 2021-09-17 | 广东美云智数科技有限公司 | Permission data acquisition method, management device and storage medium |
| CN115599747A (en) * | 2022-04-22 | 2023-01-13 | 北京志凌海纳科技有限公司（Cn） | Metadata synchronization method, system and equipment of distributed storage system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116633946B (en) | 2023-11-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112099958B (en) | Distributed multi-task management method and device, computer equipment and storage medium | |
| CN108920272B (en) | Data processing method, device, computer equipment and storage medium | |
| CN105468718B (en) | Data consistency processing method, device and system | |
| CN111399764B (en) | Data storage method, data reading device, data storage equipment and data storage medium | |
| CN110231915B (en) | Data management method, system, device, computer equipment and storage medium | |
| CN112256433B (en) | Partition migration method and device based on Kafka cluster | |
| CN111291062B (en) | Data synchronous writing method and device, computer equipment and storage medium | |
| CN111552701A (en) | Method for determining data consistency in distributed cluster and distributed data system | |
| CN113742135A (en) | Data backup method and device and computer readable storage medium | |
| CN112685333A (en) | Heap memory management method and device | |
| CN113268540B (en) | Data synchronization method and device | |
| CN116633946B (en) | Cluster state synchronous processing method and system based on distributed protocol | |
| CN114564281A (en) | Container scheduling method, device, equipment and storage medium | |
| CN111913927A (en) | Data writing method and device and computer equipment | |
| CN109547253B (en) | File downloading method and device, computer equipment and storage medium | |
| CN110275777B (en) | Resource scheduling system | |
| CN116388916A (en) | Time stamp distribution method, device, storage medium and system | |
| CN111506400A (en) | Computing resource allocation system, method, device and computer equipment | |
| CN115203255A (en) | Data query method and device, electronic equipment and storage medium | |
| CN112527561B (en) | Data backup method and device based on Internet of things cloud storage | |
| CN112291299B (en) | Synchronization method, device, equipment and storage medium based on AI Station inference platform | |
| CN109525675B (en) | Northbound server file downloading method and device, computer equipment and storage medium | |
| CN114172903A (en) | Node capacity expansion method, device, equipment and medium of slarm scheduling system | |
| CN114328711A (en) | Data processing method and device, computer equipment and storage medium | |
| CN113438274A (en) | Data transmission method and device, computer equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |