CN110071978B - Cluster management method and device - Google Patents
Cluster management method and device Download PDFInfo
- Publication number
- CN110071978B CN110071978B CN201910351626.XA CN201910351626A CN110071978B CN 110071978 B CN110071978 B CN 110071978B CN 201910351626 A CN201910351626 A CN 201910351626A CN 110071978 B CN110071978 B CN 110071978B
- Authority
- CN
- China
- Prior art keywords
- node
- monitored
- hash
- key value
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/10—Active monitoring, e.g. heartbeat, ping or trace-route
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- Cardiology (AREA)
- General Health & Medical Sciences (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The embodiment of the application provides a cluster management method and device, relates to the technical field of communication, and aims to solve the problem that service delay is long due to the restarting process when a newly added node or a separated node of a cluster needs to be restarted. The scheme of the embodiment of the application comprises the following steps: acquiring a survival node list, wherein the survival node list comprises Internet Protocol (IP) addresses of all nodes which are connected with a main node, and then calculating hash values of the IP addresses of all nodes in the survival node list; according to the Hash value of the IP address of each node, each node is mapped to a position on a Hash ring, then the key value to be monitored corresponding to the data to be monitored is obtained, the Hash value corresponding to each key value to be monitored is determined, and then the monitoring range of each node is determined according to the position of the Hash ring to which the Hash value corresponding to each key value to be monitored is mapped.
Description
Technical Field
The present application relates to the field of communications technologies, and in particular, to a method and an apparatus for cluster management.
Background
In the era of everything interconnection, massive data formatting processing is the basis of big data analysis and machine learning, and the fluent log collection framework is a data collection tool, has the functions of data collection, filtration, processing, writing and the like, and can perform various complex processing on data in a memory. At present, in the face of mass data in a big data environment, a multiple fluent instance technology based on consistency appears, that is, a cluster includes multiple fluent nodes, one fluent node is a master node, and the rest fluent nodes can be accessed to the master node as slave nodes.
When a new node is accessed to the master node or a slave node is separated from the master node, the master node needs to balance the load of each node in the cluster again, namely: the hash value of each node and the hash value of a key value to be monitored corresponding to each data to be monitored are recalculated, and since the parameters used in calculating the hash values include the number of nodes and the number of the nodes changes, the calculated hash values of each node and the calculated hash values of the key values to be monitored all change, so that after load balancing is performed again, the data to be monitored of each node all change, more data migration and other work can be generated, the cluster restart time is longer, and service delay is generated.
Disclosure of Invention
In view of this, embodiments of the present application provide a method and an apparatus for cluster management, so as to reduce service latency caused by a restart process when a cluster has a newly added node or a detached node that needs to be restarted. The specific technical scheme is as follows:
in a first aspect, an embodiment of the present application provides a cluster management method, where the method is applied to a master node in a cluster, and the method includes:
acquiring a survival node list, wherein the survival node list comprises Internet Protocol (IP) addresses of all nodes which are connected with the main node;
calculating the hash value of the IP address of each node in the survival node list;
respectively mapping each node to a position on a hash ring according to the hash value of the IP address of each node;
acquiring key values to be monitored corresponding to data to be monitored, and determining hash values corresponding to the key values to be monitored;
and mapping the hash value corresponding to each key value to be monitored to the position of the hash ring, and determining the monitoring range of each node.
In a possible implementation manner, the determining the monitoring range of each node according to the position where the hash value corresponding to each key value to be monitored is mapped to the hash ring includes:
mapping each key value to be monitored to a position on the hash ring according to the hash value corresponding to each key value to be monitored;
and for each key value to be monitored, determining the first node searched according to the anticlockwise sequence as the node for monitoring the key value to be monitored from the position of the key value to be monitored on the hash ring.
In one possible implementation, the method further includes:
when detecting that a new node is accessed, adding the new node into the surviving node list to obtain an updated surviving node list; and/or the presence of a gas in the gas,
and when a disconnected node which is disconnected with the main node is detected, deleting the disconnected node from the survival node list to obtain an updated survival node list.
In a possible implementation manner, after obtaining the updated surviving node list, the method further includes:
calculating the hash value of the IP address of each node in the updated surviving node list;
respectively mapping each node to a position on the hash ring according to the hash value of the IP address of each node in the updated survival node list;
and determining the monitoring range of each node in the updated surviving node list according to the position of the hash value mapping to the hash ring corresponding to each key value to be monitored.
In one possible implementation, after the obtaining the surviving node list, the method further includes:
respectively setting a preset number of virtual nodes for each node in the surviving node list, wherein each virtual node corresponding to one node is respectively identified by the IP address of the node and the label of the virtual node;
calculating the hash value of the identifier of each virtual node;
and respectively mapping each virtual node to a position on the hash ring according to the hash value of the IP address of each virtual node.
In a possible implementation manner, the determining the monitoring range of each node according to the position where the hash value corresponding to each key value to be monitored is mapped to the hash ring includes:
mapping each key value to be monitored to a position on the hash ring according to the hash value corresponding to each key value to be monitored;
for each key value to be monitored, searching nodes and virtual nodes in a counterclockwise sequence from the position of the key value to be monitored on the hash ring;
if the first node searched according to the anticlockwise sequence is a virtual node, determining that the key value to be monitored is monitored by the node corresponding to the searched virtual node; alternatively, the first and second electrodes may be,
and if the first node searched according to the anticlockwise sequence is not the virtual node, determining that the key value to be monitored is monitored by the searched node.
In a possible implementation manner, after determining the listening range of each node according to the position where the hash value corresponding to each key value to be listened is mapped to the hash ring, the method further includes:
and respectively sending the key value to be monitored, which is required to be monitored by the node, to each node, so that each node can process data according to the key value to be monitored, which is required to be monitored.
In a second aspect, an embodiment of the present application provides an apparatus for cluster management, where the apparatus is applied to a master node in a cluster, and the apparatus includes:
an obtaining module, configured to obtain a surviving node list, where the surviving node list includes internet protocol IP addresses of nodes that establish connection with the host node;
the calculation module is used for calculating the hash value of the IP address of each node in the survival node list;
the mapping module is used for mapping each node to a position on a hash ring according to the hash value of the IP address of each node calculated by the calculating module;
the acquisition module is also used for acquiring the key values to be monitored and determining the hash value corresponding to each key value to be monitored;
and the determining module is used for determining the monitoring range of each node according to the position of the hash ring mapped by the hash value corresponding to each key value to be monitored.
In a possible implementation manner, the determining module is specifically configured to:
mapping each key value to be monitored to a position on the hash ring according to the hash value corresponding to each key value to be monitored;
and for each key value to be monitored, determining the first node searched according to the anticlockwise sequence as the node for monitoring the key value to be monitored from the position of the key value to be monitored on the hash ring.
In one possible implementation, the apparatus further includes:
the updating module is used for adding the newly added node into the surviving node list when the access of the newly added node is detected, so as to obtain an updated surviving node list; and/or deleting the detached node from the surviving node list when the detached node which is disconnected with the main node is detected, so as to obtain an updated surviving node list.
In a possible implementation manner, the calculating module is further configured to calculate a hash value of an IP address of each node in the updated surviving node list;
the mapping module is further configured to map each node to a position on the hash ring according to the hash value of the IP address of each node in the updated surviving node list;
the determining module is further configured to determine a monitoring range of each node in the updated surviving node list according to a position where the hash value corresponding to each key value to be monitored is mapped to the hash ring.
In one possible implementation, the apparatus further includes: setting a module;
the setting module is used for setting a preset number of virtual nodes for each node in the surviving node list, and each virtual node corresponding to one node is identified by the IP address of the node and the label of the virtual node;
the calculation module is further configured to calculate a hash value of the identifier of each virtual node;
the mapping module is further configured to map each virtual node to a position on the hash ring according to the hash value of the IP address of each virtual node.
In a possible implementation manner, the determining module is specifically configured to:
mapping each key value to be monitored to a position on the hash ring according to the hash value corresponding to each key value to be monitored;
for each key value to be monitored, searching nodes and virtual nodes in a counterclockwise sequence from the position of the key value to be monitored on the hash ring;
if the first node searched according to the anticlockwise sequence is a virtual node, determining that the key value to be monitored is monitored by the node corresponding to the searched virtual node; alternatively, the first and second electrodes may be,
and if the first node searched according to the anticlockwise sequence is not the virtual node, determining that the key value to be monitored is monitored by the searched node.
In one possible implementation, the apparatus further includes:
and the sending module is used for sending the key value to be monitored, which is required to be monitored by the node, to each node respectively, so that each node can process data according to the key value to be monitored, which is required to be monitored by each node.
In a third aspect, an embodiment of the present application provides a server, where the server is a master node in a cluster, and the server includes: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor, the processor being caused by the machine-executable instructions to: the method of cluster management as described in the first aspect is implemented.
In a fourth aspect, this application further provides a computer-readable storage medium, in which a computer program is stored, and when executed by a processor, the computer program implements the method for cluster management in the first aspect.
In a fifth aspect, embodiments of the present application further provide a computer program product containing instructions, which when run on a computer, cause the computer to perform the method for cluster management described in the first aspect.
Therefore, by using the method provided by the embodiment of the application, the master node may obtain the surviving node list, calculate the hash value of the IP address of each node in the surviving node list, then map each node to a position on the hash ring according to the hash value, obtain the key value to be monitored, determine the hash value corresponding to each key value to be monitored, and further determine the monitoring range monitored by each node according to the position where the hash value corresponding to each key value to be monitored is mapped to the hash ring.
Compared with the prior art, the master node determines the monitoring range of each node according to the hash ring in the embodiment of the application. Even if the nodes are newly added or separated and load balancing is carried out again, the monitoring range of each node is determined by the positions of the nodes and the key values to be monitored on the hash ring, and the nodes which are not changed can still be mapped to the same positions on the hash ring. And the position of mapping each key value to be monitored to the hash ring can not be changed, so the monitoring range of the unchanged node can not be changed generally. Only nodes adjacent to the detached node may need to monitor the key value to be monitored, which is responsible for monitoring by the detached node; or a part of key values to be monitored, which are required to be monitored by nodes adjacent to the newly added node, are monitored by the newly added node instead. Only the range of listening for nodes adjacent to the newly added node or the node away from it may change. Compared with the prior art that the key values to be monitored corresponding to the nodes are changed after load balancing is carried out again, the method provided by the embodiment of the application has small influence on the nodes which are not changed in the surviving node list when the cluster is changed, and compared with the prior art, the service delay can be reduced.
Of course, not all advantages described above need to be achieved at the same time in the practice of any one product or method of the present application.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is an exemplary diagram of a cluster provided in an embodiment of the present application;
fig. 2 is a flowchart of a method for cluster management according to an embodiment of the present application;
fig. 3 is an exemplary diagram of a hash ring according to an embodiment of the present disclosure;
fig. 4 is a flowchart of another cluster management method according to an embodiment of the present application;
fig. 5 is an exemplary diagram of another hash ring provided in an embodiment of the present application;
fig. 6 is an exemplary diagram of another hash ring provided in an embodiment of the present application;
fig. 7 is an exemplary diagram of another hash ring provided in an embodiment of the present application;
fig. 8 is a schematic structural diagram of a device for cluster management according to an embodiment of the present application;
fig. 9 is a schematic structural diagram of a server according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
For convenience in understanding, an application scenario of the embodiment of the present application is first described, and after a cluster is started or restarted, if a master node does not exist in the cluster, the master node is elected by applying the method of the embodiment of the present application. Or, in the cluster operation process, each slave node in the cluster periodically receives the heartbeat message sent by the master node, so as to determine whether the master node is alive, and if the slave node does not receive the heartbeat message sent by the master node in at least one continuous period, the master node is considered to be not alive, and the master node needs to be elected again.
It is understood that other methods of determining whether a master node is alive in the prior art are also applicable to the embodiments of the present application.
The election principle of the main node is as follows: and selecting the node with the highest version as the master node from all the nodes in the cluster, and if a plurality of nodes with the highest versions exist or the versions of all the nodes are the same, selecting the node with the smallest group number as the master node.
Each node in the cluster is a server in the cluster, and the master node has higher authority and can manage each node as a slave node. Each node, which is a slave node, may establish a connection with the master node, and the master node may then maintain information for each node and determine the listening range of each node.
For example, as shown in fig. 1, the management node serves as a master node, and each of node a, node B, node C, and node D accesses the master node, which is four slave nodes of the master node. Wherein, each node in fig. 1 may be a node installed with a fluent tool.
It should be noted that "listening" may refer to acquiring data of other devices. In the embodiment of the application, the data to be monitored is data which needs to be acquired by nodes in the cluster, and the key value to be monitored is used for identifying a group of data to be monitored. For example, if data generated by a server is used as data to be monitored, the key value to be monitored corresponding to the data to be monitored generated by the server may be the IP address of the server.
The monitoring range of the node refers to the range of the data to be monitored, which is responsible for monitoring by the node, and the monitoring range can be represented by the key value to be monitored. For example, if the monitoring range of the node a is the key value 1 to be monitored and the key value 2 to be monitored, it indicates that the node a needs to monitor the data to be monitored corresponding to the key value 1 to be monitored and the data to be monitored corresponding to the key value 2 to be monitored.
In the embodiment of the application, each node in the cluster is used as a data processing tool for monitoring and processing data to be monitored. The data to be monitored can be configured according to business requirements, and the key value to be monitored corresponding to the data to be monitored is stored in the database in advance. After monitoring the data to be monitored, each node may process the data to be monitored according to an actual service scenario, and the method for processing the data to be monitored may refer to related technologies, which is not limited in the embodiment of the present application.
As an example, if the cluster is responsible for collecting log data for each game server for a game, the IP address of each game server may be stored in the database as the key value to be listened to. Therefore, the main node can obtain the key value to be monitored from the database according to the method provided by the embodiment of the application, determine the key value to be monitored of each node, and send the key value to be monitored obtained from the database to each node, so that each node monitors the log data generated by the server corresponding to the key value to be monitored according to the key value to be monitored distributed by the main node.
For example, if it is determined that the key value to be monitored received by node C is the IP address of game server 1 and the IP address of game server 2, node C monitors the log data generated by game server 1 and game server 2 according to the IP address of game server 1 and the IP address of game server 2. In other words, the data to be monitored by the node C is log data generated by the game server 1 and the game server 2.
In order to simplify cluster management, in combination with the scenario shown in fig. 1, an embodiment of the present application provides a method for cluster management, where the method is applied to a master node in a cluster, and as shown in fig. 2, the method includes:
s201, acquiring a survival node list.
The surviving node list includes Internet Protocol (IP) addresses of nodes that establish connection with the host node.
For example, in connection with fig. 1, the master node may determine that the IP addresses of node a, node B, node C, and node D are included in the surviving node list.
S202, calculating the hash value of the IP address of each node in the survival node list.
And S203, respectively mapping each node to a position on the hash ring according to the hash value of the IP address of each node.
Wherein the master node may construct a hash ring, which may include 2, as an example31-1 location, upon determining the hash value of the IP address of each node, the master node may map each node to a location on the hash ring based on the hash value.
Illustratively, as shown in fig. 3, the node a, the node B, the node C, and the node D in fig. 1 may be mapped to a position in the hash ring, and the four nodes are simply referred to as the node a, the node B, the node C, and the node D in fig. 3.
S204, acquiring the key values to be monitored corresponding to the data to be monitored, and determining the hash value corresponding to each key value to be monitored.
In one embodiment, the data to be listened to by each node is in the form of a key-value.
For example, if the cluster needs to process log data, and the log data includes an audit log, an update log, and a system log, the identifier of each log may be used as a key value, and the log data of each log may be used as a value. It can be understood that the database stores the identifiers of various logs, and the key value to be monitored acquired from the database by the master node is the identifier of each log.
For another example, if the cluster needs to process log data of 100 servers, the IP address of each server may be used as a key value, and the log data generated by each server may be used as a value. It can be understood that the database stores the IP addresses of the servers, and the key value to be monitored acquired from the database by the master node is the IP address of each server.
The main node can read each key value needing monitoring from the database and calculate the hash value of each key value.
S205, mapping the hash value corresponding to each key value to be monitored to the position of the hash ring, and determining the monitoring range of each node.
After the main node determines the hash value corresponding to each key value to be monitored, the position of each hash value on the hash ring can be determined, and then each key value to be monitored is mapped to one position on the hash ring. And then, for each key value to be monitored, determining the first node searched according to the anticlockwise sequence as the node for monitoring the key value to be monitored from the position of the key value to be monitored on the hash ring.
For example, assuming that the key values to be listened to are keys 1 to 9, as shown in fig. 3, keys 1 to 9 may be mapped onto the hash ring according to the hash values of keys 1 to 9, respectively.
Taking key1 as an example, if the first node found from key1 on the hash ring in a counterclockwise order is node a, the node for listening to the data corresponding to key1 may be determined to be node a. According to this approach, as can be seen in fig. 3, node a may listen for keys 1 and key 2; node B may listen for key3, key4, and key 5; node C may listen for key6, key7, and key 8; node D may listen for key 9.
It can be understood that, after the master node determines the monitoring range of each node, the master node may send the key value to be monitored to each node in the surviving node list, so that each node monitors the value corresponding to the acquired key value to be monitored.
By adopting the method provided by the embodiment of the application, the main node can obtain the surviving node list, calculate the hash value of the IP address of each node in the surviving node list, map each node to a position on the hash ring according to the hash value, obtain the key value to be monitored, determine the hash value corresponding to each key value to be monitored, and further map the hash value corresponding to each key value to be monitored to the position of the hash ring according to the hash value, and determine the monitoring range monitored by each node.
Compared with the prior art, the master node determines the monitoring range of each node according to the hash ring, and even if the nodes are newly added or separated and load balancing is carried out again, because the monitoring range of each node is determined by the nodes and the positions of the key values to be monitored on the hash ring, the unchanged nodes can still be mapped to the same positions on the hash ring, and the positions of the key values to be monitored mapped to the hash ring can not be changed, so the monitoring range of the unchanged nodes can not be changed generally. Only nodes adjacent to the detached node may need to monitor the key value to be monitored, which is responsible for monitoring by the detached node; or a part of key values to be monitored, which are required to be monitored by nodes adjacent to the newly added node, are monitored by the newly added node instead. Only the range of listening for nodes adjacent to the newly added node or the node away from it may change. Compared with the prior art that the key values to be monitored corresponding to the nodes are changed after load balancing is carried out again, the method provided by the embodiment of the application has small influence on the nodes which are not changed in the surviving node list when the cluster is changed, and compared with the prior art, the service delay can be reduced.
In addition, in the prior art, the master node needs to maintain metadata (meta) information of the whole cluster, the meta information records index information of each node in the current cluster, configuration information, persistent state information and the like of each node, the data volume of the meta information is large, once the cluster needs to be restarted, each node needs to report own information to the master node again, so that the master node can regenerate the meta information, and it can be seen that cluster management is heavy. By adopting the method of the embodiment of the application, the information needing interaction among all the nodes is less, even if the cluster is restarted, the main node only needs to acquire the IP address of each node, and does not need to acquire the index information and the configuration information of each node, namely the persistent information and the like, namely the meta information of the whole cluster does not need to be maintained, the cluster management work is simplified, the light-weight cluster management work can be realized, and the load balancing and restarting work can be quickly realized.
When a node in the cluster changes, for example, a new node or a node that is separated from the cluster occurs, the monitoring range of the node in the cluster needs to be adjusted, and the following method for performing load balancing again after updating the surviving node list is described, as shown in fig. 4, specifically includes the following steps:
s401, when detecting that a newly added node is accessed, adding the newly added node into the survival node list to obtain an updated survival node list.
When the new node exists in the cluster, the new node requests to access the host node, and after the host node establishes connection with the new node, the IP address of the new node can be added into the surviving node list.
S402, when a disconnected node which is disconnected with the main node is detected, the disconnected node is deleted from the survival node list, and an updated survival node list is obtained.
After each node accesses the master node, each node periodically sends a heartbeat message to the master node, and the master node can determine whether a detached node exists or not based on the heartbeat message.
In one embodiment, the master node may determine at regular intervals whether each node in the list of surviving nodes has detached. For example, if the cycle of sending the heartbeat message by each node is 60 seconds, if it is determined that the heartbeat message of a certain node is not received within 100 seconds, it is determined that the node has disconnected from the master node, that is, the node is a detached node, and the IP address of the detached node may be deleted from the surviving node list.
It should be noted that S401 and S402 are executed when the node accessing the master node changes, for example, a new node or a node leaving the master node exists, and therefore S401 and S402 may not exist at the same time. For example, if there is a new node but there is no off node, only S401 is executed, and S402 is not executed; if the node exists, but the newly added node does not exist, only executing S402, not executing S401; if there are both new nodes and detached nodes, both S401 and S402 need to be executed, but the execution sequence of S401 and S402 is not limited in the embodiment of the present application.
And S403, calculating the hash value of the IP address of each node in the updated surviving node list.
S404, according to the hash value of the IP address of each node in the updated survival node list, each node is mapped to a position on a hash ring.
For example, as shown in fig. 3 and 5, if a node E is added to the surviving node list, the updated surviving node list includes the IP addresses of the node a, the node B, the node C, the node D, and the node E. It can be understood that the IP addresses of the node a, the node B, the node C, and the node D are not changed, so the corresponding hash values are also not changed, and therefore the positions mapped onto the hash ring are also not changed. After the host node calculates the hash value of the newly added IP address of the node E, the node E may be mapped to a corresponding position of the hash ring according to the hash value.
S405, determining the monitoring range of each node in the updated surviving node list according to the position of the hash value mapping to the hash ring corresponding to each key value to be monitored.
The master node may determine, for each key value to be monitored, a first node found in a counterclockwise order from the position of the key value to be monitored on the hash ring as a node for monitoring the key value to be monitored, where the hash value corresponding to each key value to be monitored does not change, and therefore the position on the hash ring does not change.
For example, as shown in FIG. 5, key1 is still being listened to by node A, and since node E is newly added, the first node found starting from the position of key2 on the hash ring in a counterclockwise order is node E, so the node that is used to listen to key2 can be determined to be node E. As seen from fig. 5, the listening ranges of the other nodes except for node a are unchanged, and node B may listen to key3, key4 and key 5; node C may listen for key6, key7, and key 8; node D may listen for key 9.
As can be seen from comparison between fig. 3 and fig. 5, when a new node appears in the cluster by using the method of the embodiment of the present application, although the embodiment of the present application also needs to perform load balancing again, since the positions of nodes a to D on the hash ring are not changed, and the positions of keys 1 to 9 on the hash ring are also not changed. Therefore, in fig. 5, the monitoring ranges of the node B, the node C and the node D are not changed, and only the key2 originally responsible for monitoring by the node a is changed to be monitored by the newly added node E, so that more data migration work cannot be generated, the cluster restarting speed is higher, and the influence on the service delay is smaller.
In a possible implementation manner, in order to more uniformly distribute the key values to be monitored to the nodes, in this embodiment of the present application, a virtual node may be further set for each node, and the specific method includes:
and respectively setting a preset number of virtual nodes for each node in the survival node list, then calculating the hash value of the identifier of each virtual node, and respectively mapping each virtual node to a position on the hash ring according to the hash value of the IP address of each virtual node.
Each virtual node corresponding to one node is respectively identified by the IP address of the node and the label of the virtual node.
For example, if the IP address of node 1 is 10.1.1.1, and node 1 corresponds to two virtual nodes, namely virtual node a and virtual node B, the identifier of virtual node a may be set to 10.1.1.1: a, setting the identification of the virtual node B to be 10.1.1.1: B.
after the virtual nodes are set, the method for determining the monitoring range of each node in the surviving node list comprises the following steps:
and mapping each key value to be monitored to a position on the hash ring according to the hash value corresponding to each key value to be monitored, and searching nodes and virtual nodes in a counterclockwise sequence from the position of the key value to be monitored on the hash ring for each key value to be monitored.
If the first node searched according to the anticlockwise sequence is a virtual node, determining that the key value to be monitored is monitored by the node corresponding to the searched virtual node; alternatively, the first and second electrodes may be,
and if the first node searched according to the anticlockwise sequence is not the virtual node, determining that the key value to be monitored is monitored by the searched node.
For example, if the found first node is node 2, it is determined that the key value to be monitored is monitored by node 2. Or, if the found first node is the virtual node a and the virtual node a is the virtual node of the node 1, determining that the key value to be monitored is monitored by the node 1.
It should be noted that the number of virtual nodes corresponding to each node may be preset, and the smaller the number of nodes in the cluster, the larger the number of virtual nodes set for each node; conversely, the larger the number of nodes in the cluster, the smaller the number of virtual nodes set for each node.
It should be noted that the virtual nodes are set when the number of nodes is less than what value, and the number of the set virtual nodes may be set according to an empirical value, which is not limited in the present application. By way of example and not limitation, the number of nodes and virtual nodes on the hash ring may be 32 or more.
The following describes a method for setting a virtual node with reference to a specific example. As shown in fig. 6, as an example, there are node a and node B in the cluster, and as can be seen from fig. 6, node a needs to listen to 6 key values, and node B only needs to listen to 2 key values, that is, when there are fewer nodes in the cluster, the load balancing effect is not good enough.
Therefore, virtual nodes can be set for node a and node B respectively, for example, node a corresponds to 2 virtual nodes, namely virtual node a1 and virtual node a2, the identification of virtual node a1 is the IP address of node a: 1, the identification of virtual node a2 is the IP address of node a: 2.
similarly, node B also corresponds to 2 virtual nodes, virtual node B1 and virtual node B2, where the identification of virtual node B1 is the IP address of node B: 1, the identification of virtual node B2 as the IP address of node B: 2.
the master node may calculate hash values of the identifiers of the virtual nodes, and then map the virtual nodes onto the hash ring according to the hash values, where the mapped positions are shown in fig. 7.
As can be seen from fig. 7, Key1 is still monitored by node a, and the first node found counterclockwise for Key2 and Key3 is virtual node a1, and then Key2 and Key3 are monitored by node a.
The first nodes found counterclockwise for keys 4 and keys 5 are all virtual node B1, so keys 4 and keys 5 are monitored by node B. And key6 is also listened to by node B. The first node looked up counter-clockwise for key7 is virtual node B2, so key7 is still listened to by node B.
After the virtual nodes are set, the node A monitors 4 key values, the node B monitors 4 key values, the number of the key values monitored by each node is the same, and the load balancing effect is better.
Based on the same inventive concept, an embodiment of the present application further provides an apparatus for cluster management, where the apparatus is applied to a master node in a cluster, and as shown in fig. 8, the apparatus includes: an acquisition module 801, a calculation module 802, a mapping module 803, and a determination module 804.
An obtaining module 801, configured to obtain a surviving node list, where the surviving node list includes internet protocol IP addresses of nodes that establish connection with a host node;
a calculating module 802, configured to calculate a hash value of an IP address of each node in the surviving node list;
a mapping module 803, configured to map each node to a position on a hash ring according to the hash value of the IP address of each node calculated by the calculating module 802;
the obtaining module 801 is further configured to obtain key values to be monitored corresponding to the data to be monitored, and determine hash values corresponding to the key values to be monitored;
the determining module 804 is configured to determine the monitoring range of each node according to the position where the hash value corresponding to each key value to be monitored is mapped to the hash ring.
In a possible implementation manner, the determining module 804 is specifically configured to:
mapping each key value to be monitored to a position on a hash ring according to the hash value corresponding to each key value to be monitored;
and for each key value to be monitored, determining the first node searched according to the anticlockwise sequence as the node for monitoring the key value to be monitored from the position of the key value to be monitored on the hash ring.
In one possible implementation, the apparatus further includes: and updating the module.
The updating module is used for adding the newly added node into the surviving node list when the access of the newly added node is detected, so as to obtain an updated surviving node list; and/or deleting the detached node from the live node list when the detached node which is disconnected with the main node is detected, so as to obtain an updated live node list.
In a possible implementation manner, the calculating module 802 is further configured to calculate a hash value of an IP address of each node in the updated surviving node list;
the mapping module 803 is further configured to map each node to a position on the hash ring according to the hash value of the IP address of each node in the updated surviving node list;
the determining module 804 is further configured to determine the monitoring range of each node in the updated surviving node list according to the position where the hash value corresponding to each key value to be monitored is mapped to the hash ring.
In one possible implementation, the apparatus further includes: and setting a module.
The setting module is used for setting a preset number of virtual nodes for each node in the survival node list, and each virtual node corresponding to one node is identified by the IP address of the node and the label of the virtual node;
the calculating module 802 is further configured to calculate a hash value of the identifier of each virtual node;
the mapping module 803 is further configured to map each virtual node to a position on the hash ring according to the hash value of the IP address of each virtual node.
In another possible implementation manner, the determining module 804 is specifically configured to:
mapping each key value to be monitored to a position on a hash ring according to the hash value corresponding to each key value to be monitored;
for each key value to be monitored, searching nodes and virtual nodes in a counterclockwise sequence from the position of the key value to be monitored on the hash ring;
if the first node searched according to the anticlockwise sequence is a virtual node, determining that the key value to be monitored is monitored by the node corresponding to the searched virtual node; alternatively, the first and second electrodes may be,
and if the first node searched according to the anticlockwise sequence is not the virtual node, determining that the key value to be monitored is monitored by the searched node.
In one possible implementation, the apparatus further includes: and a sending module.
And the sending module is used for sending the key value to be monitored, which is required to be monitored by the node, to each node respectively, so that each node can process data according to the key value to be monitored, which is required to be monitored by each node.
The embodiment of the present application further provides a server, which may be a master node in a cluster, as shown in fig. 9, and includes a processor 901, a communication interface 902, a memory 903, and a communication bus 904, where the processor 901, the communication interface 902, and the memory 903 complete communication with each other through the communication bus 904,
a memory 903 for storing computer programs;
the processor 901 is configured to implement the steps executed by the master node in the above method embodiments when executing the program stored in the memory 903.
The communication bus mentioned in the above server may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the server and other devices.
The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.
In yet another embodiment provided by the present application, a computer-readable storage medium is further provided, in which a computer program is stored, and the computer program, when executed by a processor, implements the steps of any of the above-mentioned cluster management methods.
In yet another embodiment provided by the present application, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the method of cluster management of any of the above embodiments.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the apparatus embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The above description is only for the preferred embodiment of the present application, and is not intended to limit the scope of the present application. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application are included in the protection scope of the present application.
Claims (12)
1. A method for cluster management, the method being applied to a master node in a cluster, the method comprising:
acquiring a survival node list, wherein the survival node list comprises Internet Protocol (IP) addresses of all nodes which are connected with the main node;
calculating the hash value of the IP address of each node in the survival node list;
respectively mapping each node to a position on a hash ring according to the hash value of the IP address of each node;
acquiring key values to be monitored corresponding to data to be monitored, and determining hash values corresponding to the key values to be monitored;
mapping the hash value corresponding to each key value to be monitored to the position of the hash ring, and determining the monitoring range of each node;
when detecting that a new node is accessed, adding the new node into the surviving node list to obtain an updated surviving node list; and/or the presence of a gas in the gas,
when a disconnected node which is disconnected with the main node is detected, deleting the disconnected node from the survival node list to obtain an updated survival node list;
after the obtaining the surviving node list, the method further comprises:
respectively setting a preset number of virtual nodes for each node in the surviving node list, wherein each virtual node corresponding to one node is respectively identified by the IP address of the node and the label of the virtual node;
calculating the hash value of the identifier of each virtual node;
and respectively mapping each virtual node to a position on the hash ring according to the hash value of the IP address of each virtual node.
2. The method according to claim 1, wherein the determining the listening range of each node according to the position of the hash value mapping to the hash ring corresponding to each key value to be listened comprises:
mapping each key value to be monitored to a position on the hash ring according to the hash value corresponding to each key value to be monitored;
and for each key value to be monitored, determining the first node searched according to the anticlockwise sequence as the node for monitoring the key value to be monitored from the position of the key value to be monitored on the hash ring.
3. The method according to claim 1, wherein after obtaining the updated list of surviving nodes, the method further comprises:
calculating the hash value of the IP address of each node in the updated surviving node list;
respectively mapping each node to a position on the hash ring according to the hash value of the IP address of each node in the updated survival node list;
and determining the monitoring range of each node in the updated surviving node list according to the position of the hash value mapping to the hash ring corresponding to each key value to be monitored.
4. The method according to claim 1, wherein the determining the listening range of each node according to the position of the hash value mapping to the hash ring corresponding to each key value to be listened comprises:
mapping each key value to be monitored to a position on the hash ring according to the hash value corresponding to each key value to be monitored;
for each key value to be monitored, searching nodes and virtual nodes in a counterclockwise sequence from the position of the key value to be monitored on the hash ring;
if the first node searched according to the anticlockwise sequence is a virtual node, determining that the key value to be monitored is monitored by the node corresponding to the searched virtual node; alternatively, the first and second electrodes may be,
and if the first node searched according to the anticlockwise sequence is not the virtual node, determining that the key value to be monitored is monitored by the searched node.
5. The method according to any one of claims 1 to 4, wherein after determining the listening range of each node according to the position of the hash ring mapped to the hash value corresponding to each key value to be listened to, the method further comprises:
and respectively sending the key value to be monitored, which is required to be monitored by the node, to each node, so that each node can process data according to the key value to be monitored, which is required to be monitored.
6. An apparatus for cluster management, the apparatus being applied to a master node in a cluster, the apparatus comprising:
an obtaining module, configured to obtain a surviving node list, where the surviving node list includes internet protocol IP addresses of nodes that establish connection with the host node;
the calculation module is used for calculating the hash value of the IP address of each node in the survival node list;
the mapping module is used for mapping each node to a position on a hash ring according to the hash value of the IP address of each node calculated by the calculating module;
the acquisition module is further used for acquiring the key values to be monitored corresponding to the data to be monitored and determining the hash values corresponding to the key values to be monitored;
the determining module is used for mapping the hash value corresponding to each key value to be monitored to the position of the hash ring and determining the monitoring range of each node;
the updating module is used for adding the newly added node into the surviving node list when the access of the newly added node is detected, so as to obtain an updated surviving node list; and/or deleting the disconnected node from the surviving node list when the disconnected node disconnected with the main node is detected, so as to obtain an updated surviving node list;
the device further comprises: setting a module;
the setting module is used for setting a preset number of virtual nodes for each node in the surviving node list, and each virtual node corresponding to one node is identified by the IP address of the node and the label of the virtual node;
the calculation module is further configured to calculate a hash value of the identifier of each virtual node;
the mapping module is further configured to map each virtual node to a position on the hash ring according to the hash value of the IP address of each virtual node.
7. The apparatus of claim 6, wherein the determining module is specifically configured to:
mapping each key value to be monitored to a position on the hash ring according to the hash value corresponding to each key value to be monitored;
and for each key value to be monitored, determining the first node searched according to the anticlockwise sequence as the node for monitoring the key value to be monitored from the position of the key value to be monitored on the hash ring.
8. The apparatus of claim 6,
the calculation module is further configured to calculate a hash value of an IP address of each node in the updated surviving node list;
the mapping module is further configured to map each node to a position on the hash ring according to the hash value of the IP address of each node in the updated surviving node list;
the determining module is further configured to determine a monitoring range of each node in the updated surviving node list according to a position where the hash value corresponding to each key value to be monitored is mapped to the hash ring.
9. The apparatus of claim 6, wherein the determining module is specifically configured to:
mapping each key value to be monitored to a position on the hash ring according to the hash value corresponding to each key value to be monitored;
for each key value to be monitored, searching nodes and virtual nodes in a counterclockwise sequence from the position of the key value to be monitored on the hash ring;
if the first node searched according to the anticlockwise sequence is a virtual node, determining that the key value to be monitored is monitored by the node corresponding to the searched virtual node; alternatively, the first and second electrodes may be,
and if the first node searched according to the anticlockwise sequence is not the virtual node, determining that the key value to be monitored is monitored by the searched node.
10. The apparatus according to any one of claims 6-9, further comprising:
and the sending module is used for sending the key value to be monitored, which is required to be monitored by the node, to each node respectively, so that each node can process data according to the key value to be monitored, which is required to be monitored by each node.
11. A server comprising a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor, the processor being caused by the machine-executable instructions to: carrying out the method steps of any one of claims 1 to 5.
12. A machine-readable storage medium having stored thereon machine-executable instructions that, when invoked and executed by a processor, cause the processor to: carrying out the method steps of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910351626.XA CN110071978B (en) | 2019-04-28 | 2019-04-28 | Cluster management method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910351626.XA CN110071978B (en) | 2019-04-28 | 2019-04-28 | Cluster management method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110071978A CN110071978A (en) | 2019-07-30 |
| CN110071978B true CN110071978B (en) | 2022-04-22 |
Family
ID=67369357
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910351626.XA Active CN110071978B (en) | 2019-04-28 | 2019-04-28 | Cluster management method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110071978B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110971480B (en) * | 2019-10-18 | 2022-08-19 | 平安科技（深圳）有限公司 | Computer network condition monitoring method and device, computer equipment and storage medium |
| CN110841282B (en) * | 2019-10-30 | 2020-12-11 | 腾讯科技（深圳）有限公司 | Data processing method and device, storage medium and electronic device |
| CN113127409A (en) * | 2019-12-31 | 2021-07-16 | 深圳云天励飞技术有限公司 | Management method, device, equipment and medium of operation cluster for data real-time archiving |
| CN111478953B (en) * | 2020-03-27 | 2022-09-06 | 北京金山云网络技术有限公司 | Self-construction method, device, system, equipment and storage medium of server cluster |
| CN111781890A (en) * | 2020-07-14 | 2020-10-16 | 厦门海为科技有限公司 | Multi-equipment engineering communication method and system in configuration engineering |
| CN112231080B (en) * | 2020-09-27 | 2024-01-26 | 武汉物易云通网络科技有限公司 | Task scheduling method and device based on multiple hash rings with different precision |
| CN112492008B (en) * | 2020-11-19 | 2022-05-20 | 深圳壹账通智能科技有限公司 | Node position determination method and device, computer equipment and storage medium |
| CN113821340A (en) * | 2021-08-27 | 2021-12-21 | 济南浪潮数据技术有限公司 | Dynamic balancing method, system, terminal and storage medium of distributed system |
| CN115314558A (en) * | 2022-08-09 | 2022-11-08 | 中国电信股份有限公司 | Resource allocation method and device in computational power network, storage medium and electronic equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105187547A (en) * | 2015-09-25 | 2015-12-23 | 浪潮(北京)电子信息产业有限公司 | Load balancing method and system |
| CN108551474A (en) * | 2018-03-26 | 2018-09-18 | 南京邮电大学 | A kind of load-balancing method of server cluster |
| CN108768985A (en) * | 2018-05-17 | 2018-11-06 | 成都致云科技有限公司 | A kind of accessed node access distribution method and device |
| CN109040212A (en) * | 2018-07-24 | 2018-12-18 | 苏州科达科技股份有限公司 | Equipment access server cluster method, system, equipment and storage medium |
-
2019
- 2019-04-28 CN CN201910351626.XA patent/CN110071978B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105187547A (en) * | 2015-09-25 | 2015-12-23 | 浪潮(北京)电子信息产业有限公司 | Load balancing method and system |
| CN108551474A (en) * | 2018-03-26 | 2018-09-18 | 南京邮电大学 | A kind of load-balancing method of server cluster |
| CN108768985A (en) * | 2018-05-17 | 2018-11-06 | 成都致云科技有限公司 | A kind of accessed node access distribution method and device |
| CN109040212A (en) * | 2018-07-24 | 2018-12-18 | 苏州科达科技股份有限公司 | Equipment access server cluster method, system, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110071978A (en) | 2019-07-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110071978B (en) | Cluster management method and device | |
| US10997211B2 (en) | Systems and methods for database zone sharding and API integration | |
| US20180314750A1 (en) | Systems and methods for database zone sharding and api integration | |
| US9367261B2 (en) | Computer system, data management method and data management program | |
| CA2929777C (en) | Managed service for acquisition, storage and consumption of large-scale data streams | |
| US8990243B2 (en) | Determining data location in a distributed data store | |
| AU2014346366B2 (en) | Partition-based data stream processing framework | |
| US20180189367A1 (en) | Data stream ingestion and persistence techniques | |
| US20170357703A1 (en) | Dynamic partitioning techniques for data streams | |
| US9847907B2 (en) | Distributed caching cluster management | |
| US10462250B2 (en) | Distributed caching cluster client configuration | |
| CA2896865C (en) | Method and system for using a recursive event listener on a node in hierarchical data structure | |
| US10860604B1 (en) | Scalable tracking for database udpates according to a secondary index | |
| US9529772B1 (en) | Distributed caching cluster configuration | |
| US12001290B2 (en) | Performing a database backup based on automatically discovered properties | |
| US11500755B1 (en) | Database performance degradation detection and prevention | |
| CN111708755A (en) | Data migration method, device, system, electronic equipment and readable storage medium | |
| US8447730B1 (en) | Probe system for replication monitoring | |
| KR102665749B1 (en) | Method and apparatus for ensuring continued device operational reliability in cloud-degraded mode | |
| JP2013025425A (en) | Distributed data management system, distributed data management method, and distributed data management program | |
| CN111221857B (en) | Method and apparatus for reading data records from a distributed system | |
| WO2019120629A1 (en) | On-demand snapshots from distributed data storage systems | |
| WO2022222665A1 (en) | Request recognition method and apparatus, and device and storage medium | |
| US20240022627A1 (en) | Domain name system based global server load balancing service | |
| CN111723146B (en) | Method for monitoring database, management system and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |