CN113127409A - Management method, device, equipment and medium of operation cluster for data real-time archiving - Google Patents
Management method, device, equipment and medium of operation cluster for data real-time archiving Download PDFInfo
- Publication number
- CN113127409A CN113127409A CN201911421888.5A CN201911421888A CN113127409A CN 113127409 A CN113127409 A CN 113127409A CN 201911421888 A CN201911421888 A CN 201911421888A CN 113127409 A CN113127409 A CN 113127409A
- Authority
- CN
- China
- Prior art keywords
- cluster
- node
- operation node
- effective
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007726 management method Methods 0.000 title claims abstract description 91
- 238000000034 method Methods 0.000 claims abstract description 11
- 238000004590 computer program Methods 0.000 claims description 30
- 238000004891 communication Methods 0.000 claims description 22
- 238000007405 data analysis Methods 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 238000012544 monitoring process Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/113—Details of archiving
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Mathematical Physics (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a management method, a device, equipment and a medium of an operation cluster for real-time archiving of data, wherein the method comprises the following steps: when the operation nodes of the operation cluster are monitored to change, determining effective operation nodes currently contained in the operation cluster, and acquiring first identification information of each effective operation node; respectively carrying out hash operation on the first identification information of each effective operation node and the second identification information of each file data stored in the file database to generate a hash ring; determining the archive data to be stored currently by each effective operation node according to the positions of the first identification information and the second identification information on the hash ring; and respectively aiming at each effective operation node, managing the effective operation node according to the file data which needs to be stored currently by the effective operation node, so that all the file data in the file database are stored in the effective operation node currently contained in the operation cluster. The invention can improve the usability of the operation cluster.
Description
Technical Field
The invention relates to the technical field of data archiving, in particular to a management method, a management device, management equipment and a management medium of an operation cluster for data real-time archiving.
Background
With the progress of society, the circulation of personnel is more common, and the difficulty of personnel management is increased. Based on this, some departments or systems manage personnel by creating archival data. At present, a human face big data analysis system is generally used for analyzing a human face picture collected in real time, and the human face picture collected in real time is compared with archive data in an archive database to complete archiving. At present, the intermediate message component kafka (kafka is a distributed stream processing system) is generally adopted for processing data in a streaming mode, and collected data is archived in real time. Because real-time archive data can be stressed in the processing process by the total amount of database (i.e. archive database) and the pressure of real-time data traffic, it is common to complete real-time archive by combining with operation cluster service. When the operation nodes in the operation cluster are changed, if a certain operation node goes down, single-point data abnormity occurs (namely, the archive data stored by the operation node is lost), so that the accuracy of real-time archiving is affected. It can be seen that the availability of such an operation cluster for real-time archiving is currently not high.
Disclosure of Invention
The invention provides a management method, a device, equipment and a medium of an operation cluster for real-time data archiving, and aims to solve the problem that the usability of the operation cluster for real-time archiving is not high.
In order to achieve the above object, in a first aspect, an embodiment of the present invention provides a management method for an operation cluster for real-time archiving of data, where the management method includes:
when the change of the operation nodes of the operation cluster is monitored, determining effective operation nodes currently contained in the operation cluster, and acquiring first identification information of each effective operation node;
respectively carrying out hash operation on the first identification information of each effective operation node and the second identification information of each file data stored in the file database by using a consistent hash algorithm to generate a hash ring;
determining the archive data to be stored currently by each effective operation node according to the positions of the first identification information and the second identification information on the hash ring;
and respectively aiming at each effective operation node, managing the effective operation node according to the file data which needs to be stored currently by the effective operation node, so that all the file data in the file database are stored in the effective operation node currently contained in the operation cluster.
In a second aspect, an embodiment of the present invention further provides a management device for an operation cluster for real-time archiving of data, where the management device includes:
the first determining module is used for determining effective operation nodes currently contained in the operation cluster and acquiring first identification information of each effective operation node when the operation node of the operation cluster is monitored to change;
the generating module is used for respectively carrying out hash operation on the first identification information of each effective operation node and the second identification information of each file data stored in the file database by utilizing a hash algorithm to generate a hash ring;
the second determining module is used for determining the archive data to be stored currently by each effective operation node according to the positions of the first identification information and the second identification information on the hash ring;
and the management module is used for respectively aiming at each effective operation node and managing the effective operation node according to the file data which needs to be stored currently by the effective operation node, so that all the file data in the file database are stored in the effective operation node contained currently by the operation cluster.
In a third aspect, an embodiment of the present invention further provides a management device for an operation cluster for real-time data archiving, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the management method for the operation cluster for real-time data archiving when executing the computer program.
In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the computer program implements the steps of the above method for managing an operation cluster for real-time data archiving.
The scheme of the invention has at least the following beneficial effects:
in the embodiment of the invention, the operation nodes of the operation cluster are monitored, when the operation nodes of the operation cluster are monitored to be changed, the effective operation nodes currently contained in the operation cluster are determined, and the first identification information of each effective operation node is obtained; then, performing hash operation on the first identification information of each effective operation node and the second identification information of each archival data stored in the archival database respectively by using a consistent hash algorithm to generate a hash ring, and determining the archival data currently required to be stored by each effective operation node according to the positions of the first identification information and the second identification information on the hash ring; and finally, respectively aiming at each effective operation node, managing the effective operation node according to the file data which needs to be stored currently by the effective operation node, so that all the file data in the file database are stored in the effective operation nodes currently contained in the operation cluster, thereby ensuring that the file data cannot be lost due to the crash of a certain operation node and further improving the usability of the operation cluster.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart of a management method of an operation cluster for real-time data archiving according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a partial structure of a human face big data analysis system according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a management apparatus of an operation cluster for real-time data archiving according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a management device of an operation cluster for real-time data archiving according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first" and "second" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present invention, "a plurality" means two or more unless specifically defined otherwise.
As shown in fig. 1, an embodiment of the present invention provides a management method for an operation cluster for real-time archiving of data, where the management method is applicable to a management node of the operation cluster. Specifically, the management method comprises the following steps:
In the embodiment of the present invention, the operation cluster is mainly used for data real-time archiving, and the operation cluster includes a plurality of operation nodes for archiving a face picture acquired in real time. And the operation nodes of the operation cluster are considered to be changed by the aid of the operation nodes which are down in the operation cluster or the newly added operation nodes.
Specifically, in the embodiment of the present invention, the monitoring of the operation cluster may be completed by monitoring whether an operation node in the operation cluster is down and whether an operation node is newly added. More specifically, when communication disconnection with a first operation node of an operation cluster is detected, the first operation node is determined to be down, and changes of the operation node of the operation cluster are determined to be monitored. In addition, when a communication connection request which is sent by a second operation node and used for requesting to establish communication connection is received, the second operation node is determined to be a newly added operation node of the operation cluster, and the operation node of the operation cluster is determined to be monitored to change. It should be noted that, in the embodiment of the present invention, the operation cluster may be monitored through ZooKeeper (ZooKeeper is a distributed, open-source distributed application coordination service). The ZooKeeper monitors whether a down operation node occurs in the operation cluster by monitoring the heartbeat signals of all operation nodes of the operation cluster, and determines that the operation node is down when the heartbeat signal of a certain operation node is not monitored; when an operation node is added, the operation node sends a communication connection request (the communication connection request carries the first identification information of the operation node) for requesting to establish communication connection to the ZooKeeper so as to establish communication connection with the ZooKeeper.
After determining that the operation node of the operation cluster changes, it is necessary to determine an effective operation node currently included in the operation cluster, that is, an operation node currently in a working state in the operation cluster. It should be noted that, when the operation node of the operation cluster changes due to the operation node that is down in the operation cluster, the management node may obtain the first identification information of each effective operation node from a cache (in which the first identification information of each operation node of the operation cluster is stored); when the operation node of the operation cluster is changed due to the addition of the operation node in the operation cluster, the first identification information of the added operation node can be obtained from the ZooKeeper.
As a preferred example, the first identification information of the valid operation node may be an Internet Protocol (IP) address of the valid operation node, and may also be other information of the valid operation node, and may uniquely identify the valid operation node, such as a number.
And step 12, performing hash operation on the first identification information of each effective operation node and the second identification information of each archive data stored in the archive database respectively by using a consistent hash algorithm to generate a hash ring.
In the embodiment of the present invention, a consistent hash (hash) algorithm is used to perform a hash operation on each first identification information and each second identification information, so that it is ensured that archive data is uniformly distributed to each effective operation node in the following process, which is helpful to improve the availability of an operation cluster.
As a preferable example, the second identification information of the archive data is information that can uniquely identify the archive data, and may specifically be information such as a number and a name of the archive data.
And step 13, determining the archive data currently required to be stored by each effective operation node according to the positions of the first identification information and the second identification information on the hash ring.
In the embodiment of the present invention, in a hash (hash) ring generated by using a consistent hash algorithm, each first identification information and each second identification information have corresponding positions in the hash ring, so according to a hash ring principle, according to the positions of each first identification information and each second identification information on the hash ring, the first identification information to which each second identification information belongs can be determined, and the second identification information corresponds to archive data, and the first identification information corresponds to an effective operation node, so that the archive data currently required to be stored by each effective operation node can be determined.
And step 14, respectively aiming at each effective operation node, managing the effective operation node according to the file data which needs to be stored currently by the effective operation node, so that all the file data in the file database are stored in the effective operation node currently contained in the operation cluster.
In the embodiment of the present invention, when determining the file data currently required to be stored in each valid operation node, the valid operation node is managed according to the file data currently required to be stored in the valid operation node, so as to store all the file data in the file database (i.e. the base database) in each valid operation node in a balanced manner, thereby avoiding the loss of the file data.
It is worth mentioning that, in the embodiment of the present invention, the operation nodes of the operation cluster are monitored, and when the operation nodes of the operation cluster are monitored to be changed, the effective operation nodes currently included in the operation cluster are determined, and the first identification information of each effective operation node is obtained; then, respectively performing hash operation on the first identification information of each effective operation node and the second identification information of each archival data stored in the archival database by using a consistent hash algorithm to generate a hash ring, and determining archival data which needs to be stored currently by each effective operation node according to the positions of each first identification information and each second identification information on the hash ring; and finally, respectively aiming at each effective operation node, managing the effective operation node according to the file data which needs to be stored currently by the effective operation node, so that all the file data in the file database are stored in the effective operation nodes currently contained in the operation cluster, thereby ensuring that the file data cannot be lost due to the crash of a certain operation node and further improving the usability of the operation cluster.
It should be noted that, since the storage space of each effective operation node is fixed, the remaining capacity of each effective operation node needs to be detected, so that when the archive data amount of the base database increases, the newly added archive data is distributed and stored, and the accuracy of real-time archiving is further ensured.
Specifically, in an embodiment of the present invention, the management method further includes the following steps:
when third target archive data which needs to be stored in the effective operation nodes currently contained in the operation cluster are received, the remaining capacity of each effective operation node currently contained in the operation cluster is detected. When the residual capacity of a target effective operation node in the effective operation nodes currently contained in the operation cluster is detected to be smaller than the preset capacity, the target effective operation node is removed from the hash ring, the third identification information of the third target archive data is subjected to hash operation by using the consistency hash algorithm, the effective operation node of the third target archive data in the hash ring is determined, and finally the third target archive data is loaded into the determined effective operation node.
The third target archive data can be newly added archive data in the base data; the preset capacity can be set according to actual conditions. That is, in the embodiment of the present invention, when third target archive data needs to be stored in an effective operation node currently included in an operation cluster, if a target effective operation node whose remaining capacity is smaller than a preset capacity exists at this time, the target effective operation node is removed from the hash ring, a hash operation is performed on third identification information (which may be information such as a code and a name) of the third target archive data by using a consistent hash algorithm, then an effective operation node to which the third target archive data belongs in the hash ring is determined according to a position of the third identification information in the hash ring and a position of first identification information of the remaining effective operation node in the hash ring, and finally the third target archive data is loaded into the determined effective operation node.
In order to facilitate understanding of the operation cluster, the operation cluster is further described herein with reference to a partial structure of a human face big data analysis system (the human face big data analysis system is mainly used for analyzing a human face picture collected in real time, and comparing the human face picture collected in real time with archive data in an archive database to complete archiving). As shown in fig. 2, the human face big data analysis system includes an intermediate message component kafka, a management node of an operation cluster, a ZooKeeper, and each operation node in the operation cluster. The operation nodes are used for storing archive data in an archive database, the face pictures collected in real time are transmitted to the management node through the intermediate message component kafka and then transmitted to the operation nodes by the management node, the face pictures collected in real time are compared with the archive data stored by the operation nodes respectively to complete archiving, and an archiving result is sent to the kafka to be transmitted to the next process service processing. The ZooKeeper can be used for monitoring the operation cluster and transmitting the monitoring result to the management node in time so as to manage each operation node. It should be noted that, because the filing of the event (i.e. the face picture collected in real time) is based on the filing of the database data, the same person files the same file by comparing the database data, so the real-time filing needs to enter the archive of the whole system. Specifically, when the management node is initially started, each archive data in the archive database (i.e., the base database) is transmitted to the management node through the intermediate message component kafka, and then the management node allocates to each operation node according to the hash ring.
Next, a specific implementation of managing valid operation nodes according to the reason why the operation nodes of the operation cluster change will be described.
When communication disconnection with a first operation node of an operation cluster is detected, namely the first operation node in the operation cluster is down, determining an effective operation node currently contained in the operation cluster, wherein the step comprises the following steps: and deleting the first operation node from the operation cluster, and taking other operation nodes except the first operation node in the operation cluster as effective operation nodes currently contained in the operation cluster.
Correspondingly, for any effective operation node currently contained in the operation cluster, according to the archive data which needs to be stored currently by the effective operation node, the concrete implementation mode for managing the effective operation node comprises the following steps:
the first step is that first target archive data stored in the first operation node is determined according to a hash ring generated in the previous management of the operation nodes of the operation cluster.
It should be noted that, whenever the operation node of the operation cluster changes, the operation node of the operation cluster needs to be managed again, so as to ensure that all the archive data in the archive database can be stored in the operation node currently included in the operation cluster after the operation node of the operation cluster changes. Each management step is the above step 11 to step 14, so that when any effective operation node currently included in the operation cluster is managed, a hash ring generated when the operation node of the operation cluster is managed in the previous time, that is, when the operation node of the operation cluster in the previous time changes, the hash ring generated when the operation node of the operation cluster is managed, and thus the first target archive data stored in the first operation node can be determined according to the hash ring.
And secondly, comparing the first target file data with the file data which needs to be stored currently by the effective operation node, and determining the file data which needs to be loaded to the effective operation node in the first target file data.
And thirdly, loading the determined file data to the effective operation node.
That is, in the embodiment of the present invention, after determining the first target archive data stored in the first operation node and the archive data currently to be stored by the effective operation node, the first target archive data and the archive data currently to be stored by the effective operation node are compared to determine the archive data to be loaded to the effective operation node in the first target archive data, and the management of the effective operation node is completed by loading the determined archive data to the effective operation node, so that the first target archive data stored in the first operation node is prevented from being lost, and the accuracy of real-time archiving is ensured.
For example, assuming that the computing cluster originally includes A, B, C computing nodes, the computing node C is down, and the active computing nodes currently included in the computing cluster are the computing node a and the computing node B, at this time, the manner of managing the computing node a and the computing node B is the same, and the management of the computing node a is taken as an example here. Specifically, the first target archive data stored in the operation node C can be determined in a hash ring generated in the previous management of the operation node of the operation cluster, then the first target archive data is compared with the archive data currently required to be stored by the operation node a, the archive data required to be loaded to the operation node a in the first target archive data is determined, and the determined archive data is loaded to the operation node a to complete the management of the operation node a.
When a communication connection request for requesting to establish communication connection sent by a second operation node is received, namely a second operation node is newly added in an operation cluster, the step of determining an effective operation node currently contained in the operation cluster comprises the following steps: and taking the second operation node and the operation node originally contained in the operation cluster as the effective operation node currently contained in the operation cluster.
Correspondingly, for any effective operation node currently contained in the operation cluster, according to the archive data which needs to be stored currently by the effective operation node, the concrete implementation mode for managing the effective operation node comprises the following steps:
and when the effective operation node is the second operation node, acquiring the file data which needs to be stored currently by the effective operation node from the file database, and loading the acquired file data into the effective operation node.
When the effective operation node is the operation node originally included in the operation cluster, determining second target archive data originally stored in the operation node originally included in the effective operation node according to a hash ring generated when the operation node of the operation cluster is managed in the previous time (namely, when the operation node of the operation cluster in the previous time is changed, the hash ring generated when the operation node of the operation cluster is managed can determine the second target archive data stored in the operation node originally included in the operation cluster according to the hash ring); and then comparing the second target file data with the file data which needs to be stored currently by the effective operation node, determining other file data in the second target file data except the file data which needs to be stored currently by the effective operation node, and deleting the determined other file data from the effective operation node.
That is, in the embodiment of the present invention, when a second operation node is newly added in the operation cluster, for the management of the second operation node, the file data that needs to be stored by the second operation node at present may be directly obtained from the file database, and the obtained file data is loaded into the second operation node. For the operation nodes originally contained in the operation cluster, second target archive data originally stored in the effective operation nodes are determined according to a hash ring generated in the previous management of the operation nodes of the operation cluster; and then comparing the second target file data with the file data which needs to be stored currently by the effective operation node, determining other file data in the second target file data except the file data which needs to be stored currently by the effective operation node, and deleting the determined other file data from the effective operation node to complete the management of the effective operation node.
For example, suppose that the operation cluster originally includes A, B operation nodes and a new operation node C, and the effective operation nodes currently included in the operation cluster are operation node a, operation node B and operation node C. At this time, for the operation node C, the archive data currently required to be stored by the operation node C can be directly obtained from the archive database, and the obtained archive data is loaded into the operation node C; the same way of managing the operation node a and the operation node B is used, and the operation node a is taken as an example here. Specifically, the second target archive data stored in the operation node a can be determined in the hash ring generated in the previous management of the operation node of the operation cluster, and then the second target archive data is compared with the archive data currently required to be stored by the operation node a, so that other archive data except the archive data currently required to be stored by the operation node a in the second target archive data is determined, and the determined other archive data is deleted from the operation node a, so that the management of the operation node a is completed.
It is worth mentioning that the processing speed of the human face big data analysis system can be greatly improved and the accuracy of real-time filing is ensured by adding new operation nodes in the operation cluster.
As shown in fig. 3, an embodiment of the present invention further provides a management device for an operation cluster for real-time archiving of data, where the management device includes: a first determining module 31, a generating module 32, a second determining module 33 and a managing module 34.
The first determining module 31 is configured to determine, when it is monitored that an operation node of an operation cluster changes, an effective operation node currently included in the operation cluster, and acquire first identification information of each effective operation node;
a generating module 32, configured to perform a hash operation on the first identification information of each valid operation node and the second identification information of each archive data stored in the archive database by using a hash algorithm, respectively, so as to generate a hash ring;
a second determining module 33, configured to determine, according to the positions of the first identification information and the second identification information on the hash ring, archive data currently to be stored in each effective operation node;
the management module 34 is configured to manage, for each effective operation node, the effective operation node according to the archive data that the effective operation node needs to store currently, so that all the archive data in the archive database are stored in the effective operation node included in the operation cluster currently.
In the embodiment of the present invention, the management device 30 of the operation cluster for real-time data archiving is a device corresponding to the above-described management method of the operation cluster for real-time data archiving, and can improve the usability of the operation cluster.
It should be noted that the management device 30 of the operation cluster for real-time data archiving includes all modules or units for implementing the management method of the operation cluster for real-time data archiving, and in order to avoid excessive repetition, details of each module or unit of the management device 30 of the operation cluster for real-time data archiving are not described herein.
As shown in fig. 4, an embodiment of the present invention further provides a management device for an operation cluster for real-time data archiving, including a memory 41, a processor 42, and a computer program 43 stored in the memory 41 and executable on the processor 42, where the processor 42 implements the steps of the management method for an operation cluster for real-time data archiving when executing the computer program 43.
Specifically, when the processor 42 of the management device 40 for an operation cluster for real-time data archiving executes the computer program 43, the following steps are implemented: when the change of the operation nodes of the operation cluster is monitored, determining effective operation nodes currently contained in the operation cluster, and acquiring first identification information of each effective operation node; respectively performing hash operation on the first identification information of each effective operation node and the second identification information of each file data stored in the file database by using a consistent hash algorithm to generate a hash ring; determining the current archive data to be stored of each effective operation node according to the positions of the first identification information and the second identification information on the hash ring; and respectively aiming at each effective operation node, managing the effective operation node according to the file data which needs to be stored currently by the effective operation node, so that all the file data in the file database are stored in the effective operation node currently contained in the operation cluster.
Optionally, when the processor 42 of the management device 40 for an operation cluster for real-time data archiving executes the computer program 43, the following steps are further implemented: when communication disconnection with a first operation node of an operation cluster is detected, determining that the first operation node is down, and determining that the operation node of the operation cluster changes; or when a communication connection request which is sent by a second operation node and used for requesting to establish communication connection is received, determining that the second operation node is a new operation node of an operation cluster, and determining that the operation node of the operation cluster changes.
Optionally, when detecting that the communication with the first operation node of the operation cluster is disconnected, the processor 42 of the management device 40 of the operation cluster for real-time data archiving executes the computer program 43 to further implement the following steps: and deleting the first operation node from the operation cluster, and taking other operation nodes except the first operation node in the operation cluster as effective operation nodes currently contained in the operation cluster.
Optionally, when the processor 42 of the management device 40 for an operation cluster for real-time data archiving executes the computer program 43, the following steps are further implemented: determining first target archive data stored in the first operation node according to a hash ring generated in the previous management of the operation nodes of the operation cluster; comparing the first target file data with the file data which needs to be stored currently by the effective operation node, and determining the file data which needs to be loaded to the effective operation node in the first target file data; and loading the determined file data to the effective operation node.
Optionally, when receiving a communication connection request for requesting to establish a communication connection sent by the second operation node, the processor 42 of the management device 40 for an operation cluster for real-time data archiving executes the computer program 43 to further implement the following steps: and taking the second operation node and the operation node originally contained in the operation cluster as the effective operation node currently contained in the operation cluster.
Optionally, when the processor 42 of the management device 40 for an operation cluster for real-time data archiving executes the computer program 43, the following steps are further implemented: when the effective operation node is the second operation node, acquiring the file data which needs to be stored currently by the effective operation node from the file database, and loading the acquired file data into the effective operation node; when the effective operation node is an operation node originally contained in the operation cluster, determining second target archive data originally stored in the effective operation node according to a hash ring generated in the previous management of the operation node of the operation cluster; comparing the second target file data with the file data which needs to be stored currently by the effective operation node, and determining other file data in the second target file data except the file data which needs to be stored currently by the effective operation node; and deleting the determined other file data from the effective operation node.
Optionally, when the processor 42 of the management device 40 for an operation cluster for real-time data archiving executes the computer program 43, the following steps are further implemented: when third target archive data which needs to be stored in effective operation nodes currently contained in the operation cluster are received, detecting the residual capacity of each effective operation node currently contained in the operation cluster; when detecting that the residual capacity of a target effective operation node in effective operation nodes currently contained in the operation cluster is smaller than a preset capacity, removing the target effective operation node from the hash ring, and performing hash operation on third identification information of third target archive data by using the consistency hash algorithm to determine an effective operation node to which the third target archive data belongs in the hash ring; and loading the third target file data into the determined effective operation node.
That is, in the embodiment of the present invention, when the processor 42 of the management device 40 for an arithmetic cluster for real-time data archiving executes the computer program 43, the steps of the above-described management method for an arithmetic cluster for real-time data archiving are implemented, and the usability of the arithmetic cluster can be improved.
Illustratively, the above-described computer program 43 may be partitioned into one or more modules/units, which are stored in the memory 41 and executed by the processor 42 to implement the present invention. And the one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, which are used for describing the execution process of the computer program 43 in the management device 40 of the operation cluster for archiving data in real time.
The management device 40 of the operation cluster for real-time data archiving may be a desktop computer, a notebook, a palm computer, a cloud server, or other computing devices. The management device 40 of the operation cluster for real-time data archiving includes, but is not limited to, a processor 42 and a memory 41. It will be understood by those skilled in the art that the schematic diagram is merely an example of the management device 40 of the operation cluster for real-time data archiving, and does not constitute a limitation to the management device 40 of the operation cluster for real-time data archiving, and may include more or less components than those shown, or combine some components, or different components, for example, the management device 40 of the operation cluster for real-time data archiving may further include an input and output device, a network access device, a bus, and the like.
The Processor 42 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, and the processor 42 is a control center of the management device 40 of the operation cluster for data real-time filing, and various interfaces and lines are used to connect the respective parts of the management device 40 of the entire operation cluster for data real-time filing.
The memory 41 may be used to store the computer program 43 and/or the module, and the processor 42 may implement various functions of the management device 40 of the operation cluster for real-time data archiving by running or executing the computer program 43 and/or the module stored in the memory 41 and calling the data stored in the memory 41. Specifically, the memory 41 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required by at least one function, and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the cellular phone, and the like. Further, the memory 41 may include a high speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), at least one magnetic disk storage device, a Flash memory device, or other volatile solid state storage device.
It should be noted that, since the processor 42 of the management device 40 for an operation cluster for real-time data archiving executes the computer program 43 to implement the steps of the management method for an operation cluster for real-time data archiving, all embodiments of the management method for an operation cluster for real-time data archiving can be applied to the management device 40 for an operation cluster for real-time data archiving, and can achieve the same or similar beneficial effects.
Furthermore, an embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored, and the computer program, when executed by a processor, implements the steps of the above-mentioned management method for an operation cluster for real-time data archiving.
That is, in an embodiment of the present invention, when being executed by a processor, a computer program of a computer-readable storage medium implements the steps of the above-mentioned method for managing an operation cluster for real-time data archiving, so as to improve the usability of the operation cluster.
Illustratively, the computer program of the computer-readable storage medium comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, and the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present invention, and are intended to be included within the scope of the present invention.
Claims (10)
1. A management method of an operation cluster for real-time archiving of data is characterized by comprising the following steps:
when the change of the operation nodes of the operation cluster is monitored, determining effective operation nodes currently contained in the operation cluster, and acquiring first identification information of each effective operation node;
respectively carrying out hash operation on the first identification information of each effective operation node and the second identification information of each file data stored in the file database by using a consistent hash algorithm to generate a hash ring;
determining the archive data to be stored currently by each effective operation node according to the positions of the first identification information and the second identification information on the hash ring;
and respectively aiming at each effective operation node, managing the effective operation node according to the file data which needs to be stored currently by the effective operation node, so that all the file data in the file database are stored in the effective operation node currently contained in the operation cluster.
2. The management method according to claim 1, wherein the management method further comprises:
when communication disconnection with a first operation node of an operation cluster is detected, determining that the first operation node is down, and determining that the operation node of the operation cluster changes; or
When a communication connection request which is sent by a second operation node and used for requesting to establish communication connection is received, the second operation node is determined to be a newly added operation node of an operation cluster, and the operation node of the operation cluster is determined to be monitored to change.
3. The method of claim 2, wherein when a communication disconnection with a first compute node of a compute cluster is detected;
the step of determining the valid operation nodes currently included in the operation cluster includes:
and deleting the first operation node from the operation cluster, and taking other operation nodes except the first operation node in the operation cluster as effective operation nodes currently contained in the operation cluster.
4. The method according to claim 3, wherein the step of managing the active operation node according to the archive data currently required to be stored by the active operation node comprises:
determining first target archive data stored in the first operation node according to a hash ring generated in the previous management of the operation nodes of the operation cluster;
comparing the first target file data with the file data which needs to be stored currently by the effective operation node, and determining the file data which needs to be loaded to the effective operation node in the first target file data;
and loading the determined file data to the effective operation node.
5. The management method according to claim 2, wherein when receiving a communication connection request sent by the second computing node for requesting to establish a communication connection;
the step of determining the valid operation nodes currently included in the operation cluster includes:
and taking the second operation node and the operation node originally contained in the operation cluster as the effective operation node currently contained in the operation cluster.
6. The method according to claim 5, wherein the step of managing the active operation node according to the archive data currently required to be stored by the active operation node comprises:
when the effective operation node is the second operation node, acquiring the file data which needs to be stored currently by the effective operation node from the file database, and loading the acquired file data into the effective operation node;
when the effective operation node is an operation node originally contained in the operation cluster, determining second target archive data originally stored in the effective operation node according to a hash ring generated in the previous management of the operation node of the operation cluster;
comparing the second target file data with the file data which needs to be stored currently by the effective operation node, and determining other file data in the second target file data except the file data which needs to be stored currently by the effective operation node;
and deleting the determined other file data from the effective operation node.
7. The management method according to claim 1, wherein the management method further comprises:
when third target archive data which needs to be stored in effective operation nodes currently contained in the operation cluster are received, detecting the residual capacity of each effective operation node currently contained in the operation cluster;
when detecting that the residual capacity of a target effective operation node in effective operation nodes currently contained in the operation cluster is smaller than a preset capacity, removing the target effective operation node from the hash ring, and performing hash operation on third identification information of third target archive data by using the consistent hash algorithm to determine an effective operation node to which the third target archive data belongs in the hash ring;
and loading the third target file data into the determined effective operation node.
8. A management apparatus for an operation cluster for real-time archiving of data, the management apparatus comprising:
the first determining module is used for determining effective operation nodes currently contained in the operation cluster and acquiring first identification information of each effective operation node when the operation node of the operation cluster is monitored to change;
the generating module is used for respectively carrying out hash operation on the first identification information of each effective operation node and the second identification information of each file data stored in the file database by utilizing a hash algorithm to generate a hash ring;
the second determining module is used for determining the archive data to be stored currently by each effective operation node according to the positions of the first identification information and the second identification information on the hash ring;
and the management module is used for respectively aiming at each effective operation node and managing the effective operation node according to the file data which needs to be stored currently by the effective operation node, so that all the file data in the file database are stored in the effective operation node contained currently by the operation cluster.
9. A management device for an operation cluster for real-time data archiving, comprising a memory, a processor and a computer program stored in the memory and operable on the processor, wherein the processor implements the steps of the management method for an operation cluster for real-time data archiving according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, in which a computer program is stored, which, when being executed by a processor, implements the steps of the method for managing an operational cluster for real-time archiving of data according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911421888.5A CN113127409A (en) | 2019-12-31 | 2019-12-31 | Management method, device, equipment and medium of operation cluster for data real-time archiving |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911421888.5A CN113127409A (en) | 2019-12-31 | 2019-12-31 | Management method, device, equipment and medium of operation cluster for data real-time archiving |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113127409A true CN113127409A (en) | 2021-07-16 |
Family
ID=76770682
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911421888.5A Pending CN113127409A (en) | 2019-12-31 | 2019-12-31 | Management method, device, equipment and medium of operation cluster for data real-time archiving |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113127409A (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080319956A1 (en) * | 2006-04-11 | 2008-12-25 | Brother Kogyo Kabushiki Kaisha | Tree-type broadcast system, reconnection process method, node device, node process program, server device, and server process program |
| CN104391863A (en) * | 2014-10-23 | 2015-03-04 | 中国建设银行股份有限公司 | Data storage method and device |
| CN105516231A (en) * | 2014-10-20 | 2016-04-20 | 中国电信股份有限公司 | Distributed cache method and system based on improved consistent hashing |
| US20160342588A1 (en) * | 2015-05-19 | 2016-11-24 | Morgan Stanley | Topology aware distributed storage system |
| CN106572153A (en) * | 2016-10-21 | 2017-04-19 | 乐视控股（北京）有限公司 | Data storage method and device of cluster |
| WO2017122922A1 (en) * | 2016-01-11 | 2017-07-20 | 충북대학교 산학협력단 | Load balancing system using data replication and data migration in distributed in-memory environment |
| CN108132830A (en) * | 2016-12-01 | 2018-06-08 | 北京金山云网络技术有限公司 | A kind of method for scheduling task, apparatus and system |
| CN109040212A (en) * | 2018-07-24 | 2018-12-18 | 苏州科达科技股份有限公司 | Equipment access server cluster method, system, equipment and storage medium |
| CN110071978A (en) * | 2019-04-28 | 2019-07-30 | 新华三信息安全技术有限公司 | A kind of method and device of cluster management |
-
2019
- 2019-12-31 CN CN201911421888.5A patent/CN113127409A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080319956A1 (en) * | 2006-04-11 | 2008-12-25 | Brother Kogyo Kabushiki Kaisha | Tree-type broadcast system, reconnection process method, node device, node process program, server device, and server process program |
| CN105516231A (en) * | 2014-10-20 | 2016-04-20 | 中国电信股份有限公司 | Distributed cache method and system based on improved consistent hashing |
| CN104391863A (en) * | 2014-10-23 | 2015-03-04 | 中国建设银行股份有限公司 | Data storage method and device |
| US20160342588A1 (en) * | 2015-05-19 | 2016-11-24 | Morgan Stanley | Topology aware distributed storage system |
| WO2017122922A1 (en) * | 2016-01-11 | 2017-07-20 | 충북대학교 산학협력단 | Load balancing system using data replication and data migration in distributed in-memory environment |
| CN106572153A (en) * | 2016-10-21 | 2017-04-19 | 乐视控股（北京）有限公司 | Data storage method and device of cluster |
| CN108132830A (en) * | 2016-12-01 | 2018-06-08 | 北京金山云网络技术有限公司 | A kind of method for scheduling task, apparatus and system |
| CN109040212A (en) * | 2018-07-24 | 2018-12-18 | 苏州科达科技股份有限公司 | Equipment access server cluster method, system, equipment and storage medium |
| CN110071978A (en) * | 2019-04-28 | 2019-07-30 | 新华三信息安全技术有限公司 | A kind of method and device of cluster management |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108566290B (en) | Service configuration management method, system, storage medium and server | |
| CN109218133B (en) | Network speed testing system, method, device and computer readable storage medium | |
| CN109656873B (en) | Block chain-based data archiving method and device and terminal equipment | |
| CN109885786B (en) | Data caching processing method and device, electronic equipment and readable storage medium | |
| US8959229B1 (en) | Intelligently provisioning cloud information services | |
| CN110708212B (en) | Method and device for tracking call link in distributed system | |
| CN110471749B (en) | Task processing method, device, computer readable storage medium and computer equipment | |
| CN111459749A (en) | Prometous-based private cloud monitoring method and device, computer equipment and storage medium | |
| EP3958122A1 (en) | Memory management method, apparatus, and system | |
| CN112965879A (en) | Data processing method and device, electronic equipment and readable storage medium | |
| CN109522043B (en) | Method and device for managing configuration data and storage medium | |
| US20140196044A1 (en) | SYSTEM AND METHOD FOR INCREASING THROUGHPUT OF A PaaS SYSTEM | |
| CN109462507B (en) | Configuration updating method, device and system and electronic equipment | |
| CN109348434B (en) | Scene information sending method, sending device and terminal equipment | |
| CN111405052A (en) | P2P penetration method based on port prediction, electronic device and medium | |
| US11592809B2 (en) | System and method for measurement data management in a distributed environment | |
| CN111444074A (en) | Data monitoring method and device, electronic equipment and readable storage medium | |
| CN113127409A (en) | Management method, device, equipment and medium of operation cluster for data real-time archiving | |
| CN107422991B (en) | Storage strategy management system | |
| CN113905092B (en) | Method, device, terminal and storage medium for determining reusable agent queue | |
| CN111190858B (en) | Method, device, equipment and storage medium for storing software information | |
| CN109462592B (en) | Data sharing method, device, equipment and storage medium | |
| CN110554904A (en) | scaling strategy processing method, device, equipment and storage medium | |
| CN108805741B (en) | Fusion method, device and system of power quality data | |
| US20200117522A1 (en) | Lightweight application programming interface (api) creation and management |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210716 |