CN114218236A - Database cluster metadata management method - Google Patents
Database cluster metadata management method Download PDFInfo
- Publication number
- CN114218236A CN114218236A CN202111463092.3A CN202111463092A CN114218236A CN 114218236 A CN114218236 A CN 114218236A CN 202111463092 A CN202111463092 A CN 202111463092A CN 114218236 A CN114218236 A CN 114218236A
- Authority
- CN
- China
- Prior art keywords
- metadata
- cluster
- node
- metadata management
- request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007726 management method Methods 0.000 title abstract description 116
- 238000004458 analytical method Methods 0.000 claims abstract description 13
- 238000000034 method Methods 0.000 claims description 10
- 230000002159 abnormal effect Effects 0.000 claims description 4
- 230000000977 initiatory effect Effects 0.000 claims description 4
- 238000004891 communication Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
- G06F11/1464—Management of the backup or restore process for networked environments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/273—Asynchronous replication or reconciliation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/907—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/80—Database-specific techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/82—Solving problems relating to consistency
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Quality & Reliability (AREA)
- Library & Information Science (AREA)
- Computing Systems (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a database cluster metadata management method, relates to the technical field of database cluster metadata management, and solves the technical problem that cluster metadata are inconsistent under the conditions of high concurrency scenes and cluster node failure in the conventional scheme; the invention sets up the metadata management intermediate and cluster node; the metadata management request is sent to a metadata management intermediate after the validity of the metadata management request is checked by the cluster node, the metadata management intermediate updates the metadata of the cluster node or the cluster backup node firstly, and finally completes the metadata updating of all the cluster nodes and the cluster backup node, so that the problem of inconsistent cluster metadata can be effectively solved; the metadata management intermediate carries out consistency analysis on the metadata of the cluster nodes and the cluster backup nodes at regular time, and generates logs generated by consistency analysis results and stores the logs in the metadata management intermediate, so that the consistency of the metadata of each node is guaranteed in an all-around manner.
Description
Technical Field
The invention belongs to the technical field of database cluster metadata management, and particularly relates to a database cluster metadata management method.
Background
The database cluster is characterized in that two or more database services are utilized to form a virtual single database logical image and provide transparent data services for a client; in database systems, metadata is used to describe data attributes, such as indicating storage locations, historical data, resource lookups, file records, and the like.
The existing database cluster architectures mainly comprise two types, namely a cluster architecture with a central node and a cluster architecture without the central node; in the database cluster architecture, the existing cluster metadata management method is easy to cause inconsistency of cluster metadata in a high concurrency scene, and causes the problem that a cluster cannot manage metadata information under the condition that cluster nodes fail; therefore, a method for ensuring efficient management of database cluster metadata in a high-concurrency scenario is needed.
Disclosure of Invention
The present invention is directed to solving at least one of the problems of the prior art; therefore, the invention provides a database cluster metadata management method, which is used for solving the technical problem of inconsistent cluster metadata caused by the failure of cluster nodes in a high concurrency scene in the existing scheme.
To achieve the above object, an embodiment according to a first aspect of the present invention provides a database cluster metadata management method, including:
when the cluster node receives the metadata management request, the validity of the metadata management request is checked and then the metadata management request is sent to a metadata management intermediate;
the metadata management intermediate serializes the metadata management request, generates a unique request ID, and simultaneously acquires the states of all cluster nodes; wherein the cluster node state comprises valid and invalid;
the metadata management intermediate updates the metadata of the cluster nodes by combining the metadata management request, and sends an update completion signal to a client initiating the metadata management request after updating is completed; the metadata management intermediate at least comprises one cluster backup node.
Preferably, the metadata management intermediate is in communication connection with a plurality of cluster nodes and at least one cluster backup node; the cluster nodes and the cluster backup nodes have the same structure.
Preferably, the metadata management request completes updating of the metadata of the cluster node in combination with the backup node of the cluster, and includes:
sending the metadata management request to a cluster backup node to execute and complete metadata updating, and associating a request ID of the metadata management request with updated metadata;
and sending a backup updating completion signal to the metadata management intermediate, sending the metadata corresponding to the request ID to an effective cluster node connected with the metadata management intermediate, and completing metadata updating by the effective cluster node.
Preferably, when a cluster node is invalid, the metadata management intermediary integrates request IDs of metadata management requests, which are not executed by the invalid cluster node, to generate a first ID set;
when the cluster node changes from invalid to valid, the metadata updating of the cluster node is immediately completed according to the first ID set.
Preferably, the update completion signal is sent after all cluster nodes complete the metadata update.
Preferably, the updating of the cluster node metadata by the metadata management intermediary in combination with the metadata management request includes:
sending the metadata management request to an effective cluster node to execute metadata updating, and associating the request ID of the metadata management request with the metadata updated by the effective cluster node;
sending the metadata corresponding to the request ID to a cluster backup node to complete metadata updating;
acquiring an unexecuted metadata management request of an invalid cluster node and integrating to generate a second ID set;
and when the invalid cluster node is recovered, completing the metadata updating of the cluster node by the cluster node completing the metadata updating or the cluster backup node combining the second ID set.
Preferably, the consistency analysis is carried out on the metadata of the cluster nodes and the cluster backup nodes at regular time;
when the metadata are not consistent, updating the metadata of the cluster nodes and the cluster backup nodes through the standard metadata; the standard metadata is metadata in a cluster node or a cluster backup node with the latest updating time;
and generating a consistency analysis log and sending the consistency analysis log to a metadata management intermediate.
Preferably, the checking the validity of the metadata management request includes:
checking a syntax of the metadata management request; when the grammar of the metadata management request does not conform to the grammar specification of the database cluster, directly sending a metadata grammar abnormal signal to a corresponding client;
checking the performability of the metadata management request; and when the content of the metadata management request conflicts with the metadata in the database cluster, directly returning a metadata management failure signal to the corresponding client.
Compared with the prior art, the invention has the beneficial effects that:
1. the invention sets up the metadata management intermediate and cluster node; the metadata management request is sent to the metadata management intermediate after the validity of the metadata management request is checked by the cluster node, the metadata management intermediate updates the metadata of the cluster node or the cluster backup node firstly, and finally the metadata updating of all the cluster nodes and the cluster backup node is completed, so that the problem of inconsistent cluster metadata can be effectively solved.
2. The metadata management intermediate carries out consistency analysis on the metadata of the cluster nodes and the cluster backup nodes at regular time, and generates logs generated by consistency analysis results and stores the logs in the metadata management intermediate, so that the consistency of the metadata of each node is guaranteed in an all-around manner.
Drawings
FIG. 1 is a schematic diagram of the process steps of the present invention;
fig. 2 is a schematic diagram of the system of the present invention.
Detailed Description
The technical solutions of the present invention will be described clearly and completely with reference to the following embodiments, and it should be understood that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1-2, with the rapid development of the information technology, the number of users and the data volume of an enterprise both increase explosively, and while the traffic volume increases, the access volume and the data volume of the database increase correspondingly to the processing capacity and the calculation strength of the database, so that the database at a single node cannot be assumed at all, and therefore, it becomes important to efficiently manage the metadata of the database cluster.
There are two major database cluster architectures: a cluster architecture with a central node and an MPP architecture without a central node; the cluster architecture with the central node is characterized in that only one piece of cluster metadata is provided, the central node is in uniform charge of management, the application client side is uniformly accessed into the database cluster from the central node, and the cluster metadata and the service provided by the database cluster are obtained; the MP architecture without the central node is characterized in that the cluster metadata comprises a plurality of shares, each cluster node is completely the same, each cluster node is maintained by the cluster node, and the application client can access the database cluster from any cluster node and obtain consistent cluster metadata and service provided by the database cluster.
For the two data cluster architectures, when a plurality of management metadata requests initiated by a single client or a plurality of clients initiate different management metadata requests, the existing cluster metadata management method can ensure that metadata stored by each cluster node is the same; however, in a high concurrency scenario, if the same metadata management request is initiated from the clients accessed from different cluster nodes at the same time, the metadata of the cluster nodes is easily inconsistent, so the present invention provides a database cluster metadata management method.
The invention provides a database cluster metadata management method which comprises a metadata management intermediate and a plurality of cluster nodes connected with the metadata management intermediate.
The metadata management intermediate has functions equivalent to a server, including data processing, data storage and other functions, and at least comprises one cluster backup node.
It should be noted that the cluster backup node of the present application is consistent with the cluster node, but the cluster backup node is mainly used for backing up metadata, so as to avoid the loss of metadata.
The invention provides a database cluster metadata management method, which comprises the following steps:
when the cluster node receives the metadata management request, the validity of the metadata management request is checked and then the metadata management request is sent to a metadata management intermediate;
the metadata management intermediate serializes the metadata management request, generates a unique request ID, and simultaneously acquires the states of all cluster nodes;
and the metadata management intermediate updates the metadata of the cluster nodes by combining the metadata management request, and sends an update completion signal to the client initiating the metadata management request after the update is completed.
It is understood that the state of the cluster nodes connected to the metadata management middlebox includes valid and invalid; the active cluster node can directly execute the metadata management request to complete the metadata updating, but the inactive cluster node cannot directly execute the metadata management request until the abnormal state is eliminated, and the metadata management request can not be executed to complete the metadata updating.
In one embodiment, the metadata management intermediate is in communication connection with a plurality of cluster nodes and at least one cluster backup node, and the number of the cluster nodes and the number of the cluster backup nodes can be determined according to the data volume of the metadata; it can be understood that the cluster nodes and the cluster backup nodes have the same structure, but the specific use is different, the cluster backup nodes are equivalent to backup metadata, and are not necessarily updated in time according to the metadata management request, and it can also be understood that the main role of the metadata management request is the cluster nodes.
In one embodiment, the validity check on the metadata management request comprises:
checking a syntax of the metadata management request; when the grammar of the metadata management request does not conform to the grammar specification of the database cluster, directly sending a metadata grammar abnormal signal to a corresponding client;
checking the performability of the metadata management request; and when the content of the metadata management request conflicts with the metadata in the database cluster, directly returning a metadata management failure signal to the corresponding client.
It can be understood that the validity check on the metadata management request does not only include syntax and feasibility checks, and any factor affecting the execution of the metadata management request by the cluster node or the cluster backup node needs to be checked, so as to ensure the reasonable effectiveness of the metadata request and the update efficiency of the metadata.
In a specific embodiment, the metadata management request, in combination with the cluster backup node, completes the update of the cluster node metadata, including:
sending the metadata management request to a cluster backup node to execute and complete metadata updating, and associating a request ID of the metadata management request with updated metadata;
and sending a backup updating completion signal to the metadata management intermediate, sending the metadata corresponding to the request ID to an effective cluster node connected with the metadata management intermediate, and completing metadata updating by the effective cluster node.
The main idea in this embodiment is to update the cluster backup data in the metadata management intermediate according to the metadata management request, and associate the updated metadata with the request ID corresponding to the metadata management request, that is, the updated part of data is easily obtained; and the updated data is sent to the cluster nodes and other cluster backup nodes to complete metadata updating, so that the metadata consistency of each cluster node and each cluster backup node can be ensured.
When an invalid cluster node occurs in the embodiment, recording a request ID corresponding to the invalid cluster node as update metadata through a metadata management intermediate, and integrating to generate a first ID set; when the cluster node is changed from invalid to valid, the metadata are acquired according to the first ID set and sent to the cluster node to complete metadata updating, so that all the cluster nodes can timely and effectively complete metadata updating, and the consistency of the metadata of each node is further improved.
In a specific embodiment, the metadata management intermediary, in conjunction with the metadata management request, performs the update of the cluster node metadata, including:
sending the metadata management request to an effective cluster node to execute and complete metadata updating, and associating the request ID of the metadata management request with the metadata updated by the effective cluster node;
sending the metadata corresponding to the request ID to a cluster backup node to complete metadata updating;
acquiring an unexecuted metadata management request of an invalid cluster node and integrating to generate a second ID set;
and when the invalid cluster node is recovered, completing the metadata updating of the cluster node by the cluster node completing the metadata updating or the cluster backup node combining the second ID set.
The main idea of this embodiment is to perform metadata update on an effective cluster node according to a metadata management request, associate the updated metadata with a request ID of the metadata management request, and send metadata corresponding to the request ID to a cluster backup node to complete metadata update on the cluster backup node; and when the invalid cluster node is recovered, updating the metadata of the cluster node according to the metadata corresponding to the second ID set, and also ensuring the consistency of the metadata of each node.
In one embodiment, the update completion signal is sent after all cluster nodes complete metadata update, and before all metadata update is completed, the statistics of metadata update may also be sent to the client, such as how many cluster nodes are updated and how many cluster nodes are not updated.
In one embodiment, the metadata management intermediate periodically performs consistency analysis on the metadata of the cluster nodes and the cluster backup nodes;
when the metadata are not consistent, updating the metadata of the cluster nodes and the cluster backup nodes through the standard metadata; the standard metadata is metadata in a cluster node or a cluster backup node with the latest updating time;
and generating a consistency analysis log and sending the consistency analysis log to a metadata management intermediate.
It is to be understood that standard metadata may be understood as metadata after the last metadata update request was performed, i.e. the latest version of metadata.
The working principle of the invention is as follows:
and when the cluster node receives the metadata management request, the validity of the metadata management request is checked and then the metadata management request is sent to the metadata management intermediate.
And the metadata management intermediate serializes the metadata management request, generates a unique request ID and simultaneously acquires the states of all cluster nodes.
And the metadata management intermediate updates the effective and ineffective cluster node metadata respectively by combining the metadata management request, and sends an update completion signal to the client initiating the metadata management request after the update is completed.
Although the present invention has been described in detail with reference to the preferred embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted for elements thereof without departing from the spirit and scope of the present invention.
Claims (8)
1. A method for managing metadata of a database cluster is characterized by comprising the following steps:
when the cluster node receives the metadata management request, the validity of the metadata management request is checked and then the metadata management request is sent to a metadata management intermediate;
the metadata management intermediate serializes the metadata management request, generates a unique request ID, and simultaneously acquires the states of all cluster nodes; wherein the cluster node state comprises valid and invalid;
the metadata management intermediate updates the metadata of the cluster nodes by combining the metadata management request, and sends an update completion signal to a client initiating the metadata management request after updating is completed; the metadata management intermediate at least comprises one cluster backup node.
2. The method of claim 1, wherein the metadata management broker is communicatively coupled to a plurality of cluster nodes and at least one cluster backup node; the cluster nodes and the cluster backup nodes have the same structure.
3. The method of claim 1, wherein the metadata management request in conjunction with the cluster backup node completes the update of the cluster node metadata comprises:
sending the metadata management request to a cluster backup node to execute and complete metadata updating, and associating a request ID of the metadata management request with updated metadata;
and sending a backup updating completion signal to the metadata management intermediate, sending the metadata corresponding to the request ID to an effective cluster node connected with the metadata management intermediate, and completing metadata updating by the effective cluster node.
4. The method according to claim 3, wherein when a cluster node is invalid, the metadata management broker integrates request IDs of metadata management requests that are not executed by the invalid cluster node to generate a first ID set;
when the cluster node changes from invalid to valid, the metadata updating of the cluster node is immediately completed according to the first ID set.
5. The method of claim 1, wherein the metadata management intermediary performs the update of the cluster node metadata in conjunction with the metadata management request, and comprises:
sending the metadata management request to an effective cluster node to execute metadata updating, and associating the request ID of the metadata management request with the metadata updated by the effective cluster node;
sending the metadata corresponding to the request ID to a cluster backup node to complete metadata updating;
acquiring an unexecuted metadata management request of an invalid cluster node and integrating to generate a second ID set;
and when the invalid cluster node is recovered, completing the metadata updating of the cluster node by the cluster node completing the metadata updating or the cluster backup node combining the second ID set.
6. The method of claim 1, wherein the update completion signal is sent after all cluster nodes complete the metadata update.
7. The method for managing the metadata of the database cluster according to claim 1, wherein the consistency analysis is performed on the metadata of the cluster nodes and the cluster backup nodes at regular time;
when the metadata are not consistent, updating the metadata of the cluster nodes and the cluster backup nodes through the standard metadata; the standard metadata is metadata in a cluster node or a cluster backup node with the latest updating time;
and generating a consistency analysis log and sending the consistency analysis log to a metadata management intermediate.
8. The method according to claim 1, wherein checking the validity of the metadata management request comprises:
checking a syntax of the metadata management request; when the grammar of the metadata management request does not conform to the grammar specification of the database cluster, directly sending a metadata grammar abnormal signal to a corresponding client;
checking the performability of the metadata management request; and when the content of the metadata management request conflicts with the metadata in the database cluster, directly returning a metadata management failure signal to the corresponding client.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202111463092.3A CN114218236A (en) | 2021-12-02 | 2021-12-02 | Database cluster metadata management method |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202111463092.3A CN114218236A (en) | 2021-12-02 | 2021-12-02 | Database cluster metadata management method |
Publications (1)
Publication Number | Publication Date |
---|---|
CN114218236A true CN114218236A (en) | 2022-03-22 |
Family
ID=80699578
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202111463092.3A Pending CN114218236A (en) | 2021-12-02 | 2021-12-02 | Database cluster metadata management method |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN114218236A (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2024119980A1 (en) * | 2022-12-07 | 2024-06-13 | 华为云计算技术有限公司 | Data analysis method and related device |
-
2021
- 2021-12-02 CN CN202111463092.3A patent/CN114218236A/en active Pending
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2024119980A1 (en) * | 2022-12-07 | 2024-06-13 | 华为云计算技术有限公司 | Data analysis method and related device |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10929428B1 (en) | Adaptive database replication for database copies | |
US11468060B2 (en) | Automatic query offloading to a standby database | |
JP6404907B2 (en) | Efficient read replica | |
CN110572281B (en) | Credible log recording method and system based on block chain | |
US10691722B2 (en) | Consistent query execution for big data analytics in a hybrid database | |
JP6254606B2 (en) | Database streaming restore from backup system | |
JP2558052B2 (en) | Transaction processing system using hypothetical commit two-phase commit protocol and operating method thereof | |
US8862936B2 (en) | Apparatus and method of managing databases of active node and standby node of main memory database management system | |
US10831741B2 (en) | Log-shipping data replication with early log record fetching | |
US10366106B2 (en) | Quorum-based replication of data records | |
AU2016271618A1 (en) | Disconnected operation within distributed database systems | |
CN108616574B (en) | Management data storage method, device and storage medium | |
CN101334797A (en) | Distributed file systems and its data block consistency managing method | |
CN111506592B (en) | Database upgrading method and device | |
CN109388481A (en) | A kind of transmission method of transaction information, device, calculates equipment and medium at system | |
CN111045648A (en) | Micro-service development tool | |
US7752225B2 (en) | Replication and mapping mechanism for recreating memory durations | |
JP2023541298A (en) | Transaction processing methods, systems, devices, equipment, and programs | |
CN113326272A (en) | Distributed transaction processing method, device and system | |
CN111475480A (en) | Log processing method and system | |
CN110348826A (en) | Strange land disaster recovery method, system, equipment and readable storage medium storing program for executing mostly living | |
CN114218236A (en) | Database cluster metadata management method | |
CN109726211B (en) | Distributed time sequence database | |
CN114422331A (en) | Disaster tolerance switching method, device and system | |
CN111404737B (en) | Disaster recovery processing method and related device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
WD01 | Invention patent application deemed withdrawn after publication | ||
WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20220322 |