CN114218236A - Database cluster metadata management method - Google Patents

Database cluster metadata management method Download PDF

Info

Publication number
CN114218236A
CN114218236A CN202111463092.3A CN202111463092A CN114218236A CN 114218236 A CN114218236 A CN 114218236A CN 202111463092 A CN202111463092 A CN 202111463092A CN 114218236 A CN114218236 A CN 114218236A
Authority
CN
China
Prior art keywords
metadata
cluster
node
metadata management
request
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111463092.3A
Other languages
Chinese (zh)
Inventor
蔡成招
肖磊
毕明曼
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Individual
Original Assignee
Individual
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Individual filed Critical Individual
Priority to CN202111463092.3A priority Critical patent/CN114218236A/en
Publication of CN114218236A publication Critical patent/CN114218236A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23Updating
    • G06F16/2365Ensuring data consistency and integrity
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14Error detection or correction of the data by redundancy in operation
    • G06F11/1402Saving, restoring, recovering or retrying
    • G06F11/1446Point-in-time backing up or restoration of persistent data
    • G06F11/1448Management of the data involved in backup or backup restore
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14Error detection or correction of the data by redundancy in operation
    • G06F11/1402Saving, restoring, recovering or retrying
    • G06F11/1446Point-in-time backing up or restoration of persistent data
    • G06F11/1458Management of the backup or restore process
    • G06F11/1464Management of the backup or restore process for networked environments
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23Updating
    • G06F16/2358Change logging, detection, and notification
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G06F16/273Asynchronous replication or reconciliation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/907Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2201/00Indexing scheme relating to error detection, to error correction, and to monitoring
    • G06F2201/80Database-specific techniques
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2201/00Indexing scheme relating to error detection, to error correction, and to monitoring
    • G06F2201/82Solving problems relating to consistency

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Quality & Reliability (AREA)
  • Library & Information Science (AREA)
  • Computing Systems (AREA)
  • Computer Security & Cryptography (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a database cluster metadata management method, relates to the technical field of database cluster metadata management, and solves the technical problem that cluster metadata are inconsistent under the conditions of high concurrency scenes and cluster node failure in the conventional scheme; the invention sets up the metadata management intermediate and cluster node; the metadata management request is sent to a metadata management intermediate after the validity of the metadata management request is checked by the cluster node, the metadata management intermediate updates the metadata of the cluster node or the cluster backup node firstly, and finally completes the metadata updating of all the cluster nodes and the cluster backup node, so that the problem of inconsistent cluster metadata can be effectively solved; the metadata management intermediate carries out consistency analysis on the metadata of the cluster nodes and the cluster backup nodes at regular time, and generates logs generated by consistency analysis results and stores the logs in the metadata management intermediate, so that the consistency of the metadata of each node is guaranteed in an all-around manner.

Description

Database cluster metadata management method
Technical Field
The invention belongs to the technical field of database cluster metadata management, and particularly relates to a database cluster metadata management method.
Background
The database cluster is characterized in that two or more database services are utilized to form a virtual single database logical image and provide transparent data services for a client; in database systems, metadata is used to describe data attributes, such as indicating storage locations, historical data, resource lookups, file records, and the like.
The existing database cluster architectures mainly comprise two types, namely a cluster architecture with a central node and a cluster architecture without the central node; in the database cluster architecture, the existing cluster metadata management method is easy to cause inconsistency of cluster metadata in a high concurrency scene, and causes the problem that a cluster cannot manage metadata information under the condition that cluster nodes fail; therefore, a method for ensuring efficient management of database cluster metadata in a high-concurrency scenario is needed.
Disclosure of Invention
The present invention is directed to solving at least one of the problems of the prior art; therefore, the invention provides a database cluster metadata management method, which is used for solving the technical problem of inconsistent cluster metadata caused by the failure of cluster nodes in a high concurrency scene in the existing scheme.
To achieve the above object, an embodiment according to a first aspect of the present invention provides a database cluster metadata management method, including:
when the cluster node receives the metadata management request, the validity of the metadata management request is checked and then the metadata management request is sent to a metadata management intermediate;
the metadata management intermediate serializes the metadata management request, generates a unique request ID, and simultaneously acquires the states of all cluster nodes; wherein the cluster node state comprises valid and invalid;
the metadata management intermediate updates the metadata of the cluster nodes by combining the metadata management request, and sends an update completion signal to a client initiating the metadata management request after updating is completed; the metadata management intermediate at least comprises one cluster backup node.
Preferably, the metadata management intermediate is in communication connection with a plurality of cluster nodes and at least one cluster backup node; the cluster nodes and the cluster backup nodes have the same structure.
Preferably, the metadata management request completes updating of the metadata of the cluster node in combination with the backup node of the cluster, and includes:
sending the metadata management request to a cluster backup node to execute and complete metadata updating, and associating a request ID of the metadata management request with updated metadata;
and sending a backup updating completion signal to the metadata management intermediate, sending the metadata corresponding to the request ID to an effective cluster node connected with the metadata management intermediate, and completing metadata updating by the effective cluster node.
Preferably, when a cluster node is invalid, the metadata management intermediary integrates request IDs of metadata management requests, which are not executed by the invalid cluster node, to generate a first ID set;
when the cluster node changes from invalid to valid, the metadata updating of the cluster node is immediately completed according to the first ID set.
Preferably, the update completion signal is sent after all cluster nodes complete the metadata update.
Preferably, the updating of the cluster node metadata by the metadata management intermediary in combination with the metadata management request includes:
sending the metadata management request to an effective cluster node to execute metadata updating, and associating the request ID of the metadata management request with the metadata updated by the effective cluster node;
sending the metadata corresponding to the request ID to a cluster backup node to complete metadata updating;
acquiring an unexecuted metadata management request of an invalid cluster node and integrating to generate a second ID set;
and when the invalid cluster node is recovered, completing the metadata updating of the cluster node by the cluster node completing the metadata updating or the cluster backup node combining the second ID set.
Preferably, the consistency analysis is carried out on the metadata of the cluster nodes and the cluster backup nodes at regular time;
when the metadata are not consistent, updating the metadata of the cluster nodes and the cluster backup nodes through the standard metadata; the standard metadata is metadata in a cluster node or a cluster backup node with the latest updating time;
and generating a consistency analysis log and sending the consistency analysis log to a metadata management intermediate.
Preferably, the checking the validity of the metadata management request includes:
checking a syntax of the metadata management request; when the grammar of the metadata management request does not conform to the grammar specification of the database cluster, directly sending a metadata grammar abnormal signal to a corresponding client;
checking the performability of the metadata management request; and when the content of the metadata management request conflicts with the metadata in the database cluster, directly returning a metadata management failure signal to the corresponding client.
Compared with the prior art, the invention has the beneficial effects that:
1. the invention sets up the metadata management intermediate and cluster node; the metadata management request is sent to the metadata management intermediate after the validity of the metadata management request is checked by the cluster node, the metadata management intermediate updates the metadata of the cluster node or the cluster backup node firstly, and finally the metadata updating of all the cluster nodes and the cluster backup node is completed, so that the problem of inconsistent cluster metadata can be effectively solved.
2. The metadata management intermediate carries out consistency analysis on the metadata of the cluster nodes and the cluster backup nodes at regular time, and generates logs generated by consistency analysis results and stores the logs in the metadata management intermediate, so that the consistency of the metadata of each node is guaranteed in an all-around manner.
Drawings
FIG. 1 is a schematic diagram of the process steps of the present invention;
fig. 2 is a schematic diagram of the system of the present invention.
Detailed Description
The technical solutions of the present invention will be described clearly and completely with reference to the following embodiments, and it should be understood that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1-2, with the rapid development of the information technology, the number of users and the data volume of an enterprise both increase explosively, and while the traffic volume increases, the access volume and the data volume of the database increase correspondingly to the processing capacity and the calculation strength of the database, so that the database at a single node cannot be assumed at all, and therefore, it becomes important to efficiently manage the metadata of the database cluster.
There are two major database cluster architectures: a cluster architecture with a central node and an MPP architecture without a central node; the cluster architecture with the central node is characterized in that only one piece of cluster metadata is provided, the central node is in uniform charge of management, the application client side is uniformly accessed into the database cluster from the central node, and the cluster metadata and the service provided by the database cluster are obtained; the MP architecture without the central node is characterized in that the cluster metadata comprises a plurality of shares, each cluster node is completely the same, each cluster node is maintained by the cluster node, and the application client can access the database cluster from any cluster node and obtain consistent cluster metadata and service provided by the database cluster.
For the two data cluster architectures, when a plurality of management metadata requests initiated by a single client or a plurality of clients initiate different management metadata requests, the existing cluster metadata management method can ensure that metadata stored by each cluster node is the same; however, in a high concurrency scenario, if the same metadata management request is initiated from the clients accessed from different cluster nodes at the same time, the metadata of the cluster nodes is easily inconsistent, so the present invention provides a database cluster metadata management method.
The invention provides a database cluster metadata management method which comprises a metadata management intermediate and a plurality of cluster nodes connected with the metadata management intermediate.
The metadata management intermediate has functions equivalent to a server, including data processing, data storage and other functions, and at least comprises one cluster backup node.
It should be noted that the cluster backup node of the present application is consistent with the cluster node, but the cluster backup node is mainly used for backing up metadata, so as to avoid the loss of metadata.
The invention provides a database cluster metadata management method, which comprises the following steps:
when the cluster node receives the metadata management request, the validity of the metadata management request is checked and then the metadata management request is sent to a metadata management intermediate;
the metadata management intermediate serializes the metadata management request, generates a unique request ID, and simultaneously acquires the states of all cluster nodes;
and the metadata management intermediate updates the metadata of the cluster nodes by combining the metadata management request, and sends an update completion signal to the client initiating the metadata management request after the update is completed.
It is understood that the state of the cluster nodes connected to the metadata management middlebox includes valid and invalid; the active cluster node can directly execute the metadata management request to complete the metadata updating, but the inactive cluster node cannot directly execute the metadata management request until the abnormal state is eliminated, and the metadata management request can not be executed to complete the metadata updating.
In one embodiment, the metadata management intermediate is in communication connection with a plurality of cluster nodes and at least one cluster backup node, and the number of the cluster nodes and the number of the cluster backup nodes can be determined according to the data volume of the metadata; it can be understood that the cluster nodes and the cluster backup nodes have the same structure, but the specific use is different, the cluster backup nodes are equivalent to backup metadata, and are not necessarily updated in time according to the metadata management request, and it can also be understood that the main role of the metadata management request is the cluster nodes.
In one embodiment, the validity check on the metadata management request comprises:
checking a syntax of the metadata management request; when the grammar of the metadata management request does not conform to the grammar specification of the database cluster, directly sending a metadata grammar abnormal signal to a corresponding client;
checking the performability of the metadata management request; and when the content of the metadata management request conflicts with the metadata in the database cluster, directly returning a metadata management failure signal to the corresponding client.
It can be understood that the validity check on the metadata management request does not only include syntax and feasibility checks, and any factor affecting the execution of the metadata management request by the cluster node or the cluster backup node needs to be checked, so as to ensure the reasonable effectiveness of the metadata request and the update efficiency of the metadata.
In a specific embodiment, the metadata management request, in combination with the cluster backup node, completes the update of the cluster node metadata, including:
sending the metadata management request to a cluster backup node to execute and complete metadata updating, and associating a request ID of the metadata management request with updated metadata;
and sending a backup updating completion signal to the metadata management intermediate, sending the metadata corresponding to the request ID to an effective cluster node connected with the metadata management intermediate, and completing metadata updating by the effective cluster node.
The main idea in this embodiment is to update the cluster backup data in the metadata management intermediate according to the metadata management request, and associate the updated metadata with the request ID corresponding to the metadata management request, that is, the updated part of data is easily obtained; and the updated data is sent to the cluster nodes and other cluster backup nodes to complete metadata updating, so that the metadata consistency of each cluster node and each cluster backup node can be ensured.
When an invalid cluster node occurs in the embodiment, recording a request ID corresponding to the invalid cluster node as update metadata through a metadata management intermediate, and integrating to generate a first ID set; when the cluster node is changed from invalid to valid, the metadata are acquired according to the first ID set and sent to the cluster node to complete metadata updating, so that all the cluster nodes can timely and effectively complete metadata updating, and the consistency of the metadata of each node is further improved.
In a specific embodiment, the metadata management intermediary, in conjunction with the metadata management request, performs the update of the cluster node metadata, including:
sending the metadata management request to an effective cluster node to execute and complete metadata updating, and associating the request ID of the metadata management request with the metadata updated by the effective cluster node;
sending the metadata corresponding to the request ID to a cluster backup node to complete metadata updating;
acquiring an unexecuted metadata management request of an invalid cluster node and integrating to generate a second ID set;
and when the invalid cluster node is recovered, completing the metadata updating of the cluster node by the cluster node completing the metadata updating or the cluster backup node combining the second ID set.
The main idea of this embodiment is to perform metadata update on an effective cluster node according to a metadata management request, associate the updated metadata with a request ID of the metadata management request, and send metadata corresponding to the request ID to a cluster backup node to complete metadata update on the cluster backup node; and when the invalid cluster node is recovered, updating the metadata of the cluster node according to the metadata corresponding to the second ID set, and also ensuring the consistency of the metadata of each node.
In one embodiment, the update completion signal is sent after all cluster nodes complete metadata update, and before all metadata update is completed, the statistics of metadata update may also be sent to the client, such as how many cluster nodes are updated and how many cluster nodes are not updated.
In one embodiment, the metadata management intermediate periodically performs consistency analysis on the metadata of the cluster nodes and the cluster backup nodes;
when the metadata are not consistent, updating the metadata of the cluster nodes and the cluster backup nodes through the standard metadata; the standard metadata is metadata in a cluster node or a cluster backup node with the latest updating time;
and generating a consistency analysis log and sending the consistency analysis log to a metadata management intermediate.
It is to be understood that standard metadata may be understood as metadata after the last metadata update request was performed, i.e. the latest version of metadata.
The working principle of the invention is as follows:
and when the cluster node receives the metadata management request, the validity of the metadata management request is checked and then the metadata management request is sent to the metadata management intermediate.
And the metadata management intermediate serializes the metadata management request, generates a unique request ID and simultaneously acquires the states of all cluster nodes.
And the metadata management intermediate updates the effective and ineffective cluster node metadata respectively by combining the metadata management request, and sends an update completion signal to the client initiating the metadata management request after the update is completed.
Although the present invention has been described in detail with reference to the preferred embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted for elements thereof without departing from the spirit and scope of the present invention.

Claims (8)

1. A method for managing metadata of a database cluster is characterized by comprising the following steps:
when the cluster node receives the metadata management request, the validity of the metadata management request is checked and then the metadata management request is sent to a metadata management intermediate;
the metadata management intermediate serializes the metadata management request, generates a unique request ID, and simultaneously acquires the states of all cluster nodes; wherein the cluster node state comprises valid and invalid;
the metadata management intermediate updates the metadata of the cluster nodes by combining the metadata management request, and sends an update completion signal to a client initiating the metadata management request after updating is completed; the metadata management intermediate at least comprises one cluster backup node.
2. The method of claim 1, wherein the metadata management broker is communicatively coupled to a plurality of cluster nodes and at least one cluster backup node; the cluster nodes and the cluster backup nodes have the same structure.
3. The method of claim 1, wherein the metadata management request in conjunction with the cluster backup node completes the update of the cluster node metadata comprises:
sending the metadata management request to a cluster backup node to execute and complete metadata updating, and associating a request ID of the metadata management request with updated metadata;
and sending a backup updating completion signal to the metadata management intermediate, sending the metadata corresponding to the request ID to an effective cluster node connected with the metadata management intermediate, and completing metadata updating by the effective cluster node.
4. The method according to claim 3, wherein when a cluster node is invalid, the metadata management broker integrates request IDs of metadata management requests that are not executed by the invalid cluster node to generate a first ID set;
when the cluster node changes from invalid to valid, the metadata updating of the cluster node is immediately completed according to the first ID set.
5. The method of claim 1, wherein the metadata management intermediary performs the update of the cluster node metadata in conjunction with the metadata management request, and comprises:
sending the metadata management request to an effective cluster node to execute metadata updating, and associating the request ID of the metadata management request with the metadata updated by the effective cluster node;
sending the metadata corresponding to the request ID to a cluster backup node to complete metadata updating;
acquiring an unexecuted metadata management request of an invalid cluster node and integrating to generate a second ID set;
and when the invalid cluster node is recovered, completing the metadata updating of the cluster node by the cluster node completing the metadata updating or the cluster backup node combining the second ID set.
6. The method of claim 1, wherein the update completion signal is sent after all cluster nodes complete the metadata update.
7. The method for managing the metadata of the database cluster according to claim 1, wherein the consistency analysis is performed on the metadata of the cluster nodes and the cluster backup nodes at regular time;
when the metadata are not consistent, updating the metadata of the cluster nodes and the cluster backup nodes through the standard metadata; the standard metadata is metadata in a cluster node or a cluster backup node with the latest updating time;
and generating a consistency analysis log and sending the consistency analysis log to a metadata management intermediate.
8. The method according to claim 1, wherein checking the validity of the metadata management request comprises:
checking a syntax of the metadata management request; when the grammar of the metadata management request does not conform to the grammar specification of the database cluster, directly sending a metadata grammar abnormal signal to a corresponding client;
checking the performability of the metadata management request; and when the content of the metadata management request conflicts with the metadata in the database cluster, directly returning a metadata management failure signal to the corresponding client.
CN202111463092.3A 2021-12-02 2021-12-02 Database cluster metadata management method Pending CN114218236A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111463092.3A CN114218236A (en) 2021-12-02 2021-12-02 Database cluster metadata management method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111463092.3A CN114218236A (en) 2021-12-02 2021-12-02 Database cluster metadata management method

Publications (1)

Publication Number Publication Date
CN114218236A true CN114218236A (en) 2022-03-22

Family

ID=80699578

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111463092.3A Pending CN114218236A (en) 2021-12-02 2021-12-02 Database cluster metadata management method

Country Status (1)

Country Link
CN (1) CN114218236A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024119980A1 (en) * 2022-12-07 2024-06-13 华为云计算技术有限公司 Data analysis method and related device

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024119980A1 (en) * 2022-12-07 2024-06-13 华为云计算技术有限公司 Data analysis method and related device

Similar Documents

Publication Publication Date Title
US10929428B1 (en) Adaptive database replication for database copies
US11468060B2 (en) Automatic query offloading to a standby database
JP6404907B2 (en) Efficient read replica
CN110572281B (en) Credible log recording method and system based on block chain
US10691722B2 (en) Consistent query execution for big data analytics in a hybrid database
JP6254606B2 (en) Database streaming restore from backup system
JP2558052B2 (en) Transaction processing system using hypothetical commit two-phase commit protocol and operating method thereof
US8862936B2 (en) Apparatus and method of managing databases of active node and standby node of main memory database management system
US10831741B2 (en) Log-shipping data replication with early log record fetching
US10366106B2 (en) Quorum-based replication of data records
AU2016271618A1 (en) Disconnected operation within distributed database systems
CN108616574B (en) Management data storage method, device and storage medium
CN101334797A (en) Distributed file systems and its data block consistency managing method
CN111506592B (en) Database upgrading method and device
CN109388481A (en) A kind of transmission method of transaction information, device, calculates equipment and medium at system
CN111045648A (en) Micro-service development tool
US7752225B2 (en) Replication and mapping mechanism for recreating memory durations
JP2023541298A (en) Transaction processing methods, systems, devices, equipment, and programs
CN113326272A (en) Distributed transaction processing method, device and system
CN111475480A (en) Log processing method and system
CN110348826A (en) Strange land disaster recovery method, system, equipment and readable storage medium storing program for executing mostly living
CN114218236A (en) Database cluster metadata management method
CN109726211B (en) Distributed time sequence database
CN114422331A (en) Disaster tolerance switching method, device and system
CN111404737B (en) Disaster recovery processing method and related device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
WD01 Invention patent application deemed withdrawn after publication
WD01 Invention patent application deemed withdrawn after publication

Application publication date: 20220322