CN114328374A - Snapshot method, device, related equipment and database system - Google Patents
Snapshot method, device, related equipment and database system Download PDFInfo
- Publication number
- CN114328374A CN114328374A CN202011052827.9A CN202011052827A CN114328374A CN 114328374 A CN114328374 A CN 114328374A CN 202011052827 A CN202011052827 A CN 202011052827A CN 114328374 A CN114328374 A CN 114328374A
- Authority
- CN
- China
- Prior art keywords
- snapshot
- storage node
- transaction log
- node
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 70
- 238000003860 storage Methods 0.000 claims abstract description 190
- 239000012634 fragment Substances 0.000 claims description 36
- 238000004891 communication Methods 0.000 claims description 13
- 230000008569 process Effects 0.000 abstract description 16
- 230000007246 mechanism Effects 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 8
- 238000013467 fragmentation Methods 0.000 description 8
- 238000006062 fragmentation reaction Methods 0.000 description 8
- 230000002085 persistent effect Effects 0.000 description 8
- 238000004590 computer program Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 238000011084 recovery Methods 0.000 description 7
- 238000012545 processing Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000009826 distribution Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Retry When Errors Occur (AREA)
Abstract
The application provides a snapshot method, a snapshot device, related equipment and a database system, and particularly relates to a method for generating a transaction log and a snapshot command aiming at data in a storage node for a computing node in the database system, and then adding the snapshot command to the transaction log and submitting the transaction log. Therefore, the storage node can obtain the snapshot command from the transaction log submitted by the computing node to complete the corresponding snapshot process. Therefore, depending on a reliability mechanism after the transaction log is submitted, the success rate of the data snapshot of the storage node can be effectively improved, and the reliability of the database system can be improved.
Description
Technical Field
The present application relates to the field of snapshot technologies, and in particular, to a snapshot method, an apparatus, a device, a database system, and a computer-readable storage medium.
Background
With the development of information technology, reliability and usability of data are more and more emphasized. At present, snapshot backup technology is generally adopted to backup stored data, so that the reliability and availability of the data can reach a high level. A snapshot is a complete, available copy of a specified data set, the copy including a static image of the specified data set at the time of copying, and the snapshot may be considered a copy or duplication of the specified data set.
In some database systems, a compute node typically sends a snapshot command to each storage node in the database system through a command channel to cause the storage node to snapshot the stored data. However, the method for performing snapshot on the data in the storage node has a low success rate, thereby affecting the reliability of the database system.
Disclosure of Invention
The application provides a snapshot method, which is used for improving the success rate of snapshot of data in storage nodes and improving the reliability of a database system. In addition, the application also provides a device, equipment, a database system, a computer readable storage medium and a computer program product corresponding to the method.
In a first aspect, the present application provides a snapshot method, where the method is applied to a database system, where the database system includes a compute node and a storage node, and the method includes:
the computing node generates a transaction log and a snapshot command for data in the storage node;
the computing node adds the snapshot command to the transaction log to obtain a target transaction log;
the computing node commits the target transaction log.
In this embodiment, after the computing node successfully submits the target transaction log, the target transaction log is generally inevitably received by the normally operating storage node, so that the storage node can execute the snapshot command, and therefore, the computing node can consider that creating the snapshot is successful after submitting the target transaction log, without waiting for the storage node to return a result of the snapshot success within a specified time length. Meanwhile, for the storage node, even if a heavier data reading/writing task exists at present, the storage node can perform data snapshot processing according to the target transaction log when the load is smaller; or the storage node fails, and persistent storage is also performed on the target transaction log submitted by the computing node, so that after the storage node completes fault recovery, the corresponding data snapshot process can be completed according to the target transaction log of the persistent storage. Therefore, the success rate of the data snapshot of the storage nodes can be effectively improved, and the reliability of the database system can be improved.
In one possible embodiment, the storage node includes a plurality of shards, and the method further includes:
and the computing node carries out snapshot on the fragment information, wherein the fragment information comprises the storage of the data in the storage node in the plurality of fragments.
In one possible embodiment, the method further comprises:
the computing node takes a snapshot of the target transaction log.
In one possible embodiment, the transaction log includes a start log sequence number LSN and an end log sequence number LSN, and the end LSN is used to indicate an end position where the storage node takes a snapshot of data.
In a second aspect, an embodiment of the present application provides a snapshot method, where the method is applied to a database system, where the database system includes a compute node and a storage node, and the method includes:
the storage node receives a target transaction log submitted by the computing node, wherein the target transaction log comprises a snapshot command aiming at data in the storage node;
and the storage node carries out snapshot on the stored data according to the snapshot command in the target transaction log.
In one possible embodiment, the method further comprises:
the storage node receives an uploading instruction;
and the storage node responds to the uploading instruction and uploads the stored data and/or the snapshot to the cloud.
In a third aspect, an embodiment of the present application further provides a snapshot apparatus, where the snapshot apparatus is applied to a compute node in a database system, the database system further includes a storage node, and the snapshot apparatus includes:
the generating module is used for generating a transaction log and a snapshot command aiming at the data in the storage node;
the adding module is used for adding the snapshot command to the transaction log to obtain a target transaction log;
and the submitting module is used for submitting the target transaction log.
In a possible implementation manner, the storage node includes a plurality of shards, and the snapshot apparatus further includes:
and the snapshot module is used for carrying out snapshot on the fragment information, wherein the fragment information comprises the storage of the data in the storage node in the plurality of fragments.
In a possible implementation manner, the snapshot module is further configured to snapshot the target transaction log.
In one possible embodiment, the transaction log includes a start log sequence number LSN and an end log sequence number LSN, and the end LSN is used to indicate an end position where the storage node takes a snapshot of data.
In a fourth aspect, an embodiment of the present application further provides a snapshot apparatus, where the snapshot apparatus is applied to a storage node in a database system, where the database system includes a compute node, and the snapshot apparatus includes:
a communication module, configured to receive a target transaction log submitted by the compute node, where the target transaction log includes a snapshot command for data in the storage node;
and the snapshot module is used for carrying out snapshot on the stored data according to the snapshot command in the target transaction log.
In a possible implementation, the communication module is further configured to:
receiving an uploading instruction;
and responding to the uploading instruction, and uploading the stored data and/or the snapshot to the cloud.
In a fifth aspect, the present application provides an apparatus comprising a processor and a memory. The processor and the memory are in communication with each other. The processor is configured to execute the instructions stored in the memory to cause the apparatus to perform the snapshot method as in the first aspect or any implementation manner of the first aspect.
In a sixth aspect, the present application provides an apparatus comprising a processor and a memory. The processor and the memory are in communication with each other. The processor is configured to execute the instructions stored in the memory to cause the device to perform the snapshot method as in the second aspect or any implementation manner of the second aspect.
In a seventh aspect, the present application provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and the instructions instruct a device to perform the snapshot method according to the first aspect or any implementation manner of the first aspect.
In an eighth aspect, the present application provides a computer-readable storage medium, in which instructions are stored, and the instructions instruct a device to execute the snapshot method according to the second aspect or any implementation manner of the second aspect.
In a ninth aspect, the present application provides a computer program product comprising instructions that, when run on a device, cause the device to perform the snapshot method of the first aspect or any of the implementations of the first aspect, or cause the device to perform the snapshot method of the second aspect or any of the implementations of the second aspect.
The present application can further combine to provide more implementations on the basis of the implementations provided by the above aspects.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art according to the drawings.
FIG. 1 is a block diagram of an exemplary database system according to an embodiment of the present application;
FIG. 2 is a schematic structural diagram of a compute node in an embodiment of the present application;
FIG. 3 is a schematic structural diagram of a storage node according to an embodiment of the present application;
FIG. 4 is a flow chart illustrating a snapshot method according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of a snapshot apparatus applied to a compute node 110 in an embodiment of the present application;
fig. 6 is a schematic structural diagram of a snapshot apparatus applied to the storage node 120 in the embodiment of the present application.
Detailed Description
The terms "first" and "second" in the embodiments of the present application are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.
Referring to fig. 1, a schematic diagram of an exemplary database system according to an embodiment of the present application is shown.
As shown in FIG. 1, database system 100 may include at least compute node 110 and storage node 120. Wherein, the computing node 110 may be connected with the storage node 120, for example, the computing node 110 may be wirelessly connected with the storage node 120 through the routing device 130 as shown in fig. 1, and the like.
For example, the computing node 110 may also be a node, or may include a plurality of nodes (in fig. 1, 1 node is taken as an example for illustration); the storage node 120 may be a single node, or may include a plurality of nodes as shown in fig. 1, which are a storage node 121, a storage node 122, a storage node 123, and a storage node 124.
Wherein the storage node 120 may be used to store data, and the compute node 110 may manage the data storage of the storage node 120. For example, the computing node 110 may send a snapshot command to the storage node 120 to cause the storage node 120 to snapshot the data it stores, and so on.
In some examples, the database system 100 may further include an upper application 140, so that a user may perform corresponding operations on the database system through the upper application 140, for example, the user may instruct the database system 100 to perform tasks such as data reading, writing, snapshot, and the like through the upper application 140.
During the process of executing the task, the database system 100 may further generate a corresponding log, so as to record the execution process and the result of the task based on the log, and perform operations such as rollback based on the log. The generated logs may be stored by the log storage node 150 in the database system 100. The log storage node 150 may specifically include one or more nodes. The log storage node 150 may store one or more log copies when storing the log, and the plurality of log copies may be distributed over different log storage nodes. The database system 100 shown in fig. 1 is described by way of example as including the log storage node 151, the log storage node 152, and the log storage node 153, and in actual application, more or fewer storage nodes may be included.
In general, the computing node 110 may establish a command channel with the storage node 120, and the computing node 110 may send a snapshot command to the storage node 120 through the command channel and wait for the storage node 120 to successfully execute the snapshot command within a certain period of time after sending the snapshot command. However, in this way, the implementation of the snapshot on the data in the storage node 120 has a low snapshot success rate.
For example, storage node 120 may have a high operation load for some time period, for example, storage node 120 may have a heavy data read and/or write task, which makes it difficult for storage node 120 to respond to the snapshot command and take a snapshot of the currently stored data within a specified time period, so that computing node 110 determines that the snapshot of storage node 120 fails because it waits too long for storage node 120 to respond to the snapshot command. For another example, when there is a failure of a part of storage nodes, for example, when storage node 121 fails, computing node 110 may not be able to send a snapshot command to storage node 121, or storage node 121 may have difficulty responding to a snapshot command issued by computing node 110 within a specified time period, which results in that data is not snapshot because the snapshot command is not received even after storage node 121 fails and recovers.
Also, in some database systems, each storage node may contain multiple shards, and each shard may have multiple copies and be distributed across different storage nodes. Since snapshot data can be used for the storage node 120 to maintain data consistency when the storage node fails to recover, when taking a snapshot, the snapshot command is executed successfully only if all copies of the shard can complete the snapshot, and when any copy does not execute (e.g., does not execute due to a storage node failure, etc.), the snapshot command is determined to have failed to execute.
Therefore, when determining that the data stored in the storage node 120 needs to be snapshot, the computing node 110 may generate a transaction log (redo log) and simultaneously generate a snapshot command. Then, the computing node 110 adds the snapshot command to the generated transaction log instead of sending the snapshot command directly to the storage node through the command channel, so as to obtain the transaction log (hereinafter referred to as the target transaction log) containing the snapshot command. Finally, the compute node 110 may commit the target transaction log. Since the storage node 120 can receive the successful transaction log submitted by the computing node 110 in the database system, the storage node 120 that normally operates can snapshot the stored data according to the snapshot command in the received target transaction log based on the reliability mechanism of the transaction log.
Thus, for the computing node 110, after the target transaction log is successfully submitted, since the computing node can be inevitably received by the normally operating storage node and executes the snapshot command, the computing node 110 can consider that creating the snapshot is successful, and does not need to wait for the storage node 120 to return a result of the snapshot success within a specified time, so that the dependency relationship between the computing node 110 and the storage node 120 can be reduced, and decoupling is realized. In practice, the process may be completed within a time period of the order of milliseconds, so that the process of taking a snapshot of the data in the database system 100 may be considered to be completed within the order of milliseconds.
Meanwhile, for the storage node 120, even if a heavier data read/write task currently exists, the storage node can perform data snapshot processing according to the target transaction log when the load is small; or the storage node 120 fails, persistent storage is also performed on the target transaction log submitted by the computing node 110, so that after the storage node 120 completes failure recovery, the corresponding data snapshot process can be completed according to the target transaction log of the persistent storage. Therefore, various fault scenes of the storage node 120 can be dealt with, the success rate of data snapshot of the storage node 120 is effectively improved, and the reliability of the database system can be improved.
It is noted that the architecture of the database system shown in fig. 1 is only an example, and in practical applications, the adaptation and transformation can be performed based on the database system shown in fig. 1. For example, in other possible embodiments, the database system may include a greater or lesser number of storage nodes 120, or the storage nodes 120 and the log storage node 150 are merged into the same node, and the like, which is not limited in this embodiment.
A hardware implementation of the compute node 110 and the storage node 120 is illustrated below.
As shown in fig. 2, computing node 110 may include a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, memory 204, and communication interface 203 communicate via a bus 201. The bus 201 may be a Peripheral Component Interconnect (PCI) bus, a peripheral component interconnect express (PCIe) or Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 2, but it is not intended that there be only one bus or one type of bus. The communication interface 203 is used for communicating with the outside, for example, receiving a snapshot indication or the like sent by the upper layer application 140.
The processor 202 may be a Central Processing Unit (CPU). Memory 204 may include volatile memory (volatile memory), such as Random Access Memory (RAM). The memory 204 may also include a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory, an HDD, or an SSD.
The memory 204 stores programs or instructions that the processor 202 executes to perform the snapshot method described above. Of course, the memory 204 may also store data, such as a generated transaction log and/or a target transaction log.
As shown in fig. 3, storage node 120 includes a bus 301, a processor 302, a communication interface 303, and a memory 304. The processor 302, memory 304, and communication interface 303 communicate over a bus 301. The memory 304 has stored therein programs or instructions. The memory 304 has stored therein programs or instructions. The processor 302 executes the program or instructions to perform the snapshot method described above. Of course, the memory 304 may also store data, such as user-provided data.
In some possible implementations, the above-described computing node 110 functions may also be implemented by one or more chips. Each chip includes a processor and a chip interface. The chip interface is used for receiving the instruction and transmitting the instruction to the processor. The processor executes the above instructions to perform the steps of the snapshot method described above. Certainly, when the functions of the computing node 110 are implemented by using multiple chips, the chip interface of each chip receives a part of instructions, and the processor of each chip executes a part of steps of the snapshot method according to the part of instructions, so that the multiple processors can implement all steps of executing the snapshot method.
In order to make the technical solution of the present application clearer and easier to understand, the following describes in detail a resource scheduling method according to an embodiment of the present application from the perspective of interaction between the computing node 110 and the storage node 120.
Referring to fig. 4, a flow chart of a snapshot method is shown, the method comprising:
s401: the compute node 110 generates a transaction log.
In some possible embodiments, the upper layer application 140 may send an indication to the compute node 110 to snapshot data, and the compute node 110 triggers snapshot service execution based on the received snapshot indication. Specifically, the computing node 110 may generate a transaction that creates the snapshot-generating instruction and generate a corresponding transaction log for the transaction. The snapshot instruction may instruct the storage node to snapshot a piece of data, or may instruct the storage node to snapshot all data stored thereon.
In other possible embodiments, the computing node 110 may also periodically generate snapshot commands to periodically snapshot backup the data in the storage node 120.
In this embodiment, the transaction log generated by the computing node 110 may have a starting Log Sequence Number (LSN) and an ending LSN.
The end LSN may be used to indicate an end position of the storage node 120 for performing snapshot on the data, that is, may indicate which data stored on the storage node 120 is to be subjected to snapshot.
The starting LSN may be used to identify transaction logs, and transaction logs having LSNs smaller than the starting LSN (i.e., transaction logs before) have all completed committing and persisted, as may be written to log storage node 150 shown in FIG. 1. For example, when the storage node 120 includes multiple slices, the starting LSN may specifically be a Global Slice persistent log sequence number (Global Slice persistent LSN) which is a maximum value of Slice persistent log sequence numbers (Slice persistent LSNs) corresponding to all slices in the storage node 120, and the Slice Psersist LSN is a maximum LSN at which logs of some Slice data are continuously landed to the log storage node. Accordingly, the log of all sharded data prior to this LSN has been written to the log storage node. A shard is an independent data area in the storage node 120, and the area may be physically isolated from other shards or logically isolated from other shards.
S402: the compute node 110 adds the snapshot command to the transaction log, resulting in a target transaction log containing the snapshot command.
In a specific implementation, a row of command data may be added in multiple rows of data in the transaction log, where the command data is a snapshot command written into the transaction log, so that the transaction log including the snapshot command may be obtained, and for convenience of description, the transaction log is hereinafter referred to as a target transaction log.
S403: the computing node 110 commits the target transaction log.
Since the database system usually commits the transaction log after determining that the transaction is completed, and persistently stores the committed transaction log, the target transaction log is not lost due to long-time non-reception of the storage node like a snapshot instruction, but a snapshot command may be executed when the storage node 120 has the capability of executing a snapshot service. Thus, typically, after the compute node 110 commits the target transaction log, the storage node 120 must subsequently be able to complete the snapshot process.
To this end, in some possible embodiments, the computing node 110 may feed back the result of the snapshot success after submitting the target transaction log, for example, may feed back the result of creating the snapshot success to the upper-layer application 140.
S404: the storage node 120 receives the target transaction log.
In practical applications, the storage node 120 may obtain the target transaction log from the log storage node 150, for example, after the storage node 120 completes fault recovery, the storage node 120 may read a plurality of logs stored in the log storage node 150, and snapshot the data and recover the data to a snapshot position indicated by an end LSN in the target transaction log according to the target transaction log in the plurality of logs. Of course, in the case that the storage node 120 is operating normally, the computing node 110 may send the target transaction log to the storage node 120 through the routing device 130. In this embodiment, a specific implementation process of how the storage node 120 obtains the target transaction log is not limited, and the target transaction log may also be obtained in other possible manners.
S405: the storage node 120 performs snapshot on the stored data according to the snapshot command in the target transaction log.
After obtaining the target transaction log, the storage node 120 may parse the snapshot command from the target transaction log, for example, the storage node 120 may parse the snapshot command from a specific column of the target transaction log. In this way, storage node 120 may obtain a snapshot command, which may be executed to implement a snapshot of data. In this embodiment, a specific implementation process of the storage node 120 to execute the snapshot instruction is not limited.
In practical applications, the storage node 120 may include multiple fragments, and therefore, in order to manage the data of each fragment in the storage node 120, the computing node 110 may generate a corresponding fragment list, where the fragment list may include identifiers of the fragments in the storage node 120, fragments that each storage node 120 has, and identifiers of data stored by different fragments at a certain time. In this way, the computing node 110 may determine which data of which shards to snapshot based on the shard list.
Meanwhile, in a further possible implementation manner, the compute node 110 may also perform snapshot on the fragment information of the storage node 120, so as to record a fragment state at a time when the compute node 110 triggers data snapshot. Illustratively, the fragmentation information may be, for example, the above-mentioned fragmentation list, or be presented in other forms, etc. In addition, the computing node 110 may also take a snapshot of the target transaction log at the current time. In this way, when the database system 100 is performing failure recovery or log rollback as a whole, the database system can be recovered to the state at that time according to the snapshot fragmentation list, the target transaction log, and the snapshot data. In this manner, the reliability of the database system 100 may be improved. When performing fault recovery, the recovered time point may be a time point corresponding to the end LSN included in the target transaction log.
It is noted that, when the target transaction log includes the start LSN and the end LSN, the compute node 110 is a process for triggering the snapshot process at a time corresponding to the start LSN and continuing to execute the snapshot fragmentation information. In practical application, since a certain time is required in the process of snapshotting the fragment information by the computing node 110, the fragment information at the time corresponding to the starting LSN may be different from the fragment information snapshotted by the computing node 110, for example, in the process of snapshotting the fragment information by the computing node 110, part of data may be newly written, deleted, or changed in a part of the fragments of the storage node 120. Therefore, if the storage node 120 uses the start LSN as the end position of the data snapshot, the snapshot data of the storage node 120 and the data on the slice may not be consistent with the slice information corresponding to the start LSN. In this way, when the database system 100 performs failure recovery based on the fragment information snapshotted by the compute node 110 and the data snapshotted by the storage node 120, a situation occurs that the data recorded by the compute node 110 is inconsistent with the data actually stored by the storage node 120, so that there is an abnormality in failure recovery, which affects the reliability of the database system.
To this end, the computing node 110 may record the end LSN when the snapshot fragmentation information is completed, so that the storage node 120 may determine an end position where the data is snapshot according to the end LSN. Since the time consumed for the computing node 110 to record the end LSN is short, each piece of fragment data on the storage node 120 indicated by the piece information snapshot by the computing node 110 is consistent with the piece of fragment data snapshot by the storage node 120, so that the reliability of the database system can be improved.
Further, the fragment information of the snapshot of the compute node 110, the target transaction log, and the data of the snapshot of the storage node 120 may also be backed up to the cloud. In this way, the computing node 110 and the storage node 120 may release the storage space occupied by the data of the current snapshot, so that more new data may be stored.
In particular implementations, database system 100 may also include a backup device (not shown in FIG. 1), which may be a computing node 110 or a device separate from computing node 110. The upper-layer application 140 may send a backup instruction to the backup device, which is used to instruct the database system 100 to perform cloud backup on the snapshot data in the storage node 120 and the compute node 110. The backup device may send an upload command for uploading snapshot data to the storage node 120 through the routing device 130, for example, the upload command may be issued through a command channel. When the storage node 120 includes a plurality of segments and each segment has a plurality of copies, the backup device may determine distribution of data to be backed up to the cloud in each segment in the storage node 120, so that a corresponding segment may be selected as an upload object from the distribution, and an upload backup command is sent to the storage node where the segment is located, and only one copy may be uploaded for the same data. Therefore, when data backup is achieved, a plurality of storage nodes do not need to upload a plurality of same copies, network bandwidth needed by backup data can be reduced, the situation that a cloud end stores a plurality of same copies of data can be avoided, and consumption of storage space of the cloud end is reduced.
The storage node 120 may determine the fragment data to be uploaded according to the received upload command, and upload the fragment data to the cloud. Meanwhile, the backup device may obtain the fragment information of the snapshot from the compute node 110 and obtain the target transaction log of the backup from the log storage node 150, and then the backup device may upload the fragment information to the cloud in the form of the target transaction log. Of course, when the backup devices are the same devices as the computing nodes 110, the backup devices may directly upload the fragment information stored in the backup devices to the cloud.
It should be noted that the embodiment shown in fig. 4 is only used as an exemplary illustration, and is not intended to limit the implementation of the steps of the embodiment of the present application in practical applications to the example shown in fig. 4, for example, more other steps may be added on the basis of the steps shown in fig. 4, such as in the process of generating the target transaction log, the snapshot fragmentation information and the log, and the like.
The database system and the snapshot method provided in the present application are described in detail above with reference to fig. 1 to 4, and the apparatus provided in the present application will be described below with reference to the accompanying drawings.
Referring to fig. 5, an embodiment of the present application further provides a snapshot apparatus applied to the above-mentioned computing node 110, where the apparatus 500 may include:
a generating module 501, configured to generate a transaction log and a snapshot command for data in the storage node 120;
an adding module 502, configured to add the snapshot command to the transaction log to obtain a target transaction log;
a commit module 503, configured to commit the target transaction log.
In a possible implementation manner, the storage node 120 includes a plurality of shards, and the snapshot apparatus 500 further includes:
a snapshot module 504, configured to snapshot fragmentation information, where the fragmentation information includes storage of data in the storage node in the multiple fragments.
In a possible implementation manner, the snapshot module 504 is further configured to snapshot the target transaction log.
In one possible implementation, the transaction log includes a start log sequence number LSN and an end log sequence number LSN, and the end LSN is used to indicate an end position where the storage node 120 takes a snapshot of data.
The snapshot apparatus 500 according to the embodiment of the present application may correspond to perform the method described in the embodiment of the present application, and the above and other operations and/or functions of each module/unit of the snapshot apparatus 500 are respectively for implementing the corresponding flow of each method in the embodiment shown in fig. 4, and are not described herein again for brevity.
Referring to fig. 6, an embodiment of the present application further provides a snapshot apparatus applied to the storage node 120, where the apparatus 600 is applied to the storage node 120, and the snapshot apparatus 600 may include:
a communication module 601, configured to receive a target transaction log submitted by the computing node, where the target transaction log includes a snapshot command for data in the storage node;
the snapshot module 602 is configured to snapshot stored data according to a snapshot command in the target transaction log.
In a possible implementation, the communication module 601 is further configured to:
receiving an uploading instruction;
and responding to the uploading instruction, and uploading the stored data and/or the snapshot to the cloud.
The snapshot apparatus 600 according to the embodiment of the present application may correspond to perform the method described in the embodiment of the present application, and the above and other operations and/or functions of each module/unit of the snapshot apparatus 600 are respectively for implementing corresponding flows of each method performed by the storage node 120 in the embodiment shown in fig. 4, and are not described herein again for brevity.
The embodiment of the present application further provides a computer-readable storage medium, which includes instructions for instructing a computer to execute the above snapshot method applied to the computing node 110 or the storage node 120.
The embodiment of the present application further provides a computer program product, and when the computer program product is executed by a computer, the computer executes any one of the foregoing snapshot methods. The computer program product may be a software installation package which may be downloaded and executed on a computer in the event that any of the aforementioned snapshot methods are required.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present application. Thus, if such modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to encompass such modifications and variations.
Claims (16)
1. A snapshot method, applied to a database system, the database system comprising a compute node and a storage node, the method comprising:
the computing node generates a transaction log and a snapshot command for data in the storage node;
the computing node adds the snapshot command to the transaction log to obtain a target transaction log;
the computing node commits the target transaction log.
2. The method of claim 1, wherein the storage node comprises a plurality of shards, the method further comprising:
and the computing node carries out snapshot on the fragment information, wherein the fragment information comprises the storage of the data in the storage node in the plurality of fragments.
3. The method according to claim 1 or 2, characterized in that the method further comprises:
the computing node takes a snapshot of the target transaction log.
4. The method of any of claims 1 to 3, wherein the transaction log comprises a start Log Sequence Number (LSN) and an end Log Sequence Number (LSN) indicating an end location where the storage node takes a snapshot of data.
5. A snapshot method, applied to a database system, the database system comprising a compute node and a storage node, the method comprising:
the storage node receives a target transaction log submitted by the computing node, wherein the target transaction log comprises a snapshot command aiming at data in the storage node;
and the storage node carries out snapshot on the stored data according to the snapshot command in the target transaction log.
6. The method of claim 5, further comprising:
the storage node receives an uploading instruction;
and the storage node responds to the uploading instruction and uploads the stored data and/or the snapshot to the cloud.
7. A snapshot apparatus, wherein the snapshot apparatus is applied to a compute node in a database system, the database system further includes a storage node, and the snapshot apparatus includes:
the generating module is used for generating a transaction log and a snapshot command aiming at the data in the storage node;
the adding module is used for adding the snapshot command to the transaction log to obtain a target transaction log;
and the submitting module is used for submitting the target transaction log.
8. The snapshot apparatus in accordance with claim 7, wherein the storage node comprises a plurality of shards, the snapshot apparatus further comprising:
and the snapshot module is used for carrying out snapshot on the fragment information, wherein the fragment information comprises the storage of the data in the storage node in the plurality of fragments.
9. The snapshot apparatus according to claim 7 or 8, wherein the snapshot module is further configured to snapshot the target transaction log.
10. The snapshot apparatus of any one of claims 7 to 9, wherein the transaction log comprises a start Log Sequence Number (LSN) and an end Log Sequence Number (LSN), and wherein the end LSN is used to indicate an end position where the storage node snapshots data.
11. A snapshot apparatus, wherein the snapshot apparatus is applied to a storage node in a database system, the database system includes a compute node, and the snapshot apparatus includes:
a communication module, configured to receive a target transaction log submitted by the compute node, where the target transaction log includes a snapshot command for data in the storage node;
and the snapshot module is used for carrying out snapshot on the stored data according to the snapshot command in the target transaction log.
12. The snapshot apparatus of claim 11, wherein the communication module is further configured to:
receiving an uploading instruction;
and responding to the uploading instruction, and uploading the stored data and/or the snapshot to the cloud.
13. An apparatus, comprising a processor and a memory;
the processor is to execute instructions stored in the memory to cause the device to perform the method of any of claims 1 to 4.
14. An apparatus, comprising a processor and a memory;
the processor is to execute instructions stored in the memory to cause the device to perform the method of any of claims 5 to 6.
15. A database system, characterized in that the database system comprises a compute node according to any of claims 1 to 4 and a storage node according to any of claims 5 to 6.
16. A computer-readable storage medium comprising instructions that direct a device to perform the method of any of claims 1-4 or to perform the method of any of claims 5-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011052827.9A CN114328374A (en) | 2020-09-29 | 2020-09-29 | Snapshot method, device, related equipment and database system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011052827.9A CN114328374A (en) | 2020-09-29 | 2020-09-29 | Snapshot method, device, related equipment and database system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114328374A true CN114328374A (en) | 2022-04-12 |
Family
ID=81010710
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011052827.9A Pending CN114328374A (en) | 2020-09-29 | 2020-09-29 | Snapshot method, device, related equipment and database system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114328374A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115269563A (en) * | 2022-09-26 | 2022-11-01 | 北京奥星贝斯科技有限公司 | Method and device for performing log analysis on database system |
-
2020
- 2020-09-29 CN CN202011052827.9A patent/CN114328374A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115269563A (en) * | 2022-09-26 | 2022-11-01 | 北京奥星贝斯科技有限公司 | Method and device for performing log analysis on database system |
| CN115269563B (en) * | 2022-09-26 | 2022-12-09 | 北京奥星贝斯科技有限公司 | Method and device for performing log analysis on database system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10817386B2 (en) | Virtual machine recovery method and virtual machine management device | |
| US7437603B2 (en) | Method for restoring snapshot in a storage system | |
| CN101258472B (en) | Snapshot restore method and apparatus | |
| US8140790B2 (en) | Failure management method in thin provisioning technology for storage | |
| US7823008B2 (en) | Maintaining consistency in a remote copy data storage system | |
| US8555012B2 (en) | Data storage apparatus | |
| EP3147797B1 (en) | Data management method, node and system for database cluster | |
| US11803412B2 (en) | Containerized application management system and management method | |
| US11733874B2 (en) | Managing replication journal in a distributed replication system | |
| CN116107516B (en) | Data writing method and device, solid state disk, electronic equipment and storage medium | |
| US11307944B2 (en) | Automated failover for asynchronous remote copy | |
| CN112199240A (en) | Method for switching nodes during node failure and related equipment | |
| CN110377664B (en) | Data synchronization method, device, server and storage medium | |
| CN116501259A (en) | Disk group dual-activity synchronization method and device, computer equipment and storage medium | |
| CN115729749A (en) | Data backup method and system | |
| US10078558B2 (en) | Database system control method and database system | |
| US20070061613A1 (en) | Restart method for operating system | |
| CN114328374A (en) | Snapshot method, device, related equipment and database system | |
| US10210060B2 (en) | Online NVM format upgrade in a data storage system operating with active and standby memory controllers | |
| CN109165117B (en) | Data processing method and system | |
| US9952941B2 (en) | Elastic virtual multipath resource access using sequestered partitions | |
| US10089202B1 (en) | Providing data high availability to a set of host computers via automatic failover | |
| CN108599982B (en) | Data recovery method and related equipment | |
| CN110389713B (en) | Data synchronization method, apparatus and computer readable medium | |
| CN112328429A (en) | Hard disk snapshot method and device based on Openstack |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |