CN116521066A

CN116521066A - Fragment transfer method and device

Info

Publication number: CN116521066A
Application number: CN202310396007.9A
Authority: CN
Inventors: 徐虎; 杨敏; 赵裕众; 韩富晟; 王万红
Original assignee: Beijing Oceanbase Technology Co Ltd
Current assignee: Beijing Oceanbase Technology Co Ltd
Priority date: 2023-04-13
Filing date: 2023-04-13
Publication date: 2023-08-01

Abstract

The embodiment of the specification provides a fragment transfer method and a fragment transfer device, and relates to the technical field of databases. The method can be used for transferring the first slice between a source end log stream and a target end log stream in the same node, and comprises the following steps: creating a second shard in the target-side log stream, the second shard referencing shard data in the first shard; the sliced data is added to the second slice to complete the transfer of the first slice. The method may also be used to transfer shards between a source side log stream and a target side log stream at different nodes, comprising: creating a temporary log stream on a node where a source log stream is located; transferring the fragments in the source log stream to a temporary log stream; migrating the temporary log stream to a node where the target-end log stream is located; and transferring the fragments in the temporary log stream to the target-end log stream to finish the transfer of the fragments. The scheme provided by the specification realizes the transfer of the fragments among different log streams on the premise of not influencing the read-write performance of the user.

Description

Fragment transfer method and device

Technical Field

One or more embodiments of the present disclosure relate to the field of database technologies, and in particular, to a method and an apparatus for transferring slices.

Background

Because the log stream of the database can contain a plurality of fragments, when a single log stream is too large, part of fragments in the log stream need to be transferred to a new log stream so as to realize load balancing of the database.

In the related art, when the shard transfer is performed, a chase strategy is generally adopted, the shard stock data is copied into a new log stream, and then the shard increment data is written, so that the writing performance of the database is seriously affected.

Disclosure of Invention

In view of this, one or more embodiments of the present disclosure provide a method and apparatus for transferring slices.

In order to achieve the above object, one or more embodiments of the present disclosure provide the following technical solutions:

according to a first aspect of one or more embodiments of the present disclosure, a slice transfer method is provided, where the slice transfer method is used for transferring a first slice in a source log stream to a target log stream, and the source log stream and the target log stream are located at the same node, and the method includes:

creating a second shard in the target-side log stream, the second shard referencing shard data in the first shard;

the sliced data is added to the second slice to complete the transfer of the first slice.

According to a second aspect of one or more embodiments of the present specification, there is provided a shard transfer method for transferring shards in a source log stream into a target log stream, the source log stream and the target log stream being located at different nodes, the method comprising:

creating a temporary log stream on a node where a source log stream is located;

transferring the fragments in the source log stream to a temporary log stream;

migrating the temporary log stream to a node where the target-end log stream is located;

and transferring the fragments in the temporary log stream to the target-end log stream to finish the transfer of the fragments.

According to a third aspect of one or more embodiments of the present specification, there is provided a slice transferring apparatus for transferring a first slice in a source side log stream into a target side log stream, the source side log stream and the target side log stream being located at the same node, the apparatus comprising:

the creating module is used for creating a second fragment in the target-end log stream, and the second fragment refers to fragment data in the first fragment;

and the transfer module is used for adding the slice data into the second slice to finish the transfer of the first slice.

According to a fourth aspect of one or more embodiments of the present specification, there is provided a shard transferring apparatus for transferring shards in a source side log stream into a destination side log stream, the source side log stream and the destination side log stream being located at different nodes, the apparatus comprising:

the creating module is used for creating a temporary log stream on a node where the source log stream is located;

the first transfer module is used for transferring the fragments in the source log stream to the temporary log stream;

the migration module is used for migrating the temporary log stream to the node where the target end log stream is located;

and the second transfer module is used for transferring the fragments in the temporary log stream to the target-end log stream so as to finish the transfer of the fragments.

According to a fifth aspect of one or more embodiments of the present specification, there is provided an electronic device, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor implements a method as in the first aspect or a method as in the second aspect by executing executable instructions.

According to a sixth aspect of one or more embodiments of the present description, a computer-readable storage medium is presented, on which computer instructions are stored, which instructions, when executed by a processor, implement steps as the method of the first aspect, or steps as the method of the second aspect.

According to the slice transfer method provided by the specification, the second slice is created in the target end log stream, wherein the second slice refers to slice data of the first slice in the source end log stream, so that read-write operation for the first slice can be smoothly transferred to the second slice, and the second slice provides complete read-write service for a user. The sliced data is then added to the second slice to complete the transfer of the first slice. According to the scheme provided by the specification, on the premise that the read-write performance of a user is not affected, the transfer of the fragments among different log streams is realized, so that the load balancing control of the database is realized.

Drawings

Fig. 1 is a flow chart of a method for transferring slices according to an exemplary embodiment.

Fig. 2 is a flow chart of yet another method for transferring slices according to an exemplary embodiment.

Fig. 3 is a schematic diagram of a specific application of the same node slice transfer method according to an exemplary embodiment.

Fig. 4 is a schematic application diagram of a different node slice transfer method according to an exemplary embodiment.

Fig. 5 is a schematic diagram of an apparatus according to an exemplary embodiment.

Fig. 6 is a schematic structural view of a slice transferring apparatus according to an exemplary embodiment.

Fig. 7 is a schematic structural diagram of a slice transferring apparatus according to an exemplary embodiment.

Detailed Description

Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of the present specification. Rather, they are merely examples of apparatus and methods consistent with aspects of one or more embodiments of the present description as detailed in the accompanying claims.

It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than described in this specification. Furthermore, individual steps described in this specification, in other embodiments, may be described as being split into multiple steps; while various steps described in this specification may be combined into a single step in other embodiments.

For ease of understanding, the terms appearing in the present specification will first be explained.

A log stream is an entity automatically created and managed by a database that represents a collection of data, including a number of fragmented and ordered transaction logs.

In a distributed environment, to ensure high availability of data read-write services, a database will copy data of the same log stream to multiple nodes. In some embodiments, the copies of data of the same log stream under different nodes are referred to as replicas (replicas). Multiple copies of the same log stream may use the Paxos consistency protocol to ensure strong consistency of the copies, each log stream and its copies forming a separate Paxos group, with one log stream being a master copy (Leader) and the other log streams being slave copies (slaves). In some embodiments, the master copy has strong coherent read and write capabilities and the slave copy has weak coherent read capabilities.

A partition (Tablet) is a data storage object corresponding to a partition, and has a capability of storing data, and is a minimum unit of load balancing regulation. Each slice stores data through an ordered string table (Sorted String Table, SSTable) stored on disk and a Memory table (memtab) stored on Memory.

From the perspective of data storage, the log stream can also be abstracted into a container of the fragments, so that the addition and management of the data of the fragments are supported, and the Transfer (Transfer) of the fragments between different log streams is allowed, so that the purposes of load balancing and horizontal expansion are achieved.

It should be noted that, load balancing may be implemented by Root Service (Root Service) of the database. Because the different types of copies have different demands on resources, the factors that Root Service needs to consider when performing the balancing operation include the CPU, the disk usage, the memory usage, and the usage of disk performance (IOPS) of each resource unit. Through load balancing, all types of resources of all nodes are occupied in a relatively balanced state, and all the resources of each node are fully utilized.

When the related technology transfers the fragments, firstly, the stock data of the fragments to be transferred are copied to the fragments of the new log stream, if the increment data generated by the fragments to be transferred in the copying process is larger, the increment data is continuously copied as the stock data until the increment data is small enough, and at the moment, the writing operation for the fragments is paused (the generation of the new increment data is avoided) until all the increment data are written into the fragments of the new log stream.

However, for the slices with continuous data writing, the size of the incremental data often cannot reach the ideal situation, and in order to finish the transfer of the slices, the writing operation for the slices is suspended under the condition that a large amount of incremental data is still available, until a large amount of incremental data is written into the slices of the new log stream, which can make the slices in a state of being unable to be written for a long time and seriously affect the use of users. If the time for stopping writing of the slices is reduced, the incremental data needs to be made small enough, but in the scenario with continuous data writing, the condition cannot be satisfied, so that the slice transfer task cannot be completed.

According to the embodiment of the specification, the second shard is created in the target-end log stream, and the second shard refers to the shard data of the first shard in the source-end log stream, so that the read-write operation for the first shard can be smoothly transited to the second shard, and the long-term influence of the shard transition on the writing performance of the database is avoided.

The present exemplary embodiment will be described in detail below with reference to the accompanying drawings and examples.

First, a slice transfer method is provided in the embodiments of the present specification, and the method may be performed by any electronic device.

Fig. 1 is a flowchart of a slice transfer method according to an exemplary embodiment, where the slice transfer method is used to transfer a first slice in a source log stream to a target log stream, where the source log stream and the target log stream are located at the same node. As shown in fig. 1, the slice transfer method includes the following steps.

S101, creating a second fragment in the target-end log stream, wherein the second fragment refers to fragment data in the first fragment.

It should be noted that, the first slice in the embodiment of this description may be understood as a slice to be transferred, and the second slice may be understood as a slice after transfer. In the process of referencing the sliced data in the first sliced data by the second sliced, the sliced data is not copied but directly referenced, so that repeated occupation of storage space of the database is avoided, and meanwhile, the read-write operation for the first sliced data can be smoothly transited into the second sliced data.

And after the second fragment refers to the fragment data in the first fragment, the second fragment provides complete read-write service for the external user.

In some embodiments, the shard data includes a memtab and a first SSTable. The memory is used for storing the first partition, wherein the MemTable is data stored in the memory by the first partition, and the first SSTable is data stored in the disk by the first partition.

S102, adding the slice data into the second slice to finish the transfer of the first slice.

In some embodiments, the process of replacing the shard data into the second shard may be: the memtab is dumped to a second SSTable, and the first SSTable and the second SSTable are added to the second slice.

The dump refers to a process of storing data in memtab to a disk. The MemTable needs to be frozen first before dumping to ensure that the dumped MemTable no longer has new data writes.

It should be noted that SSTable realizes data storage by means of macro blocks. The database may split the disk into fixed-length databases (i.e., macroblocks), each SSTable being made up of several macroblocks.

In some embodiments, the source side log stream has a plurality of first copies and the target side log stream has a plurality of second copies. Before S101, it may also be checked whether the first copies and the second copies are in one-to-one correspondence. If yes, continuing to execute the subsequent fragment transfer step; if not, the transfer of the first slice is stopped.

It should be noted that, in this specification, the "one-to-one correspondence" is understood that each first copy has a second copy corresponding to the first copy, and the first copy and the second copy corresponding to each other are located at the same node.

In some embodiments, before S101, the writing operation for the first slice in the source log stream may be stopped, so that the second slice may refer to the slice data in the first slice, and provide a read-write service for the outside, so as to avoid the loss of data in the transition process from the first slice to the second slice.

In some embodiments, after the first shard transfer is completed, the first shard in the source log stream may be deleted to free up storage space in the database.

In some embodiments, the above method for transferring fragments may be performed in response to a fragment transfer request sent by a user, or may be initiated by a database system when load balancing is performed, which is not limited by the embodiments of the present disclosure. The execution subject of the above shard transfer method may be a master copy of the target-side log stream.

The embodiment solves the problem of uneven slicing transfer among log streams of the same node, and the specification further provides another slicing transfer method which is used for solving the problem of uneven slicing transfer among log streams of different nodes.

Fig. 2 is a flow chart of yet another method for transferring slices in a source log stream to a destination log stream according to an exemplary embodiment, where the source log stream and the destination log stream are located at different nodes. As shown in fig. 2, the slice transfer method includes the following steps.

S201, creating a temporary log stream on a node where the source log stream is located.

S202, transferring the fragments in the source log stream to the temporary log stream.

It should be noted that, since the source log stream and the temporary log stream are located at the same node, the method for transferring the slice from the source log stream to the temporary log stream may refer to the method provided in the embodiment of fig. 1, which is not described in detail in this disclosure.

And S203, migrating the temporary log stream to a node where the target-end log stream is located.

The migration of the temporary log stream may be understood as migrating each copy of the temporary log stream to a node corresponding to each copy of the target log stream.

S204, transferring the fragments in the temporary log stream to the target-end log stream to finish the transfer of the fragments.

It should be noted that, the migrated temporary log stream and the target log stream are located at the same node, so the method for transferring the fragments from the temporary log stream to the target log stream may refer to the method provided in the embodiment of fig. 1, which is not described in detail in this disclosure.

It should be noted that, in the embodiment of the present disclosure, the copies of the temporary log stream are in one-to-one correspondence with the copies of the source log stream, and the copies of the target log stream are in one-to-one correspondence with the copies of the temporary log stream.

According to the embodiment of the description, the temporary log stream is created, and the sliced transfer between different nodes is decomposed into the sliced transfer between two identical nodes, so that the sliced transfer between different nodes can be smoothly transited, and the influence on the read-write performance of the database is reduced.

The description of the various embodiments in this specification is intended to highlight differences between the various embodiments, and the same or similar features may be referred to each other by reference to avoid redundant description of the present specification for the sake of brevity.

The foregoing describes the method of slice transfer set forth in this specification, and for ease of understanding, will be further described below in connection with specific applications.

Fig. 3 is a schematic diagram of a specific application of the same node slice transfer method provided in an exemplary embodiment, in fig. 3, a source log stream and a target log stream are located at the same node, and the source log stream and the target log stream each have multiple copies (not shown in the figure) and are in one-to-one correspondence with each other.

At this time, since the load of the node where the primary copy of the source log stream is located is excessive, and the load of the node where the primary copy of the target log stream is located is small, in order to implement load balancing, a part of the slices (for example, the first slice) in the source log stream may be transferred to the target log stream.

The transfer process specifically comprises the following steps:

(a) The write operation to the first slice in the source log stream is stopped.

(b) And creating a second fragment in the target-side log stream, wherein the second fragment refers to the MemTable and/or SSTable of the first fragment and can provide complete read-write service for the outside.

(c) The referenced MemTable is dumped to disk, and then dump data in disk and SSTable are added to the second slice.

(d) Deleting the first slice in the source log stream, and releasing the space.

Fig. 4 is a schematic diagram of an application of a different node slice transfer method according to an exemplary embodiment, in fig. 4, a source log stream is located at a node a, a target log stream is located at a node B, and the source log stream and the target log stream each have multiple copies (not shown in the figure).

At this time, since the source log stream has too many slices, the read-write quantity of the node where the source log stream is located is too large, and the load of the node where the target log stream is located is small, so that in order to realize load balancing, the slices in the source log stream can be transferred to the target log stream.

The transfer process specifically comprises the following steps:

(a) Creating a temporary log stream in the node A, and enabling the copy of the temporary log stream to correspond to the copy of the source log stream one by one.

(b) The shards of the source log stream are transferred to the temporary log stream in node a.

(c) And migrating the temporary log stream carrying the fragments to the node B, and enabling the copy of the temporary log stream after migration to correspond to the target-end log stream one by one.

(d) And transferring the fragments carried by the temporary log stream to the target log stream in the node B.

(e) And deleting the temporary log stream in the node B and releasing the space.

Note that, in the above-mentioned transferring manners for slices in (b) and (d), reference may be made to the transferring manner for slices of the log stream of the same node provided in the present specification, and this will not be repeated in the embodiment of the present specification.

Fig. 5 is a schematic diagram of an apparatus according to an exemplary embodiment. Referring to fig. 5, at the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a nonvolatile memory 510, although other hardware may be included as needed for other services. One or more embodiments of the present description may be implemented in a software-based manner, such as by the processor 502 reading a corresponding computer program from the non-volatile storage 510 into the memory 508 and then running. Of course, in addition to software implementation, one or more embodiments of the present disclosure do not exclude other implementation manners, such as a logic device or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but may also be hardware or a logic device.

Referring to fig. 6, fig. 6 provides a slice transferring apparatus 600, which can be applied to the apparatus shown in fig. 5 to implement the technical solution of the present disclosure. The slice transferring apparatus 600 may include: the creating module 601 is configured to create a second shard in the target log stream, where the second shard references shard data in the first shard. The transfer module 602 is configured to add the slice data to the second slice to complete the transfer of the first slice.

In some embodiments, the shard data includes a memory table memtab and a first ordered string table SSTable. The transfer module 602 is further configured to dump memtab to a second SSTable; the first SSTable and the second SSTable are added to the second slice.

In some embodiments, the source side log stream has a plurality of first copies and the target side log stream has a plurality of second copies. The slice transfer apparatus 600 further includes an inspection module (not shown). The checking module is used for checking whether the first copies and the second copies are in one-to-one correspondence; if not, the transfer of the first slice is stopped.

In some embodiments, the check module is further to stop a write operation for the first slice in the source log stream.

In some embodiments, the slice transferring apparatus 600 further includes a deletion module (not shown in the figure) checking module for deleting the first slice in the source log stream.

Referring to fig. 7, fig. 7 provides another embodiment of a slice transferring apparatus 700 that may be applied to the apparatus shown in fig. 5 to implement the technical solution of the present disclosure. The slice transferring apparatus 700 may include: a creating module 701, configured to create a temporary log stream on a node where the source log stream is located. The first transfer module 702 transfers the fragments in the source log stream into the temporary log stream. And the migration module 703 is configured to migrate the temporary log stream to a node where the target log stream is located. And a second transferring module 704, for transferring the fragments in the temporary log stream to the target log stream to complete the transfer of the fragments.

In some embodiments, the first transfer module 702 transfers the fragments in the source log stream into the temporary log stream according to the method provided by the embodiment of fig. 1.

In some embodiments, the second transfer module 704 transfers the shards in the temporary log stream to the target log stream according to the method provided by the embodiment of fig. 1.

In some embodiments, the copies of the temporary log stream are in one-to-one correspondence with the copies of the source log stream, and the copies of the target log stream are in one-to-one correspondence with the copies of the temporary log stream.

The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular telephone, camera phone, smart phone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or a combination of any of these devices.

In a typical configuration, a computer includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, read only compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic disk storage, quantum memory, graphene-based storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by the computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.

The foregoing describes specific embodiments of the present disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

The terminology used in the one or more embodiments of the specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of the specification. As used in this specification, one or more embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any or all possible combinations of one or more of the associated listed items.

It should be understood that although the terms first, second, third, etc. may be used in one or more embodiments of the present description to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of one or more embodiments of the present description. The word "if" as used herein may be interpreted as "at … …" or "at … …" or "responsive to a determination", depending on the context.

The foregoing description of the preferred embodiment(s) is (are) merely intended to illustrate the embodiment(s) of the present invention, and it is not intended to limit the embodiment(s) of the present invention to the particular embodiment(s) described.

Claims

1. A shard transfer method for transferring a first shard in a source log stream to a target log stream, the source log stream and the target log stream being located at the same node, the method comprising:

2. The method of claim 1, the shard data comprising a memory table memtab and a first ordered string table SSTable;

the adding the tile data to the second tile includes:

dumping the Memable to a second SSTable;

the first SSTable and the second SSTable are added to the second slice.

3. The method of claim 1, the source side log stream having a plurality of first copies and the target side log stream having a plurality of second copies;

before the creating the second shard in the target-side log stream, the method further includes:

checking whether the first copies and the second copies are in one-to-one correspondence;

if not, stopping transferring the first fragments.

4. The method of claim 1, prior to the creating the second shard in the target-side log stream, the method further comprising:

and stopping the writing operation for the first slice in the source log stream.

5. The method of claim 1, after the adding the tile data to the second tile, the method further comprising:

and deleting the first fragment in the source log stream.

6. A shard transfer method for transferring shards in a source log stream into a target log stream, the source log stream and the target log stream being located at different nodes, the method comprising:

creating a temporary log stream on a node where the source log stream is located;

transferring the fragments in the source log stream to the temporary log stream;

7. The method of claim 6, transferring the shards in the source log stream into the temporary log stream, wherein the method employed by the shard transfer is the method of any of claims 1-5.

8. The method of claim 6, transferring the shards in the temporary log stream into the target-side log stream, wherein the method employed by the shard transfer is the method of any of claims 1-5.

9. The method of claim 6, the copies of the temporary log stream being in one-to-one correspondence with the copies of the source log stream, the copies of the target log stream being in one-to-one correspondence with the copies of the temporary log stream.

10. A shard transfer apparatus for transferring a first shard in a source log stream into a target log stream, the source log stream and the target log stream being located at the same node, the apparatus comprising:

and the transfer module is used for adding the sliced data into the second sliced data so as to finish the transfer of the first sliced data.

11. A shard transfer apparatus for transferring shards in a source log stream into a target log stream, the source log stream and the target log stream being located at different nodes, the apparatus comprising:

the creating module is used for creating a temporary log stream on the node where the source log stream is located;

the migration module is used for migrating the temporary log stream to a node where the target end log stream is located;

12. An electronic device, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to implement the method of any one of claims 1 to 5 or the method of any one of claims 6 to 9 by executing the executable instructions.

13. A computer readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the steps of the method of any of claims 1 to 5 or the steps of the method of any of claims 6 to 9.