CN115687306A

CN115687306A - Encoding method, device, electronic equipment, system and medium of distributed database

Info

Publication number: CN115687306A
Application number: CN202211445911.6A
Authority: CN
Inventors: 姜哲
Original assignee: Transwarp Technology Shanghai Co Ltd
Current assignee: Transwarp Technology Shanghai Co Ltd
Priority date: 2022-11-18
Filing date: 2022-11-18
Publication date: 2023-02-03

Abstract

The invention discloses a coding method, a device, electronic equipment, a system and a medium of a distributed database, wherein the method comprises the following steps: determining data information and a first coding format of data to be acquired of a data acquisition request; determining a second coding format of at least one intermediate data corresponding to the data to be acquired according to the data information; sending an intermediate data acquisition instruction to a storage node, and receiving intermediate data returned by the storage node based on the intermediate data acquisition instruction, wherein the intermediate data acquisition instruction is used for instructing the storage node to return the intermediate data with a second coding format; and coding the intermediate data by adopting the first coding format to obtain the data to be acquired, and sending the data to be acquired to the client. The method adopts the first coding format to code the intermediate data to obtain the data to be acquired, and can meet the data requests of different scenes of the client on the basis of not modifying a system protocol.

Description

Encoding method, device, electronic equipment, system and medium of distributed database

Technical Field

The present invention relates to the field of database technologies, and in particular, to a method, an apparatus, an electronic device, a system, and a medium for encoding a distributed database.

Background

Currently, a distributed database often adopts single coding according to different emphasis points, for example, an online transaction processing (OLTP) database usually adopts line coding, and an online analysis processing (OLAP) database usually adopts column coding. However, a single code does not efficiently cope with scenarios where transactions are mixed with analysis queries.

The existing technical scheme stores data into two formats of line type coding and column type coding, so that the storage usage amount of the whole system is doubled, and a complex concurrency control and copying protocol is required to support the consistency of multiple copies of the data.

Disclosure of Invention

The invention provides a coding method, a coding device, electronic equipment, a coding system and a coding medium of a distributed database, which are used for meeting data requests of different scenes of a client on the basis of not modifying a system protocol.

According to an aspect of the present invention, there is provided an encoding method for a distributed database, which is applied to a computing node of the distributed database, the method including:

when a data acquisition request sent by a client is received, determining data information and a first coding format of data to be acquired of the data acquisition request;

determining a second coding format of at least one intermediate data corresponding to the data to be acquired according to the data information;

sending an intermediate data acquisition instruction to a storage node, and receiving intermediate data returned by the storage node based on the intermediate data acquisition instruction, wherein the intermediate data acquisition instruction is used for instructing the storage node to return the intermediate data with a second coding format;

and coding the intermediate data by adopting the first coding format to obtain the data to be obtained, and sending the data to be obtained to the client.

According to another aspect of the present invention, there is provided an encoding method for a distributed database, which is applied to a storage node of the distributed database, the method including:

receiving an intermediate data acquisition instruction sent by a computing node, wherein the intermediate data acquisition instruction is used for indicating to return intermediate data with a second coding format;

and acquiring the intermediate data based on the second encoding format and the storage type of the storage node, and sending the intermediate data to the computing node, wherein the storage type is line storage or column storage.

According to another aspect of the present invention, there is provided an encoding apparatus for a distributed database, configured at a computing node of the distributed database, the apparatus including:

the first determining module is used for determining data information and a first coding format of data to be acquired of a data acquisition request when the data acquisition request sent by a client is received;

the second determining module is used for determining a second coding format of at least one piece of intermediate data corresponding to the data to be acquired according to the data information;

the sending module is used for sending an intermediate data acquisition instruction to a storage node and receiving intermediate data returned by the storage node based on the intermediate data acquisition instruction, wherein the intermediate data acquisition instruction is used for indicating the storage node to return the intermediate data with a second coding format;

and the coding module is used for coding the intermediate data by adopting the first coding format to obtain the data to be acquired and sending the data to be acquired to the client.

According to another aspect of the present invention, there is provided an encoding apparatus of a distributed database, configured at a storage node of the distributed database, the apparatus including:

the receiving module is used for receiving an intermediate data acquisition instruction sent by the computing node, wherein the intermediate data acquisition instruction is used for indicating that intermediate data with a second coding format is returned;

and the acquisition module is used for sending the intermediate data to the computing node, and the storage type is line storage or column storage.

According to another aspect of the present invention, there is provided an electronic apparatus including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the method of encoding a distributed database according to any of the embodiments of the present invention.

According to another aspect of the present invention, there is provided an encoding system for a distributed database, the system comprising at least one computing node and at least one storage node;

the computing node is used for determining data information and a first coding format of data to be acquired of a data acquisition request when receiving the data acquisition request sent by a client, determining a second coding format of at least one intermediate data corresponding to the data to be acquired according to the data information, sending an intermediate data acquisition instruction to a storage node, receiving the intermediate data returned by the storage node based on the intermediate data acquisition instruction, coding the intermediate data by adopting the first coding format to obtain the data to be acquired, and sending the data to be acquired to the client;

the storage node is used for receiving an intermediate data acquisition instruction sent by a computing node, acquiring the intermediate data based on the second coding format and the storage type of the storage node, and sending the intermediate data to the computing node.

According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the encoding method of the distributed database according to any one of the embodiments of the present invention when the computer instructions are executed.

The embodiment of the invention provides a coding method, a device, electronic equipment, a system and a medium of a distributed database, wherein the method is applied to a computing node of the distributed database and comprises the following steps: when a data acquisition request sent by a client is received, determining data information and a first coding format of data to be acquired of the data acquisition request; determining a second coding format of at least one intermediate data corresponding to the data to be acquired according to the data information; sending an intermediate data acquisition instruction to a storage node, and receiving intermediate data returned by the storage node based on the intermediate data acquisition instruction, wherein the intermediate data acquisition instruction is used for instructing the storage node to return the intermediate data with a second coding format; and coding the intermediate data by adopting the first coding format to obtain the data to be acquired, and sending the data to be acquired to the client. By using the technical scheme, the second coding format of at least one piece of intermediate data corresponding to the data to be acquired is determined according to the data information, so that the storage node can return the intermediate data with the second coding format, the network bandwidth overhead is reduced, meanwhile, the intermediate data is coded by adopting the first coding format to obtain the data to be acquired, and the data requests of different scenes of the client can be met on the basis of not modifying a system protocol.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present invention, nor do they necessarily limit the scope of the invention. Other features of the present invention will become apparent from the following description.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a flowchart of an encoding method for a distributed database according to an embodiment of the present invention;

fig. 2 is a flowchart of an encoding method of a distributed database according to a second embodiment of the present invention;

fig. 3 is a schematic structural diagram of an encoding method of a distributed database according to a second embodiment of the present invention;

fig. 4 is a schematic structural diagram of an encoding apparatus of a distributed database according to a third embodiment of the present invention;

fig. 5 is a schematic structural diagram of an encoding apparatus of a distributed database according to a fourth embodiment of the present invention;

fig. 6 is a schematic structural diagram of an electronic device that implements the encoding method of the distributed database according to the fourth embodiment of the present invention;

fig. 7 is a schematic structural diagram of an encoding system of a distributed database according to a sixth embodiment of the present invention.

Detailed Description

In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Example one

Fig. 1 is a flowchart of an encoding method for a distributed database according to an embodiment of the present invention, where the present embodiment is applicable to a case of performing encoding based on a distributed database, and the method may be performed by an encoding apparatus for a distributed database, where the encoding apparatus for a distributed database may be implemented in a form of hardware and/or software, and the encoding apparatus for a distributed database may be configured in an electronic device.

It is understood that, currently, distributed databases are different according to the emphasis points, and a single code, such as a line code or a column code, is often adopted. If the line-type encoding is selected, the computational power of modern CPUs (such as SIMD instructions) cannot be fully utilized when dealing with large data volume analysis queries. Meanwhile, the line coding usually occupies more memory, and has a large influence on the system capacity and the concurrency. If columnar encoding is selected, the lock granularity is too large when dealing with highly concurrent transaction requests, which can greatly reduce the throughput of the system, and for some columnar encoding, even single update is difficult to support. That is, a single code does not efficiently cope with a scenario where transactions are mixed with analysis queries.

In the prior art, part of the HTAP database supports multiple data copies, such as 1 line copy +1 column copy, and maintains consistency of distributed data through a specific replication protocol. I.e., data is stored in both row-wise and column-wise formats, the overall system storage usage is doubled and complex concurrency control and replication protocols are required to support consistency of multiple copies of data.

Based on this, the embodiment of the invention can more efficiently process the mixed scene of OLTP and OLAP by adaptively selecting different coding modes, and improve the performance of the whole system. As shown in fig. 1, the method includes:

s110, when a data acquisition request sent by a client is received, determining data information and a first coding format of data to be acquired of the data acquisition request.

The data obtaining request may be regarded as a request for obtaining data to be obtained by a client, for example, the data obtaining request may be a statement for instructing the computing node to return the data to be obtained; the data to be acquired may refer to data required by the client. The data information may refer to information included in the data acquisition request, and is used to determine data to be acquired; the first encoding format may then be considered the encoding format of the data to be acquired.

In this embodiment, when a data acquisition request sent by a client is received, it may be considered that the client needs to acquire data at this time, and then, first, data information and a first encoding format of data to be acquired of the data acquisition request may be determined, so as to perform subsequent operations. Illustratively, when the data acquisition request is to concatenate the data of table 1 and table 2 and return the concatenated data in a line code, after receiving the data acquisition request, it may be determined that the data information includes the data of table 1, the data of table 2, the concatenation, and the like, and the first encoding format is a line code format.

And S120, determining a second coding format of at least one intermediate data corresponding to the data to be acquired according to the data information.

The intermediate data may be regarded as data that needs to be acquired first in the process of acquiring the data to be acquired, so as to determine the data to be acquired according to the intermediate data. The second encoding format may be considered an encoding format for intermediate data. In this embodiment, the second encoding format may be the same as or different from the first encoding format.

It is to be understood that, after the data information is determined in the above step, the second encoding format of the at least one intermediate data corresponding to the data to be acquired may be determined based on the data information, and the manner of determining the second encoding format is not limited, and for example, the data size of the at least one intermediate data corresponding to the data to be acquired may be estimated based on the data information, and then the corresponding second encoding format may be determined based on the data size of each intermediate data. For example, when it is determined that the data information includes the data of table 1, the data of table 2, the connection, and the like, the intermediate data may be the data of table 1 and the data of table 2, and at this time, the data size of the data of table 1 and the data size of the data of table 2 may be estimated first, and then the second encoding format of the data of table 1 and the second encoding format of the data of table 2 may be determined, and the means for determining the second encoding format according to the data size may be determined according to the actual situation without further development.

S130, sending an intermediate data acquisition instruction to the storage node, and receiving intermediate data returned by the storage node based on the intermediate data acquisition instruction, wherein the intermediate data acquisition instruction is used for indicating the storage node to return the intermediate data with a second coding format.

The intermediate data fetch instruction may refer to an instruction for the computing node to request to fetch intermediate data, for instructing the storage node to return intermediate data having the second encoding format.

Specifically, after determining the second encoding format of at least one intermediate data corresponding to the data to be acquired, the computing node may send an intermediate data acquisition instruction to the storage node, and receive intermediate data returned by the storage node based on the intermediate data acquisition instruction. It is considered that, one intermediate data obtaining instruction corresponds to one intermediate data, the number of the intermediate data may be one or more, and one or more storage nodes for storing the intermediate data may also be one or more, so that the specific situation of sending the intermediate data obtaining instruction to the storage node by the computing node may be determined according to the storage situation of the intermediate data. For example, the intermediate data a may be stored in one storage node, or may be stored in a plurality of storage nodes in a distributed manner, when the intermediate data a is stored in one storage node 1, the computing node sends an intermediate data obtaining instruction to the storage node 1, and receives the intermediate data a returned by the storage node 1; when the intermediate data a is distributed and stored in the storage nodes 21 and 22, the computing node sends an intermediate data acquisition instruction to the storage nodes 21 and 22, and receives the intermediate data a returned by the storage nodes 21 and 22.

In one embodiment, the intermediate data obtaining instructions corresponding to the plurality of intermediate data may be sent to the storage node at the same time, or the plurality of intermediate data obtaining instructions may be sent in sequence.

In one embodiment, steps S120 and S130 may be performed simultaneously, or step S120 and step S130 may be performed first, or steps S120 and step S130 may be performed again after step S120 and step S130 are performed first.

For example, in one embodiment, the second encoding formats of the intermediate data m and the intermediate data n may be determined according to the data information, and then the intermediate data obtaining instruction 3 and the intermediate data obtaining instruction 4 may be sent to the storage node, and the intermediate data m returned based on the intermediate data obtaining instruction 3 and the intermediate data n returned based on the intermediate data obtaining instruction 4 may be received.

In an embodiment, the second encoding format of the intermediate data m may be determined according to the data information, then the intermediate data obtaining instruction 3 is sent to the storage node, and the intermediate data m returned based on the intermediate data obtaining instruction 3 is received; and then determining a second coding format of the intermediate data n according to the intermediate data m and the data information, sending an intermediate data acquisition instruction 4 to the storage node, and receiving the intermediate data n returned based on the intermediate data acquisition instruction 4.

S140, encoding the intermediate data by adopting the first encoding format to obtain the data to be acquired, and sending the data to be acquired to the client.

After the intermediate data returned by the storage node is obtained, the obtained intermediate data may be encoded by using the first encoding format to obtain the data to be obtained, and a specific process of obtaining the data to be obtained is not limited, for example, the obtained intermediate data may be calculated to obtain a calculation result, and the calculation result is encoded to obtain the data to be obtained.

In the encoding method for the distributed database provided by the embodiment of the present invention, when a data acquisition request sent by a client is received, data information and a first encoding format of data to be acquired of the data acquisition request are determined; determining a second coding format of at least one intermediate data corresponding to the data to be acquired according to the data information; sending an intermediate data acquisition instruction to a storage node, and receiving intermediate data returned by the storage node based on the intermediate data acquisition instruction, wherein the intermediate data acquisition instruction is used for instructing the storage node to return the intermediate data with a second coding format; and coding the intermediate data by adopting the first coding format to obtain the data to be acquired, and sending the data to be acquired to the client. By using the method, the second coding format of at least one piece of intermediate data corresponding to the data to be acquired is determined according to the data information, so that the storage node can return the intermediate data with the second coding format, the network bandwidth overhead is reduced, meanwhile, the intermediate data is coded by adopting the first coding format to obtain the data to be acquired, and the data requests of different scenes of the client can be met on the basis of not modifying a system protocol.

In one embodiment, for each intermediate data, the intermediate data includes at least one intermediate data block, and the at least one intermediate data block is encoded by using a same or different sub-encoding format as that in the second encoding format.

It is considered that, in the case that the data size of the intermediate data is too large, the storage node may return the corresponding intermediate data in blocks, so that the intermediate data received by the computing node may include a plurality of intermediate data blocks, and the encoding format between the plurality of intermediate data blocks is the same as or different from the sub-encoding format in the second encoding format.

In one embodiment, the determining, according to the data information, a second encoding format of at least one intermediate data corresponding to the data to be acquired includes:

for each intermediate data, estimating the data size of the intermediate data;

determining a second encoding format of the intermediate data to be a columnar encoding format if the data size is greater than or equal to a first data size threshold;

and if the data size is smaller than a preset data size threshold value, determining that the second encoding format of the intermediate data is a line type encoding format.

The first data size threshold may be considered as a critical value of the data size for determining the second encoding format of the intermediate data, and the specific size of the first data size threshold may be determined by empirical values, such as may be determined according to the server performance or the current network bandwidth conditions.

Specifically, in the process of determining the second encoding format of at least one intermediate data corresponding to the data to be acquired according to the data information, the data size of the intermediate data may be estimated for each intermediate data, and then the estimated data size is compared with the first data size threshold, so as to determine whether the data size is smaller than the first data size threshold, for example, when the data size is greater than or equal to the first data size threshold, it may be determined that the data size of the current intermediate data exceeds the critical value of the data size, and then the second encoding format of the intermediate data may be determined as the columnar encoding format; when the data size is smaller than the first data size threshold, it may be considered that the data size of the current intermediate data does not exceed the critical value of the data size, and the second encoding format of the intermediate data may be determined as the line encoding format. The specific method for estimating the data size of the intermediate data is not limited, for example, the computing node may predict and estimate the data according to its own statistical information.

In an embodiment, before the encoding the intermediate data by using the first encoding format to obtain the data to be acquired, the method further includes:

and converting the coding format of the target intermediate data into the first coding format, wherein the target intermediate data is intermediate data of which the adopted coding format is not the first coding format.

The target intermediate data may be understood as intermediate data in which the adopted encoding format is not the first encoding format.

It is considered that, when there is intermediate data in which the adopted encoding format is not the first encoding format, before the intermediate data is encoded to obtain the data to be acquired, the encoding format may be converted so as to convert all the intermediate data into the same encoding format, that is, the encoding format of the target intermediate data is converted into the first encoding format.

Example two

Fig. 2 is a flowchart of an encoding method for a distributed database according to a second embodiment of the present invention, where this embodiment is applicable to a case where encoding is performed based on a distributed database, and the encoding method may be performed by an encoding apparatus for a distributed database, where the encoding apparatus for a distributed database may be implemented in a form of hardware and/or software, and the encoding apparatus for a distributed database may be configured in an electronic device.

As shown in fig. 2, the method includes:

s210, receiving an intermediate data acquisition instruction sent by the computing node, wherein the intermediate data acquisition instruction is used for indicating that intermediate data with a second coding format is returned.

After the computing node sends the intermediate data obtaining instruction to the storage node, the storage node may receive the intermediate data obtaining instruction to perform subsequent return of the intermediate data having the second encoding format.

S220, acquiring the intermediate data based on the second coding format and the storage type of the storage node, and sending the intermediate data to the computing node, wherein the storage type is line storage or column storage.

The storage type may refer to a type in which data is stored in a storage node, for example, the storage type may be a line storage or a column storage.

After receiving the intermediate data acquisition instruction sent by the computing node, the intermediate data may be acquired according to the second encoding format and the storage type of the storage node, and the acquired intermediate data is sent to the computing node to complete the return of the intermediate data. The present embodiment does not limit the process of acquiring the intermediate data, as long as the intermediate data can be acquired.

The encoding method of the distributed database provided by the second embodiment of the present invention receives an intermediate data acquisition instruction sent by a computing node, where the intermediate data acquisition instruction is used to instruct to return intermediate data with a second encoding format; and acquiring the intermediate data based on the second encoding format and the storage type of the storage node, and sending the intermediate data to the computing node, wherein the storage type is line storage or column storage. By utilizing the method, the intermediate data are obtained based on the second coding format and the storage type of the storage node, so that the storage node can return the intermediate data with the second coding format, and the cost of network bandwidth is reduced; meanwhile, the data of the storage nodes are in a single storage type, so that the system requirements can be met, the occupation of a memory is reduced, and the complexity of the system is further reduced.

In one embodiment, said obtaining the intermediate data based on the second encoding format and the storage type of the storage node comprises:

if the second coding format corresponds to the storage type of the storage node, acquiring target data stored in the storage node as the intermediate data;

if the second coding format does not correspond to the storage type of the storage node, acquiring target data stored in the storage node, converting the coding format of the intermediate data into the second coding format, and taking the converted target data as the intermediate data;

wherein the target data corresponds to the intermediate data acquisition instruction.

The target data may be considered to be data stored in the storage node corresponding to the intermediate data fetch instruction.

In this embodiment, the second encoding format may be associated with the storage type of the storage node, and the intermediate data may be obtained according to the corresponding result, for example, when the second encoding format corresponds to the storage type of the storage node, it may be considered that the storage type of the storage node is consistent with the required encoding format, and at this time, the target data stored in the storage node may be directly obtained, and the target data may be used as the intermediate data; when the second encoding format does not correspond to the storage type of the storage node, it may be considered that the storage type of the storage node is not consistent with the required encoding format, and then after the target data stored in the storage node is obtained, the encoding format needs to be converted, that is, the encoding format of the intermediate data is converted into the second encoding format, and the converted target data is used as the intermediate data.

In one embodiment, said sending said intermediate data to said compute node comprises:

if the data size of the intermediate data is larger than a second data size threshold value, segmenting the intermediate data into at least two intermediate data blocks according to the data size of the intermediate data, and sending the at least two intermediate data blocks to the computing node;

and if the data size of the intermediate data is smaller than or equal to a second data size threshold value, sending the intermediate data to the computing node as an intermediate data block.

The second data size threshold may be considered as a critical value of data size, and is used for determining a transmission process of the intermediate data, and the second data size threshold may be the same as or different from the first data size threshold, which is not limited in this embodiment.

In one embodiment, when the data sizes of the intermediate data are different, the transmission processes of the intermediate data may be differentiated,

specifically, when the data size of the intermediate data is larger than the second data size threshold, the data size of the current intermediate data may be considered to exceed the critical value of the data size, the intermediate data may be segmented into at least two intermediate data blocks, and the at least two intermediate data blocks are sent to the computing node; when the data size of the intermediate data is smaller than or equal to the second data size threshold, it can be considered that the data size of the current intermediate data does not exceed the critical value of the data size, and the intermediate data can be sent to the computing node as an intermediate data block. The specific splitting rule is not limited, for example, the line number of each intermediate data block may be a set line number, or the size of each intermediate data block may be a set size, and the set line number and the set size may be set by a relevant person, for example, the set line number may be 16384 lines, and the set size may be 16MB.

In one embodiment, the encoding format between different intermediate data blocks may be the same or different sub-encoding format as in the second encoding format.

The sub-coding format may refer to a coding format in which the second coding format is specifically divided, such as when the second coding format is a column-wise coding format, the sub-coding format may include run-length coding, dictionary coding, incremental coding, and the like. The sub-coding formats of different intermediate data blocks may be the same or different, and the specific situation may be determined according to the data type of the data in the intermediate data block, for example, when the data size of the intermediate data is greater than the second data size threshold, the intermediate data may be divided into two intermediate data blocks, when the intermediate data block a is a repeated bit or character sequence, the intermediate data block a may be coded by using a run-length coding format and then sent to the computing node, and when the intermediate data block B is a digital sequence, the intermediate data block B may be coded by using an incremental coding format and then sent to the computing node.

Fig. 3 is a schematic structural diagram of an encoding method for a distributed database according to a second embodiment of the present invention, and as shown in fig. 3, a client may first send a client request (that is, a data acquisition request) to a computing node, and when the computing node receives the client request, the computing node may parse the client request and generate an execution plan (that is, determine data information and a first encoding format of data to be acquired of the data acquisition request, determine a second encoding format of at least one piece of intermediate data corresponding to the data to be acquired according to the data information, and then send a request to a storage node (that is, send an intermediate data acquisition instruction to the storage node).

After receiving the intermediate data, the storage node performs query or filtering according to the intermediate data acquisition instruction, and when the second encoding format (i.e., line encoding) corresponds to the storage type (i.e., line storage) of the storage node, the storage node maintains the line encoding, and returns the target data stored in the storage node as an intermediate result (i.e., intermediate data) to the computing node (i.e., acquires the intermediate data based on the second encoding format and the storage type of the storage node, and sends the intermediate data to the computing node).

In an embodiment, after receiving the intermediate result (i.e., receiving intermediate data returned by the storage node based on the intermediate data acquisition instruction), the computing node may transmit the intermediate result to another operator, perform formula encoding after computing by the other operator to obtain a result set, and return the result set to the client (i.e., encode the intermediate data by using the first encoding format to obtain the data to be acquired, and send the data to be acquired to the client).

In another embodiment, after the computing node receives the intermediate result, the computing node may continue to issue a request to the storage node (i.e., send an intermediate data obtaining instruction to the storage node), if the second encoding format is column-type encoding, after receiving the request, the storage node performs querying or filtering according to the intermediate data obtaining instruction, and performs column-type encoding on the target data stored in the storage node under the condition that the second encoding format (i.e., column-type encoding) does not correspond to the storage type (i.e., line-type storage) of the storage node, and returns the obtained result to the computing node (i.e., obtain the target data stored in the storage node, convert the encoding format of the intermediate data into the second encoding format, and use the converted target data as the intermediate data).

After the computing node receives the intermediate result (i.e., receives intermediate data returned by the storage node based on the intermediate data acquisition instruction), the intermediate result may be subjected to performing formula encoding after being calculated by other operators to obtain a result set, and the result set is returned to the client (i.e., the intermediate data is encoded by using the first encoding format to obtain the data to be acquired, and the data to be acquired is sent to the client).

As can be seen from the above description, the storage node according to the embodiment of the present invention adopts a certain encoding format, such as line encoding (which is common to systems with heavy OLTP); the external interface of the storage node supports a row type coding network protocol and a column type coding network protocol; the compute node may send a request to the storage node by selecting an appropriate execution plan (i.e., the corresponding intermediate result encoding format); the storage node returns a result set of the corresponding code.

In addition, for the result set with large data volume, the embodiment may also be divided into a plurality of data blocks (e.g. 16384 lines or 16MB each), and each block is encoded differently based on the pre-statistics and run-time sampling of the database. Meanwhile, each operator of the computing node supports the processing of various coding formats; and finally converting the result set into corresponding line-type or column-type codes according to the client protocol.

Therefore, the embodiment of the invention can generate the execution plan and select the coding format of the intermediate result set according to the data characteristics and the current query characteristics, and the execution engine can adopt self-adaptive inter-block coding to the intermediate result with large data volume.

In summary, the embodiment of the present invention selects a suitable encoding format, and can directly adopt line encoding for the transaction processing request, and efficiently forward the data packet of the storage engine; for the scene of analysis query, a storage engine can be supported to send a columnar code through a user-defined network protocol, the overhead of network bandwidth is greatly reduced, and efficient scanning, filtering and aggregation calculation can be performed on the basis of the columnar code. Namely, the invention does not need to modify the storage engine and the data replication protocol, and can not greatly increase the complexity of the system.

EXAMPLE III

Fig. 4 is a schematic structural diagram of an encoding apparatus of a distributed database according to a third embodiment of the present invention. As shown in fig. 4, the apparatus includes:

a first determining module 310, configured to determine, when a data obtaining request sent by a client is received, data information and a first encoding format of data to be obtained of the data obtaining request;

a second determining module 320, configured to determine, according to the data information, a second encoding format of at least one intermediate data corresponding to the data to be acquired;

a sending module 330, configured to send an intermediate data obtaining instruction to a storage node, and receive intermediate data returned by the storage node based on the intermediate data obtaining instruction, where the intermediate data obtaining instruction is used to instruct the storage node to return the intermediate data with a second encoding format;

the encoding module 340 is configured to encode the intermediate data by using the first encoding format, obtain the data to be acquired, and send the data to be acquired to the client.

In the encoding device for a distributed database provided by the third embodiment of the present invention, when a data acquisition request sent by a client is received, a first determining module 310 determines data information and a first encoding format of data to be acquired of the data acquisition request; determining a second encoding format of at least one intermediate data corresponding to the data to be acquired according to the data information through a second determining module 320; sending an intermediate data acquisition instruction to a storage node through a sending module 330, and receiving intermediate data returned by the storage node based on the intermediate data acquisition instruction, where the intermediate data acquisition instruction is used to instruct the storage node to return the intermediate data with a second encoding format; and encoding the intermediate data by adopting the first encoding format through an encoding module 340 to obtain the data to be acquired, and sending the data to be acquired to the client. By using the device, the second coding format of at least one piece of intermediate data corresponding to the data to be acquired is determined according to the data information, so that the storage node can return the intermediate data with the second coding format, the network bandwidth overhead is reduced, meanwhile, the intermediate data is coded by adopting the first coding format to obtain the data to be acquired, and the data requests of different scenes of the client can be met on the basis of not modifying a system protocol.

Optionally, for each piece of intermediate data, the intermediate data includes at least one intermediate data block, and the at least one intermediate data block is encoded by using a sub-encoding format that is the same as or different from the sub-encoding format in the second encoding format.

Optionally, the second determining module 320 is configured to:

for each intermediate data, estimating the data size of the intermediate data;

Optionally, the encoding apparatus for a distributed database provided in the embodiment of the present invention further includes:

a conversion module, configured to convert the coding format of target intermediate data into the first coding format before the intermediate data is coded in the first coding format to obtain the data to be acquired, where the target intermediate data is intermediate data whose coding format is not the first coding format.

The encoding device of the distributed database provided by the embodiment of the invention can execute the encoding method of the distributed database provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.

Example four

Fig. 5 is a schematic structural diagram of an encoding apparatus of a distributed database according to a fourth embodiment of the present invention. As shown in fig. 5, the apparatus includes:

a receiving module 410, configured to receive an intermediate data obtaining instruction sent by a computing node, where the intermediate data obtaining instruction is used to instruct to return intermediate data with a second encoding format;

an obtaining module 420, configured to obtain the intermediate data based on the second encoding format and a storage type of the storage node, where the storage type is line storage or column storage, and send the intermediate data to the computing node.

In the encoding apparatus for a distributed database provided in the fourth embodiment of the present invention, the receiving module 410 receives an intermediate data obtaining instruction sent by the computing node, where the intermediate data obtaining instruction is used to instruct to return intermediate data with a second encoding format; the storage type is line storage or column storage, through the obtaining module 420 and sending the intermediate data to the computing node. By utilizing the device, the intermediate data are acquired based on the second coding format and the storage type of the storage node, so that the storage node can return the intermediate data with the second coding format, and the cost of network bandwidth is reduced; meanwhile, the data of the storage nodes are in a single storage type, so that the system requirements can be met, the occupation of a memory is reduced, and the complexity of the system is further reduced.

Optionally, the obtaining module 410 is configured to:

Optionally, the obtaining module 420 is configured to:

if the data size of the intermediate data is larger than a second data size threshold value, segmenting the intermediate data into at least two data blocks according to the data size of the intermediate data, and sending the at least two data blocks to the computing node;

and if the data size of the intermediate data is smaller than or equal to a second data size threshold value, sending the intermediate data to the computing node as a data block.

EXAMPLE five

FIG. 6 illustrates a schematic structural diagram of an electronic device 10 that may be used to implement an embodiment of the present invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.

As shown in fig. 6, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, and the like, wherein the memory stores a computer program executable by the at least one processor, and the processor 11 can perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from a storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data necessary for the operation of the electronic apparatus 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.

A number of components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, or the like; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.

The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, or the like. The processor 11 performs the various methods and processes described above, such as encoding of a method distributed database.

In some embodiments, the encoding of the method distributed database may be implemented as a computer program tangibly embodied on a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the encoding of the method distributed database described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the encoding of the method distributed database by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

A computer program for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on a machine, as a stand-alone software package partly on a machine and partly on a remote machine or entirely on a remote machine or server.

In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.

The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solution of the present invention can be achieved.

The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

EXAMPLE six

Fig. 7 is a schematic structural diagram of an encoding system of a distributed database according to a sixth embodiment of the present invention, and as shown in fig. 7, the apparatus includes: at least one computing node 1 and at least one storage node 2 (one computing node and one storage node are taken as examples in the figure);

the computing node 1 is configured to, when receiving a data acquisition request sent by a client, determine data information and a first coding format of data to be acquired of the data acquisition request, determine, according to the data information, a second coding format of at least one intermediate data corresponding to the data to be acquired, send an intermediate data acquisition instruction to the storage node 2, receive intermediate data returned by the storage node 2 based on the intermediate data acquisition instruction, encode the intermediate data by using the first coding format, obtain the data to be acquired, and send the data to be acquired to the client;

the storage node 2 is configured to receive an intermediate data obtaining instruction sent by the computing node 1, obtain the intermediate data based on the second encoding format and the storage type of the storage node 2, and send the intermediate data to the computing node 1.

Wherein, the computing node 1 and the storage node 2 can be regarded as a device, such as a server, and the specific types of the computing node 1 and the storage node 2 are not limited.

In one embodiment, the compute node supports processing of at least two encoding formats.

The computing node 1 may support processing in at least two encoding formats, such as a line-type encoding format and a column-type encoding format, and on the basis, the efficiency and performance of the encoding system of the distributed database may be improved.

In one embodiment, the external interface of the storage node supports a row-wise encoded network protocol and/or a column-wise encoded network protocol.

It is considered that the external interface of the storage node 2 can support the line-type coding network protocol and/or the column-type coding network protocol to meet the requirements of different coding formats and adapt to more coding scenarios.

Claims

1. An encoding method for a distributed database, applied to a computing node of the distributed database, the method comprising:

and coding the intermediate data by adopting the first coding format to obtain the data to be acquired, and sending the data to be acquired to the client.

2. The method of claim 1, wherein for each intermediate data, the intermediate data comprises at least one intermediate data block, and the at least one intermediate data block is encoded in a same or different sub-encoding format as in the second encoding format.

3. The method according to claim 1, wherein the determining a second encoding format of at least one intermediate data corresponding to the data to be acquired according to the data information comprises:

for each intermediate data, estimating the data size of the intermediate data;

determining a second encoding format of the intermediate data as a column-wise encoding format if the data size is greater than or equal to a first data size threshold;

4. The method according to claim 1, wherein before the encoding the intermediate data in the first encoding format to obtain the data to be acquired, the method further comprises:

5. An encoding method for a distributed database, which is applied to a storage node of the distributed database, the method comprising:

6. The method of claim 5, wherein the obtaining the intermediate data based on the second encoding format and the storage type of the storage node comprises:

7. The method of claim 5, wherein sending the intermediate data to the computing node comprises:

8. An apparatus for encoding a distributed database, the apparatus being configured for use at a compute node of the distributed database, the apparatus comprising:

9. An apparatus for encoding a distributed database, the apparatus being configured to be disposed at a storage node of the distributed database, the apparatus comprising:

10. An electronic device, characterized in that the device comprises:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the encoding method of the distributed database of any one of claims 1-7.

11. A distributed database based coding system, the system comprising at least one compute node and at least one storage node;

12. The system of claim 11, wherein the compute node supports processing of at least two encoding formats.

13. The system according to claim 11, wherein the external interface of the storage node supports a row-wise coded network protocol and/or a column-wise coded network protocol.

14. A computer-readable storage medium storing computer instructions for causing a processor to implement the encoding method of the distributed database according to any one of claims 1 to 7 when executed.