CN114281550A

CN114281550A - Request processing method and device, computer equipment and computer readable storage medium

Info

Publication number: CN114281550A
Application number: CN202111640279.6A
Authority: CN
Inventors: 林晓芳; 张建松; 孙宇熙
Original assignee: Beijing Acme Yingtu Technology Co ltd
Current assignee: Beijing Acme Yingtu Technology Co ltd
Priority date: 2021-12-29
Filing date: 2021-12-29
Publication date: 2022-04-05

Abstract

The embodiment of the invention discloses a request processing method, a request processing device, computer equipment and a computer readable storage medium, wherein the method comprises the following steps: after any instance in the server cluster receives the processing request, acquiring the type of the data processing request, namely determining which type of the transaction processing type and the analysis processing type the request is; after the type of the request is determined, acquiring the load condition of each instance matched with the type, and acquiring the load requests of all analysis processing instances if the request is an analysis processing type; and sending the data processing request to the instance with the lowest load condition, so that the instance with the lowest load condition executes the operation corresponding to the data processing request. Therefore, the embodiment of the invention improves the data processing efficiency and improves the data throughput of the database application.

Description

Request processing method and device, computer equipment and computer readable storage medium

Technical Field

The present invention relates to the field of data processing, and in particular, to a request processing method and apparatus, a computer device, and a computer-readable storage medium.

Background

With the comprehensive development of related technologies of computing, networking, storage, virtualization, big data, cloud computing, distributed databases and memory computing, the technology update of database technologies is promoted, particularly the emergence of an HTAP (Hybrid Transactional Analytical Processing) system, and a traditional database can realize the rapid analysis of data while ensuring high availability, transaction consistency and high expandability.

However, when the HTAP system is used to process complex, variable, deeply related, and highly concurrent transaction requests corresponding to a graph database, due to problems such as system architecture, the HTAP system has difficulty in meeting business requirements at the present stage in terms of timeliness, flexibility, computing power, and algorithm capability.

Disclosure of Invention

In view of the above, the present invention provides a request processing method, an apparatus, a computer device and a computer readable storage medium, which are used to improve the current situation that it is difficult to meet the business requirements of the current stage in terms of timeliness, flexibility, computing power and algorithm capability by using an HTAP system.

In a first aspect, an embodiment of the present invention provides a request processing method, which is applied to an instance set in a server cluster, where the instance includes a transaction processing instance and an analysis processing instance, and the method includes:

responding to a data processing request, and acquiring the type of the data processing request, wherein the type comprises a transaction processing type or an analysis processing type;

obtaining a load condition of each instance matched with the type, wherein the load condition represents the number of the requests to be processed;

and sending the data processing request to the instance with the lowest load condition, so that the instance with the lowest load condition executes the operation corresponding to the data processing request.

Optionally, in an implementation manner provided by the embodiment of the present invention, after the sending the data processing request to the instance with the lowest load condition, the method further includes:

the instance with the lowest load condition updates the log information of the instance with the lowest load condition according to the data processing request, wherein the log information represents the request to be processed and the processed request;

and the example with the lowest load condition sends the log information to other examples so that the other examples finish log information synchronization.

Optionally, in an implementation manner provided by the embodiment of the present invention, the method further includes:

taking the most complete example of the log information in all the examples as a main example;

and the main instance sends the log information of the main instance to other instances so that the other instances finish log information synchronization.

acquiring the load conditions of all transaction processing instances and the load conditions of all analysis processing instances;

and when the difference value between the load condition of all the transaction processing examples and the load condition of all the analysis processing examples exceeds a preset value, migrating the resources of the examples with less load conditions to the examples with more load conditions.

Optionally, in an implementation manner provided by the embodiment of the present invention, the instance further includes a transaction and analysis mixed processing instance, and the transaction and analysis mixed processing instance and the request with the type of transaction processing and the request with the type of type analysis processing are both matched.

acquiring an example role adjustment request;

if the current instance corresponding to the instance role adjustment request is a transaction instance, setting the current instance as the analysis processing instance;

and if the current instance corresponding to the instance role adjustment request is an analysis processing instance, setting the current instance as the transaction processing instance.

In a second aspect, an embodiment of the present invention provides a request processing apparatus, applied to an instance arranged in a server cluster, where the instance includes a transaction processing instance and an analysis processing instance, and the apparatus includes:

the response module is used for responding to a data processing request and acquiring the type of the data processing request, wherein the type comprises a transaction processing type or an analysis processing type;

the load obtaining module is used for obtaining the load condition of each instance matched with the type, wherein the load condition represents the number of the requests to be processed;

and the sending module is used for sending the data processing request to the instance with the lowest load condition so as to enable the instance with the lowest load condition to execute the operation corresponding to the data processing request.

Optionally, in an implementation manner provided by the embodiment of the present invention, the apparatus further includes:

the log updating module is used for updating the log information of the instance with the lowest load condition according to the data processing request, wherein the log information represents a request to be processed and a processed request;

and the first synchronization module is used for sending the log information to other instances by the instance with the lowest load condition so as to enable the other instances to finish log information synchronization.

In a third aspect, an embodiment of the present invention provides a computer device, including a memory and a processor, where the memory stores a computer program, and the computer program, when running on the processor, executes the request processing method disclosed in any one of the first aspects.

In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when running on a processor, executes the request processing method disclosed in any one of the first aspects.

In the request processing method disclosed by the embodiment of the invention, after any one instance in the server cluster receives a data processing request, the type of the data processing request is obtained, namely the request is determined to be any one of a transaction processing type and an analysis processing type; after the type of the request is determined, acquiring the load condition of each instance matched with the type, and acquiring the load requests of all analysis processing instances if the request is an analysis processing type; and sending the data processing request to the instance with the lowest load condition, so that the instance with the lowest load condition executes the operation corresponding to the data processing request.

Therefore, the embodiment of the invention always processes the server with the lowest load condition, so that the server can respond to each request in time, the speed of the server cluster responding to the client is further ensured, and the data processing efficiency is improved; in addition, any one instance server can complete corresponding transaction operation, so that the speed of transaction processing and analysis processing is balanced, and the data throughput of the database application is improved.

Drawings

In order to more clearly illustrate the technical solution of the present invention, the drawings required to be used in the embodiments will be briefly described below, and it should be understood that the following drawings only illustrate some embodiments of the present invention, and therefore should not be considered as limiting the scope of the present invention. Like components are numbered similarly in the various figures.

Fig. 1 is a schematic flow chart illustrating a first request processing method according to an embodiment of the present invention;

fig. 2 is a flowchart illustrating a second request processing method according to an embodiment of the present invention;

FIG. 3 is a flow chart illustrating a third request processing method according to an embodiment of the present invention;

fig. 4 is a schematic structural diagram illustrating a request processing apparatus according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments.

The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present invention without making any creative effort, shall fall within the protection scope of the present invention.

Hereinafter, the terms "including", "having", and their derivatives, which may be used in various embodiments of the present invention, are only intended to indicate specific features, numbers, steps, operations, elements, components, or combinations of the foregoing, and should not be construed as first excluding the existence of, or adding to, one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing.

Furthermore, the terms "first," "second," "third," and the like are used solely to distinguish one from another and are not to be construed as indicating or implying relative importance.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which various embodiments of the present invention belong. The terms (such as those defined in commonly used dictionaries) should be interpreted as having a meaning that is consistent with their contextual meaning in the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein in various embodiments of the present invention.

It should be understood that an HTAP (Hybrid Transactional/Analytical Processing) system is used to solve the data management problem of high-concurrency transaction Processing and fast analysis of mass data in the current environment. The database application realized based on the HTAP system can execute online transaction processing and/or online analysis processing according to the request sent by the client equipment, so that the data has the advantages of high concurrency and real-time data analysis in the processing process.

However, when the database implemented based on the HTAP system executes the online analysis processing operation or the online transaction processing operation, the online analysis processing operation occupies more resources, so that the online transaction processing operation cannot be normally executed, and the throughput of the database is further reduced.

In addition, in the process of processing the transaction operation for the graph database by the database implemented based on the HTAP system, since the transaction operation corresponding to the graph database is complex and may involve more vertices and/or edges, the execution speed of the online analysis processing operation and the online transaction processing operation is further unbalanced, and the data throughput is reduced.

Therefore, embodiments of the present invention provide a request processing method, an apparatus, a computer device, and a computer-readable storage medium, which are used to improve the current situation. Specifically, referring to fig. 1, fig. 1 is a schematic flowchart illustrating a request processing method according to an embodiment of the present invention, where the request processing method according to the embodiment of the present invention is applied to an instance arranged in a server cluster, where the instance includes a transaction instance and an analysis processing instance.

That is, the request processing method provided by the embodiment of the present invention is applied to a Server in a C/S (Client/Server) architecture, that is, a Server cluster. The server cluster comprises a plurality of servers, and the plurality of servers are arranged based on a distributed technology. Each server represents an instance, which includes both transaction instances and analysis instances. All instances are used to uniformly manage data stored in one database.

Optionally, the database comprises a database.

It should be noted that, in the server cluster in the embodiment of the present invention, after a preset number of instances in the cluster are all online, each instance starts receiving an external request and executes a corresponding transaction operation according to the request; if only one instance in the server cluster is online, the online instance is used for generating metadata and/or core data; and if the number of the online examples does not exceed the preset number, only performing data synchronization on the online examples so as to enable the data of each example to be the same.

Further, the examples in the embodiments of the present invention include a transaction processing example and an analysis processing example, that is, the examples in the embodiments of the present invention are used for processing a request of which the type is a transaction processing type or analyzing a request of which the type is an analysis processing type. The request with the type of transaction processing represents that the instance executes the online transaction operation according to the request, and the request with the type of analysis processing represents that the instance executes the online analysis processing transaction operation according to the request.

It can be understood that the request processed by the transaction instance, i.e. the request whose type is transaction class, requires the server to perform the corresponding operation in real time and quickly when being processed. For example, the addition, deletion, modification and check of the vertex, the edge, the attribute corresponding to the vertex and the attribute corresponding to the edge in the graph database; also, for example, a query for a path composed of vertices and edges of a graph database or a query of the K neighborhood class. The transaction instances in embodiments of the present invention focus on real-time, highly concurrent data processing.

The request that can be processed by the analysis processing instance, that is, the request of the type being the analysis processing type, requires the server to complete relatively complex transaction operations, such as batch processing, massive data processing, full data processing, and the like, and does not require the server to return results in real time, and the results corresponding to the request may be returned asynchronously or in a streaming manner. Illustratively, the request for the type of analytics processing may be a graph algorithm class operation that is full data oriented.

Based on this, the request processing method provided by the embodiment of the invention includes:

s110, responding to the data processing request, and acquiring the type of the data processing request, wherein the type comprises a transaction processing type or an analysis processing type.

That is, the client device sends a request to the server device, that is, any one server in the server cluster, so that the server performs corresponding transaction operations on the database application according to the request, and returns a corresponding processing result.

It should be noted that, in the embodiment of the present invention, when the client device sends the request, a character that can indicate the type of the request needs to be written in the request, so that the server can determine the type of the request according to the character after parsing the request.

It should be further noted that, the embodiment of the present invention does not limit the specific execution process after the client device sends the request to the server device, and the specific execution process after the server device receives the request may be specifically set according to an actual situation.

In a feasible manner, all instances in the server cluster can select a main instance according to a preset algorithm, wherein the main instance is used for providing anti-proxy service, that is, when the client device sends a request to the server device, the client device sends the request to the main instance, so that the main instance obtains the type of the received request and forwards the request to a matched instance.

In another feasible mode, after the client device sends a request to any one server, the server sends the request to any one transaction instance, so that the transaction instance determines the type of the request, and if the type of the request is the transaction type, the transaction instance completes the request by itself; if the transaction processing instance is not the transaction processing type, the transaction processing instance sends the request to any analysis processing instance to complete the corresponding transaction operation. It will be appreciated that if the server itself is the transaction instance, then the validation of the request type is done by the server itself.

And S120, acquiring the load condition of each instance matched with the type, wherein the load condition represents the number of the requests to be processed.

It will be appreciated that any one instance may not necessarily process requests immediately upon receipt of a request, e.g., an instance may process three requests after receiving a fourth request, which may be processed after the first three requests.

Therefore, in order to ensure timely response of request processing, the embodiment of the present invention obtains the load request of each instance matching the request, that is, the number of requests that each instance needs to process, and sends the request to the instance with the lowest load condition.

Exemplarily, when the server receiving the request determines that the received request is of a transaction type, the server obtains load requests of all transaction instances to determine the transaction instance with a smaller number of requests to be processed in all transaction instances.

It should be understood that the load condition of the example is described by the number of the requests to be processed in the embodiment of the present invention, but in an actual environment, the load condition of the example may be described in other ways. As one possible approach, the load condition of an instance is described by the utilization of the instance server's central processor; in yet another possible approach, the load of the instance is described by the memory occupancy of the instance server.

S130, the data processing request is sent to the instance with the lowest load condition, so that the instance with the lowest load condition executes the operation corresponding to the data processing request.

That is, after the server receiving the request determines the lowest load instance among all the matched instances, the received request is sent to the lowest load instance to enable the request to be processed by the instances, that is, the lowest load instance executes the transaction operation corresponding to the request, so as to ensure that the request sent by the client device can be processed by the server cluster in the shortest time.

Furthermore, according to the embodiment of the invention, based on the settings of the transaction processing instance and the analysis processing instance, and the classification and identification of the request, any one server in the server cluster can send the request to the instance matched with the type of the request after receiving different types of requests, so that the instance matched with the request can complete corresponding transaction operation.

Based on this, because the instance for processing the request is always the server with the lowest load condition, the server can respond to each request in time, thereby ensuring the speed of the server cluster for responding to the client and improving the data processing efficiency. In addition, any one instance server can complete corresponding transaction operation, so that the execution speeds of transaction processing and analysis processing are not interfered with each other, and the data throughput of the database application is further improved.

Optionally, referring to fig. 2, fig. 2 is a schematic flowchart illustrating a second request processing method according to an embodiment of the present invention, that is, in an implementation manner provided by the embodiment of the present invention, after S130, the method further includes:

s140, the instance with the lowest load condition updates the log information of the instance with the lowest load condition according to the data processing request, wherein the log information represents the request to be processed and the processed request;

s150, the example with the lowest load condition sends the log information to other examples so that the other examples complete log information synchronization.

That is, in this embodiment, after any one of the example servers executes the corresponding transaction operation according to the request, it is necessary to send its own log to other servers in the server cluster, so as to synchronize the data of each server, that is, to ensure that the data in each server is uniform, and to ensure the validity of the data returned by each server.

It should be noted that, in the embodiment of the present invention, the time when the log information is sent to another instance by the instance with the lowest load condition is not limited, and may be specifically set according to an actual condition.

Illustratively, as one possibility, the instance with the lowest load updates the log information and does not send the log information to other instances immediately, but sends the log information to other servers within a predetermined instance synchronization period.

It can be understood that when log synchronization is performed, the efficiency of processing the request by the server cluster will inevitably be reduced, and if a plurality of instances update the log information and send the log information to other instances at the same time, the efficiency of processing the request by the server cluster will be further affected. Thus, the server cluster in this possible manner performs log information synchronization only during the instance synchronization period. It should be understood that, when the instance synchronization period is set, it is required to ensure that the number of requests received by the server cluster in the instance synchronization period is small, so as to avoid the untimely response of the requests.

Further, in an implementation manner provided by the embodiment of the present invention, the method further includes:

That is, in order to further ensure the consistency of the data in each instance, the embodiment of the present invention selects the master instance based on the log information of each instance, and enables the master instance to send the log information of the master instance to other instances for log information synchronization.

It will be appreciated that the most complete log information, i.e., the request handling record recorded in the log information indicating the instance, is the most recent and most abundant.

It will also be appreciated that if the master instance goes offline due to an unexpected incident, other instances will elect a new master instance in the same manner and log information synchronization will proceed again.

Optionally, referring to fig. 3, fig. 3 is a schematic flowchart illustrating a third request processing method provided in an embodiment of the present invention, that is, in an implementation manner provided in the embodiment of the present invention, the method further includes:

s160, acquiring the load conditions of all transaction processing instances and the load conditions of all analysis processing instances;

s170, when the difference value between the load condition of all the transaction processing examples and the load condition of all the analysis processing examples exceeds a preset value, the resource of the example with less load condition is migrated to the example with more load condition.

That is, according to the load conditions of all transaction processing instances and the load conditions of all analysis processing instances, the embodiments of the present invention determine the number of requests to be processed by the two types of instances, and when the difference between the number of requests to be processed by one type of instance and the number of requests to be processed by the other type of instance exceeds a preset value, migrate the resources occupied by all the instances with fewer processing requests, that is, with lower load conditions, to the instance with more processing requests, so that the instance with more load can process the requests in time.

Optionally, in a feasible manner provided by the embodiment of the present invention, besides migrating resources, some instances with less load in the embodiment of the present invention are also converted into instances with more load, for example, a transaction processing instance with less load is converted into an analysis processing instance with more load, so that the instances with more load can quickly process requests.

Further, in another possible way provided by the embodiment of the present invention, the embodiment of the present invention may further send the request to be processed by the instance with the higher load condition to the instance with the lower load condition, and convert part of the instances with the lower load condition into the instances with the higher load condition, so as to relieve the load of the instances.

Optionally, in addition to migrating the resources when the difference between the load conditions of all the transaction processing instances and the load conditions of all the analysis processing instances exceeds the preset value, the resources of the instances which do not exceed the preset load threshold may be migrated to the instances which exceed the preset load threshold when the load conditions of all the transaction processing instances or the load conditions of all the analysis processing instances exceed the preset load threshold, so as to alleviate the load conditions of the instances.

In addition, it should be noted that fig. 3 provided in the embodiment of the present invention is only used to better illustrate such an implementation provided in the embodiment of the present invention, and does not limit the execution sequence of S160 and S170 in the whole scheme, that is, S160 and S170 may be executed before or after any step.

That is, in this embodiment, the server cluster includes three types of instances, namely, a transaction instance, an analysis instance, and a transaction and analysis hybrid instance. It should be noted that the transaction and analysis hybrid processing example can process both the request of the transaction type and the request of the analysis type, and further match both the requests.

Also, since the transaction and analysis hybrid process instance can process two types of instances, the transaction and analysis hybrid process instance can be regarded as either a transaction process instance or an analysis process instance in S120, and the request may be sent to the transaction and analysis hybrid process instance in S130.

Therefore, in the embodiment of the invention, the transaction and analysis mixed processing example can flexibly complete different types of requests, dynamic matching of the example and the request is realized, and the processing efficiency of the two types of requests can be improved.

acquiring an example role adjustment request;

That is, any one server in the server cluster performs role adjustment on the current instance corresponding to the instance role adjustment request according to the instance role adjustment request sent by the client device, for example, setting the transaction instance as an analysis processing instance, or setting the analysis processing instance as a transaction processing instance.

Further, if the server cluster further includes a transaction and analysis hybrid processing instance, the user needs to specify the target role type corresponding to the current instance in an instance role adjustment request sent by the terminal. Exemplarily, if the current instance is a transaction instance, if the target role type is a transaction and analysis hybrid instance, setting the current instance as the transaction and analysis hybrid instance; and if the target role type is the analysis processing instance, setting the current instance as the analysis processing instance.

Corresponding to the request processing method, the embodiment of the present invention further provides a request processing apparatus, and specifically referring to fig. 4, fig. 4 shows a schematic structural diagram of the request processing apparatus provided in the embodiment of the present invention. The request processing device 200 provided by the embodiment of the invention is applied to the example arranged in the server cluster, the example comprises a transaction processing example and an analysis processing example, and the device comprises:

a response module 210, configured to respond to a data processing request, and obtain a type of the data processing request, where the type includes a transaction type or an analysis processing type;

a load obtaining module 220, configured to obtain a load condition of each instance matching the type, where the load condition represents the number of requests to be processed;

a sending module 230, configured to send the data processing request to the instance with the lowest load condition, so that the instance with the lowest load condition executes an operation corresponding to the data processing request.

The request processing device provided in the embodiment of the present application can implement each process of the request processing method in the method embodiment of fig. 1, and can achieve the same technical effect, and is not described here again to avoid repetition.

An embodiment of the present invention further provides a computer device, including a memory and a processor, where the memory stores a computer program, and the computer program, when running on the processor, executes the request processing method disclosed in the method embodiment shown in fig. 1.

An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when running on a processor, executes the request processing method disclosed in the method embodiment shown in fig. 1.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative and, for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

In addition, each functional module or unit in each embodiment of the present invention may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention or a part of the technical solution that contributes to the prior art in essence can be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a smart phone, a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention.

Claims

1. A request processing method applied to an instance disposed in a server cluster, the instance including a transaction instance and an analysis processing instance, the method comprising:

2. The method of claim 1, wherein after sending the data processing request to the instance with the lowest load condition, the method further comprises:

3. The method of claim 2, further comprising:

4. The method of claim 1, further comprising:

5. The method of claim 1, wherein the instances further comprise transaction and analytics hybrid instances, and wherein the transaction and analytics hybrid instances and requests of the type transaction type and requests of the type match.

6. The method of claim 1, further comprising:

acquiring an example role adjustment request;

7. A request processing apparatus applied to an instance provided in a server cluster, the instance including a transaction instance and an analysis instance, the apparatus comprising:

8. The apparatus of claim 7, further comprising:

9. A computer arrangement comprising a memory and a processor, the memory storing a computer program which, when run on the processor, performs the request processing method of any one of claims 1-6.

10. A computer-readable storage medium, having stored thereon a computer program which, when run on a processor, performs a request processing method according to any one of claims 1-6.