CN117093607A

CN117093607A - Optimization method for data statistics query in big data scene

Info

Publication number: CN117093607A
Application number: CN202310926320.9A
Authority: CN
Inventors: 李栋; 潘祥; 张燕
Original assignee: Xi'an Notice Network Technology Co ltd
Current assignee: Xi'an Notice Network Technology Co ltd
Priority date: 2023-07-26
Filing date: 2023-07-26
Publication date: 2023-11-21

Abstract

The invention discloses an optimization method for data statistics query in a big data scene, which comprises the following steps: (1) Determining a bottom data table of the business scene design according to the demand analysis; (2) Managing metadata information of the data table through a program; (3) Developing a group of independent components for controlling each statistical analysis query to accurately query according to query conditions and preset virtual table information; (4) And (3) carrying out data query again on the virtual table information of the component in the step (3) through the effective parameters to obtain a data result. Compared with the prior art, the invention has the advantages that: 1. the method can be accessed quickly to improve the data query capability; 2. the service data use scene of the change is rapidly met; 3. optimizing a data query route, quickly responding to a query result, and improving the stability of a bottom database; 4. and the stability and usability of the underlying database are ensured.

Description

Optimization method for data statistics query in big data scene

Technical Field

The invention relates to the technical field of statistical query, in particular to an optimization method for data statistical query in a big data scene.

Background

Since the entry of the big data age, massive data is not only financial, but also a burden for enterprises. The same challenges are faced with whether large or small enterprises, how to utilize the experience of large data clients, and the effect of optimizing productivity is effectively achieved. This is also an important reason why many enterprises have chosen to build modern big data analysis platforms in recent years.

However, whether it is a BI platform in the traditional sense or a data analysis product of a coupled business system, it is strongly dependent on the underlying data statistics query in nature, i.e., we often say a large SQL of query data. In general, the construction of the data platform only focuses on the quality problem of the data, and the data query stage is also the self judgment and control of users needing the data, such as a BI platform or a report tool.

Under the circumstance that the service scene is more and more complex, the situation that a plurality of tables in a database are required to be correlated and then data query is carried out through specific conditions in order to meet a certain special data query scene is unavoidable, and under the general circumstances, a common scheme is that a data logic view is made at the bottom layer of the database or ETL of data is made according to the service scene in advance.

Therefore, it is particularly important how to be able to adapt to rapidly changing business data demands, and to quickly iterate out the underlying data logic without a great deal of development intervention.

Disclosure of Invention

The invention aims to provide an optimization method for data statistics query in a big data scene, which aims to solve the problem that the quick-change business data requirement cannot be used and the underlying data logic needs to be iterated quickly.

In order to solve the technical problems, the technical scheme provided by the invention is as follows: an optimization method for data statistics query in big data scene comprises the following steps:

(1) Determining a bottom data table of the business scene design according to the demand analysis;

(2) Managing metadata information of the data table through a program;

(3) Developing a group of independent components for controlling each statistical analysis query to accurately convert the data query added by the bottom large table into a plurality of related table sub-queries and then adding the data query to the intermediate temporary data view according to the query conditions and preset virtual table information;

(4) And (3) carrying out data query again on the virtual table information of the component in the step (3) through the effective parameters to obtain a data result.

According to a third aspect of the present disclosure, there is provided an electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method as provided in the first aspect.

According to a fourth aspect of the present disclosure, there is provided a non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method provided according to the first aspect.

According to a fifth aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method provided according to the first aspect.

The method and the device for the application parameter package rendering obtain the application parameter package of the application to be deployed through receiving the application configuration parameters from the user and according to the application configuration parameters, a preset code warehouse and a configuration warehouse and through a preset application model. And finally, deploying the application to be deployed on the cloud platform according to the application parameter package. Because the user only needs to provide the configuration parameters, the application parameter package can be generated according to the configuration parameters and the application can be deployed on the cloud platform, the user does not need to directly write the application parameter package, the threshold for deploying the application on the cloud platform is greatly reduced, and a more convenient deployment mode of the cloud native application is provided.

Preferably, the separate component is capable of matching only the dataset, comprising the steps of:

(1) The front end initiates a statistical query request according to the interface parameters;

(2) Extracting data table basic information according to the data table and the fields and combining metadata information;

(3) According to basic information of the data table, dynamically splitting molecular query SQL according to query parameters, and adding sequences and conditions to the intermediate state data table serving as sub-query, wherein the virtual table is a large SQL;

(4) Performing data query on the large SQL of the virtual table according to the condition;

(5) Meanwhile, the slow query and the data magnitude change are monitored and analyzed, and are fed back to metadata for information updating, so that the sub-query addition sequence is adjusted dynamically, and the final virtual table is ensured to be an optimal intermediate state data set.

Preferably, the statistical query request comprises a data table and query conditions.

Preferably, the data table basic information comprises aggregation level and data magnitude information.

Preferably, the result of the data query on the large SQL of the virtual table is processed according to the condition and then returned to the front end for rendering.

Compared with the prior art, the invention has the advantages that: 1. the optimization strategy and the technology of the data statistics query in the big data scene are developed and realized in the background of the data statistics query, are not perceived by the data application end, and can be quickly accessed to improve the data query capability only according to a specific rule.

2. The data developer can edit and update the virtual table through the relevant interface or UI, so that the changed service data use scene can be rapidly met.

3. The method adopts the mode of intelligent splitting and query condition filling of an algorithm, dynamically disassembles and reassembles the complex added query of the bottom layer under the limited parameters of the front end, optimizes the data query route, quickly responds to the query result, improves the stability of the database of the bottom layer, and reduces unnecessary performance cost.

4. And monitoring and judging the engineering constructed by the virtual table, and rapidly intercepting and fault-tolerant processing logic which can generate slow inquiry or influence the performance of the database, so that the stability and the usability of the bottom database are ensured.

It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.

Drawings

FIG. 1 is a flow chart of the intelligent matching dataset of the present invention.

Fig. 2 is a schematic diagram of the components of the application deployment apparatus of the present invention.

Fig. 3 is a schematic block diagram of an electronic device of the present invention.

Detailed Description

Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

The method can be applied to electronic equipment, and the electronic equipment can be a smart phone, a tablet computer, a notebook computer, a desktop computer or a custom terminal. The operating systems of these devices may be Android (Android) systems, windows (Windows), apple mobile operating system (iOS), apple operating system (MacOS) or hong system (HarmonyOS), lin Nasi system (Linux), etc., which are not limiting to the present disclosure.

Fig. 2 is a schematic diagram of the composition of an application deployment device according to an embodiment of the present disclosure.

As shown in fig. 2, the apparatus may include:

the receiving module 51 is configured to receive an application configuration parameter from a user, where the application configuration parameter is used to describe an application to be deployed.

A generation module 52 for generating at least one code repository and a configuration repository from application configuration parameters.

And the rendering module 53 is configured to render, through a preset application model, an application parameter packet of the application to be deployed according to at least one code repository and a configuration repository.

The deployment module 54 is configured to deploy the application to be deployed on the cloud platform according to the application parameter package.

In some embodiments, the rendering module 53 is configured to generate a container image from at least one code repository. And acquiring the construction materials of at least one application parameter package according to the container mirror image and the configuration warehouse. And pre-rendering is carried out according to the container mirror image and the construction materials, and a pre-rendered application parameter package is obtained. And acquiring an additional modification file corresponding to the prerendered application parameter package. And merging the additional modification file and the prerendered application parameter package to obtain an application parameter package of the application to be deployed.

In some embodiments, an acquisition module 55 is also included for receiving additional modification parameters from the user based on the visual interface. And generating an additional modification file according to the additional modification parameters.

In some embodiments, the rendering module 53 is specifically configured to locate, in the pre-rendered application parameter package, a corresponding parameter according to the additional modification file. And modifying the corresponding parameters in the pre-rendered application parameter package according to the values of the parameters in the additional modification file. And packaging the modified prerendered application parameter package to obtain an application parameter package of the application to be deployed.

In the technical scheme of the disclosure, the acquisition, storage, application and the like of the related user personal information all conform to the regulations of related laws and regulations, and the public sequence is not violated.

According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

An electronic device, comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method as provided in the above embodiments.

In an exemplary embodiment, the readable storage medium may be a non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method according to the embodiments provided above.

In an exemplary embodiment, the computer program product comprises a computer program which, when executed by a processor, implements a method according to the embodiments provided above.

Fig. 3 illustrates a schematic block diagram of an example electronic device 600 that may be used to implement embodiments of the present disclosure.

Electronic devices are intended to represent various forms of digital computers, such as car computers, laptops, tablets, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 3, the apparatus 600 includes a computing unit 601 that can perform various suitable actions and processes according to a computer program stored in a Read Only Memory (ROM) 602 or a computer program loaded from a storage unit 608 into a Random Access Memory (RAM) 603. In the RAM603, various programs and data required for the operation of the device 600 may also be stored. The computing unit 601, ROM602, and RAM603 are connected to each other by a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.

Various components in the device 600 are connected to the I/O interface 605, including: an input unit 606 such as a keyboard, mouse, etc.; an output unit 607 such as various types of displays, speakers, and the like; a storage unit 608, such as a magnetic disk, optical disk, or the like; and a communication unit 609 such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.

The computing unit 601 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 601 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the respective methods and processes described above, such as a page rendering method. For example, in some embodiments, the page rendering method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 600 via the ROM602 and/or the communication unit 609. When the computer program is loaded into the RAM603 and executed by the computing unit 601, one or more steps of the page rendering method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the application deployment method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.

The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

In some embodiments, the receiving module 51 is specifically configured to receive, based on the visual interface, the standard configuration parameters and the custom configuration parameters from the user. And generating application configuration parameters according to the standard configuration parameters and the custom configuration parameters.

In combination with fig. 1, an optimization method for data statistics query in big data scene is to split complex query logic and dynamically generate a virtual table which is most in line with the current query condition through certain algorithm and logic intelligent matching in the data query background, so as to realize a component of the data statistics query function, improve the response time of each statistics analysis under the condition of the quick response requirement of the statistics query, improve the response efficiency of the system, and greatly reduce the performance consumption of the bottom database.

The method comprises the following steps:

step 1, according to demand analysis, determining a bottom data table related to a business scene

Step 2, managing metadata information of the data table through a program;

step 3, developing a group of independent components for controlling each statistical analysis query to accurately convert the data query of the bottom large table Join into a temporary data view of the intermediate state of the Join after a plurality of related table sub-queries according to the query conditions and preset virtual table information;

and 4, carrying out data query again on the virtual table constructed in the step 3 through the effective parameters to obtain a data result.

The whole logic and flow of the intelligent matching data set in the whole statistical analysis process comprises the following detailed steps:

1. the front end initiates a statistical query request according to the interface parameters, wherein the statistical query request comprises a data table and query conditions;

2. extracting data table basic information according to the data table and the fields and combining metadata information;

3. according to the basic information of the data table, the method comprises the following steps: aggregating information such as grades, data magnitude and the like, dynamically splitting molecular query SQL according to query parameters, and carrying out sequence and condition Join on intermediate state data tables serving as sub-queries to construct a virtual table large SQL;

4. performing data query on the large SQL of the virtual table according to the condition, and returning the query result to the front end for rendering after processing;

5. meanwhile, the slow query and the data magnitude change are monitored and analyzed, and are fed back to metadata for information updating, so that the sub-query addition sequence is adjusted dynamically, and the final virtual table is ensured to be an optimal intermediate state data set.

In summary, the method achieves the fastest data query and response to each statistic analysis query through the intelligent splitting of the algorithm logic and the construction of the dynamic optimal virtual table for complex data query, is convenient for rapidly coping with the change of service data requirements, improves the throughput of the system, and reduces the performance cost of the bottom database.

The invention and its embodiments have been described above with no limitation, and the actual construction is not limited to the embodiments of the invention as shown in the drawings. In summary, if one of ordinary skill in the art is informed by this disclosure, a structural manner and an embodiment similar to the technical solution should not be creatively devised without departing from the gist of the present invention.

It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel, sequentially, or in a different order, provided that the desired results of the disclosed aspects are achieved, and are not limited herein.

The above detailed description should not be taken as limiting the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure are intended to be included within the scope of the present disclosure.

Claims

1. The optimization method for the data statistics query in the big data scene is characterized by comprising the following steps:

(2) Managing metadata information of the data table through a program;

2. The optimization method for data statistics query in big data scene according to claim 1, wherein: the individual components are capable of matching only the data sets, comprising the steps of:

3. The optimization method for data statistics query in big data scene according to claim 2, wherein: the statistical query request includes a data table and query conditions.

4. The optimization method for data statistics query in big data scene according to claim 2, wherein: the data table basic information comprises aggregation level and data magnitude information.

5. The optimization method for data statistics query in big data scene according to claim 1, wherein: and the result of the data query on the large SQL of the virtual table is processed according to the condition and then returned to the front end for rendering.

6. An electronic device, comprising: at least one processor; and a memory communicatively coupled to the at least one processor;

wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

7. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-5.

8. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any of claims 1-5.