CN116401140A

CN116401140A - Data processing method, device, equipment, readable medium and software product

Info

Publication number: CN116401140A
Application number: CN202111629355.3A
Authority: CN
Inventors: 张世其; 项欢
Original assignee: Wuzhou Online E Commerce Beijing Co ltd
Current assignee: Wuzhou Online E Commerce Beijing Co ltd
Priority date: 2021-12-28
Filing date: 2021-12-28
Publication date: 2023-07-07

Abstract

The embodiment of the application provides a data processing method, a device, equipment, a readable medium and a software product, wherein the method comprises the following steps: in the process of data query, especially in the process of configuring a structured query template, the index information and the data query template can be acquired, then initial index column information can be output according to the index information and the data query template, the index column information can be analyzed, so that research personnel can know query contents corresponding to the index information, then the initial index column information can be configured, the initial index column information is adjusted to be target index column information, the index information can be more clearly described in the template through adjustment of the index column information, then the target index column information is added into the data query template, the query template is updated, and the structured query template is generated, so that data analysts can perform data query through the structured query template.

Description

Data processing method, device, equipment, readable medium and software product

Technical Field

The present invention relates to the field of data processing technology, and in particular, to a data processing method, a data processing apparatus, an electronic device, a computer readable medium, and a software product.

Background

Along with the accumulation of data in the internet industry and the entering of fine operation, more and more products, operation and technicians need to master the skills of data analysis so as to acquire the user demands in the data by checking, looking up and analyzing the data, and apply the obtained results to production by further data analysis, thereby meeting the user demands and creating more social values. In order to meet the requirements of non-professional data analyzers, the non-professional data analyzers can replace dimensions, indexes and the like to be analyzed in the data analysis process, so that multi-dimensional cross analysis is realized. However, to meet such a requirement, data research and development personnel are required to access the index of the data analysis, and a great deal of time is required to adjust the data query language, so that the cost of data query language debugging is greatly increased, and the debugging efficiency is low.

Disclosure of Invention

The embodiment of the application provides a data processing method, a data processing device, electronic equipment, a computer readable storage medium and a software product, which are used for solving or partially solving the problems of high data query language debugging cost and low debugging efficiency in the data analysis process in the related technology.

The embodiment of the application discloses a data processing method, which comprises the following steps:

acquiring index information and a data query template;

outputting initial index column information according to the index information and the data query template;

responding to an information configuration operation aiming at the initial index column information, and adjusting the initial index column information into target index column information corresponding to the information configuration operation;

and adding the target index column information to the data query template to generate a structured query template.

Optionally, the data query template includes a main query structure and a sub-query structure, and the outputting initial index column information according to the index information and the data query template includes:

inputting the index information into the data query template to generate an initial query template;

and responding to an operation instruction aiming at the initial query template, checking the initial query template, and outputting first field information corresponding to the sub-query structure and second field information corresponding to the main query structure.

Optionally, the responding to the running instruction for the initial query template checks the initial query template and outputs first field information corresponding to the main query structure and second field information corresponding to the sub-structure, including:

Responding to an operation instruction aiming at the initial query template, and acquiring a target grammar tree of the initial query template;

if the target grammar tree comprises a sub-query grammar tree, carrying out field identification on the sub-query grammar tree, and outputting first field information corresponding to the sub-query structure;

and acquiring field information of the main query structure, performing type identification on the field information, and outputting second field information corresponding to the main query structure.

Optionally, the responding to the information configuration operation for the initial index column information adjusts the initial index column information to target index column information corresponding to the information configuration operation, including:

responding to an information configuration operation aiming at the initial index column information, and acquiring configuration information corresponding to the information configuration operation, wherein the configuration information at least comprises a first field name and a first field type aiming at the first field information, and a second field name and a second field type aiming at the second field information;

adjusting the first field information into first target field information by adopting the first field name and the first field type;

And adjusting the second field information into second target field information by adopting the second field name and the second field type.

Optionally, the adding the target index column information to the data query template generates a structured query template, including:

adding the target index column information to the data query template, and updating the initial index column information;

acquiring main query information aiming at the main query structure and sub query information aiming at the sub query structure, wherein the main query information at least comprises a main query dimension and a main query identifier, and the sub query information at least comprises the sub query dimension and the sub query information;

determining a main query grammar tree corresponding to the main query structure and a sub query grammar tree corresponding to the sub query structure;

associating the primary query dimension with the primary query structure and adding the primary query identifier to the primary query syntax tree, associating the sub-query dimension with the sub-query structure and adding the sub-query identifier to the sub-query syntax tree, generating a structured query template.

Optionally, the method further comprises:

responding to data analysis operation, and acquiring query information corresponding to the data analysis operation, wherein the query information at least comprises query indexes and query dimensions;

Inputting the query index and the query dimension into the structured query template;

if the query dimension is successfully matched with the first target field information, determining a first dimension table corresponding to the sub-query dimension, and acquiring a sub-query result matched with the query dimension from the first dimension table;

if the query index is successfully matched with the second target field information, extracting a main query result matched with the query index from the query result;

and generating a data analysis result corresponding to the data analysis operation by adopting the sub-query result and the sub-query result.

The embodiment of the application also discloses a data processing device, which comprises:

the content acquisition module is used for acquiring index information and a data query template;

the index column information output module is used for outputting initial index column information according to the index information and the data query template;

an index column information adjustment module, configured to respond to an information configuration operation for the initial index column information, and adjust the initial index column information into target index column information corresponding to the information configuration operation;

and the query template generation module is used for adding the target index column information to the data query template to generate a structured query template.

Optionally, the data query template includes a main query structure and a sub-query structure, and the index column information output module includes:

the query template generation sub-module is used for inputting the index information into the data query template to generate an initial query template;

and the field information output sub-module is used for responding to the operation instruction aiming at the initial query template, checking the initial query template and outputting first field information corresponding to the sub-query structure and second field information corresponding to the main query structure.

Optionally, the field information output sub-module is specifically configured to:

Optionally, the index column information adjustment module includes:

A configuration information obtaining sub-module, configured to obtain configuration information corresponding to the information configuration operation in response to the information configuration operation for the initial index column information, where the configuration information includes at least a first field name and a first field type for the first field information, and a second field name and a second field type for the second field information;

a first field information adjustment sub-module, configured to adjust the first field information to first target field information by using the first field name and the first field type;

and the second field information adjustment sub-module is used for adjusting the second field information into second target field information by adopting the second field name and the second field type.

Optionally, the query template generating module includes:

the index column information updating sub-module is used for adding the target index column information to the data query template and updating the initial index column information;

the query information acquisition sub-module is used for acquiring main query information aiming at the main query structure and sub-query information aiming at the sub-query structure, wherein the main query information at least comprises a main query dimension and a main query identifier, and the sub-query information at least comprises a sub-query dimension and sub-query information;

A grammar tree determining sub-module, configured to determine a main query grammar tree corresponding to the main query structure and a sub-query grammar tree corresponding to the sub-query structure;

and the structured query template generation sub-module is used for associating the main query dimension with the main query structure, adding the main query identifier to the main query grammar tree, associating the sub-query dimension with the sub-query structure, and adding the sub-query identifier to the sub-query grammar tree to generate the structured query template.

Optionally, the method further comprises:

the query information acquisition module is used for responding to the data analysis operation and acquiring query information corresponding to the data analysis operation, wherein the query information at least comprises a query index and a query dimension;

the information input module is used for inputting the query indexes and the query dimensions into the structured query template;

the sub-query structure acquisition module is used for determining a first dimension table corresponding to the sub-query dimension if the query dimension is successfully matched with the first target field information, and acquiring a sub-query result matched with the query dimension from the first dimension table;

The main query structure acquisition module is used for extracting a main query result matched with the query index from the query result if the query index is successfully matched with the second target field information;

and the data analysis result generation module is used for generating a data analysis result corresponding to the data analysis operation by adopting the sub-query result and the sub-query result.

The embodiment of the application also discloses electronic equipment, which comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are communicated with each other through the communication bus;

the memory is used for storing a computer program;

the processor is configured to implement the method according to the embodiment of the present application when executing the program stored in the memory.

One or more computer-readable media having instructions stored thereon, which when executed by one or more processors, cause the processors to perform the methods described in embodiments of the present application are also disclosed.

The embodiments also disclose a software product comprising a computer program/instructions which, when executed, implement the method as described in the embodiments.

Embodiments of the present application include the following advantages:

in the embodiment of the application, in the process of data query, especially in the process of configuring a structured query template, the index information and the data query template can be acquired, then the initial index column information can be output according to the index information and the data query template, the research personnel can know the query content corresponding to the index information by analyzing the index column information, then the initial index column information can be configured, the initial index column information is adjusted to be the target index column information, the index information is more clearly described in the template by adjusting the index column information, then the target index column information is added into the data query template, the query template is updated, the structured query template is generated, and the data analysis personnel can perform data query through the structured query template, so that under the condition that the structured query template supports multi-dimensional and cross analysis, the index information is accessed into the corresponding data query template, verification and adjustment are performed, the time of data query language configuration is effectively shortened, the debugging cost is reduced, and the debugging efficiency is improved.

Drawings

FIG. 1 is a flow chart of steps of a data processing method provided in an embodiment of the present application;

fig. 2 is a schematic flow chart of index access provided in an embodiment of the present application;

FIG. 3 is a block diagram of a data processing apparatus provided in an embodiment of the present application;

FIG. 4 is a block diagram of an electronic device of the present application;

fig. 5 is a schematic diagram of a computer readable medium of the present application.

Detailed Description

In order that the above-recited objects, features and advantages of the present application will become more readily apparent, a more particular description of the invention briefly described above will be rendered by reference to specific embodiments that are illustrated in the appended drawings.

As data in the internet industry is accumulated and refined operations are entered, more and more products, operations, and technicians need to master skills of data analysts in order to determine user demands from a large amount of data by searching for the data, analyzing the data, and the like, and provide corresponding services according to the user demands. For example, in the field of logistics, the related indexes in a certain logistics application program can be analyzed, including daily active user quantity, the performance rate of a mail order, the cancellation rate of the mail order, daily receiving quantity, daily mail sending quantity and the like of the application program, and the requirements of users can be analyzed by analyzing the related indexes in the logistics process, so that more reasonable, humanized and customized services can be provided according to the requirements of the users. However, the content related to professional data analysis such as data sources, processing logics, dimension assembly of data and the like by non-professional data analysts (such as products, operators and the like) cannot be used by themselves, but can only be used by professional data research personnel through a report or a billboard and the like formed by customizing some report tools, and the data threshold analysis is high and the universality and universality are poor for the non-professional data analysts.

In order to solve the above-mentioned problems, in the related art (1), a non-professional data analyst finishes the data analysis requirements including analysis indexes, specific analysis dimensions, specific analysts, etc. at one time, then the professional data developer processes the data into the specified indexes, and then the specified indexes are presented in the form of a billboard through a related report tool, and a certain degree of pull-down screening is supported in the scope of preprocessing, however, in this way, the common indexes, the common dimensions, the common personnel, etc. cannot be precipitated, so that the data analyst cannot flexibly select the analysis dimensions for data analysis by self-service. In the related technology (2), multidimensional cross analysis is realized through placeholder indexes, specifically, indexes are defined in advance for data research personnel, SQL (Structured Query Language ) fragments are accessed, substitutes such as $ { date }, $ { crowdSQL } and the like are added in the SQL fragments, when the data analysis personnel need to select analysis dimensions and analysis crowds, the data analysis personnel splice the substitutes into specified dimension tables or crowd table fragments to form complete SQL for multidimensional cross analysis, however, in the mode, the index access cost is high, the research personnel needs to spend a large amount of time to adjust the SQL to adapt to access rules, the SQL fragments cannot directly run, the debugging cost is further improved, and the SQL debugging efficiency is low. Therefore, in the related data analysis process, data research personnel are required to access the data analysis indexes, and a large amount of time is spent for adjusting the data query language, so that the cost of data query language debugging is greatly increased, and the debugging efficiency is low.

In this regard, one of the core inventions of the embodiments of the present application is that in a data query process, particularly in a process of configuring a structured query template, index information and a data query template are obtained, where the data query template may be an SQL template, then index information is accessed into the SQL template and initial index column information is output, and by analyzing the index column information, a data developer can understand query content corresponding to the index information, then in order to describe the index information in the template more clearly, the developer may configure the index column information to obtain target index column information, and add the target index column information to the data query template, update the query template, generate the structured query template, so that a data analyzer may perform data query through the structured query template, thereby, under the condition that the structured query template supports performing multidimensional and cross analysis, by accessing the index information into the corresponding data query template, and performing verification and adjustment, the time of data query language configuration is effectively shortened, debug cost is reduced, and debug efficiency is improved.

In order to enable those skilled in the art to better understand the technical solutions of the embodiments of the present application, the following explains some technical terms related to the present application:

The index, which has a certain analysis meaning concept, may include dimension modification, time modification, metric value, and the like, for example, a single amount of a day of a personal posting of an APP, a dimension of "personal posting" of the APP, a time modification of "one day", a metric value of "single amount", and the like.

The dimension is used for modifying a description of the index, for example, taking the order conversion rate of the mail page as the index, and the related dimension can comprise different entry dimensions such as 'entering the order page from the front page', 'entering the order page from the short message connection', and the like, and also can comprise user dimensions such as 'new user', 'old user', and the like.

The crowd can be a specific user group, and in the process of data analysis, different crowds are analyzed, including 'users who have not logged in approximately 180 days', 'male users' and 'female users', and the like.

Specifically, referring to fig. 1, a flowchart illustrating steps of a data processing method provided in an embodiment of the present application may specifically include the following steps:

step 101, obtaining index information and a data query template;

in the embodiment of the application, in order to enable non-professional data analysts (such as products, operators, etc.) to use analysis tools to perform multi-dimensional cross analysis in a self-service manner, data research and development personnel can configure corresponding structured query templates first so that the data analysts perform data analysis. Optionally, a corresponding program may be run in the terminal, and the data developer may configure the data query template in the program to obtain a corresponding structured query template.

In the process of configuring the structured query template, the index information and the data query template can be acquired first, so that the index information is connected into the data query template to generate a corresponding structured query language. The index information may include an index to be analyzed, a name of the index, a description of the index, an application type of the index, a dimension for modifying the index, a time for modifying the index, a metric corresponding to the index, and the like; the data query template may be an SQL template including a preset structure, and the index information may be input to the SQL template to generate a corresponding structured query template.

It should be noted that, in the embodiment of the present application, the data query template is taken as an SQL template for example to perform an explanation, and for SQL, which may be a programming language for a special purpose, is a database query and programming language, and may be used for accessing data and querying, updating and managing a relational database system, which may implement functions such as data definition, data manipulation and data control, where in a process of performing data analysis, a data developer may implement template configuration by accessing index information into a corresponding SQL, so that a data analyzer performs multidimensional cross data analysis through the configured template.

In one example, the data query template may include a main query structure, a sub-query structure, and data table parameters at an outer layer, where a pointer value column (including a pointer name, a field of a pointer, a type of a field, etc.) to be analyzed may be configured in the main query structure; the sub-query structure can be SQL of a specific analysis dimension, and the sub-query result of the sub-query structure returns ds, user_id and the like, and for ds, the sub-query result needs to be returned in the sub-query structure for time display of the chart and the associated data table, and the user_id also needs to be returned in the sub-query structure for the associated data table and the like; the corresponding data table may be associated by a data table parameter. In addition, the sub-query structure may further include a parameter corresponding to a column returned for the outer-layer query statistical index value.

102, outputting initial index column information according to the index information and the data query template;

in the embodiment of the application, for the data query template, the index information can be accessed through the SQL of the main query structure and the sub query interface, compared with the SQL fragment mode through the alternative mode, the data query and analysis can be realized through the cooperation between the main query structure and the sub query structure without the alternative, the condition that a large amount of SQL customizing logics are needed when the index information is accessed is effectively avoided, and the SQL fragment cannot be directly operated so that whether the template has abnormal operation cannot be judged, so that the configuration and debugging efficiency of the structured query template is improved, and meanwhile, the operation stability of the template is ensured.

In a specific implementation, the index information can be input into the data query template to generate an initial query template, and then the initial query template with the index information is checked in response to an operation instruction aiming at the initial query template so as to judge whether the initial query template can normally operate. If the operation can be normally performed, the first field information corresponding to the sub-query structure and the second field information corresponding to the main query structure can be output after the operation is finished, so that the analysis content corresponding to the index information is output through the index column information corresponding to the output template, so that data research personnel can check the analysis content, and adjust and modify the analysis content as required.

Optionally, the sub-query structure may be used to correlate with the query dimension and provide the outer query statistics, and the main query structure may be used to define the index to be analyzed, so that during the data analysis process, the data query of the related dimension may be performed through the sub-query structure first, after the sub-query result is obtained, the data query of the index is performed based on the return of the sub-query result to the outer query structure (i.e. the main query structure), and the final data analysis result is obtained. For the first field information, the first field information can be a dimension field corresponding to the sub-query structure, a field name of the dimension field and a field type of the dimension field; for the second field information, the second field information can be an index field corresponding to the main query structure, a field name of the index field and a field type of the index field, so that whether the index information is successfully accessed into the SQL template or not can be detected through test operation of the initial query template, the accessed index information comprises indexes needing analysis, the indexes are analyzed from what dimension and the like, whether the SQL template can be effectively operated or not can be detected, and the operation stability of the data query template is effectively ensured.

Specifically, for the verification process of the initial query template, after the data developer inputs the operation instruction, the program may respond to the operation instruction for the initial query template to obtain the target syntax tree of the initial query template, where the target syntax tree may be the whole query SQL syntax tree corresponding to the initial query template, then the query SQL syntax tree may be detected to determine whether there is a sub-query syntax tree, if the target syntax tree includes the sub-query syntax tree, field identification is performed on the sub-query syntax tree, first field information corresponding to the sub-query structure is output, field information of the main query structure is obtained, type identification is performed on the field information, and second field information corresponding to the main query structure is output.

In one example, after the developer accesses the index information into the SQL template, a corresponding SQL parser may be created to parse the template to obtain an entire query SQL syntax tree, then obtain a From syntax tree in the syntax tree, and determine whether the From syntax tree is a sub-query syntax tree, thereby determining whether a corresponding sub-query structure exists in the template. If the sub-query structure exists, acquiring corresponding field information, identifying the field information, judging whether the field information contains fields such as ds, user_id and the like, and outputting corresponding first field information under the condition of the field information, namely the first field information can comprise ds, user_id and the like, wherein ds can represent a date and is used for representing the time analysis dimension of the index; user_id may represent a user's ID, which is used to characterize the user analysis dimension of the index, etc. Meanwhile, the field information (namely the field information of the main query structure) of the outermost query layer can be obtained, the field information is identified, the corresponding second field information can be obtained, the second field information can comprise the field types (such as character strings, numerical values, percentages, time and the like) corresponding to the indexes required to be analyzed and queried, and therefore whether the index information is successfully accessed into the SQL template or not can be detected through the test operation of the initial query template, the accessed index information comprises indexes required to be analyzed, the indexes are analyzed from what dimension and the like, whether the SQL template can be effectively operated can be detected, and the operation stability of the data query template is effectively guaranteed.

Step 103, responding to the information configuration operation aiming at the initial index column information, and adjusting the initial index column information into target index column information corresponding to the information configuration operation;

to better define and describe the use of relevant query fields in the query templates, data developers can adjust the field information output at run-time. In a specific implementation, the program may obtain configuration information corresponding to the information configuration operation in response to the information configuration operation for the initial index column information, where the configuration information includes at least a first field name and a first field type for the first field information, and a second field name and a second field type for the second field information, and then adjust the first field information to be first target field information using the first field name and the first field type, and adjust the second field information to be second target field information using the second field name and the second field type.

Optionally, the first field information and the second field information may be physical field information resolved by the query template during test run, so that in order to better define and describe the purposes of the physical fields, a data developer may adjust information such as names and types of the fields, so as to adjust the first field information to be first target field information, adjust the second field information to be second target field information, and optimize the definition and description of the fields.

In one example, when accessing a daily living user number index of an APP, after accessing an index into SQL to complete index SQL writing, a data developer can input an operation instruction, in the operation process, can analyze query column information (including indexes, dimensions, and the like related to query in a main query structure and a sub-query structure) of the index SQL, and then output physical field information of only query columns, wherein the physical field information can include ds, DAU, and the like, and the former can be used for representing a date, namely time analysis dimensions in the index; the latter can be used to characterize the daily living user index, i.e. the metric value in the index. In order to facilitate configuration by data research personnel, the field name, type and the like of the physical field information can be adjusted by performing visual display on the physical field information and then performing operation on a corresponding visual window according to the data research personnel, if the related data type is the query dimension in the index for ds, the data type can be configured as a dimension for representing that ds belongs to dimension modification, and meanwhile, the data type is characterized by a date, the corresponding field data type can be configured as a time/yyyMMdd, and the configured data field name is a date, so that ds is configured as an analysis time dimension for modifying the index; for the DAU, the related data type is a measurement value in the index, and can be configured as a measurement for representing that the DAU belongs to the measurement value, and meanwhile, the related data type is a numerical value, the corresponding field data type can be configured as a numerical value type, and the field name of the configured data field is a daily living user number, so that the index column information is adjusted

And 104, adding the target index column information to the data query template to generate a structured query template.

After the adjustment of the index column information is completed, the adjusted target index column information can be added to the data query template, and the initial index column information is updated. At this time, only the information such as the index and the dimension needed to analyze and query is defined, and the information needs to be associated with the corresponding data table to determine which dimensions need to analyze and query, specifically, the main query information aiming at the main query structure and the sub-query information aiming at the sub-query structure may be acquired first, the main query information at least includes the main query dimension and the main query identifier, the sub-query information at least includes the sub-query dimension and the sub-query information, then the main query syntax tree corresponding to the main query structure and the sub-query syntax tree corresponding to the sub-query structure may be determined, then the main query dimension and the main query structure are associated, the main query identifier is added to the main query syntax tree, the sub-query dimension and the sub-query structure are associated, and the sub-query identifier is added to the sub-query syntax tree to generate the structured query template.

The main query dimension may be used to represent a dimension of the main query structure for query, which may be associated with a corresponding data table, the sub-query dimension may be used to represent a dimension of the sub-query structure for query, which may be associated with a corresponding data table, and the data table corresponding to the main query dimension and the data table corresponding to the sub-query dimension may be the same data table or may be different data tables. In addition, the main query identifier may be a query name of the main query structure, the sub-query identifier may be a query name of the sub-query structure, and the application is not limited thereto.

In the foregoing examples of the embodiments of the present application, the performing of the index SQL configuration by using a certain index and different dimensions corresponding to the index is illustrated as an example, and it may be understood that in the related data analysis, the common dimensions and time periods of the analysis may be all exhaustive, and the common analysis methods may also be all exhaustive.

In one example, after the index column information is adjusted, SQL may be rewritten, and the main query structure and the sub-query structure in the template are associated with the corresponding data table, so that corresponding data may be obtained through the corresponding data table when data analysis is performed. Specifically, a corresponding SQL parser may be created, then a sub-query syntax tree in the query template is obtained, then a sub-query dimension and a sub-query identifier are obtained, and the sub-query dimension and the sub-query identifier are added in the sub-query syntax tree to associate the sub-query syntax tree with the corresponding data query dimension, for example, the sub-query identifier may be "xxxx_index", the sub-query dimension may include related SQL such as a circler, a data table, crowd layering, etc., and then the sub-query identifier may be added in the sub-query syntax tree, and the related SQL may be associated to the sub-query structure, so as to implement rewriting of the sub-query structure. For the outer query grammar tree (i.e. the main query structure), a corresponding main query dimension and a main query identifier can be added to the outer query grammar tree, for example, a target field can be obtained, and the target field can be used for associating the query dimension and the main query identifier of the main query structure, so that after the adjustment of the index value column is completed through the process, the template can be associated with the associated SQL, data table and the like, and meanwhile, the outer query grammar tree, the sub query grammar tree and the like are named, so that the rewriting of the index SQL template is realized, the data query template is completed, the structured query template is generated, and then, the index, the dimension and the like are precipitated in the structured query template, so that data analyst can directly select the index, the dimension and the like for data analysis, and under the condition that the structured query template supports multi-dimensional and cross analysis, the time of the configuration of the data query language is effectively shortened, the debugging cost is reduced, and the debugging efficiency is improved.

In an alternative embodiment, in the data analysis query process, a corresponding data analysis personnel can run a corresponding data analysis tool in the terminal, the data analysis tool can be configured with the structured query template involved in the process, the terminal can respond to the data analysis operation to obtain query information corresponding to the data analysis operation, the query information at least comprises a query index and a query dimension, then the query index and the query dimension are input into the structured query template, if the query dimension is successfully matched with the first target field information, a first dimension table corresponding to the sub-query dimension is determined, and a sub-query result matched with the query dimension is obtained from the first dimension table, if the query index is successfully matched with the second target field information, a main query result matched with the query index is extracted from the query result, and then the sub-query result and the sub-query result are adopted to generate a data analysis result corresponding to the data analysis operation.

In a specific implementation, a data analyzer can input indexes, dimensions and analysis modes which are required to be analyzed, then a terminal can acquire corresponding query indexes, query dimensions and the like according to the input operation of the data analyzer, then the query indexes are matched with indexes (namely second target field information) defined by a main query structure in a template, whether the query indexes are accessed in the template is judged, the query dimensions are matched with dimensions (namely first target field information) in a sub-query structure in a similar way, when the data analyzer exists in the template, corresponding data can be acquired from a corresponding data table to acquire query results corresponding to the dimensions and the indexes, when the main query structure in the template depends on the query results of the sub-query structure, the query structure acquired by the sub-query structure can be returned to the main query structure for query analysis, so that the data analysis results corresponding to the query information input by the data analyzer are acquired, and the data analyzer can select the indexes, the dimensions and the like which need to be analyzed according to own requirements for data analysis through configuration of the structured query template, the data analyzer can be prevented from needing to replace the corresponding dimensions and the data, the data analysis can be effectively matched with the data analysis threshold by the self-service analyzer, and the data analysis threshold can be ensured.

Optionally, the data processing method in the embodiment of the application can be applied to commodity shopping platforms, logistics management systems, short video platforms, music platforms, life service platforms, medical service platforms and the like, and related personnel can analyze related indexes in the platforms/systems so as to mine user requirements, thereby providing more reasonable, humanized and personalized services for users and improving user experience.

In order to enable those skilled in the art to better understand the technical solutions of the embodiments of the present application, the following explanation and description are given by way of an example:

in the data analysis process, the involved users include data research personnel and data analysis personnel, wherein the data analysis personnel can include non-professional data analysis product personnel, operators and the like. In order to enable non-professional data analysis product staff and operators to use the data analysis tools in a self-service manner so as to realize multi-dimensional cross data analysis, the data analysis can be performed by constructing the data query templates and constructing the corresponding structured query templates according to the embodiment of the application, so that the data analysis staff can select indexes, dimensions and the like to be analyzed according to own requirements.

Specifically, the process of configuring the template by the data developer may include:

1. determining an index name, description of the index, type of the index, application type, dimension for modifying the index, time for modifying the index, measurement corresponding to the index and the like;

2. accessing index information related to the process to an index SQL;

3. after the access, performing trial operation on the index SQL, detecting whether the SQL can normally operate, and analyzing a corresponding index value column;

4. Adjusting the field type and the description information of the physical field information corresponding to the index value column;

5. and accessing the correlated SQL to correlate the index SQL with the corresponding data table, and then storing the index SQL to generate the structured query template.

The process of data analysis personnel performing data query analysis may include:

1. inputting query information such as analysis names, analysis descriptions, application types, indexes, user dimensions and the like;

2. carrying out data analysis query through a structured query template;

3. and outputting a corresponding data analysis result, for example, displaying in a report visualization mode.

Optionally, in the above template configuration process, the process may include a process of index access, index SQL verification, index column analysis, index SQL rewrite, and the like, referring to fig. 2, a flow diagram of index access provided in an embodiment of the present application is shown, and a specific process may include:

1. and (3) configuring index definition and index processing information:

1.1, accessing an index into SQL for verification;

1.2, analyzing an access index SQL;

1.3, returning index column information,

2. Setting index column names, dimensions, types, etc.:

2.1, index column names, dimensions and types fall into a library;

2.2 returning the setting result

Through the process, the configuration of the structural configuration template (i.e. index SQL) can be completed, and under the condition that the structural query template supports multidimensional and cross analysis, index information is accessed into the corresponding data query template, and verification and adjustment are performed, so that the time for configuring the data query language is effectively shortened, the debugging cost is reduced, and the debugging efficiency is improved.

In the application scenario in the logistics field, the data processing method may be applied to the logistics management system, and then the data developer may configure the template in the logistics management system, including determining the names, descriptions, types, application types, dimensions for modifying the indexes, time for modifying the indexes, metrics corresponding to the indexes, and the like of indexes such as daily active user quantity, performance rate of the mail order, cancellation rate of the mail order, daily receiving quantity, daily mail quantity, and the like, and then may generate the structured query template applied to the logistics management system with reference to the related processes. After the template configuration is completed, data analysis personnel can perform data analysis on indexes such as daily active user quantity, the performance rate of a mail order, the cancellation rate of the mail order, daily delivery quantity, daily mail delivery quantity and the like on a logistics management system, and obtain corresponding analysis results, and then the user requirements can be determined according to the analysis results, so that higher-quality logistics services can be provided for the user according to the user requirements. Optionally, the method can be applied to commodity shopping platforms, short video platforms, music platforms, life service platforms, medical service platforms and the like, so as to determine the user requirements through data analysis and further provide better quality services according to the user requirements.

It should be noted that, for simplicity of description, the method embodiments are shown as a series of acts, but it should be understood by those skilled in the art that the embodiments are not limited by the order of acts described, as some steps may occur in other orders or concurrently in accordance with the embodiments. Further, those skilled in the art will appreciate that the embodiments described in the specification are all preferred embodiments and that the acts referred to are not necessarily required by the embodiments of the present application.

Referring to fig. 3, a block diagram of a data processing apparatus provided in an embodiment of the present application is shown, which may specifically include the following modules:

the content acquisition module 301 is configured to, the method comprises the steps of acquiring index information and a data query template;

the index column information output module 302 is configured to output initial index column information according to the index information and the data query template;

an index column information adjustment module 303, configured to respond to an information configuration operation for the initial index column information, and adjust the initial index column information to target index column information corresponding to the information configuration operation;

the query template generation module 304 is configured to add the target index column information to the data query template, and generate a structured query template.

In an alternative embodiment, the data query template includes a main query structure and a sub-query structure, and the index column information output module 302 includes:

In an alternative embodiment, the field information output sub-module is specifically configured to:

In an alternative embodiment, the index column information adjustment module 303 includes:

In an alternative embodiment, the query template generation module 304 includes:

In an alternative embodiment, further comprising:

For the device embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference is made to the description of the method embodiments for relevant points.

In addition, the embodiment of the application also provides an electronic device, as shown in fig. 4, which comprises a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402 and the memory 403 complete communication with each other through the communication bus 404,

a memory 403 for storing a computer program;

the processor 401, when executing the program stored in the memory 403, implements the following steps:

acquiring index information and a data query template;

In an alternative embodiment, the data query template includes a main query structure and a sub-query structure, and the outputting initial index column information according to the index information and the data query template includes:

In an optional embodiment, the responding to the operation instruction of the initial query template, verifying the initial query template, and outputting first field information corresponding to the main query structure and second field information corresponding to the sub-structure includes:

In an optional embodiment, the responding to the information configuration operation for the initial index column information adjusts the initial index column information to target index column information corresponding to the information configuration operation, and includes:

In an alternative embodiment, the adding the target index column information to the data query template generates a structured query template, including:

In an alternative embodiment, further comprising:

The communication bus mentioned by the above terminal may be a peripheral component interconnect standard (Peripheral Component Interconnect, abbreviated as PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, abbreviated as EISA) bus, etc. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.

The communication interface is used for communication between the terminal and other devices.

The memory may include random access memory (Random Access Memory, RAM) or non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor.

The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; but also digital signal processors (Digital Signal Processing, DSP for short), application specific integrated circuits (Application Specific Integrated Circuit, ASIC for short), field-programmable gate arrays (Field-Programmable Gate Array, FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

In yet another embodiment provided herein, as shown in fig. 5, there is further provided a computer readable storage medium 501 having instructions stored therein, which when run on a computer, cause the computer to perform the data processing method described in the above embodiment.

In a further embodiment provided herein, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the data processing method described in the above embodiments.

In a further embodiment provided herein, a software product is also provided, comprising a computer program/instruction, wherein the computer program/instruction, when executed, implements the data processing method as described in the above embodiments.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, by wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), etc.

It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.

The foregoing description is only of the preferred embodiments of the present application and is not intended to limit the scope of the present application. Any modifications, equivalent substitutions, improvements, etc. that are within the spirit and principles of the present application are intended to be included within the scope of the present application.

Claims

1. A method of data processing, comprising:

acquiring index information and a data query template;

2. The method of claim 1, wherein the data query template comprises a main query structure and a sub-query structure, and wherein outputting initial index column information based on the index information and the data query template comprises:

3. The method of claim 2, wherein the verifying the initial query template in response to the execution instruction for the initial query template and outputting the first field information corresponding to the main query structure and the second field information corresponding to the sub-structure comprises:

4. A method according to claim 2 or 3, wherein said adjusting the initial index column information to target index column information corresponding to the information configuration operation in response to the information configuration operation for the initial index column information comprises:

5. The method of claim 4, wherein adding the target index column information to the data query template generates a structured query template, comprising:

6. The method as recited in claim 5, further comprising:

7. A data processing apparatus, comprising:

8. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

the memory is used for storing a computer program;

the processor is configured to implement the method according to any one of claims 1-6 when executing a program stored on a memory.

9. One or more computer-readable media having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the method of any of claims 1-6.

10. A software product comprising a computer program/instruction which, when executed, implements the method of any of claims 1-6.