CN116341007A

CN116341007A - Data query method, system, equipment and storage medium

Info

Publication number: CN116341007A
Application number: CN202310323394.3A
Authority: CN
Inventors: 谢新; 罗潇
Original assignee: Alibaba China Co Ltd
Current assignee: Alibaba China Co Ltd
Priority date: 2023-03-29
Filing date: 2023-03-29
Publication date: 2023-06-27

Abstract

The embodiment of the application provides a data query method, a system, equipment and a storage medium. The received query statement can be analyzed to determine a target table source to be queried and a target data unit to be accessed in the target table source; the authority verification can be carried out on the target table source and the target data unit; the query statement can be modified according to the authority verification result, so that the modified query statement accords with the authority configuration information corresponding to the target table source; and then, inputting the modified query statement into a data query engine to perform data query. Therefore, the access control of the table source and the data unit can be finished in advance outside the data query engine, so that when the new table source is added, the configuration file inside the data query engine is not required to be rewritten, the data query engine is not required to be restarted, and the corresponding authority configuration information is added outside the data query engine, thereby effectively improving the data query efficiency.

Description

Data query method, system, equipment and storage medium

Technical Field

The present disclosure relates to the field of big data technologies, and in particular, to a data query method, system, device, and storage medium.

Background

IoT (Internet of Things ) data is a wide variety of and may contain some sensitive data, and thus access control to IoT data is required to ensure the security of IoT data.

At present, the authority of the related data table source is usually written in the configuration file in the data query engine, the configuration file is required to be loaded when the data query engine is started, and based on the configuration file, the access control on the related data table source can be performed according to the effective configuration file when the data query request is received.

However, ioT data has the characteristic of being complex and variable, and therefore, the data table sources are dynamically changing. When a new data table source appears, the configuration file inside the data query engine needs to be rewritten, and the data query engine must be restarted to realize access control to the new data table source. This severely impacts data query efficiency.

Disclosure of Invention

Aspects of the present application provide a data query method, system, device, and storage medium, for improving data query efficiency.

The embodiment of the application provides a data query method, which comprises the following steps:

analyzing a target table source to be queried and a target data unit to be accessed in the target table source from a received first query statement;

Performing authority verification on the target table source and the target data unit based on the authority configuration information corresponding to the target table source;

modifying the first query statement into a second query statement according to the authority verification result, wherein the second query statement accords with the authority configuration information corresponding to the target table source;

and inputting the second query statement into a data query engine so that the data query engine can perform data query according to the second query statement.

The embodiment of the application also provides electronic equipment, which comprises a memory and a processor;

the memory is used for storing one or more computer instructions;

the processor is coupled to the memory for executing the one or more computer instructions to implement the data query method described above.

The embodiment of the application also provides a data query system, which comprises a preprocessing node and a data query engine, wherein the data query engine comprises a scheduling node and a working node, the preprocessing node is used for realizing the data query method, so that the preprocessed query statement is input into the scheduling node, and the scheduling node is used for scheduling the working node according to the received query statement so as to perform data query.

Embodiments also provide a computer-readable storage medium storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform the aforementioned data query method.

In the embodiment of the application, the received query statement can be analyzed to determine the target table source to be queried and the target data unit to be accessed in the target table source; rights configuration information can also be maintained for each table source, and based on the rights configuration information, rights verification can be performed on the target table source and the target data unit; the query statement can be modified according to the authority verification result, so that the modified query statement accords with the authority configuration information corresponding to the target table source; and then, inputting the modified query statement into a data query engine to perform data query. Therefore, the access control of the table source and the data unit can be finished in advance outside the data query engine, so that when the new table source is added, the configuration file inside the data query engine is not required to be rewritten, the data query engine is not required to be restarted, and the corresponding authority configuration information is added outside the data query engine, thereby effectively improving the data query efficiency.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute an undue limitation to the application. In the drawings:

fig. 1 is a flowchart of a data query method according to an exemplary embodiment of the present application;

FIG. 2 is a logic diagram of a data query method according to an exemplary embodiment of the present application;

fig. 3 is a schematic diagram of an application scenario provided in an embodiment of the present application;

FIG. 4 is a schematic diagram of a data query system according to another exemplary embodiment of the present application;

fig. 5 is a schematic structural diagram of an electronic device according to another exemplary embodiment of the present application.

Detailed Description

For the purposes, technical solutions and advantages of the present application, the technical solutions of the present application will be clearly and completely described below with reference to specific embodiments of the present application and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.

Currently, when a new data table source appears, a configuration file inside the data query engine needs to be rewritten, and the data query engine must be restarted to realize access control to the new data table source, which seriously affects the data query efficiency. To this end, in some embodiments of the present application: the received query statement can be analyzed to determine a target table source to be queried and a target data unit to be accessed in the target table source; rights configuration information can also be maintained for each table source, and based on the rights configuration information, rights verification can be performed on the target table source and the target data unit; the query statement can be modified according to the authority verification result, so that the modified query statement accords with the authority configuration information corresponding to the target table source; and then, inputting the modified query statement into a data query engine to perform data query. Therefore, the access control of the table source and the data unit can be finished in advance outside the data query engine, so that when the new table source is added, the configuration file inside the data query engine is not required to be rewritten, the data query engine is not required to be restarted, and the corresponding authority configuration information is added outside the data query engine, thereby effectively improving the data query efficiency.

The following describes in detail the technical solutions provided by the embodiments of the present application with reference to the accompanying drawings.

Fig. 1 is a flowchart of a data query method according to an exemplary embodiment of the present application, where the method may be performed by a data processing device, which may be implemented as software, hardware, or a combination of software and hardware, and the data processing device may be integrated in an electronic device. Referring to fig. 1, the method may include:

step 100, analyzing a target table source of a required query and a target data unit required to be accessed in the target table source from a received first query statement;

step 101, verifying the authority of a target table source and a target data unit based on authority configuration information corresponding to the target table source;

step 102, according to the authority verification result, modifying the first query statement into a second query statement, wherein the second query statement accords with the authority configuration information corresponding to the target table source;

step 103, inputting the second query statement into the data query engine so that the data query engine can perform data query according to the second query statement.

The data query method provided in the embodiment can be applied to various scenes needing to perform data query, for example, an IoT data query scene or a log data query scene, and the embodiment is not limited to the application scene. In different application scenarios, there is a difference in the data content contained in the table source. For example, in an IoT data query scenario, the attribute information of the sensor device, the type of monitoring data supported by the sensor device, or the monitoring data collected by the sensor device, etc., may be contained in the table source.

Wherein a table is a basic unit of organizing data in a data source, in some typical definitions, a table may be interpreted as a data storage container in a data source. In this embodiment, a table as a query source will be described as a table source. It should be appreciated that because there may be differences in the organization of data among different types of data sources, the names of table sources may differ among different types of data sources, e.g., in a relational database, the names corresponding to table sources are data tables (tables); in the time sequence database, the name corresponding to the table source is a measurement (measurement). The table source may contain data units, where a data unit is a set of data contained in the table source, and, similarly, because there may be differences in data organization structures in different types of data sources, the names of the data units may be different in different types of data sources, for example, in a relational database, the names corresponding to the data units are referred to as attribute columns (columns); in the time sequence database, the corresponding name of the data unit is a measurement item (field).

In this embodiment, the data query engine refers to a processing engine that can provide a query interface to receive query sentences, and can access a data source and execute the received query sentences to generate a query result, where the data query engine can be software or a combination of software and hardware. Currently, there are many data query engines with different functions, and no example of product names is given here. In different application scenarios, a proper data query engine can be selected as required.

In this embodiment, considering the complex variability of IoT data, it is proposed that IoT data with different characteristics may be respectively stored in a suitable data source, for example, ioT data with low update frequency and high structuring degree, such as attribute information of a sensor device or a type of monitoring data supported by the sensor device, may be stored in a relational database; ioT data having time sequence, such as monitoring data collected by the sensor device, may be stored into a time sequence database; ioT data with high update frequency, such as the latest state of the monitoring data collected by the sensor device, may also be stored into a memory-implemented key-value-wise non-relational (NoSQL) database, such as redis.

Based on this, it is further proposed in this embodiment that, in an application scenario of performing data query on IoT data, a distributed SQL query engine may be selected as the data query engine used in this embodiment. In IoT data query scenarios, the target table source in this embodiment may be an IoT table source, and the distributed SQL query engine may connect to a variety of data sources, including but not limited to relational databases, temporal databases, or non-relational databases, etc., where the target table source may be stored. The distributed SQL query engine can deploy data query work to each node in the cluster in a distributed manner, which allows the distributed SQL query engine to obtain more powerful query capabilities by adding more nodes. Preferably, in this embodiment, a Trino may be selected as the data query engine used in this embodiment. The Trino is an open-source distributed SQL query engine, adopts a framework with separated computation and storage, is based on full memory computation, and processes data in a streaming pipeline mode, so that the Trino can save memory and simultaneously respond to queries more quickly. In addition, the Trino can access different data sources through various connectors, and access operations are performed on the data sources without data source dump.

Of course, in this embodiment, other types of data query engines may be used, but not limited thereto, for example, an analysis type data query engine Drios or an interactive real-time query engine Impala. It should be noted that, the implementation of the technical solution of this embodiment is not affected by selecting different types of data query engines.

Referring to FIG. 1, in step 100, a target table source of a desired query and a target data unit of desired access in the target table source may be parsed from a received first query statement. The first query statement is a query statement submitted by a user, and specifically, may be an SQL statement or an SQL-like statement. In step 100, the first query statement may be parsed, where various parsers that may occur in the present or future may be employed to parse the first query statement, the parsing principles of which are not described in detail herein. Through the parsing operation, a target table source of the required query and a target data unit required to be accessed in the target table source can be determined from the first query statement.

For example, an exemplary first query statement may be:

Select id name pwd from table01

through parsing operation, it may be determined that the target table source of the query required in the first query statement includes table01, and the target data units required to be accessed in table01 are id, name and pwd.

For another example, an exemplary first query statement may be:

Select*from table01 a

left join

table 02b

on a.id＝b.id

limit 10；

through parsing operation, it can be determined that the target table sources of the query required in the first query statement include table01 and table02. In this exemplary first query statement, a wild card for identifying data units is included, in which case all data units included in table01 may be determined to be target data units for which access is desired; all data units contained in table02 are also target data units to be accessed.

In this embodiment, it is also proposed that authority configuration information may be maintained for the table source. The permission configuration information is used to describe the permission status of the table body and each data unit contained in the table body, and the permission status can include, but is not limited to, permission of access, non-permission of access, permission of access of a specified object, and the like.

In an alternative implementation, a rights configuration interface may be provided in which a user may create rights configuration information for a table source. Based on this, in this alternative implementation, in the case of a new added table source, the rights configuration interface may be presented; and responding to the permission configuration operation aiming at the new added table source in the permission configuration interface, and generating permission configuration information corresponding to the new added table source.

Based on this, in step 101, the target table source and the target data unit may be subjected to rights verification based on the rights configuration information corresponding to the target table source. Through authority verification in step 101, access control to the first query statement can be realized, and query logic which is not allowed in the first query statement is found.

In an exemplary scheme, the authority configuration information may include authority description information of a table source dimension, which is used to describe an authority state corresponding to the table source; the permission configuration information may further include permission description information of a data unit dimension, which is used to describe a permission status corresponding to the single data unit. Based on the above, if it is determined that the target table source passes the authority verification based on the authority description information of the table source dimension contained in the authority configuration information, performing the authority verification on the target data unit based on the authority description information of the data unit dimension contained in the authority configuration information; if the target table source does not pass the authority verification, determining that the target data unit does not pass the authority verification and not executing the operation of performing the authority verification on the target data unit.

In this exemplary scenario, it may first be determined whether the target table source to be queried in the first query statement passes the permission verification based on the permission description information of the table source dimension. The target table source may be determined to not pass the permission verification if the permission status of the target table source is not allowed access, or if the permission status is allowed access to the specified object and the initiator of the first query statement is not the specified object, or else, the target table source may be determined to pass the permission verification. Then, under the condition that the target table source passes the authority verification, the authority verification can be continuously carried out on the target data unit; and under the condition that the target table source does not pass the authority verification, the authority verification is not carried out on the target data unit any more, but the default target data unit does not pass the authority verification, so that the authority verification efficiency is improved.

Of course, other schemes may be adopted in this embodiment to implement authority verification on the target table source and the target data unit based on the authority configuration information corresponding to the target table source, which is not limited to the above-mentioned exemplary scheme. For example, the rights configuration information may include only rights description information of the data unit dimension, and may directly perform rights verification on the target data unit, etc.

After the authority verification operation in step 101, an authority verification result corresponding to the first query statement may be generated. The authority verification result may include characterization information about whether the target table source and the target data unit pass the authority verification, for example, if the authority verification is passed, the corresponding characterization information may be recorded as 1, and if the authority verification is not passed, the corresponding characterization information may be recorded as 0, etc.

Referring to fig. 1, in step 102, a first query statement may be modified into a second query statement according to the authority verification result. The modification herein may include at least deleting query logic in the first query statement that does not conform to the permission configuration information corresponding to the target table source. The query logic may be deleted here by deleting the identity corresponding to the data unit and the table source that fails the authority verification from the first query statement. In practice, there may be at least two cases:

And if the first query statement contains the identification of the target data unit, deleting the identification of the data unit which does not pass the authority verification, so as to generate a second query statement.

If the first query statement contains a wild card for identifying the data unit, converting the wild card into the identification of the data unit which passes the authority verification so as to generate a second query statement. It will be appreciated that the data units that have passed the entitlement verification are some or all of the data units in the target table source. For example, select from table01 a in the above-described exemplary first query statement may be converted to Select id name from table01 a, where the permission status of data unit pwd in table01 is not allowed to be accessed.

Thus, the second query statement in step 102 is substantially a query statement generated after the access control is performed on the first query statement, and the second query statement conforms to the authority configuration information corresponding to the target table source.

On this basis, in step 103, the second query statement may be input into the data query engine for the data query engine to perform a data query in accordance with the second query statement. The data query engine can parse the second query statement through its existing parser and generate an execution plan to perform the data query.

It should be understood that steps 100-102 in this embodiment are a layer of parsing and modifying the first query statement before the data query engine to generate a second query statement; the data query engine receives the second query statement instead of the first query statement, and the data query engine continues to parse and execute the second query statement according to the existing logic. Because the second query statement is already the query statement after the access control is completed, that is, the access restricted field is no longer present in the second query statement, the access restricted problem caused by the authority problem in the process of analyzing and executing the second query statement by the data query engine is avoided, and the second query statement can be smoothly executed in the data query engine.

Preferably, in this embodiment, the authority verification function inside the data query engine may be disabled, so that the data query engine is no longer required to execute its existing authority verification logic, and the calculation amount of the data query engine may be effectively saved. Alternatively, the rights description information stored in the configuration file within the data query engine may be configured to be fully accessible, so that access-restricted problems are not caused by performing rights verification based on the query engine even if a new table source is added or the rights description information associated with an existing table source is updated. Of course, even if the data query engine is not changed, the technical effect of the embodiment is not affected, because in practical application, the update frequency of the authority description information is very low, the modification problem hardly occurs, and because the authority description information related to the newly added table source is not stored in the data query engine, and the authority verification related to the newly added table source is not triggered, the data query engine hardly generates the access limitation problem through the technical concept of completing the access control to the first query statement in advance, and the data query can be smoothly executed.

In summary, in this embodiment, the received query statement may be parsed to determine a target table source to be queried and a target data unit to be accessed in the target table source; rights configuration information can also be maintained for each table source, and based on the rights configuration information, rights verification can be performed on the target table source and the target data unit; the query statement can be modified according to the authority verification result, so that the modified query statement accords with the authority configuration information corresponding to the target table source; and then, inputting the modified query statement into a data query engine to perform data query. Therefore, the access control of the table source and the data unit can be finished in advance outside the data query engine, so that when the new table source is added, the configuration file inside the data query engine is not required to be rewritten, the data query engine is not required to be restarted, and the corresponding authority configuration information is added outside the data query engine, thereby effectively improving the data query efficiency.

Fig. 2 is a logic schematic diagram of a data query method according to an exemplary embodiment of the present application. Referring to fig. 2, in the above or below embodiments, in addition to the authority verification result in step 101 being taken as a basis for modifying the first query statement, the modification may be further performed on the first query statement in the following aspects, so that the second query statement is performed more smoothly.

First aspect:

in this embodiment, the target access location identifier mapped with the identifier of the target table source may be determined based on the mapping relationship between the table source identifier and the access location identifier, where the access location identifier is an identifier for representing an access location defined in a standard query statement supported by the data query engine; and converting the identification of the target table source in the first query statement into the identification of the target access position.

As mentioned previously, the data organization structure in the different types of data query engines may be different, and thus the type and number of access locations, etc., defined in the standard query statements they support may also be different, e.g., in a distributed SQL storage engine, such as the foregoing Trino, ANSI SQL standard query statements are typically supported, where the types of access types defined include directory category, schema, and table. An exemplary ANSI SQL standard query statement may be:

Select*from

Mysql.db01.table01 a

wherein, 3 access positions are separated by point numbers, and the access positions correspond to the catalog category, the schema and the table respectively. The catalog category is used for representing the selected data source, in the above example mysql, the schema is used for representing the selected database in the selected data source, in the above example db01, and the table is used for representing the data table of the required query in the database used in the selected data source, in the above example table01.

In this embodiment, it is proposed that a mapping relationship between the table source identifier and the access location identifier may be stored in advance. In practical application, the embodiment can provide a mapping relation configuration interface, and a user can configure the mapping relation between the table source identifier and the access position identifier in the mapping relation configuration interface. Thus, when the new table source appears, the corresponding mapping relation can be added for the new table source.

In some exemplary schemes, table source identifiers may be used as indexes, and access location identifiers having mapping relationships with the table source identifiers are recorded for each table source identifier. For example, one exemplary mapping under this exemplary scenario may be represented as: table01: { category 1: schema1: table01}. Thus, the target access location identifier mapped with the identifier of the target table source can be found from the mapping relationship stored in the present embodiment. In the above example, if the identifier of the target table source is tabella01, the identifier of the target access location found is: the plan 1 is the table01.

In other exemplary embodiments, the identifier of any access location may be used as an index, and the identifier of the other type of access location and the identifier of the table source that have a mapping relationship with each representation of such access location may be recorded separately. For example, one exemplary mapping under this exemplary scenario may be represented as: category 1: { schema1: table1, schema2: table2}. Thus, the target access location identifier mapped with the identifier of the target table source can be found from the mapping relationship stored in the present embodiment. In the above example, if the identifier of the target table source is tabella01, the identifier of the target access location found is: category 1: scheme 1: table01.

It can be understood that the specific storage form of the mapping relationship is not limited in this embodiment, and it is only required to ensure that the mapping relationship between the table source representation and the access location identifier can be correctly represented. Regardless of the form of storage, the final query results obtained are consistent.

Through the modification of the first aspect to the first query statement, the access location identifier required by the data query engine missing in the first query statement can be complemented, so as to ensure that the modified second query statement can be smoothly executed by the data query engine.

Accepting an exemplary first query statement provided above:

Select*from table01 a

left join

table 02b

on a.id＝b.id

limit 10；

according to the modification of the first aspect and the modification performed based on the authority verification result in the foregoing embodiment, after the modification of the first query statement, a second query statement may be obtained as follows:

Select id name from

mysql.db01.table01 a

left join

pstegresql.db01.table02 b

on a.id＝b.id

limit 10；

it can be understood that, after the modification, the table source identifier in the first query statement is supplemented with the complete { category. Wild card is modified to the identity id and name of the data unit that passes the rights verification.

In summary, based on the modification of the first aspect provided in this embodiment, the user does not need to precisely specify the identifier of each type of access location required by the data query engine in the first query statement, but only needs to correctly specify the identifier of the target table source to be accessed, which can effectively reduce the complexity and learning cost of writing the first query statement, and the user does not need to perceive the specific storage location (in which data source, which database, etc.) of the target table source to be queried.

Second aspect:

in this embodiment, the special characters in the table source identifier included in the first query statement may also be converted into specified characters that can be understood by the data query engine. Wherein the designated characters may include, but are not limited to, single quotation marks or double quotation marks, etc. To further ensure that the second query statement is smoothly executable by the data query engine.

Third aspect:

in this embodiment, the data types supportable by the data unit may also be stored in advance. Based on this, in this embodiment, if there is implicit conversion logic for the target data unit in the first query statement, it may be determined whether the implicit conversion logic for the target data unit in the first query statement meets the data type requirement of the target data unit according to the data type supported by the target data unit, if so, the implicit conversion logic may be retained in the second query statement, otherwise, the implicit conversion logic may be deleted from the first query statement.

It should be appreciated that the above-described further modifications in several aspects are also exemplary only, and that the present embodiment may also support modification of the first query statement from other aspects to improve the suitability between the second query statement and the data query engine, but is not limited thereto.

In the above or below embodiments, in the conventional data query engine, the configuration file also stores connection address information corresponding to each table source, and when the data query engine is started, the configuration file may be loaded, and the relevant connection address information is loaded into the memory corresponding to the data query engine to take effect. Wherein the connection address information is used to direct the data query engine to access the relevant table sources through the connectors within it. In practical application, the memory corresponding to the data query engine can maintain the connection address information through a file in a special format or a designated process. This results in that when a new table source appears, the configuration file of the data query engine needs to be modified, that is, the connection address information corresponding to the new table source is added in the configuration file, and the data query engine is restarted, so that the connection address information of the new table source can be validated. This will seriously affect the efficiency of the data query.

Referring to fig. 2, in order to solve the dilemma, the present embodiment proposes that a designated interface may be added to the data query engine, where the designated interface is configured to receive the connection address information corresponding to the table source, and load the received connection address information into the connection address configuration information that is validated in the memory corresponding to the data query engine. In practical application, an Application Programming Interface (API) can be added to the data query engine, and related functions can be written for the API so that the API has the specified interface function.

Based on this, in this embodiment, the specified interface provided by the data query engine may be invoked in the case where a new table source appears; and sending the connection address information corresponding to the new added table source to the data query engine through the designated interface so as to trigger the designated interface to load the connection address information into the effective connection address configuration information in the memory corresponding to the data query engine. That is, in this embodiment, under the condition that a new table source appears, the configuration file inside the data query engine is not required to be modified, but the connection address information corresponding to the new table source can be directly loaded into the memory to take effect by calling the designated interface.

In this embodiment, the specific implementation manner of the specified interface is not limited, and only the specified interface needs to be ensured to realize the interface functions required in the embodiment. Taking a data query engine Trino as an example, the connection address configuration information is abstracted into a ServiceAnnominator category inside the Trino. For this purpose, the Trino is provided with an internal set of operation interfaces, the CatalogManager and ConnectorServicesProvider, which specify maintenance operations (add-delete-modify-check, load, start-stop, etc.) for the connection address configuration information. A set of HTTP-based open-to-the-outside implementations may be developed for these interfaces in this embodiment. That is, in an alternative implementation, an access interface based on the HTTP manner may be created for an existing interface in the data query engine for maintaining the connection address configuration information in the memory, as the foregoing specified interface, so that by accessing the specified interface, an existing interface in the data query engine for maintaining the connection address configuration information in the memory may be invoked, thereby implementing the foregoing concept of directly loading the connection address information into the memory.

The process of loading connection address information based on the above-described designated interface may include, but is not limited to:

1. authentication, checking whether the HTTP service requested is a legal user or not;

2. analyzing the request message body, and converting the request message body into a data expression object of the link information to be updated;

3. removing (disabling) the current link information to be updated;

4. comparing and updating;

5. mounting (enabling) the updated link information;

6. a new list of available links is published.

Therefore, in this embodiment, the data query engine is not required to restart any more, so that the data query engine can be guaranteed to acquire the connection address information corresponding to the new table source, and no obstacle is caused to access the new table source.

Fig. 3 is a schematic diagram of an application scenario provided in an embodiment of the present application. Referring to fig. 3, the scheme of the present embodiment will be exemplarily described below from the viewpoint of an exemplary application scenario. The application scene is as follows: the user expects to inquire the highest and lowest temperature and humidity conditions of each granary (each granary comprises a group of temperature and humidity sensors with different numbers) in the western lake area in the past week through a data inquiry engine Trino, and the average temperature and humidity conditions determine whether the problem of improper grain storage of part of granaries exists.

In the application scenario, related IoT data may be stored in a plurality of data sources, but the user does not need to perceive the data, and the user only needs to specify the identifiers "equipment table", "on-equipment report 01" and "on-equipment report 02" of the target table source to be queried in the first query statement written by the user, wherein the attribute information of the temperature and humidity sensors deployed in each granary in the western lake area can be recorded in the "equipment table"; and the temperature and humidity data acquired by the corresponding temperature and humidity sensors can be recorded in the report form on each device. In addition, in this application scenario, the user uses wild cards in the first query statement to identify the attribute columns that need to be used in "device table", "on-device report 01", and "on-device report 02".

The contents of a part of the first query statement are as follows:

select from device table a

Left join

Report on device 01b

On a.id＝b.id

Left join

Report on device 02c

On a.id＝c.id

On the basis, authority configuration information pre-stored for the device table, the device report 01 and the device report 02 can be queried, and the attribute columns pwd in the device report 01 and the device report 02 are determined not to be allowed to be accessed.

And continuously inquiring the mapping relation between the prestored table source identifier and the access position identifier, determining that the device table is positioned in a db03 database in the data source mysql, determining that the device report 01 and the device report 02 are both positioned in the data source influx, storing the device report 01 in the db01 database in the data source influx, and storing the device report 02 in the db02 database in the data source influx.

On this basis, the first query term can be modified into the second query term as follows:

Select id name from

mysql.db03 Equipment Table a

Left join

Influx.db01. On-device report 01b

On a.id＝b.id

Left join

Influx.db02 on-device report 02c

On a.id＝c.id

Then, the second query sentence can be input into a data query engine Trino, and in the Trino, the process of performing data query according to the second query sentence is approximately as follows:

according to the technical scheme provided by the embodiment, the second query statement is sent to a scheduling node (Coordinator) in the Trino, after the scheduling node receives the second query statement, the scheduling node can conduct grammar and semantic analysis to generate a logic plan tree, then the logic plan tree is generated through planner processing to generate a physical plan tree, next, a stage (stations) with dependence is generated, each stage can contain one or more tasks (tasks), the scheduling node can distribute the tasks to a working node (worker) to execute, the working node can be connected with a corresponding data source through a connector (connector) and read a required data table from the data source, an intermediate result can be sent to other working nodes, and finally, a calculation result is obtained from the working node through the scheduling node and returned to an initiator of the first query statement.

Wherein, the above-mentioned "device table" is stored in the data source mysql, so that the relevant working node can be connected to the data source mysql through the corresponding connector, and read the "device table" from the data source mysql; the relevant working node can also be connected to the data source influx through a corresponding connector, and read from the data source influx to the device report 01 and/or the device report 02.

In addition, if a temperature and humidity sensor is newly added in the western lake region, a corresponding on-equipment report form needs to be added in the data source influx. In this case, the designated interface in the Trino of the data query engine may be called, and the connection address information of the report on the newly added device may be submitted to the designated interface, so as to trigger the designated interface to load the connection address information into the memory corresponding to the Trino to take effect. Thus, when the next query is performed, the user can add the new device report to the first query statement as one of the query objects, and the data query engine can smoothly read the new device report according to the connection address information which is already in effect in the memory. Of course, in this embodiment, the corresponding rights configuration information and the aforementioned mapping relationship information may also be stored for the report on the newly added device.

According to the application scenario, the data query scheme provided by the embodiment can perform access control, grammar semantic conversion and other processing on query sentences submitted by users before the data query engine so as to generate query sentences which can be understood by the data query engine and have completed access control, and therefore, after a new query sentence generated after modification operation is input into the data query engine, the data query engine can smoothly execute data query, and the problems of limited access, incapability of understanding sentences and the like do not occur. Moreover, by defining the designated interface for the data query engine, the connection address information corresponding to the newly added table source can be directly validated to the memory corresponding to the data query engine without restarting the data query engine under the condition that the newly added table source appears. In summary, the above aspects can effectively improve the efficiency of data query.

It should be noted that, in some of the above embodiments and the flows described in the drawings, a plurality of operations appearing in a specific order are included, but it should be clearly understood that the operations may be performed out of the order in which they appear herein or performed in parallel, the sequence numbers of the operations, such as 101, 102, etc., are merely used to distinguish between the various operations, and the sequence numbers themselves do not represent any execution order. In addition, the flows may include more or fewer operations, and the operations may be performed sequentially or in parallel. It should be noted that, the descriptions of "first" and "second" herein are used to distinguish different query sentences, and do not represent the sequence, and are not limited to the "first" and "second" being different types.

Fig. 4 is a schematic structural diagram of a data query system according to another exemplary embodiment of the present application. Referring to fig. 4, the data query system may include a preprocessing node 40 and a data query engine, the data query engine including a scheduling node 41 and a working node 42, the preprocessing node 40 being configured to:

analyzing a target table source to be queried and a target data unit to be accessed in the target table source from the received first query statement;

the second query statement is input to the scheduling node 41 in the data query engine such that the scheduling node 41 schedules the working node 42 according to the second query statement for data query.

Wherein, the scheduling node 41 in the data query engine can be used for analyzing at least the second query statement, generating an execution plan, dispatching scheduling resources and issuing tasks to the working node 42; the working node 42 in the data query engine is at least operable to perform tasks, process the calculation data, and return the calculation results to the scheduling node 41.

In an alternative embodiment, the preprocessing node 40, when parsing the target table source of the required query and the target data unit required to be accessed in the target table source from the received first query statement, is specifically configured to:

if the first query statement contains a wild card for identifying the data unit, determining all the data units contained in the target table source as target data units;

modifying the first query statement into a second query statement according to the permission verification result, including:

converting the wild card in the first query statement into an identification of the data unit which passes the authority verification so as to generate a second query statement;

wherein the data units which pass the authority verification are part or all of the data units in the target table source.

In an alternative embodiment, the preprocessing node 40 is specifically configured to, when performing authority verification on the target table source and the target data unit based on the authority configuration information corresponding to the target table source:

if the target table source passes the authority verification based on the authority description information of the table source dimension contained in the authority configuration information, performing the authority verification on the target data unit based on the authority description information of the data unit dimension contained in the authority configuration information;

If the target table source does not pass the authority verification, determining that the target data unit does not pass the authority verification and not executing the operation of performing the authority verification on the target data unit.

In an alternative embodiment, the preprocessing node 40 is specifically configured to, when converting the wild card in the first query statement into the identifier of the data unit that has passed the authority verification, generate the second query statement:

determining a target access location identifier mapped with the identifier of the target table source based on the mapping relation between the table source identifier and the access location identifier, wherein the access location identifier is defined in a standard query statement supported by a data query engine and is used for representing the access location;

converting the identification of the target table source in the first query statement into a target access position identification;

converting the wild card in the first query statement into an identification of the data units that have passed the entitlement verification to generate a second query statement.

In an alternative embodiment, the data query engine adopts a distributed SQL query engine, and the access location identifier comprises a catalog category identifier, a schema identifier and a table identifier;

the identification of the data unit uses the field name of the attribute column.

In an alternative embodiment, the preprocessing node 40 may also be configured to:

The rights verification function within the data query engine is disabled, or,

the rights description information stored in the configuration file inside the data query engine is all configured to allow access.

displaying a permission configuration interface under the condition that a new table source is added;

and responding to the permission configuration operation aiming at the new added table source in the permission configuration interface, and generating permission configuration information corresponding to the new added table source.

calling a designated interface provided by a data query engine under the condition that a new table source is added;

and sending the connection address information corresponding to the new added table source to the data query engine through the designated interface so as to trigger the designated interface to load the connection address information into the effective connection address configuration information in the memory corresponding to the data query engine.

In an alternative embodiment, the target table source is an IoT table source, and the various data sources connected to the data query engine include a time sequence database, where the target table source is stored.

It should be noted that, for the technical details about the preprocessing node 40 in each embodiment of the data query system, reference may be made to the related description in the foregoing method embodiment, which is not repeated herein for the sake of brevity, but this should not cause a loss of protection scope of the present application.

Fig. 5 is a schematic structural diagram of an electronic device according to another exemplary embodiment of the present application. As shown in fig. 5, the computing device includes: a memory 50 and a processor 51.

A processor 51 coupled to the memory 50 for executing the computer program in the memory 50 for implementing the data query method in the previous embodiment.

Further, as shown in fig. 5, the electronic device further includes: communication component 52, power supply component 53, and the like. Only some of the components are schematically shown in fig. 5, which does not mean that the electronic device only comprises the components shown in fig. 5.

It should be noted that, for the technical details of the embodiments of the electronic device, reference may be made to the related descriptions of the preprocessing node in the foregoing system embodiments or the related descriptions of the method embodiments, which are not repeated herein for the sake of brevity, but should not cause a loss of protection scope of the present application.

Accordingly, the present application also provides a computer readable storage medium storing a computer program, where the computer program is executed to implement the steps executed in the above method embodiments.

The memory of FIG. 5 described above is used to store a computer program and may be configured to store various other data to support operations on a computing platform. Examples of such data include instructions for any application or method operating on a computing platform, contact data, phonebook data, messages, pictures, videos, and the like. The memory may be implemented by any type of volatile or nonvolatile memory device or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.

The communication assembly of fig. 5 is configured to facilitate wired or wireless communication between the device in which the communication assembly is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as a mobile communication network of WiFi,2G, 3G, 4G/LTE, 5G, etc., or a combination thereof. In one exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component further comprises a Near Field Communication (NFC) module to facilitate short range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.

The power supply assembly shown in fig. 5 provides power for various components of the device in which the power supply assembly is located. The power components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the devices in which the power components are located.

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.

It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or fully authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region, and provide corresponding operation entries for the user to select authorization or rejection.

The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and changes may be made to the present application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. that fall within the spirit and principles of the present application are intended to be included within the scope of the present application.

Claims

1. A data query method, comprising:

2. The method of claim 1, the parsing, from the received first query statement, a target table source of a required query and a target data unit required to be accessed in the target table source, comprising:

If the first query statement contains a wild card for identifying the data unit, determining all the data units contained in the target table source as the target data unit;

the modifying the first query sentence into the second query sentence according to the authority verification result includes:

converting the wild card in the first query statement into an identification of a data unit which has passed authority verification to generate the second query statement;

3. The method according to any one of claims 1 or 2, wherein the performing authority verification on the target table source and the target data unit based on the authority configuration information corresponding to the target table source includes:

4. The method of claim 2, the converting the wild card in the first query statement to an identification of data units that have passed authority verification to generate the second query statement, comprising:

determining a target access position identifier mapped with the identifier of the target table source based on a mapping relation between the table source identifier and the access position identifier, wherein the access position identifier is defined in a standard query statement supported by the data query engine and is used for representing the access position;

converting the identification of the target table source in the first query statement into the identification of the target access position;

converting the wild card in the first query statement into an identification of data units that have passed authority verification to generate the second query statement.

5. The method of claim 4, wherein the data query engine employs a distributed SQL query engine, and the access location identifier comprises a catalog category identifier, a schema identifier and a table identifier;

the identification of the data unit adopts the field name of the attribute column.

6. The method of claim 1, further comprising:

disabling the rights verification function within the data query engine, or,

And configuring all the authority description information stored in the configuration file inside the data query engine to be allowed to be accessed.

7. The method of claim 1, further comprising:

and responding to the permission configuration operation which occurs in the permission configuration interface and aims at the new added table source, and generating permission configuration information corresponding to the new added table source.

8. The method of claim 1, further comprising:

calling a designated interface provided by the data query engine under the condition that a new table source is added;

and sending the connection address information corresponding to the new table source to the data query engine through the designated interface so as to trigger the designated interface to load the connection address information into the connection address configuration information which is validated in the memory corresponding to the data query engine.

9. The method of claim 1, the target table source being an IoT table source, the plurality of data sources to which the data query engine is connected including a time-series database, the target table source being stored in the time-series database.

10. An electronic device includes a memory and a processor;

The memory is used for storing one or more computer instructions;

the processor being coupled to the memory for executing the one or more computer instructions to implement the data query method of any of claims 1-9.

11. A data query system comprising a preprocessing node and a data query engine, the data query engine comprising a scheduling node and a working node, the preprocessing node being configured to implement the data query method of any one of claims 1-9, to input a preprocessed query statement into the scheduling node, the scheduling node being configured to schedule the working node for data query in accordance with the received query statement.

12. A computer-readable storage medium storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform the data query method of any of claims 1-9.