CN117216095A

CN117216095A - Structured query statement detection method, device, equipment and medium

Info

Publication number: CN117216095A
Application number: CN202311187775.XA
Authority: CN
Inventors: 鲁健翔; 李尼科; 潘斌; 刘涛; 朱慧宁; 李慧婷
Original assignee: Hunan Changyin May 8th Consumer Finance Co ltd
Current assignee: Hunan Changyin May 8th Consumer Finance Co ltd
Priority date: 2023-09-14
Filing date: 2023-09-14
Publication date: 2023-12-12

Abstract

The application discloses a method, a device, equipment and a medium for detecting a structured query statement, which relate to the technical field of computers and comprise the following steps: detecting each field of the structured query statement to be checked; the detection process comprises the following steps: if the sentence type of the structured query sentence to be checked is a data definition language type and the history field identification corresponding to the current field to be checked does not exist in the preset data standard reference library, acquiring the comprehensive similarity between the current field to be checked and the preset data standard reference library; establishing a target field identification set of a current field to be checked, and determining whether the current field to be checked meets a first preset standard condition; if the statement type of the structured query statement to be checked is the data operation language type and the target field identification corresponding to the current field to be checked exists in the preset data standard reference library, determining whether the current field to be checked meets a second preset standard condition or not based on the current field to be checked and the target field identification. The structured query statement is detected more accurately.

Description

Structured query statement detection method, device, equipment and medium

Technical Field

The present invention relates to the field of computer technologies, and in particular, to a method, an apparatus, a device, and a medium for detecting a structured query statement.

Background

Along with the increasing informatization degree of each industry, along with the increasing data generated by each information system in an enterprise, massive data can be gathered and stored by a big data technology, but how to use the massive data becomes a digital transformation problem. The digital transformation is to clean the business data of each system in the enterprise, establish data standardization, and ensure the consistency of the data, including the data field name, field type, field meaning and dictionary value consistency. The structured query language (Structured Query Language, the SQL language) is a database query and programming language for accessing data and querying, updating and managing relational database systems. In the related art, grammar specifications of a plurality of SQL languages are formulated into unified specification grammar texts, and when the development specifications of the SQL sentences are detected, the SQL sentences written by a data development engineer are checked by using the unified specification grammar texts so as to determine whether the written SQL sentences meet the specifications.

However, in a scenario where the SQL language encoding is used in an actual service, there may be SQL statements of the same meaning that do not function the same under different services. Therefore, when standard detection is performed on the SQL statement based by adopting standard language text with single content, the accuracy of the detection result may be low.

In summary, how to improve the accuracy of the detection result of the structured query statement is a problem to be solved in the art.

Disclosure of Invention

In view of the above, the present application aims to provide a method, an apparatus, a device and a medium for detecting a structured query statement, which can improve the accuracy of the detection result of the structured query statement. The specific scheme is as follows:

in a first aspect, the application discloses a method for detecting a structured query statement, which is applied to a preset verification tool and comprises the following steps:

screening a field from the structured query statement to be checked as a current field to be checked;

when the sentence type of the to-be-checked structured query sentence is a data definition language type and a history field identifier corresponding to the current to-be-checked field does not exist in a preset data standard reference library, acquiring each field name semantic similarity between the current to-be-checked field and each history field in the preset data standard reference library, and determining comprehensive similarity based on each field name semantic similarity;

Establishing a target field identification set of the current field to be checked according to the comprehensive similarity, determining whether the current field to be checked meets a first preset standard condition or not based on a first data type detection result and a dictionary value matching result between the current field to be checked and the target field identification set, and re-jumping to the step of screening one field from the structured query statement to be checked as the current field to be checked;

when the statement type of the to-be-checked structured query statement is a data operation language type and a target field identifier corresponding to the current to-be-checked field exists in a preset data standard reference library, determining whether the current to-be-checked field meets a second preset standard condition or not based on a second data type detection result and a dictionary value range detection result between the current to-be-checked field and the target field identifier, and re-jumping to the step of screening one field from the to-be-checked structured query statement as the current to-be-checked field.

Optionally, before screening a field from the to-be-verified structured query statement as the current to-be-verified field, the method further includes:

Analyzing the log to be checked to obtain an initial structured query statement to be checked;

performing format conversion on the initial structured query statement to be verified to obtain the structured query statement to be verified in a target format; wherein the target format is JSON format.

Optionally, the method for detecting the structured query statement further includes:

when the statement type of the to-be-checked structured query statement is a data definition language type and a history field identifier corresponding to the current to-be-checked field exists in a preset data standard reference library, judging that the current to-be-checked field meets the first preset standard condition, and re-jumping to the step of screening one field from the to-be-checked structured query statement as the current to-be-checked field;

when the statement type of the to-be-checked structured query statement is a data operation language type and the target field identification corresponding to the current to-be-checked field does not exist in the preset data standard reference library, judging that the current to-be-checked field does not accord with the second preset standard condition, and re-jumping to the step of screening one field from the to-be-checked structured query statement as the current to-be-checked field.

Optionally, the obtaining the semantic similarity of each field name between the current field to be checked and each history field in the preset data standard reference library includes:

performing word segmentation on the field name of the current field to be checked by using a natural language processing algorithm to obtain word groups after word segmentation;

acquiring first field name semantic similarity between the word group after word segmentation and each field name in the preset data standard reference library;

performing word segmentation on remark information of the current field to be checked by using a target word segmentation model to obtain a dictionary value, and obtaining a first English synonym set corresponding to the dictionary value;

acquiring second field name semantic similarity between the Chinese field name of the current field to be checked and each Chinese field name in the preset data standard reference library by using the natural language processing algorithm;

acquiring the semantic similarity of a third field name between the English field name of the current field to be checked and a second English synonym set in the preset data standard reference library;

acquiring a fourth field name semantic similarity between the first English synonym set and each English field name in the preset data standard reference library;

Correspondingly, before the word segmentation processing is performed on the remark information of the current field to be checked by using the target word segmentation model, the method further comprises:

the remark information of each history field in a preset data standard reference library is collected, and marking processing and data enhancement processing are carried out on the remark information of each history field so as to obtain remark training samples;

and training the initial word segmentation model by using the remark training sample to obtain a target word segmentation model.

Optionally, the determining whether the current field to be checked meets a first preset standard condition based on a first data type detection result and a dictionary value matching result between the current field to be checked and the target field identification set includes:

comparing the data type of the current field to be checked with the data type of the target field identification set to obtain a first data type detection result;

comparing the dictionary value list of the current field to be checked with the dictionary value list of the target field identification set to obtain a dictionary value matching result;

determining whether the current field to be checked meets a first preset standard condition or not based on the first data type detection result and the dictionary value matching result;

Correspondingly, the determining whether the current field to be checked meets a second preset standard condition based on a second data type detection result and a dictionary value range detection result between the current field to be checked and the target field identifier comprises the following steps:

comparing the data type of the current field to be checked with the data type of the target field identification to obtain a second data type detection result;

comparing the field value corresponding to the current field to be checked with the dictionary value list range of the target field identifier to obtain a dictionary value range detection result;

and determining whether the current field to be checked meets a second preset standard condition or not based on the second data type detection result and the dictionary value range detection result.

Optionally, the establishing the target field identifier set of the current field to be checked according to the comprehensive similarity includes:

and determining each target field identifier with the comprehensive similarity larger than a preset threshold value from the preset data standard reference library, and establishing a target field identifier set of the current field to be checked according to each target field identifier.

Optionally, when the sentence type of the to-be-verified structured query sentence is a data definition language type and the history field identifier corresponding to the current to-be-verified field does not exist in the preset data standard reference library, obtaining semantic similarity of each field name between the current to-be-verified field and each history field in the preset data standard reference library includes:

When the statement type of the to-be-checked structured query statement is a data definition language type and a current service benchmark mapping table in a preset data standard benchmark library does not have a history field identifier corresponding to the current to-be-checked field, acquiring semantic similarity of each field name between the current to-be-checked field and each history field in the preset data standard benchmark library;

correspondingly, after the target field identification set of the current field to be checked is established according to the comprehensive similarity, the method further comprises:

constructing a mapping relation between the current field to be checked and each target field identifier;

and storing the mapping relation into the current service reference mapping table, and updating the current service reference mapping table to obtain a new current service reference mapping table.

In a second aspect, the present application discloses a device for detecting a structured query statement, which is applied to a preset verification tool, and includes:

the field screening module is used for screening a field from the structured query statement to be checked to serve as a current field to be checked;

the similarity determining module is used for acquiring semantic similarity of each field name between the current field to be checked and each history field in the preset data standard reference library when the statement type of the structured query statement to be checked is a data definition language type and the history field identifier corresponding to the current field to be checked does not exist in the preset data standard reference library, and determining comprehensive similarity based on the semantic similarity of each field name;

The first jump module is used for establishing a target field identification set of the current field to be checked according to the comprehensive similarity, determining whether the current field to be checked meets a first preset standard condition or not based on a first data type detection result and a dictionary value matching result between the current field to be checked and the target field identification set, and re-jumping to the step of screening one field from the structural query statement to be checked as the current field to be checked;

and the second jump module is used for determining whether the current field to be checked meets a second preset standard condition or not based on a second data type detection result and a dictionary value range detection result between the current field to be checked and the target field identification when the statement type of the structured query statement to be checked is a data operation language type and the target field identification corresponding to the current field to be checked exists in the preset data standard reference library, and re-jumping to the step of screening one field from the structured query statement to be checked as the current field to be checked.

In a third aspect, the present application discloses an electronic device, comprising:

A memory for storing a computer program;

and a processor for executing the computer program to implement the steps of the structured query statement detection method disclosed above.

In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein the computer program when executed by a processor implements the steps of the structured query statement detection method disclosed previously.

The application has the beneficial effects that: the method is applied to a preset checking tool, and a field is screened from the structural query statement to be checked to serve as a current field to be checked; when the sentence type of the to-be-checked structured query sentence is a data definition language type and a history field identifier corresponding to the current to-be-checked field does not exist in a preset data standard reference library, acquiring each field name semantic similarity between the current to-be-checked field and each history field in the preset data standard reference library, and determining comprehensive similarity based on each field name semantic similarity; establishing a target field identification set of the current field to be checked according to the comprehensive similarity, determining whether the current field to be checked meets a first preset standard condition or not based on a first data type detection result and a dictionary value matching result between the current field to be checked and the target field identification set, and re-jumping to the step of screening one field from the structured query statement to be checked as the current field to be checked; when the statement type of the to-be-checked structured query statement is a data operation language type and a target field identifier corresponding to the current to-be-checked field exists in a preset data standard reference library, determining whether the current to-be-checked field meets a second preset standard condition or not based on a second data type detection result and a dictionary value range detection result between the current to-be-checked field and the target field identifier, and re-jumping to the step of screening one field from the to-be-checked structured query statement as the current to-be-checked field. Therefore, when the statement type of the structured query statement to be checked is the data definition language type and the history field identification corresponding to the current field to be checked does not exist in the preset data standard reference library, the comprehensive similarity between the current field to be checked and each history field in the preset data standard reference library can be determined, and then the target field identification set is established according to the comprehensive similarity, so that the current field standard to be checked can be accurately judged later, and the structured query statement of the data definition language type and the data operation language type can be detected, and the detection type is more comprehensive.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present application, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow chart of a method for detecting structured query language according to the present disclosure;

FIG. 2 is a flowchart of a specific method for detecting a structured query term according to the present disclosure;

FIG. 3 is a flowchart of another exemplary structured query statement detection method of the present disclosure;

FIG. 4 is a schematic diagram of a structural query statement detection device according to the present disclosure;

fig. 5 is a block diagram of an electronic device according to the present disclosure.

Detailed Description

The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

Along with the increasing informatization degree of each industry, along with the increasing data generated by each information system in an enterprise, massive data can be gathered and stored by a big data technology, but how to use the massive data becomes a digital transformation problem. The digital transformation is to clean the business data of each system in the enterprise, establish data standardization, and ensure the consistency of the data, including the data field name, field type, field meaning and dictionary value consistency. The structured query language is a database query and programming language used for accessing data and querying, updating and managing a relational database system. In the related art, grammar specifications of a plurality of SQL languages are formulated into unified specification grammar texts, and when the development specifications of the SQL sentences are detected, the SQL sentences written by a data development engineer are checked by using the unified specification grammar texts so as to determine whether the written SQL sentences meet the specifications.

Therefore, the application correspondingly provides a structural query statement detection scheme which can improve the accuracy of the structural query statement detection result.

Referring to fig. 1, an embodiment of the application discloses a method for detecting a structured query statement, which is applied to a preset verification tool and comprises the following steps:

step S11: and screening a field from the structured query statement to be checked as a current field to be checked.

It will be appreciated that where the various business systems within an enterprise are developed by different project groups, the definition of the same meaning field may not be consistent, even though the same project group, the definition of the same meaning field may not be consistent by different project members, such as shown in the following table:

list one

From the above, it is clear that the definition of the "certificate type" table field may not be consistent in the database tables of different systems, so that the structured query statement with consistent information such as field names cannot be directly combined, and standard verification is required.

The preset checking tool in this embodiment includes an automatic checking Pre-module (Biz-Detector) and an automatic checking Engine (Detect-Engine), where the automatic checking Pre-module includes a slave node monitoring unit (DB-Monitor), a Pre-processing unit (Pre-Processor), and a Post-processing unit (Post-Processor). Before detection, a corresponding pre-preparation is required:

1) The Binary Log related function of MySQL is started, and the function is mainly used for recording operations on MySQL database DDL (Data Definition Language, i.e. data definition language) and DML (Data Manipulation Language, i.e. data operation language), i.e. modifying all service verification databases MySQL configuration files my.cnf, as follows:

log-bin＝mysql-bin；

binlog_format＝row；

server-id＝1；

according to the embodiment, a master-slave replication mechanism of MySQL is utilized, script operations of all databases in a perception development test environment can be monitored in real time by simulating a slave node monitoring device, whether data changes accord with data standards or not is monitored in real time, and centralized processing is not needed until a certain specific stage is needed.

2) Presetting a data standard reference library in an automatic check engine, presetting data standard information according to national, industry and enterprise standards, wherein the data standard information comprises: field name (english), field type (int, varchar), field name abbreviation, field length, field chinese name, field dictionary value, english synonym set. The english synonym set is a set that translates a "field chinese name" into english by calling a chinese-english translation API interface (Application Programming Interface, i.e., an application programming interface), for example, as shown in table two:

Watch II

3) And collecting remark information (comments) of a large number of fields of the database, marking the remark information according to a field name and dictionary value format, using a data enhancement technology to enrich samples of the remark information, training an initial word segmentation model of the remark information by using a natural language processing algorithm (Natural Language Processing, namely NLP) deep learning framework, finally obtaining a target word segmentation model of the remark information, and obtaining word analysis of the field name, dictionary value 1 and dictionary value 2 format after word segmentation by the target word segmentation model. For example, for: the certificate type is 1, an identity card; 2, passport; and 3, acquiring the format of { certificate type 1-identity card 2-passport 3-military evidence }, after the analysis model is called.

In this embodiment, before screening a field from the to-be-verified structured query statement as the current to-be-verified field, the method further includes: analyzing the log to be checked to obtain an initial structured query statement to be checked; performing format conversion on the initial structured query statement to be verified to obtain the structured query statement to be verified in a target format; wherein the target format is JSON (JavaScript Object Notation) format. The specific process is as follows:

1) The node monitoring unit simulates MySQL slave nodes and is responsible for sending dump protocol to MySQL master nodes. After receiving the registration protocol request, the MySQL service verification database executes DDL scripts and DML scripts in the service verification database, or the system program automatically executes DDL sentences and DML sentences to generate structured query sentences to be verified, and then the MySQL service verification database starts pushing corresponding logs to be verified to the slave node monitoring unit, and the node monitoring unit analyzes the logs to be verified into initial structured query sentences to be verified;

2) The preprocessing unit is responsible for converting the initial structured query statement to be verified into JSON (JavaScript Object Notation) format data.

Such as DDL build statement:

create table T_Customer(

"ID" int (11) NOT NULL AUTO-INCREMENT COMMENT 'primary key ID',

the type of 'authentication_type' int (8) NOT NULL COMMENT 'document, 1 identity card, 2 passport, 3 officer's certificate,

the 'authentication_no' varchar (32) NOT NULL COMMENT 'certificate number',

....

)；

preprocessing and analyzing into JSON format:

such as DML statement:

after preprocessing execution is completed, traversing a param parameter list in the JSON string, calling an automatic verification engine interface, and determining a current field to be verified from a structural query statement to be verified.

Step S12: when the sentence type of the to-be-checked structured query sentence is a data definition language type and a history field identifier corresponding to the current to-be-checked field does not exist in a preset data standard reference library, acquiring semantic similarity of each field name between the current to-be-checked field and each history field in the preset data standard reference library, and determining comprehensive similarity based on the semantic similarity of each field name.

In this embodiment, when the sentence type of the structured query sentence to be checked is a data definition language type and the history field identifier corresponding to the current field to be checked does not exist in the preset data standard reference library, the semantic similarity of each field name between the current field to be checked and each history field in the preset data standard reference library is obtained, and weighted average processing is performed on the semantic similarity of each field name to obtain the comprehensive similarity.

Step S13: and establishing a target field identification set of the current field to be checked according to the comprehensive similarity, determining whether the current field to be checked meets a first preset standard condition or not based on a first data type detection result and a dictionary value matching result between the current field to be checked and the target field identification set, and re-jumping to the step of screening one field from the structured query sentence to be checked as the current field to be checked.

In this embodiment, the determining, based on the first data type detection result and the dictionary value matching result between the current field to be checked and the target field identifier set, whether the current field to be checked meets a first preset standard condition includes: comparing the data type of the current field to be checked with the data type of the target field identification set to obtain a first data type detection result; comparing the dictionary value list of the current field to be checked with the dictionary value list of the target field identification set to obtain a dictionary value matching result; and determining whether the current field to be checked meets a first preset standard condition or not based on the first data type detection result and the dictionary value matching result. Constructing a target field identification set of a current field to be checked according to fields with comprehensive similarity exceeding a preset threshold in a preset data standard reference library, comparing the data type of the current field to be checked with the data type corresponding to the target field identification set to obtain a first data type detection result, comparing a dictionary value list of the current field to be checked with a dictionary value list of the target field identification set to obtain a dictionary value matching result, and if the first data type detection result indicates that the data type of the current field to be checked is consistent with the data type corresponding to the target field identification set and the dictionary value matching result indicates that the dictionary value list of the current field to be checked is matched with the dictionary value list of the target field identification set, indicating that the current field to be checked passes checking, namely meets a first preset standard condition. For example, the dictionary value list of the current field to be checked is {1: identity card, 2: passport }, the list of dictionary values for the fields in the target field identification set is: {1: passport, 2: identity card }, the dictionary key=1 and value=identity card of the current field to be verified, but the dictionary key=1 and value=passport of the field in the target identification set, and the key=1, so that the meaning of the representation is inconsistent; the dictionary value list of the current field to be checked is {1: identity card, 2: passport }, the list of dictionary values for the fields in the target field identification set is: { 3:ID card, 4: passport, the dictionary value Key of the current field to be verified includes {1,2}, but the field Key in the target identification set is {3,4}, the dictionary value verification fails.

Step S14: when the statement type of the to-be-checked structured query statement is a data operation language type and a target field identifier corresponding to the current to-be-checked field exists in a preset data standard reference library, determining whether the current to-be-checked field meets a second preset standard condition or not based on a second data type detection result and a dictionary value range detection result between the current to-be-checked field and the target field identifier, and re-jumping to the step of screening one field from the to-be-checked structured query statement as the current to-be-checked field.

In this embodiment, the determining, based on the second data type detection result and the dictionary value range detection result between the current field to be checked and the target field identifier, whether the current field to be checked meets a second preset standard condition includes: comparing the data type of the current field to be checked with the data type of the target field identification to obtain a second data type detection result; comparing the field value corresponding to the current field to be checked with the dictionary value list range of the target field identifier to obtain a dictionary value range detection result; and determining whether the current field to be checked meets a second preset standard condition or not based on the second data type detection result and the dictionary value range detection result.

When the statement type of the structured query statement to be checked is a data operation language type and a target field identifier corresponding to the current field to be checked exists in a preset data standard reference library, the automatic check engine determines a corresponding dictionary value list range according to the target field identifier, acquires a field value of the current field to be checked, namely a field value inserted or changed by the current field to be checked, judges whether the field value of the current field to be checked belongs to the dictionary value range, namely a dictionary value range detection result, judges whether the data type of the current field to be checked is consistent with the data type of the target field identifier, namely a second data type detection result, and when the dictionary value range detection result is that the field value of the current field to be checked belongs to the dictionary value range and the second data type detection result is that the data type of the current field to be checked is consistent with the data type of the target field identifier, the current field to be checked passes the check, namely accords with a second preset standard condition, and if the dictionary value detection result and the second data type detection result are not the conditions, the current field to be checked does not pass the check, namely does not accord with the second preset standard condition. For example, the change identifier_type field value is 3, but only key=1 and value=identification card in the target field identification dictionary value list range; key=2 and value=passport, that is, 3 is not in this target field identification dictionary value list range, and thus verification fails.

It can be understood that the post-processing module stores the verification results of each field in the SQL standardized detection association table of the automatic detection database, so that the developer can carry out corresponding modification according to the fields and the structured query sentences which do not accord with the first preset standard condition and/or the second preset standard condition in the SQL standardized detection association table.

Referring to fig. 2, the embodiment of the application discloses a specific method for detecting a structured query statement, which is applied to a preset verification tool and comprises the following steps:

step S21: and screening a field from the structured query statement to be checked as a current field to be checked.

Step S22: when the sentence type of the to-be-checked structured query sentence is a data definition language type and a history field identifier corresponding to the current to-be-checked field does not exist in a preset data standard reference library, acquiring semantic similarity of each field name between the current to-be-checked field and each history field in the preset data standard reference library, and determining comprehensive similarity based on the semantic similarity of each field name.

In this embodiment, or the obtaining the semantic similarity of each field name between the current field to be checked and each history field in the preset data standard reference library includes: performing word segmentation on the field name of the current field to be checked by using a natural language processing algorithm to obtain word groups after word segmentation; acquiring first field name semantic similarity between the word group after word segmentation and each field name in the preset data standard reference library; performing word segmentation on remark information of the current field to be checked by using a target word segmentation model to obtain a dictionary value, and obtaining a first English synonym set corresponding to the dictionary value; acquiring second field name semantic similarity between the Chinese field name of the current field to be checked and each Chinese field name in the preset data standard reference library by using the natural language processing algorithm; acquiring the semantic similarity of a third field name between the English field name of the current field to be checked and a second English synonym set in the preset data standard reference library; and acquiring the semantic similarity of a fourth field name between the first English synonym set and each English field name in the preset data standard reference library. The specific process is as follows:

1) Performing word segmentation on field names of the current field to be checked by using a natural language processing algorithm, and performing synonym acquaintance calculation (the value is [0,1 ]) on the word groups subjected to word segmentation and each field name in a preset data standard reference library so as to obtain the semantic similarity of the first field name;

2) Performing word segmentation on remark information of a current field to be checked by using a target word segmentation model to obtain dictionary values, for example, a Chinese field name key1-value1 key2-value2, then calling a Chinese-English translation application programming interface, and translating the segmented Chinese field name into a plurality of corresponding synonyms to obtain a first English synonym set [ syn1, syn2, ];

3) Performing similarity calculation on the Chinese field names of the current field to be checked and each Chinese field name in a preset data standard reference library by using a natural language processing algorithm to obtain second field name semantic similarity;

4) Performing similarity calculation on the English field names of the current field to be checked and a second English synonym set in a preset data standard reference library to obtain third field name semantic similarity;

5) And semantic similarity of fourth field names between the first English synonym set and each English field name in the preset data standard reference library.

In this embodiment, determining the comprehensive similarity based on the semantic similarity of each field name includes: and carrying out weighted average processing on the first field name semantic similarity, the second field name semantic similarity, the third field name semantic similarity and the fourth field name semantic similarity to obtain comprehensive similarity. The comprehensive similarity calculation formula is as follows:

wherein similarity represents the integrated similarity, σ _i Representing weights, X _i Representing the semantic similarity of field names, i representing the sequence number of the semantic similarity of field names.

In this embodiment, before the word segmentation processing is performed on the remark information of the current field to be checked by using the target word segmentation model, the method further includes: the remark information of each history field in a preset data standard reference library is collected, and marking processing and data enhancement processing are carried out on the remark information of each history field so as to obtain remark training samples; and training the initial word segmentation model by using the remark training sample to obtain a target word segmentation model. And collecting a large amount of remark information of each history field, marking the remark information, performing data enhancement processing to obtain remark training samples, and training an initial word segmentation model by using the remark training samples to obtain a target word segmentation model.

In this embodiment, when the statement type of the to-be-verified structured query statement is a data definition language type and the history field identifier corresponding to the current to-be-verified field does not exist in the preset data standard reference library, obtaining semantic similarity of each field name between the current to-be-verified field and each history field in the preset data standard reference library includes: when the statement type of the to-be-checked structured query statement is a data definition language type and a current service benchmark mapping table in a preset data standard benchmark library does not have a history field identification corresponding to the current to-be-checked field, acquiring semantic similarity of each field name between the current to-be-checked field and each history field in the preset data standard benchmark library. It can be understood that it is necessary to determine whether a history field identifier corresponding to a current field to be checked exists in the current service reference mapping table in the preset data standard reference library, and if not, obtain semantic similarity of each field name between the current field to be checked and each history field in the preset data standard reference library.

Step S23: and determining each target field identifier with the comprehensive similarity larger than a preset threshold value from the preset data standard reference library, and establishing a target field identifier set of the current field to be checked according to each target field identifier.

It should be noted that if the current service reference mapping table does not have the history field identifier corresponding to the current field to be checked, the current service reference mapping table needs to be updated, so that the history field identifier corresponding to the current field to be checked can be found in the new current service reference mapping table, and the specific updating process is as follows: constructing a mapping relation between the current field to be checked and each target field identifier; and storing the mapping relation into the current service reference mapping table, and updating the current service reference mapping table to obtain a new current service reference mapping table. In this way, when searching is performed subsequently, the history field identifier of the field pair to be checked currently, that is, the target field identifier, can be found in the new current service reference mapping table.

Step S24: determining whether the current field to be checked meets a first preset standard condition or not based on a first data type detection result and a dictionary value matching result between the current field to be checked and the target field identification set, and re-jumping to the step of screening one field from the structured query sentence to be checked as the current field to be checked.

Step S25: when the statement type of the to-be-checked structured query statement is a data operation language type and a target field identifier corresponding to the current to-be-checked field exists in a preset data standard reference library, determining whether the current to-be-checked field meets a second preset standard condition or not based on a second data type detection result and a dictionary value range detection result between the current to-be-checked field and the target field identifier, and re-jumping to the step of screening one field from the to-be-checked structured query statement as the current to-be-checked field.

Therefore, the application can analyze remark information in the data definition language type, trains the initial word segmentation model by utilizing a natural language processing algorithm and an open source model of the deep learning framework to obtain a target word segmentation model of the remark information, is beneficial to data standardization processing of which the metadata is of a fixed dictionary value type, and can also have real-time monitoring detection analysis capability for a structured query statement to be checked of the data definition language type.

Referring to fig. 3, an embodiment of the present application discloses a specific method for detecting a structured query statement, which is applied to a preset verification tool, and includes:

Step S31: and screening a field from the structured query statement to be checked as a current field to be checked.

It can be understood that the field screening method may be to screen according to the field arrangement sequence, for example, screen according to the arrangement sequence of the positive sequence, or randomly screen a field from the structured query statement to be checked as the current field to be checked.

Step S32: when the sentence type of the to-be-checked structured query sentence is a data definition language type and a history field identifier corresponding to the current to-be-checked field does not exist in a preset data standard reference library, acquiring semantic similarity of each field name between the current to-be-checked field and each history field in the preset data standard reference library, and determining comprehensive similarity based on the semantic similarity of each field name.

In this embodiment, if the statement type of the to-be-checked structured query statement is a data definition language type and the history field identifier corresponding to the current to-be-checked field does not exist in the preset data standard reference library, it is indicated that the current to-be-checked field of the to-be-checked structured query statement is not checked, so that the current to-be-checked field needs to be checked, and whether the current to-be-checked field meets the first preset standard condition is judged.

Step S33: and establishing a target field identification set of the current field to be checked according to the comprehensive similarity, determining whether the current field to be checked meets a first preset standard condition or not based on a first data type detection result and a dictionary value matching result between the current field to be checked and the target field identification set, and re-jumping to the step of screening one field from the structured query sentence to be checked as the current field to be checked.

It should be noted that before the step of re-jumping to the step of screening a field from the to-be-verified structured query statement as the current to-be-verified field, it is required to determine whether there is an un-verified field in the to-be-verified structured query statement, if there is an un-verified field, it is required to re-jump to the step of screening a field from the to-be-verified structured query statement as the current to-be-verified field, if there is no un-verified field, i.e. each field in the to-be-verified structured query statement is verified, it is not necessary to re-jump to the step of screening a field from the to-be-verified structured query statement as the current to-be-verified field.

Step S34: when the statement type of the to-be-checked structured query statement is a data definition language type and a history field identifier corresponding to the current to-be-checked field exists in a preset data standard reference library, judging that the current to-be-checked field meets the first preset standard condition, and re-jumping to the step of screening one field from the to-be-checked structured query statement as the current to-be-checked field.

In this embodiment, if the statement type of the to-be-checked structured query statement is a data definition language type and the history field identifier corresponding to the current to-be-checked field exists in the preset data standard reference library, it is indicated that the current to-be-checked field of the to-be-checked structured query statement has been checked in advance, so that the current to-be-checked field does not need to be checked.

Step S35: when the statement type of the to-be-checked structured query statement is a data operation language type and a target field identifier corresponding to the current to-be-checked field exists in a preset data standard reference library, determining whether the current to-be-checked field meets a second preset standard condition or not based on a second data type detection result and a dictionary value range detection result between the current to-be-checked field and the target field identifier, and re-jumping to the step of screening one field from the to-be-checked structured query statement as the current to-be-checked field.

It can be understood that after determining whether the current field to be checked meets the second preset standard condition, it needs to be determined whether there is any field that is not checked in the structural query statement to be checked, and only if there is any field that is not checked, the step of screening a field from the structural query statement to be checked is performed again to be used as the current field to be checked.

Step S36: when the statement type of the to-be-checked structured query statement is a data operation language type and the target field identification corresponding to the current to-be-checked field does not exist in the preset data standard reference library, judging that the current to-be-checked field does not accord with the second preset standard condition, and re-jumping to the step of screening one field from the to-be-checked structured query statement as the current to-be-checked field.

It should be noted that, because the data operation language type premise is to ensure that the metadata of the field completes the corresponding verification and mapping through the data definition language type operation, if the statement type of the structured query statement to be verified is the data operation language type and the target field identifier corresponding to the current field to be verified does not exist in the preset data standard reference library, the exception information can be directly returned, that is, it is directly determined that the current field to be verified does not conform to the second preset standard condition.

Therefore, each field in the structured query statement to be checked is checked, the structured query statement of the data definition language type can be detected, the structured query statement of the data operation language type can be detected, and the detection type is more comprehensive.

Referring to fig. 4, an embodiment of the present application discloses a device for detecting a structured query statement, which is applied to a preset verification tool, and includes:

the field screening module 11 is configured to screen a field from the structured query statement to be checked as a current field to be checked;

the similarity determining module 12 is configured to obtain semantic similarity of each field name between the current field to be checked and each history field in the preset data standard reference library when the statement type of the structured query statement to be checked is a data definition language type and the history field identifier corresponding to the current field to be checked does not exist in the preset data standard reference library, and determine comprehensive similarity based on the semantic similarity of each field name;

a first skip module 13, configured to establish a target field identifier set of the current field to be checked according to the comprehensive similarity, determine whether the current field to be checked meets a first preset standard condition based on a first data type detection result and a dictionary value matching result between the current field to be checked and the target field identifier set, and skip to the step of screening a field from the structural query statement to be checked as the current field to be checked again;

And a second skip module 14, configured to determine, when the statement type of the to-be-verified structured query statement is a data operation language type and a target field identifier corresponding to the current to-be-verified field exists in a preset data standard reference library, whether the current to-be-verified field meets a second preset standard condition based on a second data type detection result and a dictionary value range detection result between the current to-be-verified field and the target field identifier, and skip to the step of screening a field from the to-be-verified structured query statement as the current to-be-verified field.

Further, the embodiment of the application also provides electronic equipment. Fig. 5 is a block diagram of an electronic device 20, according to an exemplary embodiment, and is not intended to limit the scope of use of the present application in any way.

Fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application. Specifically, the method comprises the following steps: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input output interface 25, and a communication bus 26. The memory 22 is configured to store a computer program, where the computer program is loaded and executed by the processor 21 to implement relevant steps in the method for detecting a structured query statement performed by an electronic device as disclosed in any of the foregoing embodiments.

In this embodiment, the power supply 23 is configured to provide an operating voltage for each hardware device on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and the external device, and the communication protocol to be followed is any communication protocol applicable to the technical solution of the present application, which is not limited herein in detail; the input/output interface 25 is used for acquiring external input data or outputting external output data, and the specific interface type thereof may be selected according to the specific application requirement, which is not limited herein.

Processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 may be implemented in at least one hardware form of DSP (Digital Signal Processing ), FPGA (Field-Programmable Gate Array, field programmable gate array), PLA (Programmable Logic Array ). The processor 21 may also comprise a main processor, which is a processor for processing data in an awake state, also called CPU (Central Processing Unit ); a coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 21 may integrate a GPU (Graphics Processing Unit, image processor) for rendering and drawing of content required to be displayed by the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence ) processor for processing computing operations related to machine learning.

The memory 22 may be a carrier for storing resources, such as a read-only memory, a random access memory, a magnetic disk, or an optical disk, and the resources stored thereon include an operating system 221, a computer program 222, and data 223, and the storage may be temporary storage or permanent storage.

The operating system 221 is used for managing and controlling various hardware devices on the electronic device and the computer program 222, so as to implement the operation and processing of the processor 21 on the mass data 223 in the memory 22, which may be Windows, unix, linux. The computer program 222 may further comprise a computer program that can be used to perform other specific tasks in addition to the computer program that can be used to perform the structured query statement detection method performed by the electronic device as disclosed in any of the previous embodiments. The data 223 may include, in addition to data received by the electronic device and transmitted by the external device, data collected by the input/output interface 25 itself, and so on.

Further, the application also discloses a computer readable storage medium for storing a computer program; the method for detecting the structured query statement comprises the steps of executing a computer program by a processor, wherein the computer program realizes the method for detecting the structured query statement disclosed by the prior art. For specific steps of the method, reference may be made to the corresponding contents disclosed in the foregoing embodiments, and no further description is given here.

In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, so that the same or similar parts between the embodiments are referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.

Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative elements and steps are described above generally in terms of functionality in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application. The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software modules may be placed in random access Memory (Random Access Memory), memory, read-Only Memory (ROM), electrically programmable EPROM (Erasable Programmable Read Only Memory), electrically erasable programmable EEPROM (Electrically Erasable Programmable Read Only Memory), registers, hard disk, removable disk, CD-ROM (CoMP 23026730act Disc Read-Only Memory), or any other form of storage medium known in the art.

Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

The above detailed description of the method, the device, the equipment and the medium for detecting the structured query statement provided by the invention applies specific examples to illustrate the principle and the implementation of the invention, and the description of the above examples is only used for helping to understand the method and the core idea of the invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.

Claims

1. The method for detecting the structured query statement is characterized by being applied to a preset checking tool and comprising the following steps of:

2. The method for detecting a structured query statement according to claim 1, wherein before screening a field from the structured query statement to be checked as a current field to be checked, the method further comprises:

3. The structured query statement detection method of claim 1, further comprising:

4. The method for detecting a structured query statement according to claim 1, wherein the obtaining semantic similarity of each field name between the current field to be checked and each history field in the preset data standard reference library includes:

5. The method according to claim 1, wherein determining whether the current field to be checked meets a first preset standard condition based on a first data type detection result and a dictionary value matching result between the current field to be checked and the target field identification set comprises:

6. The method for detecting a structured query statement according to any one of claims 1 to 5, wherein the establishing the target field identification set of the current field to be checked according to the integrated similarity includes:

7. The method for detecting a structured query sentence according to claim 6, wherein when the sentence type of the structured query sentence to be checked is a data definition language type and the history field identifier corresponding to the current field to be checked does not exist in the preset data standard reference library, obtaining semantic similarity of each field name between the current field to be checked and each history field in the preset data standard reference library includes:

8. The utility model provides a structured query statement detection device which is characterized in that is applied to the presupposition check tool, includes:

9. An electronic device, comprising:

a memory for storing a computer program;

a processor for executing the computer program to implement the steps of the structured query statement detection method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program; wherein the computer program when executed by a processor implements the steps of the structured query statement detection method as claimed in any one of claims 1 to 7.