CN112181831A - Script performance verification method, device and equipment based on keywords and storage medium - Google Patents
Script performance verification method, device and equipment based on keywords and storage medium Download PDFInfo
- Publication number
- CN112181831A CN112181831A CN202011040636.0A CN202011040636A CN112181831A CN 112181831 A CN112181831 A CN 112181831A CN 202011040636 A CN202011040636 A CN 202011040636A CN 112181831 A CN112181831 A CN 112181831A
- Authority
- CN
- China
- Prior art keywords
- data
- script
- script file
- determining
- association
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013515 script Methods 0.000 title claims abstract description 377
- 238000000034 method Methods 0.000 title claims abstract description 42
- 238000012795 verification Methods 0.000 title claims abstract description 38
- 238000012545 processing Methods 0.000 claims description 36
- 238000004590 computer program Methods 0.000 claims description 14
- 238000013473 artificial intelligence Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 238000011156 evaluation Methods 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 2
- 239000002699 waste material Substances 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3604—Software analysis for verifying properties of programs
- G06F11/3608—Software analysis for verifying properties of programs using formal methods, e.g. model checking, abstract interpretation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
Abstract
The invention relates to an artificial intelligence technology, in particular to a script performance verification method based on a keyword analysis technology, and discloses a script performance verification method based on keywords, a device, equipment and a storage medium, so that the performance condition of a script before running can be obtained in advance. The method comprises the following steps: acquiring a script file to be checked; performing script keyword analysis on the script file to obtain script keywords in the script file; determining all relevant data tables used by the script file according to the script key words, and determining the relevant conditions among all tables in all relevant data tables; correspondingly determining the association mode among all the data tables in all the associated data tables according to the association conditions; determining data information used by the script file according to the associated data table and the associated mode; determining the data volume of target data required to be read when the script file is executed according to the data information; and outputting a script file verification result comprising the data volume.
Description
Technical Field
The invention relates to the technical field of artificial intelligence, in particular to a script performance verification method, a script performance verification device, script performance verification equipment and a storage medium based on keywords.
Background
Script (Script), which is an executable file written according to a certain format using a specific descriptive language, is also called extension language, or dynamic language, is a programming language, and is usually saved in text form and is only interpreted or compiled when called. When executing a script, the computer performs a series of operations to achieve some purpose or function intended by the writer, such as a Query Language HQL script (Hibernate Query Language) for implementing a script for some data Query or processing.
It can be seen that the performance of the script is extremely important, and it is often necessary to verify or evaluate the performance of the script. At present, when performance evaluation is performed on scripts, for example, when performance evaluation is performed on an HQL script of a big data platform, evaluation is performed mainly by using experience of developers or a script execution log after execution is completed, performance conditions of the scripts cannot be obtained before the scripts are executed in the scheme, once resources are occupied too much due to problems of script performance, resources of the big data platform will be wasted, and even execution of other normal scripts is affected, so a new script performance checking or evaluating mode is urgently needed.
Disclosure of Invention
The invention provides a method, a device, equipment and a storage medium for verifying script performance based on keywords, which aim to solve the problem that the performance condition of a script cannot be obtained before the script runs and the resource occupation is excessive once the script performance is in a problem.
A script performance checking method based on keywords comprises the following steps:
acquiring a script file to be checked;
performing script keyword analysis on the script file to obtain script keywords in the script file;
determining all relevant data tables used by the script file according to the script key words, and determining the relevant conditions among all tables in all relevant data tables;
determining the association mode among all the data tables in the association data tables according to the association condition;
determining data information used by the script file according to the associated data table and the associated mode;
determining the data volume of target data required to be read when the script file is executed according to the data information;
and outputting a script file verification result comprising the data volume.
A keyword-based script performance verification apparatus, comprising:
the script acquisition module is used for acquiring a script file to be verified;
the script analysis module is used for carrying out script keyword analysis on the script file so as to obtain script keywords in the script file;
the association condition determining module is used for determining all association data tables used by the script file according to the script key words and determining association conditions among all tables in all the association data tables;
the association mode determining module is used for correspondingly determining the association modes among all the data tables in the association data tables according to the association conditions;
the data information determining module is used for determining the data information used by the script file according to the associated data table and the associated mode;
the data volume determining module is used for determining the data volume of the target data required to be read when the script file is executed according to the data information;
and the output module is used for outputting the script file verification result comprising the data volume.
A computer device comprising a memory, a processor and a computer program stored in said memory and executable on said processor, said processor implementing the steps of the script performance checking method described above when executing said computer program.
A computer-readable storage medium, storing a computer program which, when executed by a processor, implements the steps of the script performance checking method described above.
In one scheme provided by the keyword-based script performance verification method, the keyword-based script performance verification device, the computer equipment and the storage medium, a script file needing to be verified is obtained first; performing script keyword analysis on the script file to obtain script keywords in the script file; acquiring all the association modes for executing the script file based on the script keywords; then determining data information used by the script file according to the associated data table and the associated mode; determining the data volume of target data required to be read when the script file is executed according to the data information; and finally, outputting a script file verification result comprising the data volume so that a user can find the problem of the script file. In the invention, some data volume-containing conditions of the script file can be acquired and output by using the correlation mode related to the script file before the script is executed or run, so that the possible performance conditions of the script during execution can be known in advance, and the problem that the normal execution of the script is influenced by the problem of excessive resource occupation caused by the problem of the performance of the script due to the fact that the performance of the script cannot be acquired before the script is executed or run in the prior art is solved, so that the script has a high application scene.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments of the present invention will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without inventive labor.
FIG. 1 is a schematic diagram of an application environment of a script performance checking method according to an embodiment of the present invention;
FIG. 2 is a flow chart of a script performance checking method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of one embodiment of step S20 of FIG. 2;
FIG. 4 is another flowchart illustrating a script performance checking method according to an embodiment of the invention;
FIG. 5 is a diagram illustrating an embodiment of determining whether data divergence exists in an embodiment of the present invention;
FIG. 6 is a diagram illustrating an embodiment of determining whether there is data skew according to an embodiment of the present invention;
FIG. 7 is a schematic flow chart of a script performance checking method according to an embodiment of the present invention;
FIG. 8 is a schematic structural diagram of a script performance checking apparatus according to an embodiment of the present invention;
fig. 9 is a schematic structural diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The invention relates to the technical field of artificial intelligence, in particular to a script performance verification application scene, and particularly provides a keyword-based script performance verification method which can be applied to an application environment shown in figure 1, wherein a client can communicate with a server through a network. The client can be but not limited to terminal equipment with a display module, such as various personal computers, notebook computers, smart phones, tablet computers, portable wearable equipment and the like. The server may be implemented as a stand-alone server or as a server cluster consisting of a plurality of servers.
In an embodiment, as shown in fig. 2, a keyword-based script performance verification method is provided, which is described by taking the server in fig. 1 as an example, and includes the following steps:
s10: and acquiring the script file needing to be verified.
A script file is an executable file, which refers to a piece of executable code written using any one of the scripting languages. Scripts are typically saved in text form and are only interpreted or compiled when called. When the script file is executed, a computer executing the script file executes a series of operations written by the script language, so as to achieve a certain purpose or function intended by the script file writer. For example, the Query Language HQL script (Hibernate Query Language) that is currently in common use is used for scripts that implement some sort of data Query or processing. After the writer writes the script file, the performance of the script file needs to be checked, and the server can obtain the script file needing to be checked.
It should be noted that the scripting language used for writing the script file in the embodiment of the present invention may be any type, for example, common languages such as JavaScript, VBScript, Perl, PHP, Python, and the like, and is not limited specifically.
S20: and analyzing the script file to obtain the script keywords in the script file.
After acquiring a script file to be verified, when verifying the script file, firstly, analyzing script keywords of the script file to acquire the script keywords in the script file. The script keywords are sentence keywords for executing or implementing a certain purpose or function in the script code.
Illustratively, if the following script code exists in the script file:
“select form talbe_1、Join talbe_2、on talbe_1.colum1＝talbe_2.colum2”；
it is obvious that select form talbe _1, Join talbe _2, colum1 ═ talbe _2.colum2 are all script keywords in the script file.
In an embodiment, as shown in fig. 3, in step S20, that is, performing script keyword parsing on the script file to obtain a script keyword in the script file, specifically, the method includes the following steps:
s21: and receiving programming language type indication information of a user, wherein the programming language type indication information is used for indicating the programming language type adopted by the script file.
In order to conveniently and quickly acquire the script keywords in the script file, the programming language type indication information input by the user can be acquired, and the programming language type indication information is used for indicating the programming language type adopted by the script file. In one application scenario, a basic performance verification interface may be provided, which includes a programming language type option for providing a user with an input or selection of a programming language type to obtain corresponding programming language type indication information. For example, if the script file is written by using Python, the user may input Python programming language type indication information for indicating that the programming language type adopted by the current script file is Python.
S22: and acquiring the keyword information corresponding to the programming language according to the programming language type indication information.
After the programming language type indication information is obtained, the keyword information corresponding to the programming language can be obtained according to the programming language type indication information. It should be noted that, different script programming languages include different script keywords, and therefore, in order to quickly acquire a script keyword of a script file, in this embodiment, keyword information corresponding to the programming language is acquired according to the programming language type indication information. All the keyword information of the script file can be acquired from all the keyword lists corresponding to the programming language.
S23: and performing script keyword analysis on the script file according to the keyword information to obtain script keywords in the script file.
And after acquiring the keyword information corresponding to the programming language, performing script keyword analysis on the script file according to the keyword information to acquire script keywords in the script file. For example, specifically, the text of the sentence in the script file may be analyzed, and the analysis result may be matched with the keyword information, so as to obtain the script keyword in the script file, that is, the keyword in the keyword information may be directly matched with the word in the script file, so as to determine the script keyword in the script file.
S30: and determining all associated data tables used by the script file according to the script key words, and determining the associated conditions among all tables in all the associated data tables.
S40: and correspondingly determining the association modes among all the data tables in the association data tables according to the association conditions.
It is understood that when the script file is executed, the data table involved in the execution is called, in this embodiment, the associated data table. After the script keywords in the script file are obtained, the meanings expressed by various statements in the script file, including the calling relation of the data tables, are determined according to all the script keywords in the script file, so that all the associated data tables required for executing the script file and the associated conditions among all the tables in the associated data tables are determined. When executing the script file, the called different association data table usually has an association condition and an association method reflected by the association condition.
For example, as in the aforementioned code example "select form talbe _1, Join talbe _2, on talbe _1.colum1 ═ talbe _2.colum 2", it can be obtained that the association data table required when executing the script file includes talbe _1 and talbe _2, wherein Join and on talbe _1.colum1 ═ talbe _2.colum2 constitute the association condition.
It should be noted that the association manner refers to whether there is some association connection relationship between the association data tables, and the association manner can be determined by using an association condition. The associated data table required when executing the script file is assumed to include an associated data table a and an associated data table B, which are respectively as follows:
associated data Table A
| A_ID | A_NAME | … | |
| 1 | 1 | JACK | … |
| 2 | 2 | JACK1 | … |
| 3 | 3 | JACK3 | … |
| 4 | 4 | JACK4 | … |
Associated data table B
| B_ID | B_NAME | … | |
| 1 | 2 | KATE2 | … |
| 2 | 3 | KATE3 | … |
| 3 | 4 | KATE4 | … |
| 4 | 5 | KATE5 | … |
| 5 | 6 | KATE7 | … |
Taking the aforementioned two association data tables as an example, the association manner between the association data tables generally includes the following cases:
inner join-taking data present in both the two associated data tables; for example, the following code is assumed to be in the script execution file: select a, B, from a inner join B on _ ID ═ B _ ID, where select, from, inner join and on form a complete association condition, a _ ID ═ B _ ID represents an association field, the code represents to call association data table a and association data table B for internal connection, that is, association manner of association data table a and association data table B is internal connection, where the association field involved in internal connection is on a _ ID ═ B _ ID, that is, data with the same ID is taken, where the data corresponding to the same ID exists in both association data table a and association data table B, and the last internal connection result is as shown in the following table:
inner connection
| A_ID | A_NAME | … | B_ID | B_NAME | … | |
| 1 | 2 | JACK1 | … | 2 | KATE2 | … |
| 2 | 3 | JACK3 | … | 3 | KATE3 | … |
| 3 | 4 | JACK4 | 4 | KATE4 |
Left join-take all data of the left table and data that the right table can be associated with; the left connection is an external connection, for example, the script execution file is provided with the following codes: the code indicates that the association data table A and the association data table B are called to carry out left connection, namely the association mode of the association data table A and the association data table B is left connection, wherein the association field related to the left connection is on A _ ID ═ B _ ID.
Left connection
| A_ID | A_NAME | … | B_ID | B_NAME | … | |
| 1 | 2 | JACK1 | … | 2 | KATE2 | … |
| 2 | 3 | JACK3 | … | 3 | KATE3 | … |
| 3 | 4 | JACK4 | … | 4 | KATE4 | … |
| 4 | 1 | JACK | … |
Right join-take all data of the right table and data that the left table can be associated with; the right link is an external link, for example, the script execution file is provided with the following codes: the code indicates that the association data table a and the association data table B are called to perform right connection, that is, the association manner of the association data table a and the association data table B is right connection, wherein the related association field of the right connection is a _ ID ═ B _ ID.
Right connection
| A_ID | A_NAME | … | B_ID | B_NAME | … | |
| 1 | 2 | JACK1 | … | 2 | KATE2 | … |
| 2 | 3 | JACK3 | … | 3 | KATE3 | … |
| 3 | 4 | JACK4 | … | 4 | KATE4 | … |
| 4 | … | 5 | KATE5 | … | ||
| 5 | … | 6 | KATE7 | … |
Full join-take data that either of the two tables exists; for example, the following code is assumed to be in the script execution file: the code indicates that the association data table A and the association data table B are called to be fully connected, namely the association mode of the association data table A and the association data table B is fully connected, wherein the association field related to the full connection is on A _ ID or B _ ID.
Full connection
| A_ID | A_NAME | … | B_ID | B_NAME | … | |
| 1 | 2 | JACK1 | … | 2 | KATE2 | … |
| 2 | 3 | JACK3 | … | 3 | KATE3 | … |
| 3 | 4 | JACK4 | 4 | KATE4 | … | |
| 4 | … | 5 | KATE5 | … | ||
| 5 | … | 6 | KATE7 | … | ||
| 6 | 1 | JACK | … | … |
It is understood that, according to various association modes between the association data tables, the association mode of the association data table used for executing the script file can be clearly known, so that all data called or needed for executing the script file can be known.
In addition, it can be seen that different association modes between the associated data tables affect the calling or reference condition of the data tables when the script is executed, so in order to clearly know the performance when the script is executed, the association mode between the associated data tables in the script needs to be determined. It should be noted that the above association manner is only an exemplary illustration, and there may be other association manners between the association data tables, which are not examples here.
S50: and determining data information used by the script file according to the associated data table and the associated mode.
Obviously, after all the associated data tables corresponding to the script file and the associated conditions and the associated modes between the tables in all the associated data tables are obtained, the data information used by the script file can be determined according to the associated data tables and the associated modes. For example, the data information may include data amount, main foreign key of associated data table, value distribution, and the like.
S60: and determining the data volume of the target data required to be read when the script file is executed according to the data information.
S70: and outputting a script file verification result comprising the data volume.
For steps S60-S70, after determining the data information used by the script file, the data amount of the target data that needs to be read when executing the script file can be determined according to the data information, and output to the client for display, so that the user can know some performance conditions in advance when executing the script file. For example, in the code example described above, the data amount in the associated data table talbe _1 is one hundred million, the data amount in the associated data table talbe _1 is 5 million, the primary key of talbe _1 is colum1, and the primary key of talbe _2 is colum 2.
Therefore, the embodiment of the invention provides a script performance verification method based on keywords, so that some execution performance conditions of a script file can be obtained before a script is executed or run, and the problem that in the prior art, the performance conditions of the script cannot be obtained before the script is executed or run, so that the normal execution of the script is influenced by resource waste and the like is solved, and the method has a high application scenario.
In one embodiment, in order to more fully understand the performance condition of the script file during execution, the present invention further determines to obtain other information that can be used to understand the condition of the script file. As shown in fig. 4, after step S40, that is, after determining the association manner between each table in all the association data tables according to the association condition, the method specifically includes the following steps:
s80: and determining the data state when the script file is executed according to the association mode, wherein the data state comprises a data inclination state and/or a data divergence state.
The data divergence state refers to a situation that when a script file is executed, some script statements exist, and when an associated data table is called, data divergence may be caused, it should be noted that, if a problem exists in a script statement, for example, an SQL statement in a script exists, when the script file is executed, a query may not be obtained for a long time because an associated condition statement is not correct, and the reason includes the problem of data divergence. For a simple example, a number of associated data tables are typically used with SQL queries, such that the association condition for any two associated data tables is user UID, actually order ID, or user UID + date or user UID + other conditions. This may lead to a phenomenon of data divergence if the association conditions between the associated data tables are initially incomplete or erroneous. Therefore, the data divergence state when the script file is executed can be determined in an associated manner.
In an embodiment, the phenomenon of data divergence has a plurality of situations, and it is difficult to find the phenomenon in advance in practical application, and an embodiment of the present invention provides a simple way of determining the data divergence therein, wherein, as shown in fig. 5, the data divergence state is determined by the following way:
s81: and determining whether the data tables in all the associated data tables have Cartesian product relationship or not according to the association mode.
In a specific application scenario of the present invention, a cartesian product relationship is used to determine whether there is data divergence, specifically, a cartesian product is used to specifically determine whether there is a cartesian product relationship between two associated data tables, and it can be understood that the cartesian product is a quantity describing whether there is a cross-connection relationship between two associated data tables.
Taking the aforementioned associated data table a and associated data table B as an example, if the script file has the following codes: and selecting A, B from A cross join B, wherein the selecting, from and cross join form a complete association condition, and the code means that the association data table A and the association data table B are called to perform Cartesian calculation so as to obtain a Cartesian product of the two association data tables, namely the association mode of the association data table A and the association data table B is cross connection.
Cartesian product
As shown in the above table, if the script file is executed, the above Cartesian product relationship exists between the two associated data tables.
S82: if the fact that the data tables in all the associated data tables have the Cartesian product relationship is determined, determining that data divergence exists when the script file is executed;
s83: and if the data tables in all the associated data tables are determined to have no Cartesian product relationship, determining that no data divergence exists when the script file is executed.
For steps S82-S83, after determining whether the data tables in all the associated data tables have a cartesian product relationship according to the association manner, if it is determined that the data tables in all the associated data tables have a cartesian product relationship, determining that data divergence exists when executing the script file; otherwise, if it is determined that the data tables in all the associated data tables do not have the cartesian product relationship, it is determined that no data divergence exists when the script file is executed.
It will be appreciated that the cartesian product relationship between the two associated data tables results in a very computationally expensive data call when executing the script file. As another simple example, assuming that the number of data items in the associated data table 1 is 216, and the number of data items in the associated data table 2 associated with the associated data table 1 is 3509, the total number of data items in the two associated data tables will reach 216 × 3509 — 757944, if the script file is executed, the above-mentioned association call relationship exists, and if the data base of the associated data table 1 or the associated data table 2 is at a high level, for example, hundreds of thousands of levels, the total number of calls will be very large, which illustrates that there is a data divergence situation when the script file is executed.
It should be noted that, for a cluster system, generally, the cache is distributed, that is, different nodes are responsible for caching data in a certain range. The cache data is not scattered enough, so that a large amount of cache data is concentrated on one or several service nodes, which is called data skew, that is, the data skew is a data skew state caused by poor effect of load balancing implementation. In the embodiment of the present invention, the data skew specifically means that when a script file is executed, the distribution processing of data related to the associated data table is too concentrated, and when the script file is executed, there is a case where the data distribution processing is not uniform, which results in an excessively large amount of some data processing tasks, and an excessively small amount of some data processing tasks, which results in an uneven distribution of workload of the corresponding processing node, and causes the job execution time of the processing node to be much longer than that of other jobs.
In one embodiment, the data tilt status is determined by the following steps:
s84: and determining whether the distribution of the data task processing amount in the association data table has an uneven phenomenon when the script file is executed according to the association mode, wherein the uneven phenomenon comprises that the task amount of a first data processing task is a preset integer multiple of the task amount of a second data processing task.
After the association mode of the association data table related to the script file is obtained, whether the distribution of the data task processing amount in the association data table has an uneven phenomenon when the script file is executed can be determined according to the association mode, wherein the uneven phenomenon comprises that the task amount of a first data processing task is a preset integer multiple of the task amount of a second data processing task. The first data processing task and the second data processing task are two different tasks allocated by the script file when processing the data in the associated data table, wherein if the task amount of the first data processing task is found to be a preset integer multiple of the task amount of the second data processing task, wherein the preset integer multiple may be greater than 10 times, for example, 15 times, 20 times, and the like, which indicates that the data processing is too concentrated at this time, it is determined whether the allocation of the data task processing amount in the associated data table is uneven when the script file is executed.
S85: and if the distribution of the data task processing amount in the associated data table is determined to have the uneven phenomenon, determining that the data skew exists.
S86: and if the fact that the uneven phenomenon does not exist in the distribution of the data task processing amount in the associated data table is determined, determining that the data inclination does not exist.
For steps S85-S86, after determining whether there is a non-uniformity in the distribution of the processing amount of the data task in the associated data table when executing the script file according to the association manner, if it is determined that there is a non-uniformity in the distribution of the processing amount of the data task in the associated data table, determining that there is a data skew; if the fact that the distribution of the data task processing amount in the associated data table is not uneven is determined, determining that no data inclination exists
Therefore, in the embodiment, whether the data is inclined or not can be further judged, a specific judgment mode is provided, and accordingly, the states of inclination, divergence and the like of the data can be further determined, the performance condition of the script is comprehensively known, and the script is completely checked.
S90: and outputting a script file checking result comprising the data volume and the data state.
And determining the data state when the script file is executed according to the association mode, and outputting a script file verification result comprising the data volume and the data state, so that a user can more comprehensively know the performance of the executed script file, and the script file is improved in advance.
As shown in fig. 7, after step S30, that is, after determining all relevant data tables used by the script file according to the script keywords, the method further includes the following steps:
s100: and acquiring historical script execution diary information from the script execution log library.
S110: and determining the time length for calling the associated data table when other script files are executed according to the historical script execution diary information.
For steps S100 to S110, the embodiment of the present invention obtains the historical script execution journal information from the script execution journal library, where the script execution journal library has an execution journal of past script execution, and obtains the historical script execution journal information from the script execution journal library, and determines a time length for calling the associated data table when executing another script file according to the historical script execution journal information. For example, if the associated data table a is called when the history script 1 is executed, and the log of the associated data table a includes the time length for calling the associated data table, the time length for calling the associated data table a when the history script 1 is executed can be determined according to the script execution log information of the history script, and can also be determined for other associated data tables. Namely: in the embodiment of the invention, the time length for calling all the associated data tables when other script files are executed is determined according to the historical script execution diary information. For example, if a script of an associated data table a is used in the last seven days and the used resource is 50 cores, then the prediction of calling the associated data table a when the script file is executed is 50 cores.
S120: and counting the called time length of each associated data table in all the associated data tables to obtain the total time length.
And when the duration for calling all the associated data tables during the execution of other script files is determined according to the history script execution diary information, the called duration of each associated data table in all the associated data tables can be counted to obtain the total duration, so that the called duration of each related associated data table in the script file can be obtained, and the total duration can be obtained.
S130: and determining the total time length as the execution time length for executing the script file.
S140: and outputting a script file verification result comprising the data volume, the data state and the execution duration.
And determining the total time length to execute the execution time length of the script file, and outputting a script file verification result comprising the data volume, the data state and the execution time length. Therefore, in the embodiment of the present invention, the last output script verification result may be as follows: the script reads one hundred million and fifty thousand of data in total, the final output data volume is less than or equal to fifty million, and the script can not diverge, can not incline, and can estimate the information of 50 kernel-time and the like in execution time.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.
In an embodiment, a keyword-based script performance verification device is provided, and the script performance verification device corresponds to the script performance verification methods in the embodiments one to one. As shown in fig. 8, the script performance verifying apparatus includes a script obtaining module 101, a script parsing module 102, an association condition determining module 103, an association manner determining module 104, a data information determining module 105, a data amount determining module 106, and an output module 107. The functional modules are explained in detail as follows:
the script obtaining module 101 is configured to obtain a script file to be verified;
the script parsing module 102 is configured to perform script keyword parsing on the script file to obtain a script keyword in the script file;
the association condition determining module 103 is configured to determine all association data tables used by the script file according to the script keyword, and determine association conditions between tables in all the association data tables;
an association mode determining module 104, configured to correspondingly determine an association mode between each data table in all the association data tables according to the association condition;
a data information determining module 105, configured to determine, according to the associated data table and the associated manner, data information used by the script file;
a data size determining module 106, configured to determine, according to the data information, a data size of target data that needs to be read when the script file is executed;
and the output module 107 is configured to output a script file verification result including the data volume.
In one embodiment, the apparatus further comprises a data state determination module;
the data state determining module is used for determining the data state when the script file is executed according to the association mode, wherein the data state comprises a data inclination state and/or a data divergence state;
and the output module is used for outputting a script file verification result comprising the data volume and the data state.
In an embodiment, the data state determination module is specifically configured to:
determining whether the data tables in all the associated data tables have Cartesian product relationship according to the association mode;
if the fact that the data tables in all the associated data tables have the Cartesian product relationship is determined, determining that data divergence exists when the script file is executed;
and if the data tables in all the associated data tables are determined to have no Cartesian product relationship, determining that no data divergence exists when the script file is executed.
In an embodiment, the data state determination module is specifically configured to:
determining whether the distribution of the data task processing amount in the association data table has an uneven phenomenon when the script file is executed according to the association mode, wherein the uneven phenomenon comprises a preset integer multiple of the task amount of a first data processing task as the task amount of a second data processing task;
if the phenomenon that the distribution of the data task processing amount in the associated data table is uneven is determined, determining that data inclination exists when the script file is executed;
and if the fact that the distribution of the data task processing amount in the associated data table is not uneven is determined, determining that data inclination does not exist when the script file is executed.
In one embodiment, the apparatus further includes a log information obtaining module, a call duration determining module, and an execution duration determining module:
the log information acquisition module is used for acquiring historical script execution diary information from the script execution log library;
the calling duration determining module is used for determining the duration for calling the associated data table when other script files are executed according to the historical script execution diary information; counting the called time length of each associated data table in all the associated data tables to obtain the total time length;
the execution duration determining module is used for determining the execution duration of executing the script file according to the total duration;
and the output module is used for outputting a script file verification result comprising the data volume, the data state and the execution duration.
In an embodiment, the script parsing module is specifically configured to:
receiving programming language type indicating information of a user, wherein the programming language type indicating information is used for indicating the programming language type adopted by the script file;
acquiring keyword information corresponding to the programming language according to the programming language type indication information;
and performing script keyword analysis on the script file according to the keyword information to obtain script keywords in the script file.
Therefore, the script performance verifying device based on the keywords is provided by the embodiments of the present invention, so that some execution performance conditions of the script file can be obtained before the script is executed or run, and the problem that in the prior art, the performance conditions of the script cannot be obtained before the script is executed or run, which causes resource waste and the like to affect the normal execution of the script is solved, and a higher application scenario is provided.
For the specific limitation of the script performance checking device, reference may be made to the above limitation on the script performance checking method, which is not described herein again. The modules in the script performance checking apparatus may be wholly or partially implemented by software, hardware, or a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 9. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a storage medium and an internal memory. The storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and computer programs in the storage medium to run. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a script performance checking method.
In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program:
acquiring a script file to be checked;
performing script keyword analysis on the script file to obtain script keywords in the script file;
determining all relevant data tables used by the script file according to the script key words, and determining the relevant conditions among all tables in all relevant data tables;
determining the association mode among all the data tables in the association data tables according to the association condition;
determining data information used by the script file according to the associated data table and the associated mode;
determining the data volume of target data required to be read when the script file is executed according to the data information;
and outputting a script file verification result comprising the data volume.
In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:
acquiring a script file to be checked;
performing script keyword analysis on the script file to obtain script keywords in the script file;
determining all relevant data tables used by the script file according to the script key words, and determining the relevant conditions among all tables in all relevant data tables;
determining the association mode among all the data tables in the association data tables according to the association condition;
determining data information used by the script file according to the associated data table and the associated mode;
determining the data volume of target data required to be read when the script file is executed according to the data information;
and outputting a script file verification result comprising the data volume.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and can include the processes of the embodiments of the methods described above when the computer program is executed. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present invention, and are intended to be included within the scope of the present invention.
Claims (10)
1. A script performance checking method based on keywords is characterized by comprising the following steps:
acquiring a script file to be checked;
performing script keyword analysis on the script file to obtain script keywords in the script file;
determining all relevant data tables used by the script file according to the script key words, and determining the relevant conditions among all tables in all relevant data tables;
determining the association mode among all the data tables in the association data tables according to the association condition;
determining data information used by the script file according to the associated data table and the associated mode;
determining the data volume of target data required to be read when the script file is executed according to the data information;
and outputting a script file verification result comprising the data volume.
2. The script performance verifying method of claim 1, wherein after determining the association manner among the tables in all the association data tables according to the association condition, the method further comprises:
determining a data state when the script file is executed according to the association mode, wherein the data state comprises a data inclination state and/or a data divergence state;
the outputting a script file verification result including the data volume includes:
and outputting a script file checking result comprising the data volume and the data state.
3. The script performance checking method of claim 2, wherein said data divergence state is determined by:
determining whether the data tables in all the associated data tables have Cartesian product relationship according to the association mode;
if the fact that the data tables in all the associated data tables have the Cartesian product relationship is determined, determining that data divergence exists when the script file is executed;
and if the data tables in all the associated data tables are determined to have no Cartesian product relationship, determining that no data divergence exists when the script file is executed.
4. The script performance checking method of claim 2, wherein said data tilt state is determined by:
determining whether the distribution of the data task processing amount in the association data table has an uneven phenomenon when the script file is executed according to the association mode, wherein the uneven phenomenon comprises a preset integer multiple of the task amount of a first data processing task as the task amount of a second data processing task;
if the phenomenon that the distribution of the data task processing amount in the associated data table is uneven is determined, determining that data inclination exists when the script file is executed;
and if the fact that the distribution of the data task processing amount in the associated data table is not uneven is determined, determining that data inclination does not exist when the script file is executed.
5. The script performance checking method according to any one of claims 1 to 4, wherein after determining all relevant data tables used by the script file according to the script key, the method further comprises:
acquiring historical script execution diary information from a script execution log library;
determining the time length for calling the associated data table when other script files are executed according to the historical script execution diary information;
counting the called time length of each associated data table in all the associated data tables to obtain the total time length;
determining the total duration to be the execution duration for executing the script file;
the outputting a script file verification result including the data volume and the data state includes:
and outputting a script file verification result comprising the data volume, the data state and the execution duration.
6. The script performance verifying method of any one of claims 1 to 4, wherein said performing script keyword parsing on the script file to obtain a script keyword in the script file comprises:
receiving programming language type indicating information of a user, wherein the programming language type indicating information is used for indicating the programming language type adopted by the script file;
acquiring keyword information corresponding to the programming language according to the programming language type indication information;
and performing script keyword analysis on the script file according to the keyword information to obtain script keywords in the script file.
7. A script performance calibration device based on keywords is characterized by comprising:
the script acquisition module is used for acquiring a script file to be verified;
the script analysis module is used for carrying out script keyword analysis on the script file so as to obtain script keywords in the script file;
the association condition determining module is used for determining all association data tables used by the script file according to the script key words and determining association conditions among all tables in all the association data tables;
the association mode determining module is used for correspondingly determining the association modes among all the data tables in the association data tables according to the association conditions;
the data information determining module is used for determining the data information used by the script file according to the associated data table and the associated mode;
the data volume determining module is used for determining the data volume of the target data required to be read when the script file is executed according to the data information;
and the output module is used for outputting the script file verification result comprising the data volume.
8. The script performance verifying apparatus of claim 7, wherein said apparatus further comprises a data state determining module;
the data state determining module is used for determining the data state when the script file is executed according to the association mode, wherein the data state comprises a data inclination state and/or a data divergence state;
and the output module is used for outputting a script file verification result comprising the data volume and the data state.
9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the script performance checking method as claimed in any one of claims 1 to 6 when executing the computer program.
10. A computer-readable storage medium, in which a computer program is stored, which, when being executed by a processor, carries out the steps of the script performance checking method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011040636.0A CN112181831A (en) | 2020-09-28 | 2020-09-28 | Script performance verification method, device and equipment based on keywords and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011040636.0A CN112181831A (en) | 2020-09-28 | 2020-09-28 | Script performance verification method, device and equipment based on keywords and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112181831A true CN112181831A (en) | 2021-01-05 |
Family
ID=73944736
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011040636.0A Pending CN112181831A (en) | 2020-09-28 | 2020-09-28 | Script performance verification method, device and equipment based on keywords and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112181831A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113190236A (en) * | 2021-05-27 | 2021-07-30 | 中国工商银行股份有限公司 | HQL script checking method and device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2001046865A1 (en) * | 1999-12-23 | 2001-06-28 | Bull Hn Information Systems Inc. | A method and apparatus for improving the performance of a generated code cache search operation through the use of static key values |
| CN107784003A (en) * | 2016-08-26 | 2018-03-09 | 阿里巴巴集团控股有限公司 | Data query method for detecting abnormality, device, equipment and system |
| CN110287114A (en) * | 2019-06-26 | 2019-09-27 | 深圳前海微众银行股份有限公司 | A kind of method and device of database script performance test |
| CN111259012A (en) * | 2020-01-20 | 2020-06-09 | 中国平安人寿保险股份有限公司 | Data homogenizing method and device, computer equipment and storage medium |
| CN111274148A (en) * | 2020-02-04 | 2020-06-12 | 重庆特斯联智慧科技股份有限公司 | Method and device for detecting SQL code defects |
| CN111459937A (en) * | 2020-03-27 | 2020-07-28 | 中国平安人寿保险股份有限公司 | Data table association method, device, server and storage medium |
-
2020
- 2020-09-28 CN CN202011040636.0A patent/CN112181831A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2001046865A1 (en) * | 1999-12-23 | 2001-06-28 | Bull Hn Information Systems Inc. | A method and apparatus for improving the performance of a generated code cache search operation through the use of static key values |
| CN107784003A (en) * | 2016-08-26 | 2018-03-09 | 阿里巴巴集团控股有限公司 | Data query method for detecting abnormality, device, equipment and system |
| CN110287114A (en) * | 2019-06-26 | 2019-09-27 | 深圳前海微众银行股份有限公司 | A kind of method and device of database script performance test |
| CN111259012A (en) * | 2020-01-20 | 2020-06-09 | 中国平安人寿保险股份有限公司 | Data homogenizing method and device, computer equipment and storage medium |
| CN111274148A (en) * | 2020-02-04 | 2020-06-12 | 重庆特斯联智慧科技股份有限公司 | Method and device for detecting SQL code defects |
| CN111459937A (en) * | 2020-03-27 | 2020-07-28 | 中国平安人寿保险股份有限公司 | Data table association method, device, server and storage medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113190236A (en) * | 2021-05-27 | 2021-07-30 | 中国工商银行股份有限公司 | HQL script checking method and device |
| CN113190236B (en) * | 2021-05-27 | 2024-02-02 | 中国工商银行股份有限公司 | HQL script verification method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108197306B (en) | SQL statement processing method and device, computer equipment and storage medium | |
| CN113127347B (en) | Interface testing method, device, equipment and readable storage medium | |
| CN110851159B (en) | Business rule updating method and device, computer equipment and storage medium | |
| CN108415998B (en) | Application dependency relationship updating method, terminal, device and storage medium | |
| CN107247665B (en) | Method and system for testing C interface function | |
| CN111881477A (en) | Indexing method and device of data content, computer equipment and storage medium | |
| CN112181430B (en) | Code change statistical method, device, electronic equipment and storage medium | |
| CN112286934A (en) | Database table importing method, device, equipment and medium | |
| CN111797026A (en) | Test case generation method and device, computer equipment and storage medium | |
| CN111488736B (en) | Self-learning word segmentation method, device, computer equipment and storage medium | |
| CN111885184A (en) | Method and device for processing hot spot access keywords in high concurrency scene | |
| CN109656947B (en) | Data query method and device, computer equipment and storage medium | |
| CN112181831A (en) | Script performance verification method, device and equipment based on keywords and storage medium | |
| CN117271481A (en) | Automatic database optimization method and equipment | |
| CN117130979A (en) | Service resource migration method and device and electronic equipment | |
| CN115757172A (en) | Test execution method and device, storage medium and computer equipment | |
| CN113010550B (en) | Batch object generation and batch processing method and device for structured data | |
| CN111079390B (en) | Method and device for determining selection state of check box list | |
| CN114398178A (en) | Task execution method and device and electronic equipment | |
| CN111352852A (en) | Regression test case selection method and device | |
| CN111858366A (en) | Test case generation method, device, equipment and storage medium | |
| CN116069464B (en) | Optimization method and device based on distributed storage call data execution | |
| CN113177224B (en) | Block chain based data sealing method, device, equipment and storage medium | |
| CN115016835A (en) | Parameter optimization method, device, equipment and medium | |
| CN113886260A (en) | Automatic testing method, system, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |