CN112948251B

CN112948251B - Automatic software testing method and device

Info

Publication number: CN112948251B
Application number: CN202110233769.8A
Authority: CN
Inventors: 李宇胜; 李伟良; 李俊; 刘欢
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2021-03-03
Filing date: 2021-03-03
Publication date: 2024-06-21
Anticipated expiration: 2041-03-03
Also published as: CN112948251A

Abstract

The embodiment of the application provides a method and a device for automatically testing software, which can be used in the technical field of artificial intelligence, and the method comprises the following steps: inputting keywords of a current target program into a preset software test scheme matching model, and acquiring a software test scheme corresponding to the target program according to the output of the software test scheme matching model, wherein the software test scheme matching model comprises a preset Conditional Random Field (CRF) model; and automatically testing the target program based on the software testing scheme. The application can effectively improve the degree of automation, the degree of intellectualization and the efficiency of software testing scheme matching, and can effectively improve the degree of automation, the efficiency and the reliability of software testing.

Description

Automatic software testing method and device

Technical Field

The application relates to the technical field of data processing, in particular to the technical field of artificial intelligence, and particularly relates to a method and a device for automatically testing software.

Background

In the whole life process of software development, software testing is to write various testing methods and testing cases, wherein the software testing modes generally comprise black box testing, white box testing and gray box testing, and the case design methods comprise methods such as equivalence class, boundary value, causal graph, decision table, scene method, orthogonal method, enumeration method and the like.

The existing software test mode is to manually select a test method or a design method of a manual selection case, and then to maintain the case, so that the maintenance mode has high manual maintenance cost and low efficiency. Some existing case design methods are universal, but under different scenes, the method needs to be manually maintained again to redesign the case, so that the cost of software test engineering is increased, and the development efficiency needs to be further improved.

Disclosure of Invention

Aiming at the problems in the prior art, the application provides the automatic software testing method and the device, which can effectively improve the degree of automation, the degree of intellectualization and the efficiency of software testing scheme matching, and can effectively improve the degree of automation, the efficiency and the reliability of software testing.

In order to solve the technical problems, the application provides the following technical scheme:

in a first aspect, the present application provides a method for automatically testing software, including:

inputting keywords of a current target program into a preset software test scheme matching model, and acquiring a software test scheme corresponding to the target program according to the output of the software test scheme matching model, wherein the software test scheme matching model comprises a preset Conditional Random Field (CRF) model;

and automatically testing the target program based on the software testing scheme.

Further, before the keyword of the current target program is input into the preset software testing scheme matching model, the method further comprises the following steps:

Acquiring a plurality of historical program files;

Extracting keywords of each historical program file and receiving software test scheme identifiers corresponding to the keywords, wherein the keywords are used for representing program types of corresponding programs;

Respectively setting labels corresponding to the historical program files according to the software test scheme identifications corresponding to the keywords of the historical program files respectively so as to form corresponding training sets;

Based on a preset conditional random field CRF algorithm, training according to the training set to obtain a conditional random field CRF model for outputting a software test scheme identifier corresponding to a program.

Performing model verification on the conditional random field CRF model obtained through training, and performing tuning processing on the conditional random field CRF model according to a corresponding model verification result, wherein the tuning processing comprises: tag tuning and/or model tuning.

Receiving each software automatic test request, wherein the software automatic test request comprises a target program and a corresponding keyword;

And adding the keywords of the target programs corresponding to the software automatic test requests into an index.

Further, the obtaining the software test scheme corresponding to the target program according to the output of the software test scheme matching model includes:

And storing the software test scheme identifiers of the target program output by the software test scheme matching model into a batch timing processing task, so as to periodically establish corresponding index files for the software test scheme identifiers in the batch timing processing task and determine the software test schemes corresponding to the software test scheme identifiers.

Further, after the automatic test of the target program based on the software test scheme, the method further comprises:

And outputting a corresponding test result after the automatic test of the target program so as to obtain an evaluation result of the conditional random field CRF model based on the test result.

Further, the software testing scheme includes: at least one of a black box test, a white box test, and a gray box test.

In a second aspect, the present application provides an automatic software testing apparatus, comprising:

The model prediction module is used for inputting keywords of a current target program into a preset software test scheme matching model and obtaining a software test scheme corresponding to the target program according to the output of the software test scheme matching model, wherein the software test scheme matching model comprises a preset Conditional Random Field (CRF) model;

And the automatic test module is used for automatically testing the target program based on the software test scheme.

In a third aspect, the present application provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the software automatic test method when executing the program.

In a fourth aspect, the present application provides a computer readable storage medium having stored thereon a computer program which when executed by a processor implements the software automatic test method.

According to the technical scheme, the automatic software testing method and device provided by the application comprise the following steps: inputting keywords of a current target program into a preset software test scheme matching model, and acquiring a software test scheme corresponding to the target program according to the output of the software test scheme matching model, wherein the software test scheme matching model comprises a preset Conditional Random Field (CRF) model; the automatic test is carried out on the target program based on the software test scheme, the automatic selection of the software test scheme of the target program is realized by applying a Conditional Random Field (CRF) model, the degree of automation and the degree of intellectualization of the matching of the software test scheme can be effectively improved, the matching efficiency of the software test scheme can be effectively provided, and the labor cost and the time cost can be effectively saved; the automatic test is carried out on the target program based on the software test scheme, so that the automation degree of the software test can be further improved, the efficiency and the reliability of the software test can be effectively improved, the method can be applied to the software research and development process, and the test case can be efficiently and automatically provided; the test engineer can automatically match the software test scheme according to the program submitted by development, so as to rapidly verify the test engineering and improve the efficiency of developing the whole software.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic flow chart of a software automatic test method according to an embodiment of the application.

FIG. 2 is a second flow chart of the software automatic test method according to the embodiment of the application.

FIG. 3 is a third flow chart of a software automatic test method according to an embodiment of the application.

Fig. 4 is a fourth flowchart of a software automatic test method according to an embodiment of the present application.

Fig. 5 is a fifth flowchart of a software automatic test method according to an embodiment of the present application.

FIG. 6 is a sixth flowchart of an automatic software testing method according to an embodiment of the present application.

Fig. 7 is a schematic structural diagram of an automatic software testing device in an embodiment of the application.

FIG. 8 is a flow chart of the method for automatically testing software in an application example of the application.

Fig. 9 is a schematic structural diagram of an electronic device in an embodiment of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

It should be noted that the method and the device for automatically testing software disclosed by the application can be used in the technical field of artificial intelligence, and can also be used in any field except the technical field of artificial intelligence, and the application field of the method and the device for automatically testing software disclosed by the application is not limited.

In order to solve the problems of poor automation degree and low efficiency of matching or selecting a software test mode and the like in the existing software test method, the application respectively provides a software automatic test method, a software automatic test device, electronic equipment and a computer readable storage medium, and the automatic selection of a software test scheme of a target program is realized by applying a Conditional Random Field (CRF) model, so that the automation degree and the intelligent degree of matching the software test scheme can be effectively improved, the matching efficiency of the software test scheme can be effectively provided, and the labor cost and the time cost can be effectively saved; the automatic test is carried out on the target program based on the software test scheme, so that the automation degree of the software test can be further improved, the efficiency and the reliability of the software test can be effectively improved, the method can be applied to the software research and development process, and the test case can be efficiently and automatically provided; the test engineer can automatically match the software test scheme according to the program submitted by development, so as to rapidly verify the test engineering and improve the efficiency of developing the whole software.

Based on the above, the present application further provides a software automatic testing device for implementing the software automatic testing method provided in one or more embodiments of the present application, where the software automatic testing device may be connected to a client device by itself or through communication between a third party server and the like, so as to receive each software automatic testing request sent by each client device, where the software automatic testing request includes a target program and a corresponding keyword; inputting keywords of a current target program into a preset software test scheme matching model, and acquiring a software test scheme corresponding to the target program according to the output of the software test scheme matching model, wherein the software test scheme matching model comprises a preset Conditional Random Field (CRF) model; and automatically testing the target program based on the software testing scheme, sending a testing result corresponding to the automatic test to corresponding client equipment, so that users such as research personnel and development personnel give out an evaluation result of the conditional random field CRF model according to the testing result displayed in the held client equipment, and sending the evaluation result of the conditional random field CRF model to the software automatic testing device through the client equipment, so that the software automatic testing paper judges whether to return to readjust the conditional random field CRF model according to the received evaluation result of the conditional random field CRF model.

In one or more embodiments of the present application, the conditional random field CRF (conditional random field algorithm) combines the features of the maximum entropy model and the hidden markov model, which is an undirected graph model, and has achieved good results in recent years in the task of sequence labeling such as word segmentation, part-of-speech labeling, and named entity recognition. Conditional random fields are a typical discriminant model whose joint probabilities can be written in the form of a number of potential function multiplications, with linear chain member random fields being the most common.

It is understood that the client device may include a smart phone, a tablet electronic device, a network set top box, a portable computer, a desktop computer, a Personal Digital Assistant (PDA), an in-vehicle device, a smart wearable device, etc. Wherein, intelligent wearing equipment can include intelligent glasses, intelligent wrist-watch, intelligent bracelet etc..

In another practical application, the foregoing part of the software automatic testing apparatus for performing the software automatic test may be executed in a server as described above, or all operations may be completed in the client device. Specifically, the selection may be made according to the processing capability of the client device, and restrictions of the use scenario of the user. The application is not limited in this regard. If all operations are performed in the client device, the client device may further include a processor for specific processing of the software automatic test.

The client device may have a communication module (i.e. a communication unit) and may be connected to a remote server in a communication manner, so as to implement data transmission with the server. The server may include a server on the side of the task scheduling center, and in other implementations may include a server of an intermediate platform, such as a server of a third party server platform having a communication link with the task scheduling center server. The server may include a single computer device, a server cluster formed by a plurality of servers, or a server structure of a distributed device.

Any suitable network protocol may be used between the server and the client device, including those not yet developed on the filing date of the present application. The network protocols may include, for example, TCP/IP protocol, UDP/IP protocol, HTTP protocol, HTTPS protocol, etc. Of course, the network protocol may also include, for example, RPC protocol (Remote Procedure Call Protocol ), REST protocol (Representational STATE TRANSFER) or the like used above the above-described protocol.

The following embodiments and application examples are described in detail.

In order to improve the automation degree and efficiency of software test scheme selection and application, the application provides an embodiment of a software automatic test method, referring to fig. 1, the software automatic test method based on the execution of the software automatic test device specifically comprises the following contents:

Step 100: inputting keywords of a current target program into a preset software testing scheme matching model, and acquiring a software testing scheme corresponding to the target program according to output of the software testing scheme matching model, wherein the software testing scheme matching model comprises a preset Conditional Random Field (CRF) model.

It is understood that the keyword refers to the program type of the corresponding target program or history program, and may also be the program type or phenomenon of the target program or history program input by the user.

Before executing step 100, firstly determining a test scheme, performing label setting on a historical software file based on the type of the test scheme, and then training according to the historical software file provided with the label to obtain the conditional random field CRF model, wherein the test mode is classified as follows: through research of common methods in the industry, common testing methods in testing engineering, such as black box testing, white box testing, gray box testing and the like, are selected; the case design method is used for collecting and classifying: according to the classified attribute of the test method, collecting samples of the case design method, wherein the common case design method is classified as follows: the design method of the black box test case comprises the following steps: equivalence classes, boundary values, causal graphs; the white box test case design method comprises the following steps: statement overlay, predicate overlay, condition overlay, predicate/condition overlay, multiple condition overlay, path overlay, logic overlay comparison; ash box testing, and the like.

Step 200: and automatically testing the target program based on the software testing scheme.

As can be seen from the above description, the automatic software testing method provided by the embodiment of the application realizes the automatic selection of the software testing scheme of the target program by applying the conditional random field CRF model, can effectively improve the automation degree and the intelligent degree of the matching of the software testing scheme, can effectively provide the matching efficiency of the software testing scheme, and can effectively save the labor cost and the time cost; the automatic test is carried out on the target program based on the software test scheme, so that the automation degree of the software test can be further improved, the efficiency and the reliability of the software test can be effectively improved, the method can be applied to the software research and development process, and the test case can be efficiently and automatically provided; the test engineer can automatically match the software test scheme according to the program submitted by development, so as to rapidly verify the test engineering and improve the efficiency of developing the whole software.

In order to pre-train the conditional random field CRF model, in one embodiment of the software automatic test method provided in the present application, referring to fig. 2, before step 100 in the software automatic test method, the method specifically includes the following steps:

Step 011: acquiring a plurality of historical program files;

step 012: extracting keywords of each historical program file and receiving software test scheme identifiers corresponding to the keywords, wherein the keywords are used for representing program types of corresponding programs;

Step 013: respectively setting labels corresponding to the historical program files according to the software test scheme identifications corresponding to the keywords of the historical program files respectively so as to form corresponding training sets;

Step 014: based on a preset conditional random field CRF algorithm, training according to the training set to obtain a conditional random field CRF model for outputting a software test scheme identifier corresponding to a program.

Specifically, the related attribute of the filter program is first filtered to define related labels, such as JAVA program, and the related label is verified by judging/condition coverage method with if, else and other keywords in the program. The program may be tagged with tags defining a predicate/conditional overlay. Then, selecting a model algorithm, wherein machine automatic learning is needed, a cited deep learning PADDLEPADDLE framework is built, PADDLEPADDLE covers a plurality of fields such as natural language processing, computer vision, recommendation engines and the like, a plurality of leading pre-training Chinese models are opened, simultaneously ultra-large-scale deep learning parallel training of dense parameter and sparse parameter scenes is supported, efficient parallel training of billions of scale parameters and hundreds of points is supported, the stability and backward compatibility of an API are realized, and the method has perfect Chinese-English bilingual use documents, so that the method has the technical characteristics of easy learning and easy use, simplicity and high efficiency; the model algorithm adopts a conditional random field CRF algorithm, combines the characteristics of a maximum entropy model and a hidden Markov model, and is an undirected graph model. By tagging a series of programs, the CRF layer may add some constraints to the last predicted tag to ensure that the predicted tag is legitimate. During training of training data, these constraints may be automatically learned by the CRF layer. But the data efficiency is not high when training only using CRF algorithm. Also referenced BiLSTM is a combination of forward LSTM and backward LSTM. The LSTM combines the word representations into sentence representations, and can adopt an addition method, namely, adding all word representations, or averaging and other methods, and the LSTM model can be used for better capturing long-distance dependency relations. Since LSTM can learn which information to memorize and which information to forget through the training process, the bi-directional semantic dependence can be better captured through BiLSTM. Therefore, the machine learning is modeled by adopting BILSTM +CRF algorithm.

The relevant attributes of the relevant programs are arranged in the attribute comb of the programs, relevant labels are designed for the attributes, if, else and else in the programs are taken as examples, and a case design method of judging-condition coverage can be adopted, such as a method that a program can judge coverage in the programs with while.

And (3) completing label collection by program engineers and test engineers in the lottery software project development group, enabling the whole research and development team to assemble program sentences into labels designed in a case according to the program sentences, collecting the labels, and completing label marking according to the label design in the last step. Meanwhile, the existing cases in the case asset library are marked with specific design methods. This corresponds to a method of distinguishing specific case designs from related cases by different program types.

The PADDLEPADDLE framework is cited with a CRF algorithm and a BILSTM model, a user can directly call the model, the model is divided into a plurality of parts, a first layer PADDLEPADDLE framework layer, a second layer BILSTM layer and a third layer CRF layer; the first layer is a framework layer for converting discrete vectors in test cases and methods into continuous vectors, the second layer uses BILSTM algorithm to calculate a transmission matrix, each value in the matrix represents how likely the current character is which program label, and the third layer uses CRF algorithm to obtain context information; the collected program labels can be manually divided into a verification set and a training set, the batch size is set to 128 for accelerating the training speed, 20 epoch training is carried out simultaneously, the trained model becomes a usable model, and the next tuning is carried out after the model.

As can be seen from the above description, according to the software automatic test method provided by the embodiment of the present application, the conditional random field CRF model for outputting the software test scheme identifier corresponding to the program is obtained by training according to the training set based on the preset conditional random field CRF algorithm in advance, so that the reliability and accuracy of the application of the conditional random field CRF model can be effectively improved, and further, the degree of automation, the degree of intelligence and the efficiency of software test scheme matching can be further improved.

In order to provide a model verification tuning manner, in one embodiment of the software automatic test method provided by the present application, referring to fig. 3, before step 100 and after step S014 in the software automatic test method, the following are specifically included:

Step 020: performing model verification on the conditional random field CRF model obtained through training, and performing tuning processing on the conditional random field CRF model according to a corresponding model verification result, wherein the tuning processing comprises: tag tuning and/or model tuning.

Specifically, the data of the verification set is used to complete tuning of the model, the model is called to identify the verification set, the identification condition of the related named entity, namely the program attribute is observed, the identification condition of different program attributes is observed, algorithm tuning is performed on the attributes which cannot be identified, and the tuning has two methods, namely, the tuning of the related label is as follows: if the entity recognition degree of continuous is low, firstly observing whether labeling of the existing label by judging to cover the label is accurate or not, and secondly considering whether a related label needs to be newly added or not; the second method is to adjust the model parameters, increase or decrease the frequency parameters to reach the optimal value for the user using method and the frequency of the cases, and then order the cases according to the order of the using frequency from high to low. The applicability of the application is further improved by establishing a new model relation through the two methods.

As can be seen from the above description, according to the software automatic test method provided by the embodiment of the present application, by performing model verification on the conditional random field CRF model obtained by training, and performing tuning processing on the conditional random field CRF model according to the corresponding model verification result, the reliability and accuracy of the application of the conditional random field CRF model can be further improved, and further the automation degree, the intelligence degree and the efficiency of software test scheme matching can be further improved.

In order to further improve the software testing efficiency, in one embodiment of the software automatic testing method provided by the present application, referring to fig. 4, before step 100 in the software automatic testing method, the method further specifically includes the following:

Step 031: receiving each software automatic test request, wherein the software automatic test request comprises a target program and a corresponding keyword;

step 032: and adding the keywords of the target programs corresponding to the software automatic test requests into an index.

As can be seen from the above description, the automatic software testing method provided by the embodiment of the application can count the codes in the program, and if the index is added in the key words in the program, the efficiency of matching case asset library is higher when the user submits the code, and the bottleneck of query efficiency caused by the use of a large number of users is avoided.

In order to further improve the software testing efficiency, in one embodiment of the software automatic testing method provided by the present application, referring to fig. 5, step 100 in the software automatic testing method further specifically includes the following:

Step 110: and inputting the keywords of the current target program into a preset software testing scheme matching model so that the software testing scheme matching model outputs the software testing scheme identification of the target program.

Step 120: and storing the software test scheme identifiers of the target program output by the software test scheme matching model into a batch timing processing task, so as to periodically establish corresponding index files for the software test scheme identifiers in the batch timing processing task and determine the software test schemes corresponding to the software test scheme identifiers.

As can be seen from the above description, the automatic software testing method provided by the embodiment of the present application establishes batch timing tasks, and establishes a unified index file for the screened program labels at intervals of a period of time, so that the time consumption can be reduced and the automation degree, efficiency and reliability of the software testing can be further improved without increasing the complexity and code amount of the program.

In order to further improve the accuracy of the software test, in one embodiment of the automatic software test method provided by the present application, referring to fig. 6, after step 200 in the automatic software test method, the method specifically further includes the following contents:

step 300: and outputting a corresponding test result after the automatic test of the target program so as to obtain an evaluation result of the conditional random field CRF model based on the test result.

Specifically, the user judges whether the query case result meets the requirements according to the result returned by the system, if so, the query is ended, and if not, the query result is fed back to the model tuning step to further tune the data model.

As can be seen from the above description, according to the software automatic test method provided by the embodiment of the present application, the conditional random field CRF model is optimized by obtaining the evaluation result of the conditional random field CRF model according to the output test result, so that the reliability and accuracy of the application of the conditional random field CRF model can be further improved, and further, the degree of automation, the degree of intelligence and the efficiency of software test scheme matching can be further improved.

In order to further improve the applicability and comprehensiveness of the software test, in one embodiment of the software automatic test method provided by the application, the software test scheme in the software automatic test method comprises the following steps: at least one of a black box test, a white box test, and a gray box test.

As can be seen from the above description, the software automatic test method provided by the embodiment of the present application, by providing the type of the software test scheme, can effectively improve the applicability and comprehensiveness of the software automatic test, and further can further improve the automation degree, efficiency and reliability of the software automatic test.

In order to improve the automation degree and efficiency of software testing scheme selection and application, the present application provides an embodiment of a software automatic testing device for executing all or part of the content in the software automatic testing method, referring to fig. 7, the software automatic testing device specifically includes the following contents:

The model prediction module 10 is configured to input a keyword of a current target program into a preset software test scheme matching model, and obtain a software test scheme corresponding to the target program according to an output of the software test scheme matching model, where the software test scheme matching model includes a preset conditional random field CRF model.

Before executing the model prediction module 10, firstly, determining a test scheme, performing label setting on a historical software file based on the type of the test scheme, and then training according to the historical software file provided with the label to obtain the conditional random field CRF model, wherein the test mode is classified as follows: through research of common methods in the industry, common testing methods in testing engineering, such as black box testing, white box testing, gray box testing and the like, are selected; the case design method is used for collecting and classifying: according to the classified attribute of the test method, collecting samples of the case design method, wherein the common case design method is classified as follows: the design method of the black box test case comprises the following steps: equivalence classes, boundary values, causal graphs; the white box test case design method comprises the following steps: statement overlay, predicate overlay, condition overlay, predicate/condition overlay, multiple condition overlay, path overlay, logic overlay comparison; ash box testing, and the like.

An automatic test module 20, configured to automatically test the target program based on the software test scheme.

The embodiment of the automatic software testing device provided by the application can be particularly used for executing the processing flow of the embodiment of the automatic software testing method in the embodiment, and the functions of the embodiment of the automatic software testing device are not repeated herein, and reference can be made to the detailed description of the embodiment of the method.

As can be seen from the above description, the automatic software testing device provided by the embodiment of the application realizes the automatic selection of the software testing scheme of the target program by applying the conditional random field CRF model, can effectively improve the automation degree and the intelligent degree of the matching of the software testing scheme, can effectively provide the matching efficiency of the software testing scheme, and can effectively save the labor cost and the time cost; the automatic test is carried out on the target program based on the software test scheme, so that the automation degree of the software test can be further improved, the efficiency and the reliability of the software test can be effectively improved, the method can be applied to the software research and development process, and the test case can be efficiently and automatically provided; the test engineer can automatically match the software test scheme according to the program submitted by development, so as to rapidly verify the test engineering and improve the efficiency of developing the whole software.

In order to pre-train the conditional random field CRF model, in one embodiment of the software automatic test apparatus provided by the present application, the software automatic test apparatus is further specifically configured to perform the following:

Step 011: acquiring a plurality of historical program files;

Specifically, the relevant attribute of the filter program is first filtered to perform relevant label definition, such as JAVA program, and the referenced device can be verified by the device covered by the decision/condition. The program may be tagged with tags defining a predicate/conditional overlay. Then, selecting a model algorithm, wherein machine automatic learning is needed, a cited deep learning PADDLEPADDLE framework is built, PADDLEPADDLE covers a plurality of fields such as natural language processing, computer vision, recommendation engines and the like, a plurality of leading pre-training Chinese models are opened, simultaneously ultra-large-scale deep learning parallel training of dense parameter and sparse parameter scenes is supported, efficient parallel training of billions of scale parameters and hundreds of points is supported, the stability and backward compatibility of an API are realized, and the method has perfect Chinese-English bilingual use documents, so that the method has the technical characteristics of easy learning and easy use, simplicity and high efficiency; the model algorithm adopts a conditional random field CRF algorithm, combines the characteristics of a maximum entropy model and a hidden Markov model, and is an undirected graph model. By tagging a series of programs, the CRF layer may add some constraints to the last predicted tag to ensure that the predicted tag is legitimate. During training of training data, these constraints may be automatically learned by the CRF layer. But the data efficiency is not high when training only using CRF algorithm. Also referenced BiLSTM is a combination of forward LSTM and backward LSTM. The LSTM combines the word representations into sentence representations, and an adding device, that is, a device that adds all word representations, or an averaging device, may be used to better capture long-distance dependency relationships using the LSTM model. Since LSTM can learn which information to memorize and which information to forget through the training process, the bi-directional semantic dependence can be better captured through BiLSTM. Therefore, the machine learning is modeled by adopting BILSTM +CRF algorithm.

The relevant attributes of the relevant program are arranged in the attribute comb of the program, relevant labels are designed for the attributes, if, else and else in the program are taken as examples, and a case design device for judging-condition coverage can be adopted, such as a device for judging coverage of the program with a while in the program.

And (3) completing label collection by program engineers and test engineers in the lottery software project development group, enabling the whole research and development team to assemble program sentences into labels designed in a case according to the program sentences, collecting the labels, and completing label marking according to the label design in the last step. Meanwhile, the existing cases in the case asset library are marked with specific design devices. This corresponds to a device and related cases that distinguish specific case designs by different program types.

The PADDLEPADDLE framework is cited with a CRF algorithm and a BILSTM model, a user can directly call the model, the model is divided into a plurality of parts, a first layer PADDLEPADDLE framework layer, a second layer BILSTM layer and a third layer CRF layer; the first layer is a framework layer for converting discrete vectors in test cases and devices into continuous vectors, the second layer uses BILSTM algorithm to calculate a transmission matrix, each value in the matrix represents how likely the current character is which program label, and the third layer uses CRF algorithm to obtain context information; the collected program labels can be manually divided into a verification set and a training set, the batch size is set to 128 for accelerating the training speed, 20 epoch training is carried out simultaneously, the trained model becomes a usable model, and the next tuning is carried out after the model.

As can be seen from the above description, the software automatic testing device provided by the embodiment of the application trains the conditional random field CRF model for outputting the software testing scheme identifier corresponding to the program according to the training set by pre-setting the conditional random field CRF algorithm, so that the application reliability and accuracy of the conditional random field CRF model can be effectively improved, and further the automation degree, the intelligent degree and the efficiency of software testing scheme matching can be further improved.

In order to provide a model verification tuning way, in one embodiment of the automatic software testing device provided by the application, the automatic software testing device is further specifically configured to execute the following:

Specifically, the data of the verification set is used to complete tuning of the model, the model is called to identify the verification set, the identification condition of the related named entity, namely the program attribute is observed, the identification condition of different program attributes is observed, algorithm tuning is performed on the attributes which cannot be identified, the tuning has two devices, and the first device is tuning of the related label, for example: if the entity recognition degree of continuous is low, firstly observing whether labeling of the existing label by judging to cover the label is accurate or not, and secondly considering whether a related label needs to be newly added or not; the second device is to adjust the model parameters, for the frequency of the user using device and cases, increase or decrease the frequency parameters to reach the optimal value, and then order the cases in order from high to low. The applicability of the application is further improved by establishing a new model relationship through the two devices.

As can be seen from the above description, the software automatic test device provided by the embodiment of the present application performs model verification on the conditional random field CRF model obtained by training, and performs tuning processing on the conditional random field CRF model according to the corresponding model verification result, so that the reliability and accuracy of the application of the conditional random field CRF model can be further improved, and further the automation degree, the intelligence degree and the efficiency of software test scheme matching can be further improved.

In order to further improve the software testing efficiency, in one embodiment of the software automatic testing device provided by the application, the software automatic testing device is further specifically configured to perform the following:

As can be seen from the above description, the automatic software testing device provided by the embodiment of the application can count the codes in the program, and if the index is added in the key words in the program, the efficiency of matching case asset library is higher when the user submits the code, and the bottleneck of query efficiency caused by the use of a large number of users is avoided.

In order to further improve the software testing efficiency, in one embodiment of the automatic software testing device provided by the present application, the model prediction module 10 in the automatic software testing device is further specifically configured to perform the following:

As can be seen from the above description, the automatic software testing device provided by the embodiment of the application establishes batch timing tasks, and establishes a unified index file for the screened program labels at intervals of a period of time, so that the time consumption can be reduced and the automation degree, efficiency and reliability of the software testing can be further improved without increasing the complexity and code amount of the program.

In order to further improve the accuracy of software testing, in one embodiment of the automatic software testing device provided by the application, the automatic software testing device is further specifically configured to perform the following:

As can be seen from the above description, the automatic software testing device provided by the embodiment of the present application obtains the evaluation result of the conditional random field CRF model according to the output test result to perform tuning processing on the conditional random field CRF model, so that the reliability and accuracy of the application of the conditional random field CRF model can be further improved, and further the automation degree, the intelligent degree and the efficiency of software testing scheme matching can be further improved.

In order to further improve the applicability and comprehensiveness of the software test, in one embodiment of the software automatic test device provided by the application, the software test scheme in the software automatic test device comprises: at least one of a black box test, a white box test, and a gray box test.

As can be seen from the above description, the software automatic test device provided by the embodiment of the application can effectively improve the applicability and comprehensiveness of the software automatic test by providing the type of the software test scheme, and further can further improve the automation degree, efficiency and reliability of the software automatic test.

In order to further explain the book scheme, the application also provides a specific application example of the software automatic test method. The user inputs the program type or phenomenon, automatically matches the best test method for the user to select, machine learns to select the test method which is most suitable for the program according to the label provided by the user, and automatically generates a case. Meanwhile, the selection sequence is gradually optimized in machine learning according to the selection frequency of the user, and the user selects the high-frequency priority order.

The application example effectively performs data analysis, accurately predicts the case required by the case designer through the machine learning technology, and finally generates the case to the user, so that the software testing efficiency is improved, and the development period of the software is further shortened.

Referring to fig. 8, the method for automatically testing software provided by the application example of the present application specifically includes the following:

step 1: screening the related attribute of the program keywords, and storing the program related keywords into a database;

Step 2: allowing the machine to learn by establishing a model;

step3: according to the program library, establishing an index for the existing program according to the model;

step 4: judging a machine learning method and selecting a case design;

Step 5: and according to whether the keyword extraction case meets the expectations or not.

Specifically, the technical scheme of the application example of the application mainly comprises the following steps:

Step one, classifying a testing method: through research of common methods in the industry, common testing methods in testing engineering, such as black box testing, white box testing, gray box testing and the like, are selected; the case design method is used for collecting and classifying: according to the classified attribute of the test method, collecting samples of the case design method, wherein the common case design method is classified as follows: the design method of the black box test case comprises the following steps: equivalence classes, boundary values, causal graphs; the white box test case design method comprises the following steps: statement overlay, predicate overlay, condition overlay, predicate/condition overlay, multiple condition overlay, path overlay, logic overlay comparison; ash box testing, and the like.

And step two, relevant attributes of the screening program are subjected to relevant label definition, such as JAVA programs, keywords such as if, else and the like are provided in the programs, and the quoted method can be verified by a judging/condition coverage method. The program may be tagged with tags defining a predicate/conditional overlay. By way of illustration in table 1, different method analyses, and different case combinations related thereto, are cited.

TABLE 1

1. Sentence covering method

For example, the test case may refer to statements in statement block 1 through statement block 4.

{ X=2, y=2 } can execute to statement block 1 and statement block 4, the path taken: a-b-e-f

{ X= -4, y=0 } can be performed to statement block 2, statement block 3 and statement block 4, the path taken: a-c-d-f

Thus, the statement coverage criteria are met by two test cases, which, of course, are not unique.

Assuming "≡ &" in the first judgment statement if (x >0 ++y > 0) is wrongly written by the programmer as "||", i.e., if (x >0||y > 0), the test is performed using the set of test cases designed above, and still 100% statement coverage can be achieved, so that the statement coverage cannot find the above logical error.

2. Determining coverage

Enough test cases are designed such that each judged "true", "false" branch in the program under test is executed at least once. In this example, there are two determinations if (x >0& & y > 0) (denoted as P1) and if (logic < 0) (denoted as P2) for covering the corresponding test cases, see table 2.

TABLE 2

Data	P1	P2	Path
				{x＝3,y＝3}	T	F	a-b-e-f
{x＝-3,y＝0}	F	T	a-c-d-f

Both the true and false branches of the determination have been performed, so the criteria for determining coverage are met.

Sufficiency of test

Assuming "≡ &" in the first judgment statement if (x >0 ++y > 0) is wrongly written by the programmer as "||", i.e., if (x >0||y > 0), the test is performed using the set of test cases designed above, and still 100% of judgment coverage can be achieved, so the judgment coverage cannot find the above logical error either.

Compared with statement coverage: since the executable statement is not on the true branch of the predicate or on the false branch, the statement coverage criteria must be met as long as the predicate coverage criteria are met, and vice versa.

3. Condition coverage

Enough test cases are designed such that the possible value of each logic condition in each judgment statement in the program under test is satisfied at least once. It may also be described as designing enough test cases such that the possible values for each logic condition in the program under test are met at least once.

In this example, there are two determinations if (x >0& & y > 0) (denoted as P1) and if (magic < 0) (denoted as P2), and three conditions x >0 (denoted as C1), y >0 (denoted as C2) and magic <0 (denoted as C3) in total. Test cases for conditional coverage are shown in table 3.

TABLE 3 Table 3

Data

C1

C2

C3

P1

P2

Path

{x＝3,y＝3}

T

F

a-b-e-f

{x＝-3,y＝0}

F

T

a-c-d-f

The three conditions all meet the various possible values once, thus reaching the standard of 100% condition coverage.

Sufficiency of test

The above test cases also reach the 100% criterion coverage criteria at the same time, but there is no guarantee that all test cases (groups) that reach the 100% conditional coverage criteria reach the 100% criterion coverage criteria, for example as shown in table 4:

TABLE 4 Table 4

Data

C1

C2

C3

P1

P2

Path

{x＝3,y＝0}

T

F

T

F

a-c-e-f

{x＝-3,y＝5}

F

T

F

a-c-e-f

Since the conditional coverage criterion cannot reach 100% of the criterion for judging coverage, it is not necessarily possible to reach 100% of the sentence coverage criterion.

4. Decision/condition coverage

The test cases are designed to be sufficiently large so that the determination result (true or false) of each determination itself in the program under test is satisfied at least once, and at the same time, the possible value of each logic condition is also satisfied at least once. I.e. meets both the criteria of 100% decision coverage and 100% conditional coverage. Test cases for decision/condition coverage are shown in table 5.

TABLE 5

Data

C1

C2

C3

P1

P2

Path

{x＝3,y＝3}

T

F

a-b-e-f

{x＝-3,y＝0}

F

T

a-c-d-f

All possible values of the conditions are satisfied once, and all the judging results of the judgment are satisfied once.

Sufficiency of test

Reaching the 100% predicate-condition coverage criteria can certainly reach 100% condition coverage, 100% predicate coverage, and 100% statement coverage.

Step three, establishing a model, and optimizing by using a machine learning algorithm

① Firstly, selecting a model algorithm, wherein machine automatic learning is needed, a cited deep learning PADDLEPADDLE framework is built, PADDLEPADDLE covers a plurality of fields such as natural language processing, computer vision, recommendation engines and the like, a plurality of leading pre-training Chinese models are opened, simultaneously ultra-large-scale deep learning parallel training of dense parameter and sparse parameter scenes is supported, efficient parallel training of trillion scale parameters and hundreds of points is supported, the stability and backward compatibility of an API are realized, and the method has perfect Chinese-English bilingual use documents, so that the method has the technical characteristics of easy learning and easy use, simplicity and high efficiency; the model algorithm adopts a Conditional Random Field (CRF) algorithm, combines the characteristics of a maximum entropy model and a hidden Markov model, and is an undirected graph model. By tagging a series of programs, the CRF layer may add some constraints to the last predicted tag to ensure that the predicted tag is legitimate. During training of training data, these constraints may be automatically learned by the CRF layer. But the data efficiency is not high when training only using CRF algorithm. Also referenced BiLSTM is a combination of forward LSTM and backward LSTM. The LSTM combines the word representations into sentence representations, and can adopt an addition method, namely, adding all word representations, or averaging and other methods, and the LSTM model can be used for better capturing long-distance dependency relations. Since LSTM can learn which information to memorize and which information to forget through the training process, the bi-directional semantic dependence can be better captured through BiLSTM. Therefore, the machine learning is modeled by adopting BILSTM +CRF algorithm.

② Program label design

③ The relevant attributes of the relevant program are arranged in the attribute comb of the program, relevant labels are designed for the attributes, if, else and else in the program are taken as examples, and a case design method of judging-condition coverage can be adopted, such as a method that the program can judge coverage in the program with while.

④ Program label collection and processing

⑤ Model building and training

⑥ Model tuning

The data of the verification set is used for completing the tuning of the model, the model is called for identifying the verification set, the identification condition of related named entities, namely program attributes is observed, the identification condition of different program attributes is observed, algorithm tuning is carried out on the attributes which cannot be identified, the tuning has two methods, and the first method is the tuning of related labels, for example: if the entity recognition degree of continuous is low, firstly observing whether labeling of the existing label by judging to cover the label is accurate or not, and secondly considering whether a related label needs to be newly added or not; the second method is to adjust the model parameters, increase or decrease the frequency parameters to reach the optimal value for the user using method and the frequency of the cases, and then order the cases according to the order of the using frequency from high to low. The applicability of the application is further improved by establishing a new model relation through the two methods.

⑦ Interface and encapsulation of services

The invocation of the model is implemented using a micro-service method, i.e., the entry field of the service is a key of the program, such as while in the program, the service passes the statement into the model and returns the output of the model to the user.

Step four, creating an index file

The codes in the program can be said to be counted, if the key words in the program are used for adding the index, the efficiency of matching the case asset library when the user submits the key words in the use is higher, and the bottleneck of query efficiency caused by the use of a large number of users is avoided. And establishing batch timing tasks, and establishing uniform index files for the screened program labels at intervals of a period of time, so that time consumption is not increased due to the increase of complexity and code quantity of the program.

Step five, method judgment and case design selection

For inspection of program input conditions, output results can be expected, then causal analysis methods can be used, and reference can be made to cases involved in the case asset library. If various combinations of input conditions must be considered at the time of testing, the number of possible combinations will be astronomical numbers, and thus the design of the test case must be considered in a form suitable for describing the combination of conditions, yielding a corresponding plurality of actions.

Specific examples are as follows:

Identity: if ci is 1, then ei is also 1; otherwise ei is 0.

Non-: if ci is 1, then ei is 0; otherwise ei is 1.

Or: if c1 or c2 or c3 is 1, ei is 1; otherwise ei is 0. Or may have any number of inputs.

And (2) the following steps: if c1 and c2 are both 1, then ei is 1; otherwise ei is 0. There may also be any number of inputs to the AND.

Step six, judging the query result

And the user judges whether the query case result meets the requirements according to the result returned by the system, if so, the query is ended, and if not, the query result is fed back to the model tuning step to further tune the data model.

In summary, the present application provides a software automatic test method, which provides a method for directly referencing a program according to self-learning and high-frequency cases and methods by using a machine learning technology, and can be applied to the software development process to efficiently and automatically provide test cases. By the method, a test engineer can automatically match the case and the method according to the program submitted by development, so that the verification of the test engineering is rapidly carried out, and the research and development efficiency of the whole software is improved.

In order to improve the automation degree and efficiency of software test scheme selection and application from the hardware level, the application provides an embodiment of an electronic device for realizing all or part of contents in the software automatic test method, wherein the electronic device specifically comprises the following contents:

Fig. 9 is a schematic block diagram of a system configuration of an electronic device 9600 according to an embodiment of the present application. As shown in fig. 9, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. Notably, this fig. 9 is exemplary; other types of structures may also be used in addition to or in place of the structures to implement telecommunications functions or other functions.

In one embodiment, the software auto-test function may be integrated into the central processor. Wherein the central processor may be configured to control:

As can be seen from the above description, the electronic device provided by the embodiment of the present application realizes automatic selection of the software test scheme of the target program by applying the conditional random field CRF model, so that the degree of automation and the degree of intellectualization of matching the software test scheme can be effectively improved, the matching efficiency of the software test scheme can be effectively provided, and the labor cost and the time cost can be effectively saved; the automatic test is carried out on the target program based on the software test scheme, so that the automation degree of the software test can be further improved, the efficiency and the reliability of the software test can be effectively improved, the method can be applied to the software research and development process, and the test case can be efficiently and automatically provided; the test engineer can automatically match the software test scheme according to the program submitted by development, so as to rapidly verify the test engineering and improve the efficiency of developing the whole software.

In another embodiment, the software automatic test equipment may be configured separately from the central processing unit 9100, for example, the software automatic test equipment may be configured as a chip connected to the central processing unit 9100, and the software automatic test function is implemented by control of the central processing unit.

As shown in fig. 9, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is noted that the electronic device 9600 need not include all of the components shown in fig. 9; in addition, the electronic device 9600 may further include components not shown in fig. 9, and reference may be made to the related art.

As shown in fig. 9, the central processor 9100, sometimes referred to as a controller or operational control, may include a microprocessor or other processor device and/or logic device, which central processor 9100 receives inputs and controls the operation of the various components of the electronic device 9600.

The memory 9140 may be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information about failure may be stored, and a program for executing the information may be stored. And the central processor 9100 can execute the program stored in the memory 9140 to realize information storage or processing, and the like.

The input unit 9120 provides input to the central processor 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used for displaying display objects such as images and characters. The display may be, for example, but not limited to, an LCD display.

The memory 9140 may be a solid state memory such as Read Only Memory (ROM), random Access Memory (RAM), SIM card, etc. But also a memory which holds information even when powered down, can be selectively erased and provided with further data, an example of which is sometimes referred to as EPROM or the like. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application/function storage portion 9142, the application/function storage portion 9142 storing application programs and function programs or a flow for executing operations of the electronic device 9600 by the central processor 9100.

The memory 9140 may also include a data store 9143, the data store 9143 for storing data, such as contacts, digital data, pictures, sounds, and/or any other data used by an electronic device. The driver storage portion 9144 of the memory 9140 may include various drivers of the electronic device for communication functions and/or for performing other functions of the electronic device (e.g., messaging applications, address book applications, etc.).

The communication module 9110 is a transmitter/receiver 9110 that transmits and receives signals via an antenna 9111. A communication module (transmitter/receiver) 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, as in the case of conventional mobile communication terminals.

Based on different communication technologies, a plurality of communication modules 9110, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, etc., may be provided in the same electronic device. The communication module (transmitter/receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and to receive audio input from the microphone 9132 to implement usual telecommunications functions. The audio processor 9130 can include any suitable buffers, decoders, amplifiers and so forth. In addition, the audio processor 9130 is also coupled to the central processor 9100 so that sound can be recorded locally through the microphone 9132 and sound stored locally can be played through the speaker 9131.

The embodiment of the present application also provides a computer-readable storage medium capable of implementing all the steps in the software automatic test method in the above embodiment, the computer-readable storage medium storing a computer program thereon, the computer program implementing all the steps in the software automatic test method in which the execution subject in the above embodiment is a server or a client when executed by a processor, for example, the processor implementing the following steps when executing the computer program:

As can be seen from the above description, the computer readable storage medium provided by the embodiment of the present application realizes automatic selection of the software test scheme of the target program by applying the conditional random field CRF model, so that the degree of automation and the degree of intellectualization of matching the software test scheme can be effectively improved, the efficiency of matching the software test scheme can be effectively provided, and the labor cost and the time cost can be effectively saved; the automatic test is carried out on the target program based on the software test scheme, so that the automation degree of the software test can be further improved, the efficiency and the reliability of the software test can be effectively improved, the method can be applied to the software research and development process, and the test case can be efficiently and automatically provided; the test engineer can automatically match the software test scheme according to the program submitted by development, so as to rapidly verify the test engineering and improve the efficiency of developing the whole software.

It will be apparent to those skilled in the art that embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein. The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks. These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The principles and embodiments of the present invention have been described in detail with reference to specific examples, which are provided to facilitate understanding of the method and core ideas of the present invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.

Claims

1. A method for automatically testing software, comprising:

Inputting a keyword of a current target program into a preset software test scheme matching model, and acquiring a software test scheme corresponding to the target program according to the output of the software test scheme matching model, wherein the software test scheme matching model comprises a preset Conditional Random Field (CRF) model, and the keyword is used for representing the program type of the corresponding program;

automatically testing the target program based on the software testing scheme;

Before the keyword of the current target program is input into the preset software testing scheme matching model, the method further comprises the following steps:

Adding the keywords of the target programs corresponding to the software automatic test requests into an index;

Wherein, the software test scheme includes: at least one of a black box test, a white box test, and a gray box test, wherein the black box test case design method comprises: equivalence classes, boundary values, and causal graphs; the white box test case design method comprises the following steps: statement overlay, predicate overlay, condition overlay, predicate/condition overlay, multiple condition overlay, path overlay, and logical overlay comparison.

2. The method for automatically testing software according to claim 1, further comprising, before said inputting the keyword of the current object program into the preset software test scheme matching model:

Acquiring a plurality of historical program files;

Extracting keywords of each historical program file and receiving software test scheme identifiers corresponding to the keywords respectively;

3. The automatic software testing method according to claim 2, further comprising, before said inputting the keyword of the current target program into the preset software testing scheme matching model:

4. The method for automatically testing software according to claim 1, wherein the obtaining the software test scheme corresponding to the target program according to the output of the software test scheme matching model comprises:

5. The automatic software testing method according to claim 1, further comprising, after the automatic testing of the target program based on the software testing scheme:

6. An automatic software testing device, comprising:

The model prediction module is used for inputting keywords of a current target program into a preset software test scheme matching model, and acquiring a software test scheme corresponding to the target program according to the output of the software test scheme matching model, wherein the software test scheme matching model comprises a preset Conditional Random Field (CRF) model, and the keywords are used for representing the program types of the corresponding programs;

the automatic test module is used for automatically testing the target program based on the software test scheme;

The automatic software testing device is also used for executing the following contents:

7. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the software automatic test method of any one of claims 1 to 5 when executing the computer program.

8. A computer readable storage medium having stored thereon a computer program, characterized in that the computer program, when executed by a processor, implements the software automatic test method of any of claims 1 to 5.