CN114647578B

CN114647578B - System testing method, device, equipment and storage medium

Info

Publication number: CN114647578B
Application number: CN202210268260.1A
Authority: CN
Inventors: 杨洋
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2022-03-18
Filing date: 2022-03-18
Publication date: 2024-06-25
Anticipated expiration: 2042-03-18
Also published as: CN114647578A

Abstract

The disclosure provides a system testing method, device, equipment and storage medium, and relates to the technical field of computer software, in particular to the technical field of software testing. The specific implementation scheme is as follows: obtaining a target data processing strategy of a service provided by a system to be tested and strategy parameters of the target data processing strategy; generating a first matched tuple set covering all target data processing strategies and all strategy parameters, wherein each first matched tuple comprises the data processing strategies and the strategy parameters, and the included strategy parameters are the strategy parameters of all the included data processing strategies; obtaining a test sample for each first matching tuple in the set of first matching tuples; and testing the system to be tested by adopting the obtained test sample. By applying the scheme provided by the embodiment of the disclosure to test the system, the coverage rate of the data processing strategy can be ensured.

Description

System testing method, device, equipment and storage medium

Technical Field

The present disclosure relates to the field of computer software technology, and in particular, to the field of software testing technology.

Background

With the development of software technology, various systems have been developed that are capable of providing a variety of services to their users. Each service realizes data processing according to the corresponding data processing strategy. In the system application process, a developer may improve the system due to factors such as system demand change and problems, and in order to ensure that the improved system can normally operate, the improved system needs to be tested before being on line.

In the related art, part of data is generally randomly selected from a large amount of test data to be used as a test sample, and the system is tested.

Disclosure of Invention

The present disclosure provides a system testing method, apparatus, device, and storage medium.

According to an aspect of the present disclosure, there is provided a system testing method, including:

Obtaining a target data processing strategy of a service provided by a system to be tested and strategy parameters of the target data processing strategy;

Generating a first matched tuple set covering all target data processing strategies and all strategy parameters, wherein each first matched tuple comprises the data processing strategies and the strategy parameters, and the included strategy parameters are the strategy parameters of all the included data processing strategies;

for each first matching tuple in the set of first matching tuples, a test sample is obtained.

And testing the system to be tested by adopting the obtained test sample.

According to another aspect of the present disclosure, there is provided a system testing apparatus, comprising:

the strategy and parameter obtaining module is used for obtaining a target data processing strategy and strategy parameters of the target data processing strategy of the service provided by the system to be tested;

The set generation module is used for generating a first matched tuple set covering all target data processing strategies and all strategy parameters, wherein each first matched tuple comprises the data processing strategies and the strategy parameters, and the included strategy parameters are the strategy parameters of all the included data processing strategies;

a first test sample obtaining module, configured to obtain a test sample for each first matching tuple in the first matching tuple set;

And the system testing module is used for testing the system to be tested by adopting the obtained test sample.

According to another aspect of the present disclosure, there is provided an electronic device including:

at least one processor; and

A memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the system testing method described above.

According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the above-described system test method.

According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the system test method described above.

From the above, when the scheme provided by the embodiment of the present disclosure is applied to perform system testing, the test sample used is not randomly selected in the existing test data, but after the first set of matching tuples covering all the target data processing policies and all the policy parameters is generated, the test sample obtained for each matching tuple in the first set of matching tuples is obtained, so that the obtained test sample can cover the target data processing policies and all the policy parameters contained in each matching tuple in the first set of matching tuples. On the basis, the system to be tested is tested by using the test sample generated in the mode, and the system to be tested can be tested aiming at each target data processing strategy, so that the coverage rate of the data processing strategy in the test process can be effectively ensured.

It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.

Drawings

The drawings are for a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

FIG. 1 is a flow chart of a first system testing method provided by an embodiment of the present disclosure;

FIG. 2 is a flow chart of a second system testing method provided by an embodiment of the present disclosure;

FIG. 3 is a flow chart of a third system testing method provided by an embodiment of the present disclosure;

FIG. 4 is a flow chart of a fourth system testing method provided by an embodiment of the present disclosure;

FIG. 5 is a schematic diagram of a system testing device according to an embodiment of the present disclosure;

fig. 6 is a block diagram of an electronic device for implementing a system test method of an embodiment of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

Application scenarios of the embodiments of the present disclosure are described first below.

In one scenario, the map application may display information of the place on the map, such as the name, address, etc., to the user. This information may be provided through a map POI (Point of Interest ) system. Specifically, the map POI system may be an LBS (Location Based Services, location-based service) map POI system. The map POI system can provide a plurality of services to the map application software, and each service has different data processing strategies for processing data to obtain information of places displayed to a user by the map application software. When the map POI system is changed, testing is required to be carried out on the changed content in the map POI system, and the map POI system can normally operate after the change. For example, a new service is added to the map POI system in one upgrade process, and the new service has a new data processing policy, so that the system test method provided by the scheme can be used for testing the map POI system with the new service, and whether the new service can normally execute the data processing policy in the operation process of the map POI system is detected.

Therefore, in the above scenario, the system to be tested is a map POI system, so that the map POI system can apply the system test scheme disclosed in the embodiment of the present disclosure to perform system test. Under the condition, whether the service in the map POI system can normally run or not can be detected, and further normal operation of the map POI system is guaranteed.

Different systems to be tested may provide different services. In another scenario, the system to be tested may be a campus information management system. The service provided by the system to be tested can be a course selection service provided for students, etc. The service of the campus information management system may be changed, for example, in the course selection module of the system, a new course is selected and selected in a new learning period, so that the course selection service is changed, and the changed course selection service can provide new selectable courses for students. In this case, the campus information management system after service change can also be tested by using the scheme provided by the embodiment of the disclosure, and whether the changed service can normally run or not is detected.

It should be noted that, the embodiment of the present disclosure only uses the above examples as an application scenario, but is not limited to a specific application scenario of the embodiment of the present disclosure, and a system capable of providing services to the outside may be used as a system to be tested and tested by using the solution provided by the embodiment of the present disclosure.

Concepts related to the embodiments of the present disclosure are described below.

Data processing strategies: the data processing policy may be understood as an implementation manner of a service provided by the system, and may be a screening policy, a calculation policy, and the like of data. For example, if the system to be tested is a map POI system, the service provided by the map POI system may be to generate field information about a target location on a map. The field information may indicate the name of the place, for example, the scenic spot names such as "hometown", "jojoss", or the address of the place; in this case, the data processing policy of the service may be: the most accurate description of each field information of the target site is screened from the map POI data in the map POI system, for example, the most accurate description of the name field of the target site screened from the field data of 'hometown', 'museum', 'palace' and the like in the original map POI data is as follows: "Imperial palace".

Policy parameters: policy parameters are input parameters used when executing a data processing policy, and the parameters may have different parameter values. In the system testing process, test samples with different parameter values can be constructed according to the strategy parameters, and then the operation condition of service under the condition that the strategy parameters take the different parameter values can be tested. One data processing policy may have a plurality of policy parameters, and one policy parameter may also be an input parameter of the plurality of data processing policies, respectively. Different data processing policies may have different kinds of policy parameters, and thus the kind of policy parameters that the data processing policies have reflects the characteristics of the data processing policies.

The system test method provided by the embodiment of the present disclosure is described in detail below through specific embodiments.

In one embodiment of the present disclosure, referring to FIG. 1, a flow diagram of a first system testing method is provided, the method comprising the following steps S101-S104.

Step S101: and obtaining the target data processing strategy of the service provided by the system to be tested and the strategy parameters of the target data processing strategy.

The target data processing policy in this step may be a preset data processing policy that needs to be covered by the test sample under test.

In one embodiment of the present disclosure, the target data processing policy includes a data processing policy for a service for which an update exists in the system to be tested. During the process of upgrading and maintaining the system to be tested, the system operation staff can change the functions of the existing service in the system or add new service, and the changed or added service can be called as the service with updated existence. In this case, the data processing policy of the service for which the update exists may be regarded as the target data processing policy. In this case, the data processing policy of the updated service is covered, so that the situation that the updated service executes the data processing policy of the updated service can be tested, and the normal operation of the updated service can be ensured.

In another embodiment of the present disclosure, the target data processing policy includes a data processing policy for a specified service in the system to be tested. The specified service may be a preset service that needs to be tested each time the system is tested, such as a service that provides core functionality in the system. In this case, the data processing policy of the specified service is covered, so that the condition that the specified service executes the data processing policy of the specified service after the system to be tested is updated can be tested, so as to detect the possible abnormal influence of the update of the system to be tested on the specified service.

In yet another embodiment of the present disclosure, the target data processing policy may include a data processing policy of a service with an update, and a data processing policy of a specified service, so that when testing, the data processing policy of the service with an update and the data processing policy of the specified service are overlaid in the system to be tested, the types of the tested services are more, the types of the overlaid data processing policies are more, and the test is more comprehensive.

The policy parameters of the target data processing policy are input parameters used when executing the target data processing policy. One target data processing policy may have a plurality of policy parameters, and one policy parameter may be input to the plurality of target data processing policies, respectively.

Step S102: a first set of matching tuples is generated that cover all target data processing policies and all policy parameters.

Each first matching tuple comprises a data processing policy and policy parameters, and the included policy parameters are the policy parameters of all the included data processing policies. The first matching tuple can comprise a plurality of data processing strategies at the same time or a plurality of strategy parameters at the same time, and any strategy parameter in the same first matching tuple is the input of any data processing strategy.

For example, a first matching tuple formed by the data processing policies A1, A2 and the policy parameters B1 may be represented by (A1, A2, B1), where B1 is both the policy parameters of A1 and A2. The first matching tuple formed by the data processing policy A1 and the policy parameters B1, B2 may be denoted (A1, B2), the policy parameters of A1 comprising both B1 and B2.

The first set of matching tuples may include one or more first matching tuples.

If any target data processing policy exists in at least one first matching tuple in the first set of matching tuples, policy parameters of any target data processing policy also exist in at least one first matching tuple in the first set of matching tuples, the first set of matching tuples may be considered to cover all target data processing policies and all policy parameters.

The specific generation of the first set of matching tuples may be referred to in the following steps in the embodiment of fig. 2, which is not described in detail here.

Alternatively, the first set of matching tuples may be generated by an exhaustive approach. Specifically, a set of all first matching tuples generated as follows may be regarded as a first matching tuple set:

Assuming M target data strategies and N strategy parameters;

for m=1, 2, … … M, the following steps are performed: selecting m target processing strategies, sharing Seed case;

For n=1, 2, … … N, the following steps are performed: selecting n policy parameters, sharing Seed case;

And forming a first matching tuple based on the m target processing strategies selected in each case and the n strategy parameters selected in each case.

Step S103: for each first matching tuple in the set of first matching tuples, a test sample is obtained.

The test sample may be data that can trigger execution of different data processing policies and can provide policy parameters corresponding to the data processing policies.

For each first matching tuple, data meeting the sample condition may be obtained as a test sample. The sample condition described above refers to that the selected data can trigger any one of the data processing policies in the first matching tuple to be executed and can provide all of the policy parameters in the first matching tuple.

Specifically, the test sample obtaining mode may be: and selecting data meeting the sample conditions from the existing test data as a test sample, or generating the data meeting the sample conditions as the test sample.

In generating test samples for each first matching tuple, the test samples can be generated for some first matching tuple components and the test samples can not be successfully generated for other first matching tuples under the influence of different triggering conditions of the data processing strategy, different existence conditions of the strategy parameters and the like.

The first matching tuple that failed to generate the test sample is described below.

In one case, if the first matching tuple includes mutually exclusive data processing policies, that is, there is a data processing policy that cannot be executed simultaneously, the number of samples obtained for the first matching tuple is 0.

For example, the data processing policy A1 and the data processing policy A2 have a common policy parameter B1, in which case the first matching tuple formed may be represented by (A1, A2, B1). If B1 is a preset threshold, A1 represents data processing of test data when the number of test data is higher than the B1 threshold, and A2 represents data processing of test data when the number of test data is lower than the B1 threshold, in which case the number of samples that can be obtained for (A1, A2, B1) is 0 because A1 and A2 are mutually exclusive and cannot be executed by the same test sample.

In another case, if the first matching tuple includes mutually exclusive policy parameters, that is, policy parameters that cannot be used simultaneously, the number of samples obtained for the first matching tuple may be 0.

For example, in the first matching tuple (A1, B2), A1 is a data processing policy, B1, B2 are two policy parameters available for A1, and if A1 presets a limit on the number of policy parameters, e.g. only one policy parameter is allowed to be input each time A1 is executed, B1, B2 in the first matching tuple are mutually exclusive, so the number of samples that can be obtained for (A1, B2) is 0.

To reduce this, it may be considered when generating the first set of matching tuples whether there is a mutually exclusive target data processing policy or policy parameters of the target data processing policy in the first matching tuple. See also the following description of the specific manner in which the initial matching tuple is processed at step S203 in fig. 2, which will not be described in detail here.

Step S104: and testing the system to be tested by adopting the obtained test sample.

The specific test mode can be as follows: and constructing a test request by taking the data in the obtained test sample as parameters, sending the test request to a system to be tested, obtaining and operating the test sample by the system to be tested according to the test request, and obtaining corresponding output data as a test result.

In one implementation, the test sample can be used to test the system to be tested before and after updating, and the difference of different output data obtained by the system to be tested before and after updating according to the operation of the test sample is compared as a test result. Such a test method may be referred to as a Diff (Difference) test. The specific flow of such a test method can be referred to as an embodiment shown in fig. 4 below.

After determining the target data processing policy, the first set of matching tuples may be obtained in a different manner, in view of which case the disclosed embodiments provide another system testing method.

Specifically, referring to fig. 2, a flow chart of a second system testing method is provided, which includes the following steps S201-S205.

Step S201: and obtaining the target data processing strategy of the service provided by the system to be tested and the strategy parameters of the target data processing strategy.

The step S201 is the same as the step S101, and will not be described in detail here.

Step S202: and obtaining an initial matching tuple corresponding to each target data processing strategy.

Wherein each initial matching tuple comprises: a target data processing policy and a policy parameter of the target data processing policy. For example, the initial matching tuple formed by the policy parameters B1 of the target data processing policies A1 and A1 may be denoted (A1, B1), with the visible target data processing policies present in their corresponding initial matching tuples. In addition, if the target data processing policy A1 also has a policy parameter B2, the obtained initial matching tuple also includes (A1, B2).

Since each target data processing policy has its corresponding initial matching tuple in the above step, and each target data processing policy and any policy parameter thereof can form an initial matching tuple, all obtained initial matching tuples ensure that each target data processing policy exists in at least one initial matching tuple, each policy parameter also exists in at least one initial matching tuple, and all obtained initial matching tuples cover all target data processing policies and policy parameters.

Step S203: and processing the initial matching tuple to which each first object belongs to obtain a first matching tuple set.

Wherein the first object is: target data processing policies or policy parameters.

According to the description in step S202, all the initial matching tuples cover all the target data processing policies and policy parameters, and each of the target data processing policies has matching tuples with at least one of the policy parameters, and each of the policy parameters has matching tuples with at least one of the target data processing policies, so that the process of processing the initial matching tuples to which each of the first objects belongs can cover all of the target data processing policies and policy parameters with the target data processing policies or policy parameters as the first objects.

Specifically, the step S203 may process the initial matching tuple to which each first object belongs in the following manner:

Determining each initial matching tuple containing the first object as a matching tuple to be combined; merging the matched tuples to be merged, which are not mutually exclusive, of the second object to obtain a new matched tuple, wherein the second object is: information in each initial matching tuple other than the first object.

The information contained in the initial matching tuple is a target data processing strategy and a strategy parameter of the target data processing strategy, so when the first object is the target data processing strategy, the second object is a strategy parameter of the target data processing strategy; when the first object is a policy parameter, the second object is a target data processing policy to which the policy parameter belongs. In the two cases, the process of combining the matched tuples to be combined to obtain the new matched tuple is the same, and is not influenced by the selection modes of the first object and the second object. The above-described merging process will be described below by taking a first object as a policy parameter of a target data processing policy and a second object as an example of a data processing policy.

In one embodiment of the present disclosure, the target data processing policies may be consolidated with reference to policy parameters. Specifically, for a policy parameter, all target data processing policies with which a matching tuple exists may be determined, and all target data processing policies that are not mutually exclusive are placed in the same matching tuple, so as to obtain the new matching tuple. And then, for the target data processing strategies which do not form the new matching tuple, returning to execute the step of placing all the target data processing strategies which are not mutually exclusive in the same matching tuple to obtain the new matching tuple until all the target data processing strategies are used for forming the new matching tuple or the rest of the target data processing strategies cannot generate the new matching tuple with other target data processing strategies.

The process of obtaining the first set of matching tuples is illustrated below.

For example, for policy parameter B1, the four matching tuples to be merged are (A1, B1), (A2, B1), (A3, B1), (A4, B1), respectively, whereby all target data processing policies for which there are matching tuples with B1 can be determined as: a1, A2, A3, A4; it is assumed that the data processing policy A1 and the data processing policy A2 are mutually exclusive, and that no mutually exclusive relationship exists between the other data processing policies. Then A1, A3, A4 can be selected to be placed in the same matching tuple, and the policy parameters B1 for the process are added to obtain (A1, A3, A4, B1), namely a new matching tuple.

In the above example, the target data processing policy that does not form a new matching tuple is A2 only, and no new matching tuple can be generated with other data processing policies, so far as the matching tuple is combined and ended.

In this case, the resulting matching tuple comprises: the new matching tuple (A1, A3, A4, B1) and the initial matching tuple (A2, B1), i.e. the first matching tuple set comprises (A1, A3, A4, B1) and (A2, B1).

Step S204: for each first matching tuple in the set of first matching tuples, a test sample is obtained.

Step S205: and testing the system to be tested by adopting the obtained test sample.

Steps S204 to S205 are the same as steps S103 to S104, respectively, and will not be described in detail here.

From the above, each matching tuple in the first matching tuple set obtained by the scheme provided by the embodiment of the disclosure is obtained by combining initial matching tuples for each first object, and each obtained matching tuple contains one first object, so that the mutual exclusion problem between the first objects does not exist, and each obtained matching tuple is obtained by combining to-be-combined tuples which are not mutually exclusive by the second objects, so that the mutual exclusion problem between the second objects does not exist. In this case, the first object and the second object in each matching tuple in the first matching tuple set have no mutual exclusion problem, and all the first object and the second object can be covered in one test sample, so that the success rate of obtaining the test sample is improved.

Based on the first set of matching tuples comprising all new matching tuples in the above-described embodiments, a specific implementation of obtaining the test sample is described below.

In one embodiment of the present disclosure, a preset number of test samples may be obtained for each first matching tuple in the set of first matching tuples.

The specific method for obtaining the test sample is similar to the aforementioned step S103, except that the obtained test sample has a number requirement, which is not described herein.

According to the statistical principle, when the number of samples is insufficient, the obtained experimental result is insufficient in effectiveness. When the number of test samples is small, the validity of the obtained test results is also low, so that the number of test samples to be achieved can be preset, for example, 50 test samples, or 100 test samples, etc. are obtained for each matching tuple. The number of the obtained test samples reaches the preset number, so that the sufficient number of the test samples can be ensured, and the effectiveness of the test result is further improved.

For some first matching tuples, for reasons such as limited number of existing test data, when data meeting sample conditions is obtained as test samples for such first matching tuples, the number of obtained test samples cannot reach a preset number, and in this case, the test samples can be supplemented.

In view of the foregoing, embodiments of the present disclosure provide another system testing method. Specifically, referring to fig. 3, a flow chart of a third system testing method is provided, which includes the following steps S301-S309.

Step S301: and obtaining the target data processing strategy of the service provided by the system to be tested and the strategy parameters of the target data processing strategy.

Step S302: and obtaining an initial matching tuple corresponding to each target data processing strategy.

Step S303: and processing the initial matching tuple to which each first object belongs to obtain a first matching tuple set.

Step S304: for each first matching tuple in the set of first matching tuples, a test sample is obtained.

Steps S301 to S304 correspond to steps S201 to S204, respectively, and will not be described in detail here.

Step S305: and judging whether target matching tuples with the number of the obtained test samples being smaller than a preset number exist or not. If so, the following step S306 is executed, and if not, the flow is ended.

If the number of test samples obtained for the matching tuple in the first matching tuple set is smaller than the preset number, the number of test samples obtained for the matching tuple is insufficient, and for convenience of description, the matching tuple is referred to as a target matching tuple.

Step S306: a second set of matching tuples is obtained.

And each matching tuple in the second matching tuple set integrally covers the data processing strategy and strategy parameters included in the target matching tuple. That is, any target data processing policy in the target matching tuple is present in at least one matching tuple in the second set of matching tuples, and any policy parameter in the target matching tuple is present in at least one matching tuple in the second set of matching tuples.

Each matching tuple in the set of second matching tuples may be derived based on a full combination of all second objects in the target matching tuple and the first objects in the target matching tuple. Assuming that there are K second objects in the target matching tuple, the full combination of the second objects is: a combination of 1 second object is taken, a combination of 2 second objects is taken, and a combination of K second objects is taken at … …. Adding the first object to all the combinations formed, i.e. forming the matching tuples of the second set of matching tuples.

For example, the target matching tuple is (A1, A2, A3, B1), B1 is a first object, A1-A3 is a second object, a combination of 1 second objects is (A1), (A2), (A3), a combination of 2 second objects is (A1, A2), (A2, A3), (A1, A3), and a combination of 3 second objects is (A1, A2, A3) from all second objects.

In the above case, each combination and the first object, i.e. B1, may form a matching tuple.

The matching tuples formed by combining 1 object are: (A1, B1), (A2, B1), (A3, B1);

the matching tuples formed by combining 2 objects are: (A1, A2, B1), (A2, A3, B1), (A1, A3, B1);

The matching tuples formed by taking 3 objects are: (A1, A2, A3, B1) for which the target matches the tuple.

It can be seen that all the second objects A1, A2, A3 are covered for the first object B1 according to the matched tuples obtained by the full combination, and thus the formed second matched tuple set is consistent with the first object and the second object covered by the target matched tuple.

Specifically, the matching tuples in the second set of matching tuples may be determined with reference to the steps in the following embodiments, which are not described in detail herein.

Step S307: for each matching tuple in the second set of matching tuples, a preset number of test samples is obtained.

The manner of obtaining the test samples in this step is similar to the foregoing step S103, and the difference is only that the set of matching tuples is different, and the obtained test samples have a number requirement, which is not described here again.

Step S308: and judging whether the obtained test sample meets the preset sample requirement, and if not, returning to the step S306.

The sample requirements may be set for the test samples obtained for each matching tuple, or may be set for all the obtained test samples, which is not limited by the embodiments of the present disclosure.

For example, for each matching tuple, the number of obtained test samples reaches a preset number, and it is determined that the number of test samples obtained for the matching tuple meets the preset sample requirement, otherwise, the preset sample requirement is not met; or for all the matching tuples, the total number of the obtained test samples reaches the preset number, and the obtained test samples are judged to meet the preset sample requirement, otherwise, the preset sample requirement is not met.

Step S309: and testing the system to be tested by adopting the obtained test sample.

Step S309 is the same as step S205, and will not be described in detail here.

As can be seen from the above, in the solution provided by the embodiment of the present disclosure, when the number of test samples obtained according to the target matching tuple is insufficient, a new test sample may be obtained according to the matching tuple in the second matching tuple set, and the new test sample also covers the data processing policy and the policy parameters in the target matching tuple, so that the number of obtained test samples is sufficient under the condition of ensuring coverage rate.

In the foregoing step S306, the matching tuples in the second matching tuple set may be determined according to at least one of the following information:

the combined priority of the second object, the number of matching tuples in the set.

Determining a second matching tuple set according to the combination priority of the second objects, wherein the second matching tuple set can be formed by preferentially selecting the matching tuples containing the second objects with high combination priority and selecting the matching tuples containing all the second objects according to the priority order; wherein, the combination priority of each second object may be preset.

Continuing with the example of step S306, (A1, A2, B1), (A2, A3, B1), (A1, A3, B1) the highest combined priority of the second objects A1, A2, A3 is preset, the combined priority of the second objects A2, A3 is again, and (A1, A2, B1) may be selected first, and the matching tuple does not cover the second object A3, and then the matching tuple (A3, B1) is selected to form a second set of matching tuples { (A1, A2, B1), (A3, B1) }.

If the number of test samples obtained according to the matching tuples in the formed second matching tuple set is smaller than the preset number, and the test samples need to be supplemented, then (A2, A3, B1) may be selected again according to the priority order, and the second matching tuple set { (A2, A3, B1), (A1, B1) } is formed under the condition that coverage rate of the first object and the second object is ensured.

If all the second objects are covered, the selected matching tuples can form a second matching tuple set; otherwise, a new second set of matching tuples … … may be selected again (A1, A3, B1) according to the priority order, in the manner described above, until a predetermined number of matching tuples may be satisfied from the matching tuples in the formed second set of matching tuples.

Therefore, according to the combination priority of the second objects, the matching tuple set covering all the second objects can be selected, so that the coverage rate of the matching tuple set is ensured, and the second objects with high combination priority can be covered by the same matching tuple with higher probability, namely, the second objects with high combination priority can be tested by the same test sample, so that the test result can more probably reflect the influence of the combination of the second objects with high priority on the system to be tested.

The second set of matching tuples is determined according to the number of matching tuples in the set, and the second set of matching tuples with the minimum number of matching tuples is preferably selected under the condition that the whole coverage of each matching tuple in the second set of matching tuples by the data processing strategy and strategy parameters included in the target matching tuple is ensured.

For example, in the case where all the data processing policies included in the target matching tuple are A1, A2, A3, A4, and all the policy parameters included in the target matching are B1, set 1{ (A1, A2, A3, B1), (A4, B1) } and set 2{ (A1, A2, B1), (A3, B1), (A4, B1) } cover all the data processing policies and policy parameters in the target matching tuple, and set 1 is preferentially selected as the second matching tuple set because the number of matching tuples in set 1 is 2 and the number of matching tuples in set 2 is 3.

From the above, the second matching tuple set can be determined according to the number of the matching tuples, so that the number of the matching tuples in the determined second matching tuple set is smaller, the number of the obtained test samples is smaller, the test cost is reduced, and the test efficiency is improved.

Referring to fig. 4, taking the Diff test described in the foregoing step S104 as an example, the flow of the entire system test method will be described. In this embodiment, the system to be tested is a map POI system.

In the left flow, as shown in fig. 4, the environment construction platform is used for constructing two sets of environments required by Diff test, namely, a reference environment in the figure, for running the system to be tested before updating, and a test environment in the figure, for running the system to be tested after updating. Specifically, different configuration parameters can be configured for a preset environment construction tool, so that the environment construction tool generates a reference environment and a test environment, and the reference environment and the test environment are off-line environments. Under the condition, the test environment is not required to be built manually, and the validity and the correctness of the environment are ensured by configuring the corresponding configuration parameters for different environments, so that errors caused by manual operation are reduced.

The online data synchronization module and BigPipe intelligent management system are used for acquiring information required for building the environment, so that the environment building tool can complete environment building by using the acquired information. The online data synchronization module can acquire the information of the environment of the running online map POI system for the use of the offline reference environment and the offline test environment.

The newly added scene data filling module and the system data timing filling module in the figure are used for providing data required by system operation for the system to be tested in the two environments. The data to be tested in different test environments can be initialized by the filled data, so that the data processing service in the system to be tested can run according to the filled data, for example, the filled data is adopted to select the most accurate description for the place on the map, and the like. And the system to be tested is operated in the set test environment and is isolated from the on-line map POI system, so that on-line data can be prevented from being polluted.

In addition, the newly added scene data represents information of a place newly added on the map, and is used for testing functions of newly added services in the map POI system. The service obtains data representing the newly added location on the map from the data that is filled in, and forms a field representing the newly added location accordingly. In the system to be tested, a deleting function module can be preset, so that the executing function module is triggered after the test is finished, and field data of a new place are cleared. The stability of the original data in the system is not affected by the test.

The system data timing filling module can copy the POI data actually used from the running on-line POI system, and the acquired data can be stored in a classified manner according to different places described by the POI data. Wherein, the classification storage can be: the information of the different classifications is stored in different storage devices.

After the new scene data filling module and the system data timing filling module acquire POI data, the data are input into the system to be tested in the reference environment and the test environment, namely the system to be tested before and after updating. The execution pipeline module is used for running the two systems to be tested.

In the right flow, the production scene module and the branch coverage module are used for analyzing and obtaining the data processing strategy in the map POI system, and the strategy characteristic module is used for obtaining the strategy parameters of the data processing strategy. The matching tuple generating tool then generates the first set of matching tuples and the second set of matching tuples using the methods mentioned in the previous embodiments. From the obtained first set of matching tuples and second set of matching tuples, test samples, i.e. cured samples and specialized samples in the graph, can be obtained. In the above process, if the data processing policy and policy parameters for obtaining the first matching tuple set and the second matching tuple set belong to the service with update, the generated test sample is a special sample in the graph; if the data processing strategy and strategy parameters used for obtaining the first matching tuple set and the second matching tuple set belong to the specified service, the generated test sample is a solidified sample in the graph;

The Diff platform is used for testing the systems to be tested before and after updating, which run on the execution pipeline, according to the requests of the solidified sample and the special sample structure, namely, the two systems to be tested are enabled to output test results after running based on the requests of the solidified sample and the special sample structure, and the difference of the test results output by the systems to be tested before and after updating is compared, so that the Diff rate is obtained; if the Diff rate exceeds the preset threshold, the influence of the change existing in the system to be tested before and after updating is manually analyzed, otherwise, the Diff test is determined to pass.

Corresponding to the system testing method, the embodiment of the disclosure also provides a system testing device.

In one embodiment of the present disclosure, referring to fig. 5, there is provided a schematic structural diagram of a system testing device, including:

A policy and parameter obtaining module 501, configured to obtain a target data processing policy and a policy parameter of the target data processing policy of a service provided by a system to be tested;

A set generating module 502, configured to generate a first matching tuple set covering all target data processing policies and all policy parameters, where each first matching tuple includes a data processing policy and a policy parameter, and the included policy parameter is a policy parameter of all included data processing policies;

a first test sample obtaining module 503, configured to obtain a test sample for each first matching tuple in the first matching tuple set;

And the system testing module 504 is used for testing the system to be tested by adopting the obtained test sample.

In one embodiment of the present disclosure, the set generating module 502 includes:

A matching tuple obtaining unit, configured to obtain an initial matching tuple corresponding to each target data processing policy, where each initial matching tuple includes: a target data processing policy and a policy parameter of the target data processing policy;

the set obtaining unit is configured to process an initial matching tuple to which each first object belongs to obtain a first matching tuple set, where the first object is: target data processing policies or policy parameters:

In one embodiment of the present disclosure, the first test sample obtaining module 503 is specifically configured to obtain a preset number of test samples for each first matching tuple in the first matching tuple set.

The number of the obtained test samples reaches the preset number, so that the sufficient number of the test samples can be ensured, and the effectiveness of the test result is further improved.

In one embodiment of the present disclosure, the apparatus further comprises:

the set obtaining module is used for obtaining a second set of matching tuples if target matching tuples with the number of the obtained test samples being smaller than the preset number exist, wherein each matching tuple in the second set of matching tuples integrally covers a data processing strategy and strategy parameters included in the target matching tuple;

A second test sample obtaining module, configured to obtain the preset number of test samples for each matching tuple in the second matching tuple set;

and the sample requirement judging module is used for triggering the set obtaining module to re-obtain the second matched tuple set if the obtained test sample does not meet the preset sample requirement.

In one embodiment of the present disclosure, the matching tuples of the second set of matching tuples are determined from at least one of the following information:

And determining a second matched tuple set according to the number of the matched tuples, so that the number of the matched tuples in the determined second matched tuple set is smaller, the number of the obtained test samples is smaller accordingly, the test cost is reduced, and the test efficiency is improved.

In one embodiment of the present disclosure, the target data processing policy includes: a data processing strategy of updated service exists in the system to be tested;

And/or the number of the groups of groups,

The target data processing strategy comprises the following steps: and designating a data processing strategy of the service in the system to be tested.

The data processing strategy of the updated service is covered, so that the condition that the updated service executes the data processing strategy of the updated service can be tested, and the normal operation of the updated service can be ensured.

The data processing strategy of the appointed service is covered, so that the condition that the appointed service executes the data processing strategy after the system to be tested is updated can be tested, and the abnormal influence possibly caused by the updating of the system to be tested on the appointed service is detected.

When testing, the data processing strategy of the updated service and the data processing strategy of the appointed service in the system to be tested are covered, the types of the tested services are more, the types of the covered data processing strategies are more, and the testing is more comprehensive.

In one embodiment of the disclosure, the system to be tested is a map point of interest POI system.

In this case, the test may be used to determine whether the service in the map POI system can be operated normally, thereby ensuring the normal operation of the POI system.

According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

In one embodiment of the present disclosure, there is provided an electronic device including:

at least one processor; and

A memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the system test method described in the method embodiments above.

In one embodiment of the present disclosure, a non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the system test method described in the foregoing method embodiments is provided.

In one embodiment of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the system test method described in the foregoing method embodiments.

Fig. 6 illustrates a schematic block diagram of an example electronic device 600 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 6, the apparatus 600 includes a computing unit 601 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 602 or a computer program loaded from a storage unit 608 into a Random Access Memory (RAM) 603. In the RAM603, various programs and data required for the operation of the device 600 may also be stored. The computing unit 601, ROM602, and RAM603 are connected to each other by a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.

Various components in the device 600 are connected to the I/O interface 605, including: an input unit 606 such as a keyboard, mouse, etc.; an output unit 607 such as various types of displays, speakers, and the like; a storage unit 608, such as a magnetic disk, optical disk, or the like; and a communication unit 609 such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.

The computing unit 601 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 601 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as a system test method. For example, in some embodiments, the system testing method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 600 via the ROM 602 and/or the communication unit 609. When the computer program is loaded into RAM 603 and executed by computing unit 601, one or more steps of the system test method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the system test method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.

The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel, sequentially, or in a different order, provided that the desired results of the disclosed aspects are achieved, and are not limited herein.

The above detailed description should not be taken as limiting the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present disclosure are intended to be included within the scope of the present disclosure.

Claims

1. A system testing method, comprising:

generating a first matching tuple set covering all target data processing strategies and all strategy parameters, wherein each first matching tuple comprises a data processing strategy and strategy parameters, the included strategy parameters are strategy parameters of all the included data processing strategies, the strategy parameters are input parameters used when the data processing strategies are executed, and the target data processing strategies comprise data processing strategies with updated services and/or data processing strategies with appointed services in the system to be tested;

Obtaining a test sample for each first matching tuple in the set of first matching tuples;

and testing the system to be tested by adopting the obtained test sample.

2. The method of claim 1, wherein the generating a first set of matching tuples covering all target data processing policies and all policy parameters comprises:

Obtaining an initial matching tuple corresponding to each target data processing strategy, wherein each initial matching tuple comprises: a target data processing policy and a policy parameter of the target data processing policy;

Processing the initial matching tuple to which each first object belongs to obtain a first matching tuple set, wherein the first objects are as follows: target data processing policies or policy parameters:

Determining each initial matching tuple containing the first object as a matching tuple to be combined;

merging the matched tuples to be merged, which are not mutually exclusive, of the second object to obtain a new matched tuple, wherein the second object is: information in each initial matching tuple other than the first object.

3. The method of claim 2, wherein the obtaining a test sample for each first matching tuple in the first set of matching tuples comprises:

For each first matching tuple in the set of first matching tuples, a preset number of test samples is obtained.

4. A method according to claim 3, the method further comprising:

If the obtained target matching tuples with the number smaller than the preset number exist, a second matching tuple set is obtained, wherein each matching tuple in the second matching tuple set integrally covers a data processing strategy and strategy parameters included in the target matching tuple;

obtaining the preset number of test samples for each matching tuple in the second set of matching tuples;

and if the obtained test sample does not meet the preset sample requirement, returning to the step of obtaining the second matched tuple set, and re-obtaining the second matched tuple set.

5. The method of claim 4, wherein,

The matching tuples of the second set of matching tuples are determined from at least one of the following information:

6. The method according to any one of claims 1-5, wherein,

The target data processing strategy comprises the following steps: a data processing strategy of updated service exists in the system to be tested;

And/or the number of the groups of groups,

7. The method according to any one of claims 1-5, wherein,

The system to be tested is a map point of interest POI system.

8. A system testing apparatus, comprising:

The system comprises a set generation module, a data processing module and a data processing module, wherein the set generation module is used for generating a first matched tuple set covering all target data processing strategies and all strategy parameters, each first matched tuple comprises a data processing strategy and strategy parameters, the included strategy parameters are strategy parameters of all the included data processing strategies, the strategy parameters are input parameters used when the data processing strategies are executed, and the target data processing strategies comprise the data processing strategies with updated services and/or the data processing strategies with appointed services in the system to be tested;

9. The apparatus of claim 8, wherein the set generation module comprises:

10. The apparatus of claim 9, wherein,

The first test sample obtaining module is specifically configured to obtain a preset number of test samples for each first matching tuple in the first matching tuple set.

11. The apparatus of claim 10, the apparatus further comprising:

12. The apparatus of claim 11, wherein,

13. The device according to any one of claims 8-12, wherein,

And/or the number of the groups of groups,

14. The device according to any one of claims 8-12, wherein,

The system to be tested is a map point of interest POI system.

15. An electronic device, comprising:

at least one processor; and

A memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

16. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-7.

17. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any of claims 1-7.