WO2020038549A1

WO2020038549A1 - Test vector generation method for software-based control of a technical system

Info

Publication number: WO2020038549A1
Application number: PCT/EP2018/072444
Authority: WO
Inventors: Andrés Botero Halblaub; Jan Richter; Jan Götz; Jan Fischer
Original assignee: Siemens Aktiengesellschaft
Priority date: 2018-08-20
Filing date: 2018-08-20
Publication date: 2020-02-27

Abstract

An object of the present application is to provide a method for generating a set of improved test vectors used to assess an operating requirement conformity of a software, which is used to control an operation of a control device configured to control a technical system. Hereby, the method comprises a) determining an environment model describing a physical environment in which the control device and the technical system are used and describing a physical relationship how a parameter value of the technical system behaves under the control of the control device, b) determining at least one operating requirement how the parameter value of the technical system should behave under the control of the control device, c) determining algebraic equations in which the determined environment model and the operating requirements are present as constraints, d) determining a first test vector representing a first behavior of the parameter value of the technical system, and e) solving the algebraic equations taking into account the first test vector comprising solving an optimization problem, the solving providing as a result the set of improved test vectors, the set of improved test vectors representing a second behavior of the parameter value representing an ideal behavior the software should fulfill.

Description

Test vector generation method for software-based control of a technical system

TECHNICAL FIELD

Various embodiments of the invention enable techniques for generating improved test vectors that enable software tests of a software used to control an operation of a control sys tem configured to control a technical system.

BACKGROUND

For reliably enabling a desired behavior of technical systems controllable by a control device, software tests of a soft ware intended to be used by the control device of the tech nical system are considered essential. However, providing in put stimuli to such a software that investigate any malfunc tion scenario that may potentially occur to the software is a tedious, repetitive and elaborating task taking much working time of an IT specialist. As a matter of consequence, manual generated test vectors by an IT specialist are not considered sufficiently workload efficient.

According to one approach, a manual workload for generating test vectors for a software test is reduced using a randomly operating generator of test vectors. However, due to the ar bitrary character of such test vector generation means, the generated test vectors do not exhibit a sufficient coverage size for potential malfunction scenarios of the software to be tested.

Therefore, there is a demand to provide test vectors for software tests in an efficient manner, which enable a prefer ably complete investigation of the software to be tested. SUMMARY

It is an object of the present disclosure to provide means for improving efficiency of test vector generation, which en ables a preferably complete investigation of a software to be tested .

This object is achieved with the features of the independent claims .

A method for generating a set of improved test vectors as well as well as a corresponding device are provided. Further embodiments are defined in the dependent claims.

According to an embodiment, a method for generating a set of improved test vectors used to assess an operating requirement conformity of a software is disclosed. Hereby, the software is used to control an operation of a control device config ured to control a technical system. The method comprises de termining an environment model describing a physical environ ment in which the control device and the technical system are used and describing a physical relationship how a parameter value of the technical system behaves under the control of the control device. The method further comprises determining at least one operating requirement how the parameter value of the technical system should behave under the control of the control device. The method further comprises determining al gebraic equations in which the determined environment model and the operating requirements are present as constraints.

The method further comprises determining a first test vector representing a first behavior of the parameter value of the technical system. The method further comprises solving the algebraic equations taking into account the first test vector comprising solving an optimization problem, the solving providing as a result the set of improved test vectors, the set of improved test vectors representing a second behavior of the parameter value representing an ideal behavior the software should fulfill. Such an approach is based on the finding that the appearance of test vectors adapted to investigate a software behavior with respect to a plurality of different scenarios may auto matically be optimized by additionally taking the physical environment of a control device adapted to be controlled by the software to be tested into account. Based on these cir cumstances, the test vectors can be customized according to conditions, to which the technical system and the control de vice, respectively may be exposed based on the physical envi ronment including the physical laws. Due to the additional consideration of an operating requirement defining a desired behavior of the technical system, the testing size amount for sufficiently testing the software may be reduced, thereby ad ditionally improving efficiency in test vector generation for software testing.

According to another embodiment, a device for generating a set of improved test vectors useable to access an operating requirement conformity of a software, which is adapted to control an operation of a control device configured to con trol a technical system, the device comprising. The device comprises a first determination unit, which is adapted to de termine an environment model describing a physical environ ment in which the control device and the technical system are used and describing a relationship how a parameter value of the technical system behaves under the control of the control device. The device further comprises a second determination unit, which is adapted to determine at least one operating requirement how the parameter value of the technical system should behave under the control of the control device. The system further comprises a third determination unit, which is adapted to determine algebraic or differential equations in which the determined environment model and the operating re quirements are present as constraints. The device further comprises a fourth determination unit, which is adapted to determine a first test vector representing a first behavior of the parameter value of the technical system. The device further comprises a solvation unit, which is adapted to solve the algebraic equations taking into account the first test vector comprising solving an optimization problem. Hereby, the solving provides as a result the set of improved test vectors, wherein the set of improved test vectors represents a second behavior of the parameter value representing an ide al behavior the software should fulfill.

Such means enable the implementation of the method outlined above in a preferably simple and efficient manner.

A test vector within the meaning of the present disclosure may refer to any stimuli of a software, based on which at least one property of the software may be tested. Hereby, the test vector may cause a respective output response of the software to be tested, based on which the at least one prop erty of the software may be read out. As an example, the at least one test vector may be generated stochastically, in vented by an IT specialist or modified by an automated algo rithm, wherein concerning the latter case, solving an optimi zation problem may be provided.

A technical system within the meaning of the present disclo sure may refer to a system that may be controlled by a con trol device based on technical means. As an example, the technical system may refer to a vehicle, and the control de vice may contribute to control at least on property of the vehicle, such as its velocity.

An environment model within the meaning of the present dis closure may refer to any physical relationship describing an environment in which the software and/or the control device may operate. Such a physical relationship may be based on at least one property of the technical system. The environment model may include a mathematical description of the physical relationship. As an example, such a mathematical description may include at least one of a mathematical equation, a mathe matical inequation and a mathematical differential equation. The environment model may also take laws of physics into ac count .

A parameter value within the meaning of the present disclo sure may refer to any physical value depending on at least one further physical value. Hereby, the dependency of the at least one further physical value may include a time dependen cy. As an example, the parameter value may be a velocity function of the technical system over time.

An operating requirement within the meaning of the present disclosure may refer to any requirement of the technical sys tem during operation, which must be fulfilled or is at least desirable to be fulfilled during operation of the technical system. Hereby, the operating requirement may refer to a be havior of the parameter value.

In an embodiment of the method, solving the optimization problem comprises solving a dynamic optimization problem where the environment model and the operating requirements are used as algebraic constraints.

Thereby, the provision of customized test vectors based on the respective physical environment and the operating re quirements may be investigated in a preferably efficient and accurate manner.

In another embodiment of the method, solving the algebraic equations comprises using a receding horizon calculation in which the set of improved test vectors is determined based on a predictive operation of the technical system.

Using such comparison means in between required properties of a technical systems and corresponding predictions of the sys tem, customizing test vectors may be performed in a prefera bly adaptable manner. In another embodiment of the method, solving the algebraic equations using the receding horizon calculation comprises applying the first test vector to the software.

Thereby, a reference test vector for optimizing customized test vectors is provided. The first test vector provides ini tial condition for an optimization problem to be solved, thereby improving efficiency in solving the optimization problem.

In another embodiment of the method, the solving the algebra ic equations using receding horizon calculation comprises a Mixed Integer Linear Programming or a Mixed Integer Nonlinear Programming approach.

Thereby, an efficient mathematical procedure for solving the receding horizon calculation is provided.

In another embodiment of the method, the receding horizon calculation receives as an input the first test vector and generates as an output the set of improved test vectors such that the at least one operating requirement is met.

In another embodiment of the method, the parameter value re fers to a physical property describing the technical system.

Thereby, a direct correlation in between customization of the test vector and the technical system is enabled, resulting in an improved adaptability of test vectors to a given physical environment .

In another embodiment of the method, determining based on a user input whether the set of improved test vectors should meet the at least one operating requirement or not.

Thereby, a user may control restrictions of a solution space for the generation of test vectors according to the respec tive customer's needs. By reducing such a solution space, the coverage amount for generating test vectors may be improved, thereby the effort for generating essential test vectors is reduced .

In an embodiment of the method, the parameter value is a time-dependent function.

Thereby, the generation of improved test vectors is enabled for taking changes of the physical environment into account. Hereby, the testing of the software comprises an improved ad justability.

In another embodiment of the method, the determining of the first test vector is drawn from a stochastic model.

Due to such an arbitrary generation of initial conditions of test vectors, the different first test vectors exhibit a preferable huge variety in performance. Based on this, the generation of improved test vectors is configured to provide a variety of different testing inputs for a software to be tested, thereby enabling a preferably high coverage size of the software.

In another embodiment, the method further comprising output ting the set of improved test vectors to the software, where in, in response to the set of improved test vectors outputted to the software, it is determined whether the set of improved test vectors meets at least one the operating requirement as predicted or not.

Thereby, the quality of the generated improved test vectors may be evaluated by correlating these improved test vectors with the fulfilling of a certain criteria. Thus, the means for quality control of the generation of improved test vec tors are provided, which may be used for an adjustment of successive test vector generation. In another embodiment of the method, the determining of the first test vector is based on the detected test coverage of at least one test run of the software.

In another embodiment, the method further comprises repeating the test vector generation in order to maximize the test cov erage. In another embodiment of the method, the stochastic model takes into account the detected test coverage to maxim ize it by altering accordingly the parameters of the stochas tic model.

Based on such an approach, test coverage size of a software with respect to potentially occurring scenarios may be maxim ized. Thus, a preferably high number of software bugs may be detected during a software-testing period.

In another embodiment of the method, the technical system un der test refers to at least one of an automotive system, a power station system, a wind turbine and a nuclear power plant .

In another embodiment, the device is adapted to perform the method according to any of embodiments disclosed herein.

Thereby, technical means adapted to execute the method ac cording to any of the embodiments outlined above in a prefer ably simple and efficient manner are provided.

An algebraic constraint within the meaning of the present disclosure may refer to an algebraic relationship in between different parameters restricting a solution space of an opti mization problem.

A predictive operation within the meaning of the present dis closure may refer to may refer to any operation behavior of the technical, which may be predicted based on physical rela tionships. The predicted behavior of the technical system may be based on a calculation taken into account the controlla bility of the technical system by the control device.

The above summary is merely intended to give a short overview over some features of some embodiments and implementations and is not to be construed as limiting. Other embodiments may comprise other features than the ones explained above.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other elements, features, steps and character istics of the present disclosure will be more apparent from the following detailed description of embodiments with refer ence to the following figures:

Figure 1 schematically illustrates a technical system com prising a control device controllable by a software that may tested according to the various examples of the present dis closure .

Figure 2 schematically illustrates a flowchart of a method for generating a set of improved test vectors according to various examples.

Figure 3 schematically illustrates a flowchart of a method for generating a set of improved test vectors based on a software test feedback according to various examples.

Figure 4 schematically illustrates a device for generating a set of improved test vectors according to various examples.

Figures 5 schematically represents the generation of an im proved test vector according to various examples.

Figure 6 schematically represents the generation of an im proved test vector according to various examples. DETAILED DESCRIPTION OF EMBODIMENTS

In the following, embodiments of the invention will be de scribed in detail with reference to the accompanying draw ings. It is to be understood that the following description of embodiments is not to be taken in a limiting sense. The scope of the invention is not intended to be limited by the embodiments described hereinafter or by the drawings, which are taken to be illustrative only.

The drawings are to be regarded as being schematic represen tations and elements illustrated in the drawings, which are not necessarily shown to scale. Rather, the various elements are represented such that their function and general purpose become apparent to a person skilled in the art. Any connec tion or coupling between functional blocks, devices, compo nents, or other physical or functional units shown in the drawings or described herein may also be implemented by an indirect connection or coupling. A coupling between compo nents may also be established over a wireless connection. Functional blocks may be implemented in hardware, firmware, software, or a combination thereof.

Figure 1 schematically illustrates a technical system 4 com prising a control device 3 controllable by a software 2, wherein the latter may be tested according to the various ex amples of the present disclosure. Hereby, the technical sys tem 4 may be controlled by the control device 3. As an exam ple, the device may be a vehicle, and the control device 3 may include an accelerator pedal of the vehicle, which is adapted to contribute to a control of the vehicle's velocity. As a further example the device may be a temperature control system for a heating room comprising a temperature sensor and comprising as control device a valve used to control the heating of the room.

Figure 2 schematically illustrates a flowchart of a method 100 for generating a set of improved test vectors 1 according to various examples. The method 100 may be used to assess an operating requirement 14 conformity of a software 2. Hereby, the software 2 may be used to control an operation of a con trol device 3 configured to control a technical system 4. Ac cording to this, the technical system 4 may refer to at least one of an automotive system, a power station system, a wind turbine and a nuclear power plant.

At 110, the method 100 may comprise determining an environ ment model 13 describing a physical environment in which the control device 3 and the technical system 4 are used and de scribing a physical relationship how a parameter value of the technical system 4 behaves under the control of the control device 3. Hereby, the parameter value may refer to a physical property describing the technical system 4. The parameter value may further be a time-dependent function.

At 120, the method 100 may further comprise determining 120 at least one operating requirement 14 how the parameter value of the technical system 4 should behave under the control of the control device 3.

At 130, the method 100 may further comprise determining alge braic equations 15 in which the determined environment model 13 and the operating requirements 14 are present as con straints .

At 135, which is considered optionally according to the em bodiment of Figure 2, the method 100 may further comprise de termining, based on a user input 6, whether the set of im proved test vectors 1 should meet the at least one operating requirement 14 or not.

At 140, the method 100 may further comprise determining a first test vector 5 representing a first behavior of the pa rameter value of the technical system 4. Further, the deter mining 140 of the first test vector 5 may further be drawn from a stochastic model. At 150, the method 100 may further comprise solving the alge braic equations 15 taking into account the first test vector comprising solving an optimization problem, the solving 150 providing as a result the set of improved test vectors 1, the set of improved test vectors 1 representing a second behavior of the parameter value representing an ideal behavior the software 2 should fulfill. Hereby, solving the optimization problem may comprise solving a dynamic optimization problem where the environment model 13 and the operating requirements 14 may be used as algebraic constraints. Further, solving 150 the algebraic equations 15 may comprise using a receding horizon calculation in which the set of improved test vectors 1 may be determined based on a predictive operation of the technical system 4. Hereby, solving 150 the algebraic equa tions 15 using the receding horizon calculation may comprise applying the first test vector 5 to the software 2 algebraic equations. Further, solving 150 the algebraic equations 15 using receding horizon calculation may comprise a Mixed Inte ger Linear Programming or a Mixed Integer Nonlinear Program ming approach. Further, the receding horizon calculation may receive as an input the first test vector 5 and may generate as an output the set of improved test vectors 1 such that the at least one operating requirement 14 may be met.

At 160, the method 100 may further comprise outputting 160 the set of improved test vectors 1 to the software 2.

Figure 3 schematically illustrates a flowchart of a method 200 for generating a set of improved test vectors 1 based on a software test feedback according to various examples.

According to the embodiments depicted in Figure 3, 210-260 may correspond to 110-160 according to the embodiments de picted in Figure 2.

Additionally at 270, and in response to the set of improved test vectors 1 outputted to the software 2, it may be deter- mined whether the set of improved test vectors 1 meets at least one the operating requirement 14 as predicted or not. Hereby, the determining 240 of the first test vector 5 may be based on a detected test coverage of at least one test run of the software 2.

At 280, the method 200 may further comprise repeating the test vector generation in order to maximize the test cover age. Hereby, the stochastic model may take into account the detected test coverage to maximize it by altering accordingly the parameters of the stochastic model.

Figure 4 schematically illustrates a device 7 for generating a set of improved test vectors 1 according to various exam ples. Hereby, the improved test vectors 1 may be useable to access an operating requirement 14 conformity of a software 2 which may be adapted to control an operation of a control de vice 3 configured to control a technical system 4.

Hereby, the device 7 may comprise a first determination unit

8, which may be adapted to determine an environment model 13 describing a physical environment in which the control device 3 and the technical system 4 may be used, and describing a relationship how a parameter value of the technical system 4 may behave under the control of the control device 3.

The device 7 may further comprise a second determination unit

9, which may be adapted to determine at least one operating requirement 14 how the parameter value of the technical sys tem 4 should behave under the control of the control device

3.

The device 7 may further comprise a third determination unit

10, which may be adapted to determine algebraic equations 15 in which the determined environment model 13 and the operat ing requirements 4 may be present as constraints. The device 7 may further comprise a user interface 16, which may be adapted to determine, based on a user input 6, whether the set of improved test vectors 1 should meet the at least one operating requirement or not.

The device 7 may further comprise a fourth determination unit 11, which may be adapted to determine a first test vector 5 representing a first behavior of the parameter value of the technical system 4 under test.

The device 7 may further comprise a solvation unit 12, which may be adapted to solve the algebraic equations 15 taking in to account the first test vector 5 comprising solving an op timization problem, wherein the solving may provide, as a re sult, the set of improved test vectors 1. Hereby, the set of improved test vectors 1 may represent a second behavior of the parameter value representing an ideal behavior the soft ware 2 should fulfill.

Additionally the software 2 may be adapted to output a feed back of the software test. Hereby, such an output may be transferred to the fourth determination unit 11 via a feed back loop 17. Hereby, a test coverage metric, a black box op timized and a reinforcement learning may be intended to be used .

The device 7 may further be adapted to perform any of the methods 100, 200 according to Figures 2-3.

Figures 5 schematically represents the generation of an im proved test vector 1 according to various examples. According to the examples given from Figure 5, the technical system 4 may be configured as a vehicle and the control device 3 may comprise an accelerator pedal of the vehicle. Hereby, the ac celerator pedal may contribute to a time-dependent velocity (time-dependent parameter value) control of the vehicle. Ac cording to such an example, Newtonian mechanics as well as physical properties of the vehicle may be included by a re spective environmental model 13. As depicted in Figure 5, the operating requirement 5 in this case is given as follows: velocity v (t ³ 20s) ³ 5 m/s.

According to the case of Figure 5, a user may additionally determine that the operating requirement given above is met.

In such a case the improved test vector 1 depicted may be based on the first vector 5 depicted in Figure 5 and may fur ther be modified by taken the operating requirement 5 out- lined above and the environmental model 13 outlined above in to account. Hereby, the improved test vector 5 indeed fulfils the operating requirement 5 outlined above.

Figure 6 schematically represents the generation of an im- proved test vector 1 according to various examples. Basical ly, this example may correspond to the example outlined with respect to Figure 5. However, in this case a user may addi tionally determine that the operating requirement given above is not met. Such a situation may refer to a failing mode of the software test. Hereby, the improved test vector 5 indeed does not fulfil the operating requirement 5 outlined above.

Claims

1. A method (100, 200) for generating a set of improved test vectors (1) used to assess an operating requirement (14) conformity of a software (2) which is used to control an op eration of a control device (3) configured to control a tech nical system (4), the method (100, 200) comprising:

- determining (110, 210) an environment model (13) describing a physical environment in which the control device (3) and the technical system (4) are used and describing a physical relationship how a parameter value of the technical system (4) behaves under the control of the control device (3),

- determining (120, 220) at least one operating requirement (14) how the parameter value of the technical system (4) should behave under the control of the control device (3) ,

- determining (130, 230) algebraic equations (15) in which the determined environment model (13) and the operating re quirements (14) are present as constraints,

- determining (140, 240) a first test vector (5) representing a first behavior of the parameter value of the technical system (4) ,

- solving (150, 250) the algebraic equations (15) taking into account the first test vector (5) comprising solving an op timization problem, the solving (150, 250) providing as a result the set of improved test vectors (1), the set of im proved test vectors (1) representing a second behavior of the parameter value representing an ideal behavior the software (2) should fulfill.

2. The method (100, 200) according to claim 1, wherein solving the optimization problem comprises solving a dynamic optimization problem where the environment model (13) and the operating requirements (14) are used as algebraic con

straints .

3. The method (100, 200) according to claim 1 or 2, wherein solving (150, 250) the algebraic equations (15) comprises us ing a receding horizon calculation in which the set of im- proved test vectors (1) is determined based on a predictive operation of the technical system (4) .

4. The method (100, 200) according to claim 3, wherein solving (150, 250) the algebraic equations (15) using the re ceding horizon calculation comprises applying the first test vector (5) to the software (2) .

5. The method (100, 200) according claim 3 or 4, wherein the solving (150, 250) the algebraic equations (15) using re ceding horizon calculation comprises a Mixed Integer Linear Programming or a Mixed Integer Nonlinear Programming ap proach .

6. The method (100, 200) according to any of claims 3 to 5, wherein the receding horizon calculation receives as an input the first test vector (5) and generates as an output the set of improved test vectors (1) such that the at least one operating requirement (14) is met.

7. The method (100, 200) according to any of the preceding claims, wherein the parameter value refers to a physical property describing the technical system (4) .

8. The method (100, 200) according to any of the preceding claims, further determining (135, 235) based on a user input (6) whether the set of improved test vectors (1) should meet the at least one operating requirement (14) or not.

9. The method (100, 200) according to any of the preceding claims, wherein the parameter value is a time-dependent func tion .

10. The method (100, 200) according to any of the preceding claims, wherein the determining (140, 240) of the first test vector (5) is drawn from a stochastic model.

11. The method (100, 200) according to any of the preceding claims, further comprising outputting (160, 260) the set of improved test vectors (1) to the software (2), wherein, in response to the set of improved test vectors (1) outputted to the software (2), it is determined (270) whether the set of improved test vectors (1) meets at least one operating re quirement (14) as predicted or not.

12. The method (100, 200) of claim 11, wherein the determin ing (140, 240) of the first test vector (5) is based on a de tected test coverage of at least one test run of the software (2) .

13. The method (100, 200) of claim 12, further comprising repeating (280) the test vector generation in order to maxim ize the test coverage.

14. The method (100, 200) of claim 10 and at least one of claims 12 and 13, wherein the stochastic model takes into ac count the detected test coverage to maximize it by altering accordingly the parameters of the stochastic model.

15. The method (100, 200) of any of the preceding claims, wherein the technical system (4) refers to at least one of an automotive system, a power station system, a wind turbine and a nuclear power plant.

16. A device (7) for generating a set of improved test vec tors (1) useable to access an operating requirement (14) con formity of a software (2) which is adapted to control an op eration of a control device (3) configured to control a tech nical system (4), the device (7) comprising:

- a first determination unit (8), adapted to determine an en vironment model (13) describing a physical environment in which the control device (3) and the technical system (4) are used and describing a relationship how a parameter val ue of the technical system (4) behaves under the control of the control device (3) , - a second determination unit (9), adapted to determine at least one operating requirement (14) how the parameter val ue of the technical system (4) should behave under the con trol of the control device (3) ,

- a third determination unit (10), adapted to determine alge braic equations (15) in which the determined environment model (13) and the operating requirements (14) are present as constraints,

- a fourth determination unit (11), adapted to determine a first test vector (5) representing a first behavior of the parameter value of the technical system (4),

- a solvation unit (12), adapted to solve the algebraic equa tions (15) taking into account the first test vector (5) comprising solving an optimization problem, the solving providing as a result the set of improved test vectors (1), the set of improved test vectors (1) representing a second behavior of the parameter value representing an ideal be havior the software (2) should fulfill.

17. The device (7) according to claim 16, which is adapted to perform the method (100, 200) according to any of claims 2 to 15.