CN114792469B

CN114792469B - Testing method and device for sensing system and testing equipment

Info

Publication number: CN114792469B
Application number: CN202210354970.6A
Authority: CN
Inventors: 张�杰; 闫丹
Original assignee: CICTCI Technology Co Ltd
Current assignee: CICTCI Technology Co Ltd
Priority date: 2022-04-06
Filing date: 2022-04-06
Publication date: 2023-06-02
Anticipated expiration: 2042-04-06
Also published as: CN114792469A

Abstract

The invention provides a testing method, a testing device and testing equipment of a perception system, wherein the testing method comprises the following steps: obtaining test data, the test data comprising: the sensing data and the true value data corresponding to the sensing data are acquired by sensing equipment of a sensing system to be detected in a road side sensing manner; determining target information to be detected of the sensing system to be detected according to the sensing data and a sensing algorithm corresponding to the sensing system to be detected; determining target perception information of the perception system to be detected from the target information to be detected according to the truth value data; and determining the test result information of the sensing system to be tested according to the target sensing information. The scheme of the invention solves the problems of high test cost and long overall test period caused by the fact that the test mode of the sensing equipment in the prior art needs to consume a great deal of labor cost and time cost.

Description

Testing method and device for sensing system and testing equipment

Technical Field

The present invention relates to the field of communications technologies, and in particular, to a method and an apparatus for testing a sensing system, and a testing device.

Background

The vehicle-road cooperation is based on full perception calculation of roads and vehicles, and low-time-delay and high-reliability communication connection is carried out, so that the technical scheme of vehicle-vehicle and vehicle-road cooperation application is realized, and the importance of perception calculation capability in vehicle-road cooperation is seen. Therefore, the sensing device needs to be tested for its sensing capability before being put into use to ensure accuracy and reliability of data output.

At present, the test method adopted by each research and development unit mainly comprises the steps of manually driving a vehicle, driving in a test field according to driving routes of different test scenes, then manually judging data indexes by analyzing logs output by each sensor algorithm, and finally manually matching the sensing result of a road side sensor with a true value by a tester and manually outputting a test report.

The test mode does not have large-scale test capability, a lot of errors are introduced in the test process, and a lot of labor cost and time cost are consumed, so that the test cost is high, and the overall test period is long.

Disclosure of Invention

The invention provides a testing method, a testing device and testing equipment of a sensing system, which solve the problems of higher testing cost and longer overall testing period caused by the fact that a great deal of labor cost and time cost are required to be consumed in the testing mode of the sensing equipment in the prior art.

In a first aspect, an embodiment of the present invention provides a method for testing a sensing system, including:

obtaining test data, the test data comprising: the sensing data and the true value data corresponding to the sensing data are acquired by sensing equipment of a sensing system to be detected in a road side sensing manner;

determining target information to be detected of the sensing system to be detected according to the sensing data and a sensing algorithm corresponding to the sensing system to be detected;

determining target perception information of the perception system to be detected from the target information to be detected according to the truth value data;

and determining the test result information of the sensing system to be tested according to the target sensing information.

Optionally, the acquiring test data includes:

the test vehicle is used for carrying first equipment to drive according to a preset path, the truth value data is acquired through the first equipment, and,

in the process that the test vehicle runs along the preset path, road side sensing is carried out through the sensing equipment of the sensing system to be detected, and sensing data are obtained;

wherein the first device comprises a high precision positioning device.

Optionally, when the sensing device is a camera, determining, according to the sensing data and a sensing algorithm corresponding to the sensing system to be measured, target information to be measured of the sensing system to be measured includes:

Acquiring a conversion matrix corresponding to the camera according to the perception data;

and determining target information to be detected of the sensing system to be detected according to the sensing data, the conversion matrix and a sensing algorithm corresponding to the sensing system to be detected.

Optionally, in the case that the sensing device includes a camera and a second device, and the second device is at least one of a laser radar or a millimeter wave radar, determining, according to the sensing data and a sensing algorithm corresponding to the sensing system to be measured, target information to be measured of the sensing system to be measured includes:

acquiring a conversion matrix corresponding to the camera according to the perception data; the sensing data comprise first sensing data corresponding to the camera and second sensing data corresponding to the second equipment;

determining first target information to be detected according to the first perception data, the conversion matrix and a perception algorithm corresponding to the camera; determining second target information to be detected according to the second perception data and a perception algorithm corresponding to the second equipment;

and determining the target information to be detected of the perception system to be detected according to the first target information to be detected and the second target information to be detected.

Optionally, the determining, according to the truth value data, the target perception information of the perception system to be detected from the target information to be detected includes:

acquiring first truth information in the truth data frame by frame according to the output frequency of the first equipment for acquiring the truth data;

according to the target information to be detected and the first truth information, determining first target perception information corresponding to each piece of first truth information;

and determining the first target perception information as target perception information of the perception system to be detected.

Optionally, the target information to be measured includes: timestamp, coordinate information, and target ID; the first truth information includes: timestamp and coordinate information;

the determining, according to the target information to be detected and the first truth information, first target perception information corresponding to each piece of first truth information includes:

acquiring first target information to be measured from the target information to be measured according to the timestamp of the first truth value information, wherein the timestamp of the first target information to be measured is identical to the timestamp of the first truth value information;

judging whether the first identification list has the same ID information as the target ID of the first target information to be tested or not, wherein the first identification list is generated in the test process;

And determining distance relation information between the coordinate information of the first truth value information and the coordinate information of the first target information to be measured according to the first truth value information and the first target information to be measured when the judging result is that the first truth value information does not exist, wherein the distance relation information comprises: a lateral distance and a linear distance;

determining first target perception information corresponding to the first truth value information according to the distance relation information and preset threshold information;

wherein the threshold information is set according to the type of the sensing device, and the threshold information includes: the distance interval and the corresponding transverse distance threshold and straight distance threshold are preset.

Optionally, when the first truth information is the first frame in the truth data, determining, according to the distance relation information and the pre-configured threshold information, first target perception information corresponding to the first truth information includes:

according to the distance relation information and the preset threshold information, determining the first to-be-detected target information which meets a first preset condition and has the minimum linear distance as the first target perception information;

wherein, the first preset condition is: the lateral distance is greater than or equal to the lateral distance threshold and the linear distance is greater than the linear distance threshold.

Optionally, in the case that the first truth information is not the first frame in the truth data, determining, according to the distance relation information and the pre-configured threshold information, first target perception information corresponding to the first truth information includes:

judging whether the target ID of the first target information to be tested is the same as a first target ID, wherein the first target ID is the target ID of the first target perception information determined under the condition that the first truth information is a first frame in the truth data;

and determining first target perception information corresponding to the first truth value information according to the judging result, the distance relation information and the preset threshold value information.

Optionally, the determining, according to the determination result, the distance relation information and the pre-configured threshold information, first target perception information corresponding to the first truth information includes at least one of the following:

and under the condition that the judging result is the same, determining the first target information to be detected meeting a first preset condition as the first target perception information, wherein the first preset condition is as follows: the lateral distance is less than or equal to the lateral distance threshold, and the linear distance is less than the linear distance threshold;

And under the condition that the judging results are different, determining the first to-be-detected target information which meets a second preset condition and has the minimum linear distance as the first target perception information, wherein the second preset condition is as follows: the lateral distance is less than or equal to the lateral distance threshold, the linear distance is less than or equal to the linear distance threshold, and an absolute value of a time difference between a time stamp of the first target information to be measured and a time stamp of the first truth information is less than a preset time.

Optionally, after the determining that the first target information to be detected that satisfies the first preset condition is the first target perception information, the method further includes:

and storing the target ID in the first target information to be tested which does not meet the first preset condition into the first identification list.

Optionally, the test result information includes at least one of: target positioning accuracy, target detection recall rate, accuracy and tracking success rate.

In a second aspect, an embodiment of the present invention provides a testing apparatus for a sensing system, including:

the data acquisition module is used for acquiring test data, wherein the test data comprises: the sensing data and the true value data corresponding to the sensing data are acquired by sensing equipment of a sensing system to be detected in a road side sensing manner;

The first processing module is used for determining target information to be detected of the perception system to be detected according to the perception data and a perception algorithm corresponding to the perception system to be detected;

the second processing module is used for determining target perception information of the perception system to be detected from the target information to be detected according to the true value data;

and the third processing module is used for determining the test result information of the sensing system to be tested according to the target sensing information.

In a third aspect, embodiments of the present invention provide a test apparatus comprising: a transceiver, a memory, a processor and a computer program stored on the memory and executable on the processor, which processor, when executing the computer program, implements the steps of the method of testing a perception system as described in the first aspect.

In a fourth aspect, embodiments of the present invention provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method of testing a perception system as described in the first aspect.

The technical scheme of the invention has the beneficial effects that:

according to the embodiment of the invention, the corresponding sensing algorithm can be automatically invoked according to the test data to obtain the target information to be tested of the sensing system to be tested, and then the target sensing information is obtained through analysis according to the target information to be tested and the true value data, so that the automatic test of the sensing capability of the sensing system is realized, the test efficiency is improved, the labor cost and the time cost required by the test are saved, the error caused by manual analysis is reduced, and the test result information is more objective and reliable.

Drawings

FIG. 1 is a flow chart of a method of testing a sensing system according to an embodiment of the present invention;

FIG. 2 illustrates a perceptual matching process of an embodiment of the present invention;

FIG. 3 illustrates a process of acquiring a threshold value in accordance with an embodiment of the present invention;

FIG. 4 is a block diagram of a sensing system testing device according to an embodiment of the present invention;

fig. 5 shows a block diagram of the structure of the test apparatus according to the embodiment of the present invention.

Detailed Description

In order to make the technical problems, technical solutions and advantages to be solved more apparent, the following detailed description will be given with reference to the accompanying drawings and specific embodiments. In the following description, specific details such as specific configurations and components are provided merely to facilitate a thorough understanding of embodiments of the invention. It will therefore be apparent to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. In addition, descriptions of well-known functions and constructions are omitted for clarity and conciseness.

It should be appreciated that reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

In various embodiments of the present invention, it should be understood that the sequence numbers of the following processes do not mean the order of execution, and the order of execution of the processes should be determined by the functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

In addition, the terms "system" and "network" are often used interchangeably herein.

In the examples provided herein, it should be understood that "B corresponding to a" means that B is associated with a from which B may be determined. It should also be understood that determining B from a does not mean determining B from a alone, but may also determine B from a and/or other information.

In the embodiment of the invention, the form of the access network is not limited, and the access network may be an access network including a macro base station (Macro Base Station), a micro base station (Pico Base Station), a Node B (3G mobile base station name), an enhanced base station (eNB), a Home enhanced base station (Femto eNB or Home eNode B or Home eNB or HeNB), a relay station, an access point, an RRU (Remote Radio Unit, a remote radio module), an RRH (Remote Radio Head, a remote radio head), and the like. The user terminal may be a mobile phone (or handset), or other device capable of transmitting or receiving wireless signals, including user equipment, personal Digital Assistant (PDA), wireless modem, wireless communicator, handheld device, laptop, cordless phone, wireless Local Loop (WLL) station, CPE (Customer Premise Equipment, client terminal) or mobile smart hot spot capable of converting mobile signals to WiFi signals, smart home appliances, or other devices capable of spontaneously communicating with a mobile communication network without manipulation by a person, etc.

Specifically, the embodiment of the invention provides a testing method, a testing device and testing equipment of a sensing system, which solve the problems that the testing mode of the sensing equipment in the prior art needs to consume a great deal of labor cost and time cost, so that the testing cost is high and the overall testing period is long.

First embodiment

As shown in fig. 1, an embodiment of the present invention provides a method for testing a sensing system, which specifically includes the following steps:

step 11: obtaining test data, the test data comprising: the sensing data and the true value data corresponding to the sensing data are acquired by sensing equipment of a sensing system to be detected in a road side sensing mode.

It should be noted that, the test data may be obtained through data import, or may be obtained through uploading by the sensing system to be tested.

As an alternative embodiment, the step 11 may specifically include: the test vehicle is used for carrying first equipment to run along a preset path, the true value data are acquired through the first equipment, and road side sensing is carried out through sensing equipment of the sensing system to be tested in the process that the test vehicle runs along the preset path, so that the sensing data are acquired; wherein the first device comprises a high precision positioning device.

It should be noted that the first device may be a device that employs a real-time kinematic (RTK) technique (Real Time Kinematic, real-time kinematic). For example, the testing process adopts high-precision inertial navigation equipment as true value data acquisition equipment, and the positioning accuracy can reach the centimeter level. In a specific acquisition process, a target vehicle (i.e. a test vehicle) or a pedestrian can be utilized to carry high-precision inertial navigation equipment, and single-target and multi-target tests are completed according to a planned route (i.e. a preset path) of a test case, wherein data output by the high-precision inertial navigation is true value data, and one or more test vehicles can be used. Meanwhile, road side sensing is performed through sensing equipment (such as a road side camera, millimeter wave radar equipment, laser radar equipment and the like), and offline data (namely sensing data) are collected and stored in real time. Here, the offline data may include video, millimeter wave radar data, laser radar point cloud data, and the like, and it should be noted that a data format of such data needs to conform to a prescribed interface form.

Step 12: and determining target information to be detected of the sensing system to be detected according to the sensing data and a sensing algorithm corresponding to the sensing system to be detected.

In the step, sensing equipment used by the sensing system to be detected can be determined according to the sensing data, so that the determination of the target information to be detected is determined by using which sensing algorithm.

Step 13: and determining target perception information of the perception system to be detected from the target information to be detected according to the true value data.

The step can obtain the sensing point successfully matched with the true value data, namely the target sensing information of the sensing system to be detected.

As an alternative embodiment, the step 13 may specifically include:

in step 1301, according to the output frequency of the first device that collects the truth data, the first truth information in the truth data is obtained frame by frame.

For example, the output frequency of the first device is 10HZ, as shown in fig. 2, according to which the perceived target is matched frame by frame. In particular, the method comprises the steps of,based on the output frequency, the perceived target is retrieved frame by frame (i.e., the first truth information in the truth data is obtained), wherein the truth data can be represented as an image sequence { R } ₁ ,R ₂ ,R ₃ …,R _n The target position corresponding to time t can be expressed as R _t (x′ _t, y′ _t )。

Step 1302, determining first target perception information corresponding to each piece of first truth information according to the target information to be detected and the first truth information;

Step 1303, determining the first target sensing information as target sensing information of the sensing system to be detected.

Step 14: and determining the test result information of the sensing system to be tested according to the target sensing information.

In this step, after determining the test result information of the sensing system to be tested, the test result information may also be output, for example, the test result information may be displayed to the user through a screen. Here, the test result information may include at least one of: target positioning accuracy, target detection recall rate, accuracy and tracking success rate.

In the embodiment, according to the test data, a corresponding perception algorithm can be automatically invoked to obtain the target information to be detected of the perception system to be detected, and further, according to the target information to be detected and the true value data, the target perception information is obtained through analysis, so that the automatic test of the perception capability of the perception system is realized, the test efficiency is improved, the labor cost and the time cost required by the test are saved, the error caused by the manual analysis is reduced, and the test result information is more objective and reliable.

In the embodiment of the present invention, step 12 may be divided into the following two cases according to different sensing devices:

Case one: as an alternative embodiment, in the case that the sensing device is a camera, step 12 includes: acquiring a conversion matrix corresponding to the camera according to the perception data; and determining target information to be detected of the sensing system to be detected according to the sensing data, the conversion matrix and a sensing algorithm corresponding to the sensing system to be detected.

In this embodiment, according to the sensing data (for example, offline video of each test scene), an RT matrix (i.e., a conversion matrix) of the camera corresponding to the offline data may be obtained, and then a visual algorithm interface is invoked to obtain a sensing result (i.e., target information to be tested) corresponding to the sensing data, such as a time stamp, coordinates, a target ID, a type, and other parameters.

It should be noted that in the prior art, conversion among a plurality of coordinate systems needs to be realized in space matching, and the flow is complex, the calibration difficulty is high, and errors are easy to occur in evaluating positioning accuracy. When the embodiment of the invention tests the vision algorithms corresponding to different cameras, the same conversion matrix can be adopted to complete the conversion from the image to the real coordinate system, so that the error caused by the conversion matrix is reduced, and the difference between different vision algorithms can be reflected more truly.

And a second case: as an alternative embodiment, where the sensing device includes a camera and a second device, and the second device is at least one of a lidar or millimeter-wave radar, step 12 includes: acquiring a conversion matrix corresponding to the camera according to the perception data; the sensing data comprise first sensing data corresponding to the camera and second sensing data corresponding to the second equipment; determining first target information to be detected according to the first perception data, the conversion matrix and a perception algorithm corresponding to the camera; determining second target information to be detected according to the second perception data and a perception algorithm corresponding to the second equipment; and determining the target information to be detected of the perception system to be detected according to the first target information to be detected and the second target information to be detected.

For example, if the second device is a millimeter wave radar, the specific process of determining the target information to be detected of the sensing system to be detected is: calling a visual algorithm interface, and acquiring a perception result corresponding to the offline video according to the offline video (namely first perception data) of each test scene and an RT matrix (namely a conversion matrix corresponding to a camera) of the camera corresponding to the offline video; and calling a radar fusion algorithm interface, transmitting a visual perception result (namely first target information to be detected) and millimeter wave radar offline data (namely second perception data) in the same period, and acquiring the radar fusion perception result (namely target information to be detected of a perception system) such as a time stamp, coordinates, a target ID, a type and other parameters.

In this embodiment, by calling an interface of the sensing system to be tested, a large number of test cases can be automatically executed (i.e., test data is acquired to determine target sensing information of the sensing system to be tested), and output information of the sensing system to be tested is collected, so that sensing capability of sensing equipment of the sensing system to be tested is evaluated in combination with preset evaluation criteria (e.g., rules of calculating target positioning accuracy, target detection recall, accuracy, tracking success rate, etc.).

It should be noted that the invention can evaluate the sensing capability of various sensors (namely sensing equipment), and supports the automatic test of road side sensing systems such as pure vision algorithm, millimeter radar and vision fusion algorithm, millimeter radar target detection, laser radar target detection and the like; single-point testing and multi-point fusion (such as vision+millimeter wave, vision+laser radar, vision+millimeter wave+laser radar and the like) testing of vision, millimeter wave radar and laser radar are supported, testing data are rich, testing dimensions are comprehensive, single-point detection, multi-target detection and multi-target detection can be covered, and the single-point radar and multi-point radar detection system is wide in application range and high in practicability.

As shown in fig. 2, optionally, the target information to be measured includes: timestamp, coordinate information, and target ID; the first truth information includes: timestamp and coordinate information; the determining, according to the target information to be detected and the first truth information, first target perception information corresponding to each piece of first truth information includes:

step one, according to the timestamp of the first truth value information, first target information to be measured is obtained from the target information to be measured, and the timestamp of the first target information to be measured is identical to the timestamp of the first truth value information.

In this step, the truth data in the test data can be read, and the time stamp and coordinate system information in the truth data can be extracted. According to the timestamp of the first truth information, all the targets to be detected (namely the first target information) of the same timestamp can be matched, and the number of targets in each frame of the perceived image is assumed to be M _t Then all targets (i.e. first target information) of each frame of image can be expressed as

The position of the corresponding ith target at time t may be expressed as +.>

And step two, judging whether the first identification list has the same ID information as the target ID of the first target information to be tested, wherein the first identification list is generated in the testing process.

Here, the first identification list may be understood as a "blacklist" generated during the test, i.e. ID information that has been excluded by the test.

In the embodiment, the ID attribute is adopted for screening in the process of target matching, and the timestamp in the ID attribute is utilized for removing noise, so that the false detection condition is greatly reduced.

Determining distance relation information between the coordinate information of the first truth value information and the coordinate information of the first target information to be measured according to the first truth value information and the first target information to be measured when the judging result is that the first truth value information does not exist, wherein the distance relation information comprises: lateral distance and linear distance.

That is, the first target information to be detected outside the blacklist is reserved, and then the first target perception information corresponding to the first true value information is screened from the first target information. In this process, the transverse-longitudinal deviation (i.e., transverse distance, longitudinal distance) and the linear distance of the ith sensing target to be detected (i.e., first target information to be detected) and the truth target (i.e., first truth information) at the t moment need to be calculated:

the linear distance may be obtained by calculating a first formula, where the first formula may be expressed as:

Wherein (1)>

Represents the straight line distance>

Here, the ith target position at the time t in the first target information to be measured is +.>

The target position corresponding to the t time in the truth value data is R _t (x′ _t, y′ _t )；

The lateral distance may be calculated by a second formula, which may be expressed as:

wherein (1)>

Represents the lateral distance>

The straight line distance theta represents an included angle between the lane direction and the north direction;

the longitudinal distance may be calculated by a third formula, which may be expressed as:

wherein (1)>

Representing the longitudinal distance. />

Determining first target perception information corresponding to the first truth value information according to the distance relation information and preset threshold information; wherein the threshold information is set according to the type of the sensing device, and the threshold information includes: the distance interval and the corresponding transverse distance threshold and straight distance threshold are preset. And then, storing all the matched sensing point sequences P (namely the first target sensing information), and the straight line distance, the transverse distance and the longitudinal distance.

Before the automatic test, the threshold information can be written into the configuration file, and when the automatic test device is used, the corresponding transverse distance threshold and the corresponding linear distance threshold are obtained according to the equipment type and the distance interval attribute (namely the preset distance interval) by reading the configuration file. Here, different threshold information, such as different transverse distance thresholds and linear distance thresholds, can be configured according to the self characteristics and the perception ranges of different sensing device types (namely, the perception performance of the sensing device in different intervals), so that noise targets during matching are reduced to the greatest extent, blind area errors of a sensing algorithm are reduced, and mismatching and missed matching phenomena of sensing points are reduced.

Optionally, when the first truth information is the first frame in the truth data, determining, according to the distance relation information and the pre-configured threshold information, first target perception information corresponding to the first truth information includes: according to the distance relation information and the preset threshold information, determining the first to-be-detected target information which meets a first preset condition and has the minimum linear distance as the first target perception information; wherein, the first preset condition is: the lateral distance is greater than or equal to the lateral distance threshold and the linear distance is greater than the linear distance threshold.

In this embodiment, when determining the first target perceived information corresponding to the first truth information, the first truth information is the first frame in the truth data, and at this time, the perceived point that meets the threshold and is closest to the truth may be directly obtained, where the perceived point meets the following conditions:

the perceived point is stored in the sequence p= { P ₁ ,P ₂ ,P ₃ …,P _n And records the ID of the matched sensing point, i.e. the first target ID (with +.>

Representation).

As shown in fig. 3, the configuration file is read, and a lateral distance threshold is obtained according to the device type and the distance interval attribute, and is expressed as a sequence { value } ₁ ,value ₂ ,value ₃ …,value _m Acquiring a linear distance threshold value according to a distance interval, and representing the linear distance threshold value as a sequence { value } _d1 ,value _d2 ,valued ₃ …,value _dm }. Wherein value is _m R represents _t The transverse distance threshold parameter, value of the region _dm R represents _t And the linear distance threshold parameter of the region where the current frame is located, and m represents the section where the current frame is located.

step 201: and judging whether the target ID of the first target information to be tested is the same as a first target ID, wherein the first target ID is the target ID of the first target perception information determined under the condition that the first truth information is a first frame in the truth data.

That is, in the process of matching by acquiring the first truth information in the truth data frame by frame, if the first frame is successfully matched, the target ID of the first target information to be tested needs to be determined first from the second frame (i.e.

If the ID of (2) is the ID of the first frame successfully matched (i.e. the first target ID), and then can obtain the first true value information (R is used) _t Represented) distance interval, read in accordance with the distance interval Taking the linear distance threshold (value _dm Expressed) and a lateral threshold (in value) _m Representation) for further screening according to step 202. The distance intervals may be set according to specific situations, for example, as shown in fig. 3, and may be divided into four distance intervals, namely, 0 to 50m interval, 50 to 100m interval, 100 to 150m interval, and 150 to 200m interval, where different distance intervals may correspond to different lateral distance thresholds and linear distance thresholds.

Step 202: and determining first target perception information corresponding to the first truth value information according to the judging result, the distance relation information and the preset threshold value information.

As an alternative embodiment, step 202 may specifically include at least one of the following:

and determining the first target information to be detected meeting a first preset condition as the first target perception information under the condition that the judging result is the same, wherein the first preset condition is as follows: the lateral distance is less than or equal to the lateral distance threshold, and the linear distance is less than the linear distance threshold.

Specifically, if the target ID of the first target information to be measured (assumed to be expressed as

) Is identical to the first target ID (assuming that the first target information to be tested is recorded as +. >

) Continue to judge +.>

And R is R _t Straight line distance (i.e.)>

) Lateral distance (i.e.)>

) Whether a threshold is met; if it meets (i.e.)>

And is also provided with

) The perception point is saved->

And will->

The logging sequence p= { P ₁ ,P ₂ ,P ₃ …,P _n -a }; if not, then ∈>

Blacklisting and clearing the sequence P and then re-matching from the first frame.

Secondly, under the condition that the judging results are different, determining the first to-be-detected target information which meets a second preset condition and has the minimum linear distance as the first target perception information, wherein the second preset condition is as follows: the lateral distance is less than or equal to the lateral distance threshold, the linear distance is less than or equal to the linear distance threshold, and an absolute value of a time difference between a time stamp of the first target information to be measured and a time stamp of the first truth information is less than a preset time.

Specifically, if the sensing points having the same ID as the first target ID are not matched (i.e. the target ID of the first target information to be tested is different from the first target ID), the sensing points are acquired

To->

All the sensing points and R _t Is kept to meet the threshold value (i.e.)>

And->

) Is->

Then, the time stamp carried in the ID (the time stamp in the ID represents the discovery time of the target) is extracted according to the ID and recorded as

The save timestamp satisfies the condition: />

Is defined by the number of points of perception; finally, the point nearest to the true value is output +.>

The logging sequence p= { P ₁ ,P ₂ ,P ₃ …,P _n }。

It should be noted that, when evaluating the matching degree of the track output by each sensing system to be tested and the true value data, the time and space synchronization process is required. In the embodiment of the invention, different sensing system outputs to be detected are converted into real coordinates in the space position, so that the space matching of each sensor (namely sensing equipment) and the true coordinates is realized; in time, synchronous acquisition cannot be realized due to different acquisition frequencies of the sensors, and the embodiment of the invention performs data matching on similar time to targets output by different sensing devices according to the timestamps carried by the truth data, so that the matching on time is realized. In addition, the embodiment of the invention also generates real coordinates based on the same conversion matrix aiming at the visual perception test, thereby reducing matrix errors among different algorithms.

Optionally, after the determining that the first target information to be detected that satisfies the first preset condition is the first target perception information, the method further includes: and storing the target ID in the first target information to be tested which does not meet the first preset condition into the first identification list.

In the embodiment of the invention, the target positioning precision, the target detection recall rate, the accuracy, the tracking success rate and the like can be used as evaluation indexes of the sensing system to be tested, the target matching is carried out by acquiring test data, calling a sensing algorithm and the like, finally, the evaluation indexes are output according to the execution result, for example, a perfect test report can be generated, and the test report is automatically output, so that the evaluation result of the sensing capability of the sensing system to be tested is displayed to a user, and the test scene with poor sensing capability performance of the sensing system to be tested can be highlighted.

In the embodiment, the target detection and tracking algorithm of the sensing equipment of the sensing system to be tested can be tested and analyzed, the quantized index is output, the sensing capability of the sensing system can be evaluated objectively and conveniently, the algorithm capability of the sensing system to be tested under different testing scenes can be tested in a large scale, the operation is simple and convenient, and the testing result is accurate.

Specifically, the test result information may include: ID conversion times, accuracy, target detection recall, omission rate, false detection rate, horizontal and vertical deviation (namely horizontal distance and vertical distance), final Test Case coverage rate and other various indexes under each driving scene. The following is a brief explanation of some of the indicators:

ID conversion times, namely counting the ID number of all targets (sequences P) which are perceived by a perception system to be detected and recalled; the smaller the number of IDs, the better the tracking effect of the perception algorithm.

Recall rate: counting the number (sequence P) of sensing targets output by a sensing system to be detected; wherein recall = perceived recall target total/truth total;

accuracy rate: the accurate identification judgment standard is that the type and the position of the recalled target are consistent with those of the true value target, and the identification is accurate; wherein, accuracy = the accurate total number of identification/the total number of recall perceived by the perception system to be detected;

target positioning accuracy: according to different distance intervals (for example, 0-50m, 50-100m, 100-150m and >150 m) between the truth value target and the starting point of the lane, extracting test data, calculating transverse and longitudinal deviations (namely, transverse distance and longitudinal distance) at the same moment t, and respectively counting the average value of the transverse and longitudinal deviations according to the distance intervals.

According to the embodiment of the invention, the corresponding sensing algorithm can be automatically invoked according to the test data to obtain the target information to be tested of the sensing system to be tested, and then the target sensing information is obtained by analyzing according to the target information to be tested and the true value data, so that the sensing capability (namely, target detection, target tracking capability and the like) of the sensing system to be tested can be automatically evaluated, and the test efficiency and the test precision are greatly improved.

Second embodiment

As shown in fig. 4, an embodiment of the present invention provides a testing apparatus 400 of a sensing system, including:

the data acquisition module 401 is configured to acquire test data, where the test data includes: the sensing data and the true value data corresponding to the sensing data are acquired by sensing equipment of a sensing system to be detected in a road side sensing manner;

the first processing module 402 is configured to determine target information to be detected of the to-be-detected sensing system according to the sensing data and a sensing algorithm corresponding to the to-be-detected sensing system;

a second processing module 403, configured to determine target perception information of the to-be-detected perception system from the target information to be detected according to the truth value data;

and a third processing module 404, configured to determine test result information of the to-be-tested sensing system according to the target sensing information.

Optionally, the data acquisition module 401 includes:

the acquisition sub-module is used for carrying first equipment by using a test vehicle to run along a preset path, acquiring the true value data through the first equipment, and carrying out road side perception through the perception equipment of the perception system to be detected in the process of running along the preset path by using the test vehicle to acquire the perception data;

wherein the first device comprises a high precision positioning device.

Optionally, in the case where the sensing device is a camera, the first processing module 402 includes:

the first processing sub-module is used for acquiring a conversion matrix corresponding to the camera according to the perception data;

and the second processing sub-module is used for determining target information to be detected of the sensing system to be detected according to the sensing data, the conversion matrix and a sensing algorithm corresponding to the sensing system to be detected.

Optionally, where the sensing device includes a camera and a second device, and the second device is at least one of a lidar or a millimeter wave radar, the first processing module 402 includes:

the third processing sub-module is used for acquiring a conversion matrix corresponding to the camera according to the perception data; the sensing data comprise first sensing data corresponding to the camera and second sensing data corresponding to the second equipment;

The fourth processing sub-module is used for determining first target information to be detected according to the first perception data, the conversion matrix and a perception algorithm corresponding to the camera; determining second target information to be detected according to the second perception data and a perception algorithm corresponding to the second equipment;

and the fifth processing sub-module is used for determining the target information to be detected of the perception system to be detected according to the first target information to be detected and the second target information to be detected.

Optionally, the second processing module 403 includes:

a sixth processing sub-module, configured to acquire, according to an output frequency of a first device that acquires the truth data, first truth information in the truth data on a frame-by-frame basis;

a seventh processing sub-module, configured to determine, according to the target information to be detected and the first truth information, first target perception information corresponding to each piece of first truth information;

and the eighth processing sub-module is used for determining the first target perception information as target perception information of the perception system to be detected.

Wherein the seventh processing submodule includes:

the first processing unit is used for acquiring first target information to be detected from the target information to be detected according to the timestamp of the first true value information, and the timestamp of the first target information to be detected is the same as the timestamp of the first true value information;

the second processing unit is used for judging whether the first identification list has the same ID information as the target ID of the first target information to be tested or not, and the first identification list is generated in the test process;

and a third processing unit, configured to determine, when the determination result is that the first truth value information does not exist, distance relation information between coordinate information of the first truth value information and coordinate information of the first target information according to the first truth value information and the first target information, where the distance relation information includes: a lateral distance and a linear distance;

the fourth processing unit is used for determining first target perception information corresponding to the first truth value information according to the distance relation information and preset threshold information;

Optionally, the fourth processing unit includes:

the first processing subunit is configured to determine, according to the distance relation information and the preset threshold information, the first target information to be detected that satisfies a first preset condition and has the minimum linear distance as the first target perception information;

Optionally, the fourth processing unit includes:

the second processing subunit is configured to determine whether a target ID of the first target information to be tested is the same as a first target ID, where the first target ID is a target ID of the first target perception information determined when the first truth information is a first frame in the truth data;

and the third processing subunit is used for determining first target perception information corresponding to the first true value information according to the judging result, the distance relation information and the preset threshold information.

Optionally, the third processing subunit includes:

the fourth processing subunit is configured to determine, when the determination result is the same, the first target information to be detected that meets a first preset condition as the first target sensing information, where the first preset condition is: the lateral distance is less than or equal to the lateral distance threshold, and the linear distance is less than the linear distance threshold;

And a fifth processing subunit, configured to determine, as the first target sensing information, the first target information to be detected that satisfies a second preset condition and has the minimum linear distance, where the second preset condition is that: the lateral distance is less than or equal to the lateral distance threshold, the linear distance is less than or equal to the linear distance threshold, and an absolute value of a time difference between a time stamp of the first target information to be measured and a time stamp of the first truth information is less than a preset time.

Optionally, the third processing subunit further comprises:

and a sixth processing subunit, configured to store, in the first identifier list, a target ID in the first target information to be tested that does not satisfy the first preset condition.

The second embodiment of the present invention corresponds to the method of the first embodiment, and all the implementation means in the first embodiment are applicable to the embodiment of the testing device of the sensing system, so that the same technical effects can be achieved.

Third embodiment

In order to better achieve the above object, as shown in fig. 5, a third embodiment of the present invention further provides a test apparatus, including:

a processor 500; and a memory 520 connected to the processor 500 through a bus interface, the memory 520 storing programs and data used by the processor 500 in performing operations, the processor 500 calling and executing the programs and data stored in the memory 520.

Wherein the transceiver 510 is coupled to the bus interface for receiving and transmitting data under the control of the processor 500; the processor 500 is configured to read the program in the memory 520 to perform the following steps:

Wherein in fig. 5, a bus architecture may comprise any number of interconnected buses and bridges, and in particular one or more processors represented by processor 500 and various circuits of memory represented by memory 520, linked together. The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, power management circuits, etc., which are well known in the art and, therefore, will not be described further herein. The bus interface provides an interface. The transceiver 510 may be a number of elements, including a transmitter and a transceiver, providing a means for communicating with various other apparatus over a transmission medium. The user interface 530 may also be an interface capable of interfacing with an inscribed desired device for a different terminal, including but not limited to a keypad, display, speaker, microphone, joystick, etc. The processor 500 is responsible for managing the bus architecture and general processing, and the memory 520 may store data used by the processor 500 in performing operations.

Optionally, the processor 500 is specifically configured to, when acquiring the test data:

wherein the first device comprises a high precision positioning device.

Optionally, when the sensing device is a camera, the processor 500 is specifically configured to, when determining the target information to be measured of the sensing system to be measured according to the sensing data and a sensing algorithm corresponding to the sensing system to be measured:

Optionally, when the sensing device includes a camera and a second device, and the second device is at least one of a laser radar or a millimeter wave radar, the processor 500 is specifically configured to, when determining, according to the sensing data and a sensing algorithm corresponding to the sensing system to be measured, target information to be measured of the sensing system to be measured:

according to the first target information to be measured and the second target information to be measured, and determining the target information to be detected of the perception system to be detected.

Optionally, when determining the target perception information of the perception system under test from the target information under test according to the truth data, the processor 500 is specifically configured to:

the processor 500 is specifically configured to, when determining, according to the target information to be detected and the first truth information, first target perception information corresponding to each piece of first truth information:

Optionally, when the first truth information is the first frame in the truth data, the processor 500 is specifically configured to, when determining, according to the distance relation information and the pre-configured threshold information, first target perception information corresponding to the first truth information:

Optionally, when the first truth information is not the first frame in the truth data, the processor 500 is specifically configured to, when determining, according to the distance relation information and the pre-configured threshold information, first target perception information corresponding to the first truth information:

Optionally, when determining the first target perception information corresponding to the first truth value information according to the determination result, the distance relation information and the pre-configured threshold information, the processor 500 is specifically configured to:

Optionally, the processor 500 is further configured to:

According to the test equipment provided by the invention, the corresponding perception algorithm can be automatically invoked according to the test data to obtain the target information to be tested of the perception system to be tested, and then the target perception information is obtained by analyzing according to the target information to be tested and the truth value data, so that the perception capability (namely, target detection capability, target tracking capability and the like) of the perception system to be tested can be automatically evaluated, and the test efficiency and the test precision are greatly improved.

Those skilled in the art will appreciate that all or part of the steps of implementing the above-described embodiments may be implemented by hardware, or may be implemented by instructing the relevant hardware by a computer program comprising instructions for performing some or all of the steps of the above-described methods; and the computer program may be stored in a readable storage medium, which may be any form of storage medium.

In addition, a specific embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processor, implements the steps of the method as in the first embodiment described above. And the same technical effects can be achieved, and in order to avoid repetition, the description is omitted here.

Furthermore, it should be noted that in the apparatus and method of the present invention, it is apparent that the components or steps may be disassembled and/or assembled. Such decomposition and/or recombination should be considered as equivalent aspects of the present invention. Also, the steps of performing the series of processes described above may naturally be performed in chronological order in the order of description, but are not necessarily performed in chronological order, and some steps may be performed in parallel or independently of each other. It will be appreciated by those of ordinary skill in the art that all or any of the steps or components of the methods and apparatus of the present invention may be implemented in hardware, firmware, software, or a combination thereof in any computing device (including processors, storage media, etc.) or network of computing devices, as would be apparent to one of ordinary skill in the art after reading this description of the invention.

The object of the invention can thus also be achieved by running a program or a set of programs on any computing device. The computing device may be a well-known general purpose device. The object of the invention can thus also be achieved by merely providing a program product containing program code for implementing said method or apparatus. That is, such a program product also constitutes the present invention, and a storage medium storing such a program product also constitutes the present invention. It is apparent that the storage medium may be any known storage medium or any storage medium developed in the future. It should also be noted that in the apparatus and method of the present invention, it is apparent that the components or steps may be disassembled and/or assembled. Such decomposition and/or recombination should be considered as equivalent aspects of the present invention. The steps of executing the series of processes may naturally be executed in chronological order in the order described, but are not necessarily executed in chronological order. Some steps may be performed in parallel or independently of each other.

While the foregoing is directed to the preferred embodiments of the present invention, it will be appreciated by those skilled in the art that various modifications and adaptations can be made without departing from the principles of the present invention, and such modifications and adaptations are intended to be comprehended within the scope of the present invention.

Claims

1. A method for testing a sensing system, comprising:

determining test result information of the sensing system to be tested according to the target sensing information;

wherein, the determining the target perception information of the perception system to be detected from the target information to be detected according to the truth value data includes:

determining the first target perception information as target perception information of the perception system to be detected;

Wherein the target information to be measured includes: timestamp, coordinate information, and target ID; the first truth information includes: timestamp and coordinate information;

Wherein the threshold information is set according to the type of the sensing device, and the threshold information includes: presetting a distance interval and a corresponding transverse distance threshold and a linear distance threshold;

wherein, when the first truth value information is the first frame in the truth value data, the determining, according to the distance relation information and the pre-configured threshold value information, first target perception information corresponding to the first truth value information includes:

2. The method of claim 1, wherein the acquiring test data comprises:

Wherein the first device comprises a high precision positioning device.

3. The method according to claim 1, wherein, in the case where the sensing device is a camera, the determining, according to the sensing data and a sensing algorithm corresponding to the sensing system to be measured, target information to be measured of the sensing system to be measured includes:

4. The method according to claim 1, wherein, in the case where the sensing device includes a camera and a second device, and the second device is at least one of a laser radar or a millimeter wave radar, the determining, according to the sensing data and a sensing algorithm corresponding to the sensing system to be measured, target information to be measured of the sensing system to be measured includes:

5. The method according to claim 1, wherein, in the case that the first truth information is not the first frame in the truth data, the determining, according to the distance relation information and pre-configured threshold information, first target perception information corresponding to the first truth information includes:

6. The method according to claim 5, wherein determining the first target perception information corresponding to the first truth information according to the determination result, the distance relation information and the pre-configured threshold information includes at least one of:

7. The method according to claim 6, wherein after the first target information to be measured satisfying a first preset condition is determined as the first target perceived information, the method further comprises:

8. The method of claim 1, wherein the test result information comprises at least one of: target positioning accuracy, target detection recall rate, accuracy and tracking success rate.

9. A test apparatus comprising: transceiver, memory, processor and computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method for testing a perception system according to any one of claims 1 to 8 when executing the computer program.

10. A sensing system testing apparatus, comprising:

The third processing module is used for determining the test result information of the sensing system to be tested according to the target sensing information;

wherein the second processing module comprises:

an eighth processing sub-module, configured to determine the first target sensing information as target sensing information of the sensing system to be detected;

wherein the seventh processing submodule includes:

wherein the fourth processing unit includes:

11. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, carries out the steps of a method of testing a perception system as claimed in any one of claims 1 to 8.