CN111859674A - Automatic driving test image scene construction method based on semantics - Google Patents
Automatic driving test image scene construction method based on semantics Download PDFInfo
- Publication number
- CN111859674A CN111859674A CN202010712764.9A CN202010712764A CN111859674A CN 111859674 A CN111859674 A CN 111859674A CN 202010712764 A CN202010712764 A CN 202010712764A CN 111859674 A CN111859674 A CN 111859674A
- Authority
- CN
- China
- Prior art keywords
- scene
- automatic driving
- image
- semantic
- test
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012360 testing method Methods 0.000 title claims abstract description 31
- 238000010276 construction Methods 0.000 title claims abstract description 7
- 230000035772 mutation Effects 0.000 claims abstract description 36
- 238000005516 engineering process Methods 0.000 claims abstract description 12
- 238000013461 design Methods 0.000 claims abstract description 11
- 230000007547 defect Effects 0.000 claims abstract description 5
- 238000004458 analytical method Methods 0.000 claims abstract description 4
- 238000011156 evaluation Methods 0.000 claims abstract description 4
- 238000000034 method Methods 0.000 claims description 12
- 238000004088 simulation Methods 0.000 claims description 12
- 238000000605 extraction Methods 0.000 claims description 4
- 230000003321 amplification Effects 0.000 claims 1
- 230000003416 augmentation Effects 0.000 claims 1
- 238000004422 calculation algorithm Methods 0.000 claims 1
- 238000004364 calculation method Methods 0.000 claims 1
- 238000003199 nucleic acid amplification method Methods 0.000 claims 1
- 230000000717 retained effect Effects 0.000 claims 1
- 230000009286 beneficial effect Effects 0.000 abstract 1
- 238000001514 detection method Methods 0.000 abstract 1
- 239000003086 colorant Substances 0.000 description 2
- 241001464837 Viridiplantae Species 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000033001 locomotion Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/20—Design optimisation, verification or simulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3684—Test management for test design, e.g. generating new test cases
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Geometry (AREA)
- Quality & Reliability (AREA)
- Processing Or Creating Images (AREA)
Abstract
A semantic-based automatic driving test image scene construction method is characterized by a semantic-based image scene construction technology. And designing a mutation operator through semantic analysis, and applying the mutation operator to an image acquired by a sensor, thereby constructing scene data related to semantics and carrying out sufficiency detection on defects existing in automatic driving. The invention comprises three components: the system comprises a mutation operator design module, a scene generation module and a generated scene data validity evaluation module. The input of the invention is image data collected by a camera, characteristic information in the image is extracted, a scene is constructed based on the characteristic information, and the defects in the scene are detected and output. The invention has the following beneficial effects: through the variation of semantic features, a large amount of scene information can be generated, and the coverage rate of the automatic driving test case is greatly increased. By the aid of an automatic generation technology, the test period of automatic driving can be shortened, and falling of the automatic driving industry is accelerated.
Description
Technical Field
The invention belongs to the field of automatic driving test in intelligent software test, and relates to the technologies of feature extraction, mutation operator design, image scene construction and the like. After designing a mutation operator based on semantic features, applying the mutation operator to image data to generate a mutation scene, and calling a simulation engine API to adjust data and capture a picture, thereby obtaining image mutation data based on semantics.
Background
With the rapid development of artificial intelligence, intelligence has become a new trend of software system applications. Autopilot systems, which are currently the most popular intelligent software systems, are typically deployed to operate in a safety critical environment, and software defects thereof are likely to cause catastrophic consequences. Under the circumstances, a series of automatic driving test technologies are proposed by various companies and research institutes, and the field is greatly emphasized by the industry and academia.
The automatic driving system relies on various sensors including radar, a camera, a GPS and the like to sense the environment, self-positioning and driving behavior decision are completed through sensing data, the input domain of the automatic driving system is extremely large due to high complexity of weather roads and driving environments, the sensing data is obtained only through live-action road test and 3D simulation environment test to test the automatic driving system, the completeness and the sufficiency of the test are difficult to guarantee under the condition of limited resources, and new challenges are provided for quality guarantee means of the automatic driving system.
In the operation of the automatic driving system, the sensing module needs to fuse data captured by sensors such as radar and a camera and input the data into a machine learning model to complete sensing. The environmental image data captured by the camera depicts the colors and layouts of various objects in the driving scene, and the color and layout information including traffic lights, road marks and the like have a vital role in driving behavior decision. However, since the drive test and the simulation test of the automatic driving system are expensive, it is difficult to achieve high coverage in the test process for some driving scenes including congested intersections, overtaking of large trucks, other special road conditions, and the like.
Based on the analysis, the invention designs a mutation operator according to semantic information expressed by original image data acquired by a camera, generates data by using an image automatic generation technology, forms an open source image test data set by the generated data, selects an automatic driving software system of an automatic driving industrial grade to carry out experimental verification, discovers real defects and errors in the data and experimentally verifies the effectiveness of the test data generation technology.
Disclosure of Invention
The automatic driving system has huge input domain, contains rich scene semantic information and has poor internal realization logic interpretability. The invention can obtain the camera data which is automatically generated, and solves the problems of completeness and sufficiency of the test and difficult interpretation and analysis of the output result by utilizing the semantic information of the automatic driving scene to a certain extent.
The technical scheme of the invention is as follows: a method for semantic-based automated driving test image data generation uses an image automated generation technique to generate data according to designed operators of variation.
The generation method comprises the following three modules/steps:
1) and the mutation operator design module is used for designing a driving scene, wherein the driving scene image usually contains various objects including vegetation, other vehicles, pedestrians, traffic indicators and the like. In practical situations, these objects may have a variety of characteristics. For example, pedestrians may have different clothing and motions, vehicle vegetation, etc. may have completely different colors, architectural appearances may have different graffiti, etc., which may cause the determination of the image data by the automatic driving system to be incorrect. However, it is generally difficult or more computing resources are consumed to directly mutate object features in an image or simulation environment in real time, and therefore, the module designs a mutation operator in the driving scene image based on semantics to improve the diversity of the driving scene image.
2) A scene automatic generation module: the mutation operator designed by the project can semantically complete the mutation of the driving scene image characteristics, and the mutation result needs to be converted into a driving scene image to be input into an automatic driving system for testing. The application process of the mutation operator is a process for rendering the scene based on the highly structured description information. The module is used for automatically generating an image by utilizing the designed mutation operator through semantic description information.
3) An image generation data validity evaluation module: when the generated data is obtained, the validity of the generated data needs to be tested, and invalid image data cannot make any real contribution to the automatic driving system. Because the input space state domain of the automatic driving system is large, whether the driving decision of the automatic driving system in a certain scene meets the expected result cannot be determined, and the selection of the test means is difficult. The module is used for testing the effectiveness of the generated image data.
The invention is characterized in that:
1. the method is used for generating image data based on a semantic mode for the first time in the field of automatic driving test.
2. The scene description language is used for describing the semantic information of the automatic driving scene for the first time, and then data are generated.
Drawings
Fig. 1 is a general flow chart of the implementation of the present invention.
FIG. 2 is a flow chart of raw data processing.
Fig. 3 is a flow chart of data validity evaluation.
Detailed Description
The method is mainly divided into a driving scene image mutation operator design and a driving scene image automatic generation technology of semantic guidance by using a simulation engine and a domain-specific language to assist in generating image data based on semantics.
Designing a semantic-oriented driving scene image mutation operator: domain experts have devised domain-specific languages that describe scenarios of particular autodrive domains (e.g., illegal parking of automobiles, etc.) through an understanding of the autodrive domain. Given the scene characteristics of weather (sunny days, rainy and snowy days and the like), time (day, evening and the like), obstacles (pedestrians, vehicles and the like), road conditions (traffic lights, crossroads and the like), a scene description source code is compiled by using a scene description language. These feature information are described precisely by strict syntax. In the implementation process, a domain expert is introduced or a crowdsourcing form is adopted to convert the driving scene image into a source code of a structured language (such as XML and the like) with strict grammar definition and semantic description, and then a related mutation operator is designed according to semantic information to generate image data. The objects that the mutation operator considered at present can operate on include, object appearance, size, number, road indication signal status, background, etc. Based on the application of the mutation operators, the characteristic information in the image description source code can be accurately added, modified and deleted. For example, the state of the signal lights described in the source code is modified, the color of the preceding vehicle is modified, and the like.
Automatic generation technology of driving scene images: the invention aims to adopt two maximum automatic driving open-source image data sets Udacity and KITTI as original data. After the original data are obtained, a part of driving scene description image data are extracted to serve as seed data, and a scene description corpus is constructed through manual examination. Based on seed data, a corpus and a mutation operator, the mutation of the scene description source code can be realized. After mutation is completed, the source codes are input into a simulation engine and rendered into a scene image.
The invention aims to adopt CARLA as an experimental simulation engine. The CARLA simulation engine simulates a realistic dynamic virtual world and provides an interface for interacting with it. The engine is provided with an API facing Python language and supports various operations on scenes and objects in the virtual world. The simulation engine provides 3D models (buildings, grassland green plants, traffic signs, basic settings and the like) of static objects and 3D models (vehicles, pedestrians and the like) of dynamic objects, 40 different buildings, 16 vehicles and 50 pedestrian models are accumulated, the size, texture, color and the like of the objects can be changed through API, and the method is very suitable for building semantic-based test scenes. The designed mutation operator is used for mutating the scene description source code file, the mutated content is analyzed, a simulation engine API is called, the virtual scene described by the mutated source code is rendered, and then the photographing function is called through the API to generate picture data.
In the example, a driving scene image described by a structured language with strict syntactic definition and semantic description is converted into a source code, then a mutation operator is designed based on the source code, after the design of the operator is completed, the mutation operator is applied to scene image data to automatically transform an automatic driving simulation scene to generate mutation data, a sensing system is trained, tested and debugged, the performance of the sensing system is tested and faults are found under different conditions, the root cause of the faults is analyzed, and then the faults are eliminated.
Claims (5)
1. A semantic-based automatic driving test image scene construction method is characterized in that a semantic-based image scene construction technology designs a mutation operator through semantic features, applies the mutation operator to an image acquired by a sensor to construct scene data related to semantics, performs a large number of test case amplification through an automatic data generation technology, and finally performs validity evaluation on the generated scene test cases to ensure that defects existing in automatic driving are fully detected.
2. The semantic feature mutation operator design according to claim 1, wherein for a given semantic description, feature extraction is performed first, and a domain expert can describe a specific automatic driving scenario, such as weather, time, obstacles, road conditions, degree, scale, color, and other information, by using a special language designed by understanding the automatic driving technology, and design a mutation algorithm around the extracted features, and design a mutation operator for a given feature to perform augmentation of an automatic driving test scenario.
3. The method of claim 1, wherein feature extraction is performed around the image data collected by the sensor to determine a mutation operator, and then feature mutation is performed on the image data, for example, if the extracted feature is daytime, a night scene is generated, and if the extracted feature is red road, a yellow or green scene is generated.
4. The automatic generation technology of the driving scene image in claim 1 is characterized in that a scene description language is artificially generated around an open-source automatic driving image library through crowdsourcing test, corresponding mutation operators are selected based on feature extraction, CARLA is adopted as an experimental simulation engine to simulate a vivid dynamic virtual world, and a great amount of automatic driving scene image data is generated through the operation of an API facing Python language on scenes and objects in the virtual world.
5. The test case validity assessment method according to claim 1, wherein feature analysis is performed on generated scene data, and for cases which appear in the same scene for a plurality of times and have high similarity, the most representative case with the highest score is retained, and for the case where the feature disappears due to data variation, image data is directly removed, so that invalid calculation of a test system is reduced, and test efficiency is improved.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010712764.9A CN111859674A (en) | 2020-07-23 | 2020-07-23 | Automatic driving test image scene construction method based on semantics |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010712764.9A CN111859674A (en) | 2020-07-23 | 2020-07-23 | Automatic driving test image scene construction method based on semantics |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111859674A true CN111859674A (en) | 2020-10-30 |
Family
ID=72950615
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010712764.9A Pending CN111859674A (en) | 2020-07-23 | 2020-07-23 | Automatic driving test image scene construction method based on semantics |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111859674A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112859810A (en) * | 2021-01-13 | 2021-05-28 | 自行科技(武汉)有限公司 | ADAS algorithm verification method and device based on Carla platform |
| CN113836111A (en) * | 2021-08-23 | 2021-12-24 | 武汉光庭信息技术股份有限公司 | Method and system for constructing automatic driving experience database |
| CN114238097A (en) * | 2021-12-09 | 2022-03-25 | 深圳慕智科技有限公司 | Automatic driving system simulation test technology based on scene description language |
| CN116106839A (en) * | 2023-03-22 | 2023-05-12 | 武汉中关村硬创空间科技有限公司 | Vehicle-mounted radar reliability detection method, device, equipment and storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105868116A (en) * | 2016-04-15 | 2016-08-17 | 西北工业大学 | Semantic mutation operator based test case generation and optimization method |
| CN107608943A (en) * | 2017-09-08 | 2018-01-19 | 中国石油大学(华东) | Merge visual attention and the image method for generating captions and system of semantic notice |
| CN109739216A (en) * | 2019-01-25 | 2019-05-10 | 深圳普思英察科技有限公司 | The test method and system of the practical drive test of automated driving system |
| CN110197027A (en) * | 2019-05-28 | 2019-09-03 | 百度在线网络技术（北京）有限公司 | A kind of automatic Pilot test method, device, smart machine and server |
| CN110503743A (en) * | 2019-08-12 | 2019-11-26 | 浙江吉利汽车研究院有限公司 | A kind of method for detecting abnormality of automatic driving vehicle, device and equipment |
| CN110597711A (en) * | 2019-08-26 | 2019-12-20 | 湖南大学 | Automatic driving test case generation method based on scene and task |
| CN110991523A (en) * | 2019-11-29 | 2020-04-10 | 西安交通大学 | Interpretability evaluation method for unmanned vehicle detection algorithm performance |
| CN111123920A (en) * | 2019-12-10 | 2020-05-08 | 武汉光庭信息技术股份有限公司 | Method and device for generating automatic driving simulation test scene |
-
2020
- 2020-07-23 CN CN202010712764.9A patent/CN111859674A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105868116A (en) * | 2016-04-15 | 2016-08-17 | 西北工业大学 | Semantic mutation operator based test case generation and optimization method |
| CN107608943A (en) * | 2017-09-08 | 2018-01-19 | 中国石油大学(华东) | Merge visual attention and the image method for generating captions and system of semantic notice |
| CN109739216A (en) * | 2019-01-25 | 2019-05-10 | 深圳普思英察科技有限公司 | The test method and system of the practical drive test of automated driving system |
| CN110197027A (en) * | 2019-05-28 | 2019-09-03 | 百度在线网络技术（北京）有限公司 | A kind of automatic Pilot test method, device, smart machine and server |
| CN110503743A (en) * | 2019-08-12 | 2019-11-26 | 浙江吉利汽车研究院有限公司 | A kind of method for detecting abnormality of automatic driving vehicle, device and equipment |
| CN110597711A (en) * | 2019-08-26 | 2019-12-20 | 湖南大学 | Automatic driving test case generation method based on scene and task |
| CN110991523A (en) * | 2019-11-29 | 2020-04-10 | 西安交通大学 | Interpretability evaluation method for unmanned vehicle detection algorithm performance |
| CN111123920A (en) * | 2019-12-10 | 2020-05-08 | 武汉光庭信息技术股份有限公司 | Method and device for generating automatic driving simulation test scene |
Non-Patent Citations (1)
| Title |
|---|
| 贾迪;朱宁丹;杨宁华;吴思;李玉秀;赵明远;: "图像匹配方法研究综述", 中国图象图形学报, no. 05, 16 May 2019 (2019-05-16), pages 677 - 699 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112859810A (en) * | 2021-01-13 | 2021-05-28 | 自行科技(武汉)有限公司 | ADAS algorithm verification method and device based on Carla platform |
| CN113836111A (en) * | 2021-08-23 | 2021-12-24 | 武汉光庭信息技术股份有限公司 | Method and system for constructing automatic driving experience database |
| CN114238097A (en) * | 2021-12-09 | 2022-03-25 | 深圳慕智科技有限公司 | Automatic driving system simulation test technology based on scene description language |
| CN116106839A (en) * | 2023-03-22 | 2023-05-12 | 武汉中关村硬创空间科技有限公司 | Vehicle-mounted radar reliability detection method, device, equipment and storage medium |
| CN116106839B (en) * | 2023-03-22 | 2023-08-22 | 武汉中关村硬创空间科技有限公司 | Vehicle-mounted radar reliability detection method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111859674A (en) | Automatic driving test image scene construction method based on semantics | |
| CN109213126B (en) | Automatic driving automobile test system and method | |
| Ding et al. | A survey on safety-critical driving scenario generation—A methodological perspective | |
| CN106408015A (en) | Road fork identification and depth estimation method based on convolutional neural network | |
| CN109461342B (en) | Teaching system for unmanned motor vehicle and teaching method thereof | |
| CN109726426A (en) | A kind of Vehicular automatic driving virtual environment building method | |
| CN114358128A (en) | Method for training end-to-end automatic driving strategy | |
| CN112560277B (en) | Automobile automatic driving scene modeling method based on domain specific modeling language ADSML | |
| CN115292913A (en) | Vehicle-road-cooperation-oriented drive test perception simulation system | |
| Deter et al. | Simulating the Autonomous Future: A Look at Virtual Vehicle Environments and How to Validate Simulation Using Public Data Sets | |
| Seo et al. | Temporary traffic control device detection for road construction projects using deep learning application | |
| CN111882199A (en) | Automatic driving laser radar data amplification method based on rule variation | |
| CN116597413A (en) | Real-time traffic sign detection method based on improved YOLOv5 | |
| Lin et al. | 3D environmental perception modeling in the simulated autonomous-driving systems | |
| CN116958639A (en) | Method for detecting and enhancing false behavior of traffic light recognition model | |
| Li | A scenario-based development framework for autonomous driving | |
| Muller | Drivetruth: Automated autonomous driving dataset generation for security applications | |
| CN111160282A (en) | Traffic light detection method based on binary Yolov3 network | |
| Zhang et al. | A study on testing autonomous driving systems | |
| CN110134024A (en) | The construction method of distinctive mark object in Vehicular automatic driving virtual environment | |
| Grau et al. | VALERIE22-A photorealistic, richly metadata annotated dataset of urban environments | |
| CN115202479A (en) | Man-machine co-driving simulation system of man-in-loop and application thereof | |
| CN114550016A (en) | Unmanned aerial vehicle positioning method and system based on context information perception | |
| Zhuo et al. | A novel vehicle detection framework based on parallel vision | |
| Benčević et al. | Tool for automatic labeling of objects in images obtained from Carla autonomous driving simulator |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |