CN108319249B - Unmanned driving algorithm comprehensive evaluation system and method based on driving simulator - Google Patents
Unmanned driving algorithm comprehensive evaluation system and method based on driving simulator Download PDFInfo
- Publication number
- CN108319249B CN108319249B CN201711362553.1A CN201711362553A CN108319249B CN 108319249 B CN108319249 B CN 108319249B CN 201711362553 A CN201711362553 A CN 201711362553A CN 108319249 B CN108319249 B CN 108319249B
- Authority
- CN
- China
- Prior art keywords
- evaluation
- scene
- data
- module
- driving
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
- G05B23/00—Testing or monitoring of control systems or parts thereof
- G05B23/02—Electric testing or monitoring
- G05B23/0205—Electric testing or monitoring by means of a monitoring system capable of detecting and responding to faults
- G05B23/0218—Electric testing or monitoring by means of a monitoring system capable of detecting and responding to faults characterised by the fault detection method dealing with either existing or incipient faults
- G05B23/0221—Preprocessing measurements, e.g. data collection rate adjustment; Standardization of measurements; Time series or signal analysis, e.g. frequency analysis or wavelets; Trustworthiness of measurements; Indexes therefor; Measurements using easily measured parameters to estimate parameters difficult to measure; Virtual sensor creation; De-noising; Sensor fusion; Unconventional preprocessing inherently present in specific fault detection methods like PCA-based methods
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Automation & Control Theory (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Traffic Control Systems (AREA)
Abstract
The comprehensive evaluation module provides comparison of evaluation results under the same scene by design of three evaluation modes and combination of reproduction of the evaluation scene, and provides a basis for improving the direction of the algorithm. The method solves the problems that the existing unmanned driving evaluation method has single evaluation scene and cannot reliably evaluate; the algorithm performance of the unmanned vehicle is evaluated in a multi-level and systematic mode through three evaluation modes; multi-level and multi-angle evaluation is realized, the algorithm is deep, and a reliable evaluation result is obtained; the evaluation efficiency is high, the experimental process can be reproduced, and the cost is greatly reduced compared with the evaluation in a test field.
Description
Technical Field
The disclosure relates to the technical field of unmanned function evaluation, in particular to an unmanned algorithm comprehensive evaluation system and method based on a driving simulator.
Background
The unmanned vehicle integrates the functions of environmental perception, decision planning, driving control and the like, and can autonomously, safely and reliably run in a specific environment. With the development of unmanned vehicle technology in recent years, the need for evaluation of unmanned performance has also become more urgent.
Currently, the unmanned evaluation field is mainly related to an ADAS system, and related performance indexes are provided for the system and used for representing the quality of the system; the evaluation needs a test field with complete infrastructure and a plurality of evaluation devices, the cost is high, the evaluation period is long, the vehicle performance under extreme conditions cannot be evaluated, and the evaluation reliability needs to be checked.
Therefore, how to provide a more real and rich traffic scene to improve the evaluation efficiency and ensure the reliability of the evaluation result is a direction that the technicians in the field need to continue research.
BRIEF SUMMARY OF THE PRESENT DISCLOSURE
Technical problem to be solved
The present disclosure provides a comprehensive evaluation system and method for unmanned driving algorithm based on driving simulator, to at least partially solve the above-mentioned technical problems.
(II) technical scheme
According to one aspect of the present disclosure, there is provided a driving simulator-based unmanned algorithm comprehensive evaluation system, including: the comprehensive evaluation module comprises: the evaluation design unit is used for making evaluation design requirements; the scene simulation module is used for modeling an actual traffic scene according to the evaluation design requirement to obtain a scene model; in the running process of the scene model on the driving simulator, the scene simulation module collects image data representing a real-time scene of the running scene model and scene data and man-machine interaction data which are generated in the running process of the scene model and used for defining a scene/describing a vehicle state, and sends the scene data and the man-machine interaction data to the information management module; and the comprehensive evaluation module further comprises: the evaluation management unit is used for dividing the evaluation modes; and selecting an evaluation mode according to the requirement, acquiring indexes corresponding to the selected evaluation mode, and performing evaluation work under the operation of the scene model to obtain an evaluation result.
In some embodiments of the present disclosure, further comprising: the unmanned vehicle communication module is used for receiving the image data and the scene data sent by the information management module and sending the image data and the scene data to the unmanned vehicle; sensing decision data, actuator instruction data and human-computer interaction data generated by the unmanned vehicle algorithm are sent to an information management module; the information management module is used for communication among the scene simulation module, the comprehensive evaluation module and the unmanned vehicle communication module; and recording and storing data generated in the evaluation process.
In some embodiments of the present disclosure, the evaluation mode of the evaluation management unit division includes: module level evaluation, which is a minimum algorithm unit with a functional system and is used for providing guidance for algorithm deficiency and algorithm improvement; the system level evaluation is realized by establishing a system level index to finish evaluation through index integration on the basis of obtaining the module level index; the method is used for quantifying the performance of a system comprising a plurality of algorithms and researching the connection relationship between the systems; and the whole vehicle level evaluation is used for evaluating the external whole performance of the vehicle in the virtual traffic environment scene and establishing a whole vehicle level index.
In some embodiments of the present disclosure, the scene simulation module comprises: the scene modeling platform is used for establishing a scene model conforming to the real world through virtual scene software according to the requirements of the evaluation design unit or generating a combined scene model; the scene operation platform is used for ensuring that the generated scene model can run on the driving simulator in real time and providing image data, scene data and human-computer interaction data of the established model for the information management module; receiving an actuator instruction, and controlling the vehicle model to move in the scene model; and the driving simulator is used for collecting driving data in the scene model, sending the driving data to the information management module through the scene operation platform and storing the driving data.
In some embodiments of the present disclosure, the information management module comprises: an information center unit: the system is used for finishing information interaction among the scene simulation module, the unmanned vehicle communication module and the comprehensive evaluation module; and an information storage unit: the system is used for recording the experimental process, storing data generated in the evaluation process, pre-storing perception truth value data and human-computer interaction data and providing data query.
In some embodiments of the present disclosure, include: step A: determining an evaluation design requirement in an evaluation design unit, and sending the evaluation design requirement to a scene simulation module through an information management module; a scene model is built in a scene simulation module through virtual scene software; and B: selecting an evaluation mode in an evaluation management unit; the unmanned vehicle communication module correspondingly opens an unmanned vehicle communication port according to the requirements of the evaluation management module; and C: in the initial state, completing corresponding data processing according to the selected evaluation mode; step D: the scene model starts to operate on the scene operation platform, and the unmanned vehicle communication module, the scene simulation module and the comprehensive evaluation module start information interaction through the information management module; step E: the evaluation management unit obtains an evaluation result according to the internal algorithm and the interactive data, finishes evaluation, and automatically stores data generated during evaluation to the information management module; and checking the evaluation result generated by the evaluation management unit and analyzing the evaluation data.
In some embodiments of the disclosure, in the initial state, the data processing manner corresponding to the evaluation mode in step C includes: substep C1: when the whole vehicle level evaluation is selected, the driving of a scene model is completed through human beings, the scene operation platform sends the driving data of the driver to the information storage unit for storage, and the driving data of the driver is called through the information center unit and sent to the evaluation management unit; and sub-step C2: when the module level evaluation/system level evaluation is selected, the information center unit calls the pre-stored perception truth value data and sends the perception truth value data to the evaluation management unit.
In some embodiments of the present disclosure, during operation after selecting substep C1, the data sources for the full scale assessment include: the image data and the scene data of the scene model for testing are sent by the scene operation platform; human-computer interaction data of the unmanned vehicle; and collected driver driving data.
In some embodiments of the present disclosure, the data sources for the module level evaluation/system level evaluation during operation after selecting substep C2 include: the image data and the scene data of the scene model for testing are sent by the scene operation platform; perception decision data and actuator instruction data generated by an unmanned vehicle algorithm; human-computer interaction data of the unmanned vehicle; and sensing true value data stored in the information storage unit in advance.
In some embodiments of the present disclosure, the scene model built in step a is a scene model conforming to the real world, or an existing scene model is combined according to the evaluation requirement.
(III) advantageous effects
According to the technical scheme, the unmanned driving algorithm comprehensive evaluation system and method based on the driving simulator have at least one of the following beneficial effects:
(1) the unmanned driving performance of the vehicle is evaluated comprehensively by utilizing abundant traffic scenes provided by the scene simulation module.
(2) By utilizing the multi-level evaluation design of module level evaluation, system level evaluation and whole vehicle level evaluation, evaluation can be developed from the external overall performance and the internal algorithm performance of the vehicle, the evaluation coverage is wide, the evaluation result is more reliable, and the algorithm performance of the unmanned vehicle can be evaluated more systematically and effectively.
(3) In the evaluation management unit, the evaluation scene is reproduced, comparison of evaluation results in the same scene is provided, and a basis is provided for the algorithm improvement direction.
(4) The comprehensive information unit is utilized to complete information interaction among all the modules, and the system is easy to operate, high in automation degree and high in evaluation efficiency.
The method solves the problems that the existing unmanned driving evaluation method is single in evaluation scene and cannot reliably evaluate. The algorithm performance of the unmanned vehicle is evaluated in a multi-level and systematic mode through three evaluation modes. Functionally, the method realizes multi-level and multi-angle evaluation, goes deep into the algorithm and obtains a reliable evaluation result; from the overall perspective, the method has the advantages of high evaluation efficiency, reproducible experimental process and greatly reduced cost compared with the evaluation in a test field.
Drawings
Fig. 1 is a general frame diagram of the unmanned algorithm comprehensive evaluation system based on the driving simulator.
Fig. 2 is a frame diagram of the unmanned algorithm comprehensive evaluation system based on the driving simulator.
Fig. 3 is a flow chart of an evaluation method of the unmanned driving algorithm comprehensive evaluation system based on the driving simulator.
Detailed Description
The disclosure provides a comprehensive evaluation system and method for unmanned driving algorithm based on a driving simulator. A scene model with a real-world traffic scene is established through the scene simulation module, and data generated in the operation process of the scene model are sent to the information management module, so that abundant traffic scenes are provided, and the unmanned driving performance of the vehicle can be comprehensively evaluated. The comprehensive evaluation module provides comparison of evaluation results in the same scene by design of three evaluation modes and combination of reproduction of evaluation scenes, provides basis for algorithm improvement direction, and realizes multi-level and systematic evaluation of the performance of the unmanned vehicle algorithm. The information management module realizes data interaction with the unmanned vehicle through the unmanned vehicle communication module. The unmanned vehicle communication module, the scene simulation module and the comprehensive evaluation module carry out information interaction through the information management unit; and recording and storing data generated in the test process through the information management module.
For the purpose of promoting a better understanding of the objects, aspects and advantages of the present disclosure, reference is made to the following detailed description taken in conjunction with the accompanying drawings.
Certain embodiments of the present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all embodiments of the disclosure are shown. Indeed, various embodiments of the disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will satisfy applicable legal requirements.
In a first exemplary embodiment of the present disclosure, a driving simulator-based unmanned algorithm comprehensive evaluation system is provided. Fig. 1 is a general frame diagram of the unmanned algorithm comprehensive evaluation system based on the driving simulator. As shown in fig. 1, includes: the comprehensive evaluation module is responsible for finishing evaluation design requirements; dividing the evaluation modes; and performing evaluation work on the scene model, collecting data and sending the data to the information management module. The unmanned vehicle communication module is used for receiving the image data and the scene data sent by the information management module and sending the image data and the scene data to the unmanned vehicle; and sending the perception decision data and the actuator instruction data generated by the unmanned vehicle algorithm to the information management module. The scene simulation module is used for modeling a traffic scene of the real world; in the virtual scene operation process, a scene simulation module collects image data and scene data of an operated model and sends the image data and the scene data to an information management module, and the information management module utilizes the received image data and the received scene data as a criterion for evaluating the performance of a vehicle; and receiving and executing the actuator instruction data sent by the information management module, and controlling the vehicle model to complete various actions such as turning, braking, accelerating and the like in the scene. During evaluation, data acquisition can be obtained by real driver driving or man-machine driving, and is not particularly limited herein. The information management module is used for communication among the unmanned vehicle communication module, the scene simulation module and the comprehensive evaluation module; and recording and storing data generated in the evaluation process.
Fig. 2 is a frame diagram of the unmanned algorithm comprehensive evaluation system based on the driving simulator. As shown in fig. 2, the comprehensive evaluation module includes: and the evaluation design unit is used for finishing the evaluation design requirements, specifically comprising the establishment, the editing and the execution of an evaluation process, and can also be used for archiving the designed evaluation flow, reloading the stored evaluation flow and sending the evaluation flow to the information management unit. And the evaluation management unit selects an evaluation mode according to the requirement, acquires the corresponding index of the selected evaluation mode, runs an evaluation algorithm, displays the evaluation result and stores the evaluation result. The evaluation mode in the evaluation management unit specifically comprises the following steps: and the module level evaluation is a minimum algorithm unit with a certain functional system and is used for providing guidance for algorithm deficiency and algorithm improvement. For example, the indexes corresponding to the vehicle detection in the sensing system may be a vehicle detection accuracy, a vehicle trajectory prediction accuracy, and the like. The system level evaluation is realized by establishing a system level index to finish evaluation through index integration on the basis of obtaining the module level index; the method is used for quantifying the performance of a system comprising a plurality of algorithms and researching the connection relation between the systems. For example, the detection accuracy of static obstacles and the detection accuracy and detection time of dynamic obstacles in the sensing system; the man-machine co-driving harmony degree and the safety degree of the man-machine co-driving system. And the whole vehicle level evaluation is used for evaluating the external overall performance of the vehicle in the virtual traffic environment scene and establishing a whole vehicle level index. Such as accident rate, comfort level, power level, fuel economy percentage, etc.
The scene simulation module comprises: and the scene modeling platform is used for establishing a scene model which accords with reality through virtual scene software according to the requirements of the evaluation design unit, or generating a combined scene model according to the requirements of the evaluation design unit. The scene model is a scene model established according to a real road environment, a traffic environment and a weather environment. The scene operation platform is used for ensuring that the generated scene model can run on the driving simulator in real time; providing image data, scene data and man-machine interaction data of the established model to an information management module; and receiving an actuator instruction from the unmanned communication module forwarded by the information management module, and controlling the vehicle model to complete turning, braking, accelerating and other actions in the scene model. And the driving simulator is used for collecting driving data in the scene model, wherein the driving data can be driving data of a driver or man-machine driving data, and sending the driving data to the information management module through the scene operation platform and storing the driving data. The driving simulator provided by the disclosure comprises hardware such as a real accelerator pedal, a brake pedal, a clutch pedal, a gear shifting handle, a steering wheel, an instrument panel, a key switch and the like, wherein the accelerator pedal is provided with a linear displacement sensor, the brake pedal is provided with a hydraulic sensor, and the steering wheel is provided with a steering angle sensor and a pressure sensor (located at positions where both hands of the steering wheel are easy to contact).
The information management module includes: an information storage unit: the system is used for recording the experimental process, storing the data generated in the evaluation process, the pre-stored perception truth value data and the driver driving data and providing data query. And an information center unit: the system is used for finishing information interaction among the scene simulation module, the unmanned vehicle communication module and the comprehensive evaluation module. The following is the specific work that the information center unit is responsible for: and acquiring image data and scene data sent by the scene simulation module, and sending the data to the unmanned vehicle communication module. And sensing decision data sent by the unmanned vehicle communication module are collected to the information management module. And acquiring actuator instruction data sent by the unmanned vehicle communication module and sending the actuator instruction data to the scene simulation module. And warning information for the driver, which is sent by the unmanned vehicle communication module, is collected to the scene simulation module and the information management module. And collecting the evaluation design requirements of the evaluation design unit and sending the evaluation design requirements to the scene simulation module, so that the scene simulation module manually/automatically generates a scene model according to the evaluation experiment requirements. And collecting information of the steering wheel pressure sensor from the scene simulation module to the information management module. When the module level evaluation/system level evaluation is selected, the perception truth value data stored in advance by the evaluation information management module is transmitted to the evaluation management unit for evaluation. When the vehicle level evaluation is selected for taking the evaluation, the driver driving data stored by the calling information management module is sent to the evaluation management unit for vehicle level evaluation.
The modules are connected through Ethernet, and different functional modules are distributed in different network nodes to form a small distributed network. Sensor information of the driving simulator relates to displacement signals, pressure signals and the like, and the scene operation platform completes conversion of the signals and sends the signals to the information management module through the Ethernet.
The comprehensive unmanned algorithm evaluation system based on the driving simulator provided by the disclosure is introduced.
The software, hardware structure, and specific data related to the present disclosure are described in detail below, respectively.
The virtual scene modeling software related to the present disclosure mainly adopts: SCANeR DT software developed by the vehicle simulation and perception research team of Reynolds corporation, France, together with OKTAL corporation, or UC-win/Road software developed by Japan.
The driving simulator provided by the disclosure can realize man-machine driving together, can be driven by a real driver, provides driving data of a human driver in a scene model through a scene operation platform, and sends the driving data to the comprehensive evaluation module.
The image data according to the present disclosure mainly includes: the method comprises the steps of acquiring scene images with timestamps in real time in the operation process of a scene model, namely images of a front camera, a left camera, a right camera and a rear camera which are virtual in source.
The scene data related to the present disclosure mainly includes: generating data defining a scene and data describing a vehicle state during operation of the scene model; the method specifically comprises the following steps: natural environment data in the scene model, namely information such as rainfall, snow quantity, air quantity and the like; road surface environment data in the scene model, namely information such as road number, road width, road curvature, road gradient, lane line type and the like; the traffic environment data in the scene model comprises information such as traffic signs and signal lamps; driving behavior data, namely information such as an accelerator position, a brake pedal position, a gear position, a steering wheel angle and the like of a vehicle model in the scene running platform; and vehicle state data, namely information such as the vehicle speed, the acceleration, the attitude angle, the rotating speed, the distance from surrounding vehicles and obstacles, the distance from wheels to a lane line and the like of the vehicle model.
The perceptual decision data to which the present disclosure relates generally include: and sensing decision data, which are data generated during the operation of the intermediate link of the unmanned vehicle communication module, such as detected lane lines, traffic signs and traffic lights, and planned driving paths and other data.
The actuator instruction data related to the present disclosure mainly includes: the actuator commands data.
The perceptual truth data involved in the present disclosure mainly includes the following four categories: the first type is a picture representing the area where the lane line is located, the second type represents the data of the color of the traffic light corresponding to the scene number, the third type represents the data of the meaning of the traffic sign corresponding to the scene number, and the fourth type represents the data of the position of the obstacle under the corresponding scene number.
The human-computer interaction data related in the disclosure mainly includes: including warning information to the driver by the communication software and pressure sensor information of the steering wheel of the driving simulator.
The above detailed description of the software, hardware structures and specific data involved in the present disclosure is presented.
In a first exemplary embodiment of the disclosure, an evaluation method of the unmanned driving algorithm comprehensive evaluation system based on the driving simulator is also provided. Fig. 3 is a flow chart of an evaluation method of the unmanned driving algorithm comprehensive evaluation system based on the driving simulator. As shown in figure 3 of the drawings,
step A: determining an evaluation design requirement in an evaluation design unit, and sending the evaluation design requirement to a scene simulation module through an information management module; and a scene model is built in a scene simulation module through virtual scene software.
And B: selecting an evaluation mode in an evaluation management unit; and the unmanned vehicle communication module correspondingly opens the unmanned vehicle communication port according to the requirements of the evaluation management module. The specifically opened communication port includes: a communication structure for receiving image data and scene data; the communication interface is used for transmitting the actuator instruction data; and (3) a human-computer interaction data interface.
And C: when the whole vehicle level evaluation is selected, the driving of the scene model is completed through human beings, the scene operation platform sends the driving data of the driver to the information storage unit for storage, and the driving data of the driver is called through the information center unit and sent to the evaluation management unit.
Step D: the scene model starts to operate on the scene operation platform, and the unmanned vehicle communication module, the scene simulation module and the comprehensive evaluation module start information interaction through the information management module.
Step E: the evaluation management unit obtains an evaluation result according to the internal algorithm and the interactive data, the evaluation is completed, and the data generated in the evaluation period can be automatically stored in the information management module. The data source of the whole vehicle level evaluation mainly comprises image data and scene data of a scene model for testing, which are sent by a scene operation platform; human-computer interaction data of the unmanned vehicle; collected driver driving data. And checking the evaluation result generated by the evaluation management unit and analyzing the evaluation data.
The first embodiment of the comprehensive evaluation method for the unmanned driving algorithm based on the driving simulator provided by the disclosure is described above.
In a second exemplary embodiment of the present disclosure, an evaluation method of the unmanned driving algorithm comprehensive evaluation system based on the driving simulator is also provided. It is different from the first embodiment in that the evaluation mode is selected as module level evaluation/system level evaluation. Fig. 3 is a flow chart of an evaluation method of the unmanned driving algorithm comprehensive evaluation system based on the driving simulator. As shown in figure 3 of the drawings,
step A: determining an evaluation design requirement in an evaluation design unit, and sending the evaluation design requirement to a scene simulation module through an information management module; and a scene model is built in a scene simulation module through virtual scene software.
And B: selecting an evaluation mode in an evaluation management unit; and the unmanned vehicle communication module correspondingly opens the unmanned vehicle communication port according to the requirements of the evaluation management module. The specifically opened communication port includes: a communication structure for receiving image data and scene data; the communication interface is used for transmitting the actuator instruction data; a man-machine interaction data interface and a communication interface for receiving perception decision data.
And C: when the module level evaluation/system level evaluation is selected, the information center unit calls the pre-stored perception truth value data and sends the perception truth value data to the evaluation management unit.
Step D: the scene model starts to operate on the scene operation platform, and the unmanned vehicle communication module, the scene simulation module and the comprehensive evaluation module start information interaction through the information management module.
Step E: the evaluation management unit obtains an evaluation result according to the internal algorithm and the interactive data, the evaluation is completed, and the data generated in the evaluation period can be automatically stored in the information management module. The data source of the module level evaluation/system level evaluation mainly comprises image data and scene data of a scene model for testing, which are sent by a scene operation platform; perception decision data and actuator instruction data generated by an unmanned vehicle algorithm; human-computer interaction data of the unmanned vehicle; perception truth data stored in the data management unit in advance. And checking the evaluation result generated by the evaluation management unit and analyzing the evaluation data.
The above description is completed on the second embodiment of the comprehensive evaluation method for unmanned driving algorithm based on driving simulator provided by the present disclosure.
So far, the embodiments of the present disclosure have been described in detail with reference to the accompanying drawings. It is to be noted that, in the attached drawings or in the description, the implementation modes not shown or described are all the modes known by the ordinary skilled person in the field of technology, and are not described in detail. Further, the above definitions of the various elements and methods are not limited to the various specific structures, shapes or arrangements of parts mentioned in the examples, which may be easily modified or substituted by those of ordinary skill in the art.
From the above description, those skilled in the art should clearly recognize that the unmanned driving algorithm comprehensive evaluation system and method based on the driving simulator of the present disclosure.
In summary, the unmanned driving assessment method solves the problems that an existing unmanned driving assessment method is single in assessment scene and cannot reliably assess. The algorithm performance of the unmanned vehicle is evaluated in a multi-level and systematic mode through three evaluation modes. Functionally, the method realizes multi-level and multi-angle evaluation, goes deep into the algorithm and obtains a reliable evaluation result; from the overall perspective, the method has the advantages of high evaluation efficiency, reproducible experimental process and greatly reduced cost compared with the evaluation in a test field.
It should also be noted that directional terms, such as "upper", "lower", "front", "rear", "left", "right", and the like, used in the embodiments are only directions referring to the drawings, and are not intended to limit the scope of the present disclosure. Throughout the drawings, like elements are represented by like or similar reference numerals. Conventional structures or constructions will be omitted when they may obscure the understanding of the present disclosure.
And the shapes and sizes of the respective components in the drawings do not reflect actual sizes and proportions, but merely illustrate the contents of the embodiments of the present disclosure. Furthermore, in the claims, any reference signs placed between parentheses shall not be construed as limiting the claim.
Furthermore, the word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements.
The use of ordinal numbers such as "first," "second," "third," etc., in the specification and claims to modify a corresponding element does not by itself connote any ordinal number of the element or any ordering of one element from another or the order of manufacture, and the use of the ordinal numbers is only used to distinguish one element having a certain name from another element having a same name.
In addition, unless steps are specifically described or must occur in sequence, the order of the steps is not limited to that listed above and may be changed or rearranged as desired by the desired design. The embodiments described above may be mixed and matched with each other or with other embodiments based on design and reliability considerations, i.e., technical features in different embodiments may be freely combined to form further embodiments.
The algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose systems may also be used with the teachings herein. The required structure for constructing such a system will be apparent from the description above. Moreover, this disclosure is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the present disclosure as described herein, and any descriptions above of specific languages are provided for disclosure of enablement and best mode of the present disclosure.
The disclosure may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. Various component embodiments of the disclosure may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or Digital Signal Processor (DSP) may be used in practice to implement some or all of the functionality of some or all of the components in the relevant apparatus according to embodiments of the present disclosure. The present disclosure may also be embodied as apparatus or device programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing the present disclosure may be stored on a computer-readable medium or may be in the form of one or more signals. Such a signal may be downloaded from an internet website or provided on a carrier signal or in any other form.
Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Also in the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware.
Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the disclosure, various features of the disclosure are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various disclosed aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that is, the claimed disclosure requires more features than are expressly recited in each claim. Rather, as the following claims reflect, disclosed aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this disclosure.
The above-mentioned embodiments are intended to illustrate the objects, aspects and advantages of the present disclosure in further detail, and it should be understood that the above-mentioned embodiments are only illustrative of the present disclosure and are not intended to limit the present disclosure, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present disclosure should be included in the scope of the present disclosure.
Claims (6)
1. An unmanned algorithm comprehensive evaluation system based on a driving simulator comprises:
the comprehensive evaluation module comprises:
the evaluation design unit is used for making evaluation design requirements;
the scene simulation module is used for modeling an actual traffic scene according to the evaluation design requirement to obtain a scene model; in the running process of the scene model on the driving simulator, the scene simulation module collects image data representing a real-time scene of the running scene model and scene data and man-machine interaction data which are generated in the running process of the scene model and used for defining a scene/describing a vehicle state, and sends the scene data and the man-machine interaction data to the information management module; wherein the scene simulation module comprises:
the scene modeling platform is used for establishing a scene model conforming to the real world through virtual scene software according to the requirements of the evaluation design unit or generating a combined scene model;
the scene operation platform is used for ensuring that the generated scene model can run on the driving simulator in real time and providing image data, scene data and human-computer interaction data of the established model for the information management module; receiving an actuator instruction, and controlling the vehicle model to move in the scene model; and
the driving simulator is used for collecting driving data in the scene model, sending the driving data to the information management module through the scene operation platform and storing the driving data;
the comprehensive evaluation module further comprises:
the evaluation management unit is used for dividing the evaluation modes; selecting an evaluation mode according to requirements, acquiring indexes corresponding to the selected evaluation mode, and performing evaluation work under the operation of a scene model to obtain an evaluation result; wherein the evaluation mode divided by the evaluation management unit includes:
module level evaluation, which is a minimum algorithm unit with a functional system and is used for providing guidance for algorithm deficiency and algorithm improvement;
the system level evaluation is realized by establishing a system level index to finish evaluation through index integration on the basis of obtaining the module level index; the method is used for quantifying the performance of a system comprising a plurality of algorithms and researching the connection relationship between the systems; and
the whole vehicle level evaluation is used for evaluating the external whole performance of the vehicle in the virtual traffic environment scene and establishing a whole vehicle level index;
further comprising:
the unmanned vehicle communication module is used for receiving the image data and the scene data sent by the information management module and sending the image data and the scene data to the unmanned vehicle; sensing decision data, actuator instruction data and human-computer interaction data generated by the unmanned vehicle algorithm are sent to an information management module; and
the information management module is used for communication among the scene simulation module, the comprehensive evaluation module and the unmanned vehicle communication module; and recording and storing data generated in the evaluation process.
2. The unmanned algorithm comprehensive evaluation system of claim 1, wherein the information management module comprises:
an information center unit: the system is used for finishing information interaction among the scene simulation module, the unmanned vehicle communication module and the comprehensive evaluation module; and
an information storage unit: the system is used for recording the experimental process, storing data generated in the evaluation process, pre-storing perception truth value data and driver driving data and providing data query.
3. An evaluation method of an unmanned algorithm comprehensive evaluation system based on a driving simulator comprises the following steps:
step A: determining an evaluation design requirement in an evaluation design unit, and sending the evaluation design requirement to a scene simulation module through an information management module; a scene model is built in a scene simulation module through virtual scene software;
and B: selecting an evaluation mode in an evaluation management unit; the unmanned vehicle communication module correspondingly opens an unmanned vehicle communication port according to the requirements of the evaluation management module;
and C: in the initial state, completing corresponding data processing according to the selected evaluation mode; in an initial state, the data processing mode corresponding to the evaluation mode in the step C includes:
substep C1: when the whole vehicle level evaluation is selected, the driving of a scene model is completed through human beings, the scene operation platform sends the driving data of the driver to the information storage unit for storage, and the driving data of the driver is called through the information center unit and sent to the evaluation management unit; and
substep C2: when module level evaluation/system level evaluation is selected, the information center unit calls the pre-stored perception truth value data and sends the perception truth value data to the evaluation management unit;
step D: the scene model starts to operate on the scene operation platform, and the unmanned vehicle communication module, the scene simulation module and the comprehensive evaluation module start information interaction through the information management module; and
step E: the evaluation management unit obtains an evaluation result according to the internal algorithm and the interactive data, finishes evaluation, and automatically stores data generated during evaluation to the information management module; and checking the evaluation result generated by the evaluation management unit and analyzing the evaluation data.
4. The method of claim 3, wherein said vehicle class assessment data source comprises, during operation after selecting substep C1:
the image data and the scene data of the scene model for testing are sent by the scene operation platform;
human-computer interaction data of the unmanned vehicle; and
collected driver driving data.
5. The method of claim 3, wherein said module level/system level evaluation data sources during operation after selecting substep C2 comprise:
the image data and the scene data of the scene model for testing are sent by the scene operation platform;
perception decision data and actuator instruction data generated by an unmanned vehicle algorithm;
human-computer interaction data of the unmanned vehicle; and
sensing truth value data stored in the information storage unit in advance.
6. The evaluation method of the comprehensive evaluation system for unmanned algorithm according to claim 3, wherein the scene model built in the step A is a scene model conforming to the real world, or an existing scene model is combined according to the evaluation requirement.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711362553.1A CN108319249B (en) | 2017-12-15 | 2017-12-15 | Unmanned driving algorithm comprehensive evaluation system and method based on driving simulator |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711362553.1A CN108319249B (en) | 2017-12-15 | 2017-12-15 | Unmanned driving algorithm comprehensive evaluation system and method based on driving simulator |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108319249A CN108319249A (en) | 2018-07-24 |
| CN108319249B true CN108319249B (en) | 2020-10-30 |
Family
ID=62892336
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711362553.1A Active CN108319249B (en) | 2017-12-15 | 2017-12-15 | Unmanned driving algorithm comprehensive evaluation system and method based on driving simulator |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108319249B (en) |
Families Citing this family (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109470498A (en) * | 2018-09-27 | 2019-03-15 | 交控科技股份有限公司 | Fully automatic operation system integration testing method |
| CN109614886B (en) * | 2018-11-23 | 2022-06-24 | 中国人民解放军63966部队 | Performance evaluation method and device of unmanned/autonomous driving system |
| CN109557904B (en) * | 2018-12-06 | 2020-07-10 | 百度在线网络技术（北京）有限公司 | Test method, device, equipment and medium |
| CN109765060A (en) * | 2018-12-29 | 2019-05-17 | 同济大学 | A kind of automatic driving vehicle traffic coordinating virtual test system and method |
| CN109816014A (en) * | 2019-01-22 | 2019-05-28 | 天津大学 | Generate method of the deep learning target detection network training with labeled data collection |
| CN109782737A (en) * | 2019-01-28 | 2019-05-21 | 平安科技（深圳）有限公司 | Unmanned vehicle analog detection method and system |
| CN111859597B (en) * | 2019-04-18 | 2024-04-05 | 北京京东乾石科技有限公司 | Evaluation method and system of automatic driving algorithm |
| CN110046833A (en) * | 2019-05-13 | 2019-07-23 | 吉林大学 | A kind of traffic congestion auxiliary system virtual test system |
| CN110673590B (en) * | 2019-10-21 | 2021-02-05 | 安徽江淮汽车集团股份有限公司 | Method, device and equipment for judging reliability of automatic driving system and storage medium |
| CN111006879B (en) * | 2019-12-25 | 2021-11-02 | 畅加风行(苏州)智能科技有限公司 | Port environment-oriented man-machine hybrid driving automatic driving test method and test platform thereof |
| CN111983935B (en) * | 2020-08-19 | 2024-04-05 | 北京京东叁佰陆拾度电子商务有限公司 | Performance evaluation method and device |
| CN112924185B (en) * | 2021-01-22 | 2021-11-30 | 大连理工大学 | Human-computer co-driving test method based on digital twin virtual-real interaction technology |
| CN113420952A (en) * | 2021-05-17 | 2021-09-21 | 同济大学 | Automobile human-computer interaction testing and evaluating system based on simulated driving |
| CN114755035B (en) * | 2022-06-15 | 2022-09-09 | 中汽信息科技(天津)有限公司 | Intelligent driving multidimensional test method based on vehicle-mounted terminal |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8920345B2 (en) * | 2008-12-07 | 2014-12-30 | Apdm, Inc. | System and apparatus for continuous monitoring of movement disorders |
| CN105388021A (en) * | 2015-10-21 | 2016-03-09 | 重庆交通大学 | ADAS virtual development and test system |
| CN106153352A (en) * | 2016-07-04 | 2016-11-23 | 江苏大学 | A kind of automatic driving vehicle test and verification platform and method of testing thereof |
| CN106873397A (en) * | 2017-01-23 | 2017-06-20 | 同济大学 | Intelligent network joins automobile " hardware in loop " accelerated loading emulation test system |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102495631B (en) * | 2011-12-09 | 2013-08-21 | 中国科学院合肥物质科学研究院 | Intelligent control method of driverless vehicle tracking desired trajectory |
| CN202815590U (en) * | 2012-07-30 | 2013-03-20 | 中国航天科工集团第三研究院第八三五七研究所 | Control system for mini self-driving unmanned vehicle |
-
2017
- 2017-12-15 CN CN201711362553.1A patent/CN108319249B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8920345B2 (en) * | 2008-12-07 | 2014-12-30 | Apdm, Inc. | System and apparatus for continuous monitoring of movement disorders |
| CN105388021A (en) * | 2015-10-21 | 2016-03-09 | 重庆交通大学 | ADAS virtual development and test system |
| CN106153352A (en) * | 2016-07-04 | 2016-11-23 | 江苏大学 | A kind of automatic driving vehicle test and verification platform and method of testing thereof |
| CN106873397A (en) * | 2017-01-23 | 2017-06-20 | 同济大学 | Intelligent network joins automobile " hardware in loop " accelerated loading emulation test system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108319249A (en) | 2018-07-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108319249B (en) | Unmanned driving algorithm comprehensive evaluation system and method based on driving simulator | |
| US11755396B2 (en) | Generating autonomous vehicle simulation data from logged data | |
| JP6857371B2 (en) | Learning methods that support safe autonomous driving, and learning devices, test methods, and test devices that use them. | |
| US10816978B1 (en) | Automated vehicle artificial intelligence training based on simulations | |
| CN111061167B (en) | Intelligent network connection demonstration area-oriented mixed reality automatic driving test method and virtual test platform | |
| Szalay | Next generation X-in-the-loop validation methodology for automated vehicle systems | |
| CN109884916A (en) | A kind of automatic Pilot Simulation Evaluation method and device | |
| CN110103983A (en) | System and method for the verifying of end-to-end autonomous vehicle | |
| JP2022511333A (en) | Methods and systems for modifying control units of autonomous vehicles | |
| CN109726426A (en) | A kind of Vehicular automatic driving virtual environment building method | |
| US20220198107A1 (en) | Simulations for evaluating driving behaviors of autonomous vehicles | |
| CN114365108A (en) | Automatic driving vehicle simulation method under virtual environment | |
| CN112035951A (en) | Simulation platform and simulation method for automatic driving algorithm verification | |
| US20220289253A1 (en) | Method for evaluating autonomous driving system, apparatus and storage medium | |
| US20210049243A1 (en) | Hardware In Loop Testing and Generation of Latency Profiles for Use in Simulation | |
| CN112784867A (en) | Training deep neural networks using synthetic images | |
| Guvenc et al. | Simulation Environment for Safety Assessment of CEAV Deployment in Linden | |
| Artunedo et al. | Advanced co-simulation framework for cooperative maneuvers among vehicles | |
| Curiel-Ramirez et al. | Hardware in the loop framework proposal for a semi-autonomous car architecture in a closed route environment | |
| CN112597005A (en) | Closed-loop simulation test system and method for automatic driving full-stack algorithm | |
| CN116403174A (en) | End-to-end automatic driving method, system, simulation system and storage medium | |
| US20230196619A1 (en) | Validation of virtual camera models | |
| CN110134024A (en) | The construction method of distinctive mark object in Vehicular automatic driving virtual environment | |
| Miura et al. | Converting driving scenario framework for testing self-driving systems | |
| CN114578712A (en) | Multifunctional underwater autonomous vehicle cluster simulation system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |