EP4073782A1 - Perception error models - Google Patents
Perception error modelsInfo
- Publication number
- EP4073782A1 EP4073782A1 EP20900042.1A EP20900042A EP4073782A1 EP 4073782 A1 EP4073782 A1 EP 4073782A1 EP 20900042 A EP20900042 A EP 20900042A EP 4073782 A1 EP4073782 A1 EP 4073782A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- data
- vehicle
- error
- scenario
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000008447 perception Effects 0.000 title claims abstract description 106
- 238000000034 method Methods 0.000 claims abstract description 123
- 238000004088 simulation Methods 0.000 claims description 198
- 238000009826 distribution Methods 0.000 claims description 91
- 230000004044 response Effects 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims 1
- 230000008569 process Effects 0.000 description 56
- 230000007613 environmental effect Effects 0.000 description 43
- 238000004458 analytical method Methods 0.000 description 29
- 238000004891 communication Methods 0.000 description 17
- 241001465754 Metazoa Species 0.000 description 13
- 238000013179 statistical model Methods 0.000 description 13
- 238000012360 testing method Methods 0.000 description 12
- 239000003795 chemical substances by application Substances 0.000 description 9
- 230000003094 perturbing effect Effects 0.000 description 9
- 230000006870 function Effects 0.000 description 8
- 230000004807 localization Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 230000003068 static effect Effects 0.000 description 7
- 230000001133 acceleration Effects 0.000 description 6
- 230000009471 action Effects 0.000 description 6
- 230000000670 limiting effect Effects 0.000 description 5
- 230000033001 locomotion Effects 0.000 description 5
- 238000005259 measurement Methods 0.000 description 5
- 230000001960 triggered effect Effects 0.000 description 5
- 238000010200 validation analysis Methods 0.000 description 5
- 238000007796 conventional method Methods 0.000 description 4
- 238000001514 detection method Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000010206 sensitivity analysis Methods 0.000 description 4
- 230000000153 supplemental effect Effects 0.000 description 4
- 238000012549 training Methods 0.000 description 4
- 230000004075 alteration Effects 0.000 description 3
- 238000003491 array Methods 0.000 description 3
- 230000001143 conditioned effect Effects 0.000 description 3
- 230000003247 decreasing effect Effects 0.000 description 3
- 230000004888 barrier function Effects 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 230000033228 biological regulation Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000002950 deficient Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000003064 k means clustering Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000001556 precipitation Methods 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 239000013589 supplement Substances 0.000 description 2
- 238000000342 Monte Carlo simulation Methods 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 230000004931 aggregating effect Effects 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000010267 cellular communication Effects 0.000 description 1
- 238000001816 cooling Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000003066 decision tree Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000013213 extrapolation Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000003550 marker Substances 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 230000000116 mitigating effect Effects 0.000 description 1
- 230000036961 partial effect Effects 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 230000002829 reductive effect Effects 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 108700002783 roundabout Proteins 0.000 description 1
- 238000010845 search algorithm Methods 0.000 description 1
- 239000000725 suspension Substances 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3664—Environments for testing or debugging software
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3684—Test management for test design, e.g. generating new test cases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/20—Design optimisation, verification or simulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
-
- G—PHYSICS
- G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
- G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
- G09B9/00—Simulators for teaching or training purposes
- G09B9/02—Simulators for teaching or training purposes for teaching control of vehicles or other craft
- G09B9/04—Simulators for teaching or training purposes for teaching control of vehicles or other craft for teaching control of land vehicles
- G09B9/042—Simulators for teaching or training purposes for teaching control of vehicles or other craft for teaching control of land vehicles providing simulation in a real vehicle
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3696—Methods or tools to render software testable
Definitions
- An autonomous vehicle can use an autonomous vehicle controller to guide the autonomous vehicle through an environment.
- the autonomous vehicle controller can use planning methods, apparatuses, and systems to determine a drive path and guide the autonomous vehicle through the environment that contains dynamic objects (e.g., vehicles, pedestrians, animals, and the like) and static objects (e.g., buildings, signage, stalled vehicles, and the like).
- dynamic objects e.g., vehicles, pedestrians, animals, and the like
- static objects e.g., buildings, signage, stalled vehicles, and the like.
- it in order to ensure safety of the occupants, it’s important to validate the safety of the controller.
- FIG. 1 illustrates generating vehicle performance data associated with a vehicle controller based on a parameterized scenario.
- FIG. 2 illustrates the computing device(s) generating scenario data based at least in part on the log data generated by the vehicle(s), where the scenario data depicts one or more variations of a scenario.
- FIG. 3 illustrates generating error model data based at least in part on vehicle data and ground truth data.
- FIG. 4 illustrates using error model data to perturb a simulation by providing at least one of an error or an uncertainty associated with a simulated environment.
- FIG. 5 illustrates the computing device(s) generating perception error model data based at least in part on log data generated by the vehicle(s) and ground truth data.
- FIG. 6 illustrates the computing device(s) generating simulation data based at least in part on parameterized scenario data and generating safety metric data based at least in part on the simulation data.
- FIG. 7 depicts a block diagram of an example system for implementing the techniques described herein.
- FIG. 8 depicts a flow diagram of an example process for determining a safety metric associated with a vehicle controller, in accordance with examples of the disclosure.
- FIG. 9 depicts a flow diagram of an example process for determining a statistical model associated with a subsystem of an autonomous vehicle.
- FIG. 10 depicts a plurality of regions of an environment that are associated with an error probability.
- FIG. 11 illustrates vehicle data and the environment represented by the vehicle data and differences between the vehicle data and the environment.
- FIG. 12 depicts a flow diagram of an example process for determining an error model and determining perturbed simulation data.
- Simulations described herein are directed to various aspects of determining performance metrics of a system.
- such performance metrics may be determined, for example, using simulations in conjunction with other performance metric determinations.
- Simulations can be used to validate software (e.g., a vehicle controller) executed on vehicles (e.g., autonomous vehicles) and gather safety metrics to ensure that the software is able to safely control such vehicles in various scenarios.
- simulations can be used to learn about the constraints of autonomous vehicles that use the autonomous controller. For instance, simulations can be used to understand the operational space of an autonomous vehicle (e.g., an envelope in which the autonomous controller effectively controls the autonomous vehicle) in view of surface conditions, ambient noise, faulty components, etc.
- Simulations can also be useful for generating feedback for improving operations and designs of autonomous vehicles. For instance, in some examples, simulations can be useful for determining an amount of redundancy that is required in an autonomous controller, or how to modify a behavior of the autonomous controller based on what is learned through simulations. Furthermore, in additional or alternative examples, simulations can be useful to inform the hardware design of autonomous vehicles, such as optimizing placement of sensors on an autonomous vehicle.
- a scenario parameter can be used to parameterize characteristics and/or attributes of an object within the scenario and provide variations of the scenario.
- a vehicle or multiple vehicles can traverse an environment and generate log data associated with the environment.
- the log data can include sensor data captured by one or more sensors of the vehicle, perception data indicating objects identified by one or more systems onboard the vehicle (or produced during a post-processing phase), prediction data indicating an intent of objects (whether produced during the recording or subsequent thereto), and/or status data indicating diagnostic information, trajectory information, and other information generated by the vehicle.
- the vehicle can transmit the log data, via a network, to a database that stores log data and/or to a computing device that analyzes the log data.
- the computing device can, based on the log data, determine various scenarios, a frequency of the various scenarios, and regions of the environment that are associated with the various scenarios. In some instances, the computing device can group similar scenarios represented in the log data. For example, scenarios can be grouped together using, for example, k-means clustering and/or evaluating a weighted distance (e.g., Euclidian) between parameters of the environment (e.g., day time, night time, precipitation, vehicle position/velocity, object position/velocity, road segments, etc.).
- k-means clustering e.g., Euclidian
- clustering similar or like scenarios can reduce the amount of computational resources required for simulating an autonomous controller in an environment by simulating the autonomous controller in unique scenarios rather than simulating the autonomous vehicle in nearly similar scenarios which results in redundant simulation data/results.
- an autonomous controller may be expected to perform (and/or may have demonstrably performed) similarly in like scenarios.
- the computing device can determine a rate at which pedestrians appear at a crosswalk based on a number of pedestrians represented in the log data. In some instances, the computing device can determine a probability of detecting a pedestrian at the crosswalk based on the rate and a time period of operating the autonomous vehicle. Based on the log data, the computing device can determine a scenario(s) and identify a scenario parameter(s) based on the scenario(s) that can be used in simulation.
- the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data.
- the computing device can be configured to introduce inconsistencies in a scenario parameter of an object.
- an error model can indicate an error and/or an error percentage associated with a scenario parameter.
- the scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller.
- Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground- truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error.
- error models may map particular errors with probabilities/frequencies of occurrence.
- the error model can indicate that a scenario parameter such as a speed associated with an object in a simulation environment is associated with an error percentage.
- a scenario parameter such as a speed associated with an object in a simulation environment is associated with an error percentage.
- the object can be traveling in the simulated scenario at a speed of 10 meters per second and the error percentage can be 20% resulting in a range of speeds between 8 meters per second and 12 meters per second.
- the range of speeds can be associated with a probability distribution indicating that portions of the range have a higher probability of occurring than other portions of the range (e.g., 8 meters per second and 12 meters per second associated with a 15% probability, 9 meters per second and 11 meters per second associated with a 30 percent probability, and 10 meters per second associated with a 10% probability).
- a parameterized scenario can be generated.
- the parameterized scenario can provide a set of variations of a scenario. Therefore, instantiating the autonomous vehicle controller in the parameterized scenario and simulating the parameterized scenario can efficiently cover a wide range of variations of a scenario without requiring a manual enumeration of the variations.
- simulation data can indicate how the autonomous vehicle controller responded to (or will respond to) the parameterized scenario and determine a successful outcome or an unsuccessful outcome based at least in part on the simulation data.
- Aggregating the simulation data related to the parameterized scenario can provide safety metrics associated with the parameterized scenario.
- the simulation data can indicate a success rate and/or a failure rate of the autonomous vehicle controller and the parameterized scenario.
- meeting or exceeding a success rate can indicate a successful validation of the autonomous vehicle controller which can subsequently be downloaded by (or otherwise transferred to) a vehicle for further vehicle control and operation.
- a parameterized scenario can be associated with an outcome.
- the simulation data can indicate that the autonomous vehicle controller responded consistently or inconsistently with the outcome.
- the parameterized scenario can represent a simulation environment including a vehicle controlled by an autonomous vehicle controller traveling at a speed and performing a stop action before an object that is in front of the vehicle.
- the speed can be associated with a scenario parameter indicating a range of speeds of the vehicle.
- the parameterized scenario can be simulated based at least in part on the range of speeds and can generate simulation data indicating a distance between the vehicle and the object when the vehicle completes the stop action.
- the parameterized scenario can be associated with an outcome indicating that the distance between the vehicle and the object meets or exceeds a distance threshold.
- a success rate can indicate the number of times that the distance between the vehicle and the object when the vehicle completes the stop action meets or exceeds the distance threshold as compared to the total number of times that the vehicle completed the stop action.
- generating a set of unique simulated environments as many as are needed for training, testing, and/or validating systems (e.g., one or more components of an AI stack) onboard an autonomous vehicle (e.g., prior to such autonomous vehicle(s) being deployed in corresponding new real environments) — can take an inordinate amount of time, thereby limiting the ability to train, test, and/or validate such systems (e.g., one or more components of an AI stack) onboard an autonomous vehicle prior to entering into real scenarios and/or environments.
- Techniques described herein are unconventional in that they leverage sensor data collected from real environments and supplement that data with additional data to generate a substantially accurate simulated environment (e.g., relative to the corresponding real environment) more efficiently than what is available with conventional techniques.
- techniques described herein such as varying aspects of the scenario — enable the generation of many, scalable simulated scenarios in less time and with fewer computational resources than what is available with conventional techniques.
- simulated environments resulting from generation techniques described herein can be used for testing, training, and validating systems onboard an autonomous vehicle to ensure such systems can operate autonomous vehicles safely when deployed in real environments. That is, simulated environments resulting from generation techniques described herein can be used for testing, training, and validating a planner system and/or a prediction system of an autonomous vehicle controller, which can be used by an autonomous vehicle to navigate the autonomous vehicle along a trajectory in a real environment.
- a planner system and/or a prediction system of an autonomous vehicle controller which can be used by an autonomous vehicle to navigate the autonomous vehicle along a trajectory in a real environment.
- FIG. 1 illustrates an example 100 of generating vehicle performance data associated with a vehicle controller based on a parameterized scenario.
- input data 102 can be used.
- the input data 102 can include vehicle data 104 and/or additional situational data 106.
- the vehicle data 104 can include log data captured by a vehicle traveling through an environment. As discussed above, the log data can be used to identify scenarios for simulating an autonomous controller.
- the vehicle can be an autonomous vehicle configured to operate according to a Level 5 classification issued by the U. S. National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions for the entire trip, with the driver (or occupant) not being expected to control the vehicle at any time.
- the vehicle can be configured to control all functions from start to stop, including all parking functions, it can be unoccupied.
- the systems and methods described herein can be incorporated into any ground-home, airborne, or waterborne vehicle, including those ranging from vehicles that need to be manually controlled by a driver at all times, to those that are partially or fully autonomously controlled.
- the vehicle can include a computing device that includes a perception engine and/or a planner and perform operations such as detecting, identifying, segmenting, classifying, and/or tracking objects from sensor data collected from the environment.
- a computing device that includes a perception engine and/or a planner and perform operations such as detecting, identifying, segmenting, classifying, and/or tracking objects from sensor data collected from the environment.
- objects such as pedestrians, bicycles/bicyclists, motorcycles/motorcyclists, buses, streetcars, trucks, animals, and/or the like can be present in the environment.
- the sensors can capture sensor data associated with the environment.
- some of the sensor data can be associated with objects (e.g., vehicles, cyclists, and/or pedestrians).
- the sensor data can be associated with other objects including, but not limited to, buildings, road surfaces, signage, barriers, etc. Therefore, in some instances, the sensor data can be associated with dynamic objects and/or static objects.
- the dynamic objects can be, as described above, objects that are associated with a movement (e.g., vehicles, motorcycles, cyclists, pedestrians, animals, etc.) or capable of a movement (e.g., parked vehicles, standing pedestrians, etc.) within the environment.
- the static objects can be, as described above, objects that are associated with the environment such as, for example, buildings/structures, road surfaces, road markers, signage, barriers, trees, sidewalks, etc.
- the vehicle computing device can determine information about objects in the environment, such as bounding boxes, classifications, segmentation information, and the like.
- the vehicle computing device can use the sensor data to generate a trajectory for the vehicle.
- the vehicle computing device can also determine pose data associated with a position of the vehicle.
- the vehicle computing device can use the sensor data to determine position data, coordinate data, and/or orientation data of the vehicle in the environment.
- the pose data can include x-y-z coordinates and/or can include pitch, roll, and yaw data associated with the vehicle.
- the vehicle computing device can generate vehicle data 104 which can include the data discussed above.
- vehicle data 104 can include the sensor data, perception data, planning data, vehicle status data, velocity data, intent data, and/or other data generated by the vehicle computing device.
- the sensor data can include data captured by sensors such as time-of-flight sensors, location sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), lidar sensors, radar sensors, sonar sensors, infrared sensors, cameras (e.g., RGB, IR, intensity, depth, etc.), microphone sensors, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), ultrasonic transducers, wheel encoders, etc.
- sensors such as time-of-flight sensors, location sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), lidar sensors, radar sensors, sonar sensors, infrared sensors, cameras (e.g., RGB, IR, intensity, depth, etc
- the sensor data can be data captured by such sensors such as time-of-flight data, location data, lidar data, radar data, sonar data, image data, audio data, etc.
- log data may further include intermediate output by any one or more systems or subsystems of the vehicle including, but not limited to, messages indicating object detections, object tracks, predictions of future object locations, pluralities of trajectories generated in response to such detections, control signals passed to one or more systems or subsystems used to effectuate commands, and the like.
- the vehicle data 104 can include time data that is associated with the other data generated by the vehicle computing device.
- input data 102 can be used to generate a scenario.
- the input data 102 can include vehicle data 104 and/or additional situational data 106.
- the additional situational data 106 can include data such as an incident report from a third-party source.
- a third-party source can include a law enforcement agency, a department of motor vehicle, and/or a safety administration that can publish and/or store reports of activities and/or incidents.
- a report can include a type of activity (e.g., a traffic hazard such as debris on a roadway, local flooding, etc.), a location, and/or a description of the activity.
- the report can describe that a driver, while operating a vehicle, struck a fallen tree branch in a roadway while traveling at a speed of 15 meters per second.
- the report can be used to generate a similar scenario that can be used in simulation.
- the additional situational data 106 can include captured sensor data (e.g., image data).
- captured sensor data e.g., image data
- a driver of a vehicle can use a camera to capture image data while the driver operates the vehicle.
- the image data can capture activity such as an incident.
- a driver can use a dashboard camera (e.g., a camera mounted on an interior dashboard of a vehicle) to capture image data while the driver operates the vehicle. As the driver operates the vehicle, an animal can run across the roadway and the driver can immediately brake to slow the vehicle.
- the dashboard camera can capture image data of the animal running across the roadway and the vehicle slowing down.
- the image data can be used to generate a scenario of an animal running across a roadway.
- a probability that is associated with a scenario can be determined to identify scenarios for simulation based on the probability meeting or exceeding a probability threshold.
- a probability threshold of 0.001% can be used where a likelihood of encountering a scenario is less than 0.001%, then scenarios with a higher probability can be prioritized for simulation and determining safety metrics associated with the scenarios with the higher probability.
- the input data 102 can be used by a scenario editor component 108 to generate an initial scenario(s) 110.
- the input data 102 can be input into the scenario editor component 108 which can generate a synthetic environment that represents at least a portion of the input data 102 in the synthetic environment.
- Examples of generating scenarios such as initial scenario(s) 110 and data generated by a vehicle that can be included in the vehicle data 104 can be found, for example, in U.S. Patent Application No. 16/392,094 titled “Scenario Editor and Simulator” and filed April 23, 2019 which is incorporated by reference in its entirety.
- the scenario editor component 108 can be configured to scan the input data 102 to identify one or more scenarios represented in the input data 102.
- the scenario editor component 108 can determine that a portion of the input data 102 represents a pedestrian crossing a street without a right-of-way (e.g., without a crosswalk, at an intersection without a walk indication, and the like).
- the scenario editor component 108 can identify this as a scenario (e.g., a jaywalking parameter) and label (and/or categorize) the scenario as, for example, a jaywalking scenario.
- the scenario editor component 108 can use rules that define actions to generate the initial scenario(s) 110.
- a rule can define that a pedestrian crossing a road in a region that is not associated with a crosswalk is a jaywalker.
- the scenario editor component 108 can receive label data from a user of the scenario editor component 108 to associate portions of the input data 102 with labels to generate the initial scenario(s) 110.
- the scenario editor component 108 can scan other portions of the input data 102 and identify similar scenarios and label the similar scenarios with the same jaywalking label. In some instances, the scenario editor component 108 can identify scenarios that do not correspond to (or is excluded from) an existing label and generate a new label for these scenarios. In some instances, the scenario editor component 108 can generate a library of scenarios and store the library of scenarios in a database within the scenario editor component 108.
- the library of scenarios can include crosswalk scenarios, merging scenarios, lane change scenarios, and the like.
- such initial scenarios 110 may be manually specified.
- one or more users may designate certain scenarios to be tested to ensure that the vehicle is capable of safely operating when performing such scenarios, despite having never (or rarely) previously encountered the scenario.
- the parameter component 112 can determine scenario parameter(s) associated with the initial scenario(s) 110 identified by the scenario editor component 108.
- the parameter component 112 can analyze the jaywalking scenario and determine scenario parameters associated with the jaywalking scenario that include a position of the pedestrian, a pose of the pedestrian, a size of the pedestrian, a velocity of the pedestrian, a track of the pedestrian, a distance between a vehicle and the pedestrian, a velocity of the vehicle, a width of a road, and the like.
- the parameter component 112 can determine a range or a set of values associated with the scenario parameter. For example, the parameter component 112 can determine a classification associated with an object (e.g., the pedestrian) represented in the initial scenario(s) 110 and determine other objects of the same classification in the input data 102. Then the parameter component 112 can determine a range of values associated with the scenario parameter as represented by the initial scenario(s) 110.
- the scenario parameter can indicate that the pedestrian can have a velocity with a range of 0.5- 1.5 meters per second.
- the parameter component 112 can determine a probability associated with the scenario parameter.
- the parameter component 112 can associate a probability distribution such as a Gaussian distribution (also referred to as a normal distribution) with the scenario parameter.
- the parameter component 112 can determine the probability associated with the scenario parameter based on the input data 102.
- the parameter component 112 can determine a classification associated with the object represented in the input data 102 and determine other objects of the same classification in the input data 102 and/or other log data. Then the parameter component 112 can determine a probability distribution of the scenario parameter associated with the object as represented by the input data 102 and/or the other log data.
- the parameter component 112 can determine that 30% of pedestrians walk at a velocity below 0.3 meters per second, 30% of pedestrians walk at a velocity above 1.2 meters per second, and that 40% of pedestrians walk at a velocity between 0.3 and 1.2 meters per second.
- the parameter component 112 can use the distribution as a probability that the pedestrian of the j ay walking scenario will walk at a particular velocity.
- the parameter component 112 can determine that a jaywalking scenario probability of 1% which can indicate that a vehicle traversing an environment will encounter a jaywalker 5% of the time while traversing the environment.
- the scenario probability can be used to include the scenario at a rate that is associated with the scenario probability.
- the parameter component 112 can receive supplemental data 114 that is incorporated into the distribution.
- the parameter component 112 can determine scenario parameters indicating that a pedestrian can have a distance with a range of 30-60 meters to a vehicle while the vehicle travels at a velocity of 15 meters per second or alternatively represented as a time-to-collision of 2-4 seconds.
- the supplemental data 114 e.g., a regulation or guideline
- the parameter component 112 can incorporate the supplemental data 114 and determine a scenario parameter as having a 1.5-4 second time-to-collision (although any time ranges can be specified). In some instances, the parameter component 112 can use the probability distribution discussed above to determine (using interpolation and/or extrapolation techniques) a probability associated with the supplemental data 114.
- the error model component 116 can determine an error model that can indicate an error associated with a scenario parameter.
- a perception error model can produce a perception error associated with a perception parameter of a simulated object
- a prediction error model can produce a prediction error associated with a prediction parameter of a simulated object
- a sensor can produce erroneous sensor data and/or a computing device of the vehicle can incorrectly process the sensor data which can result in perception errors. Testing and simulating perception errors can help indicate an operational margin of the vehicle as it relates to potential perception errors.
- a scenario parameter such as a perception parameter, can indicate a size of an object or a range of positions of the object in an environment.
- the error model component 116 can, using a perception error model, indicate a potential error associated with the size of the object which can result in perception data of the vehicle that indicates that the object is larger or smaller than the actual object in the environment.
- the error model component 116 can determine, for example, the perception error model by comparing the input data 102 to ground truth data.
- the ground truth data can be manually labeled and/or determined from other, validated, machine-learned components.
- the input data 102 can include the sensor data and/or the perception data generated by the vehicle(s).
- the error model component 116 can compare the input data 102 with the ground truth data which can indicate the actual parameters of an object in the environment.
- the error model component 116 can determine a perception error.
- the input data 102 can indicate that a pedestrian is 1.8 meters tall while the ground truth data indicates that the pedestrian is 1.75 meters tall, and, accordingly, the perception error model can indicate a perception error of approximately 3% (e.g., [(1.8-1 75)/l .8] * 100).
- the error model component 116 can determine a classification associated with the object represented in the input data 102 and determine other objects of the same classification in the input data 102. Then the error model component 116 can determine a probability distribution (also referred to as an error distribution) associated with a range of errors of the object and associate the probability distribution with the object within the initial scenario(s) 110. By way of example and without limitation, the error model component 116 can determine that objects with a pedestrian classification have a perception error of 4%-6% and that objects with a vehicle classification have a perception error of 3%-5%. In some instances, the error model component 116 can determine a probability distribution indicating that, for example, objects greater than a threshold size are more or less likely to have an error (in classification, for example).
- the parameter component 112 can use region data 118 to determine a set of regions of an environment that are compatible with the scenario and the scenario parameter.
- the scenario can indicate that a vehicle has a velocity of 15 meters per second, that a pedestrian has a velocity of 1.5 meters per second, and that a distance between the vehicle and the pedestrian is 30 meters.
- the parameter component 112 can determine regions based on the region data 118 of an environment that conform with the scenario.
- the parameter component 112 could exclude school zones because a velocity of the vehicle in the scenario would likely exceed a speed limit associated with a school zone and therefore the scenario would not be valid in a school zone.
- a vehicle velocity e.g., 15 m/s
- such a region may be considered within the set of regions.
- the parameter component 112 can store the region data 118 that includes segments of drivable regions of an environment.
- Techniques for identifying segments and similar segments of drivable surfaces and segment classifications and/or stereotypes can be found, for example, in U.S. Patent Application No. 16/370,696 titled “Extension of Autonomous Driving Functionality to New Regions” and filed March 19, 2019, and in U.S. Patent Application No. 16/376,842 titled “Simulating Autonomous Driving Using Map Data and Driving Data” and filed April 5, 2019, which are incorporated by reference herein in their entirety.
- region data 118 of an environment can be parsed into segments and similar segments can be identified.
- a segment can include a junction segment, e.g., an intersection, a merge, or the like, or a connecting road segment, e.g., a length and/or a width of a road between junctions.
- a junction segment e.g., an intersection, a merge, or the like
- a connecting road segment e.g., a length and/or a width of a road between junctions.
- all two-lane road segments having a speed limit within a 10- mph range can be associated with the same stereotype.
- data can be associated with each of the individual segments.
- a junction segment can include a junction type, e.g., a merge, a “T,” a round-about, or the like; a number of roads meeting at the junction; a relative position of those roads, e.g., an angle between the roads meeting at the junction; information about traffic control signals at the junction; and/or other features.
- Data associated with a connecting road segment can include a number of lanes, a width of those lanes, a direction of travel in each of the lanes, an identification of parking lanes, a speed limit on the road segment, and/or other features.
- segments of the drivable surface can be grouped according to a segment classification or segment stereotype.
- some or all junction segments that conform to some range of metrics or attributes can be grouped together (e.g., using k-means, evaluating a weighted distance (e.g., Euclidian) between segment parameters, or otherwise clustering such segments based on segment parameters).
- Scenarios that include the same or similar stereotypes can be used to verify functionality of the autonomous vehicle controller.
- an autonomous vehicle can be expected to perform (and/or may have demonstrably performed) the same in the same or similar stereotypes.
- the use of stereotypes can reduce a number of comparisons to be made. For example, by identifying similar regions, a reduced number of simulation scenarios that can provide useful information.
- the techniques described herein can reduce computational complexity, memory requirements, and processing time by optimizing over specific scenarios which provide useful information for validation and testing.
- a parameterized scenario component 120 can use the data determined by the parameter component 112 (e.g., initial scenario(s) 110, scenario parameter(s), the set of regions, and/or the error model data) to generate a parameterized scenario 122.
- the initial scenario(s) 110 can indicate scenarios such as a lane change scenario, a right turn scenario, a left turn scenario, an emergency stop scenario, etc.).
- the scenario parameter(s) can indicate a speed associated with a vehicle controlled by the autonomous vehicle controller, a pose of the vehicle, a distance between the vehicle and an object, and the like.
- the scenario parameter(s) can indicate objects, positions associated with the objects, velocities associated with the objects, and the like.
- the error model (e.g., the perception error model, the prediction error model, etc.) can indicate an error associated with the scenario parameter(s) and provide a range of values and/or probabilities associated with the scenario parameter(s).
- a scenario parameter such as a speed of the vehicle can be associated with a range of speeds such as 8-12 meters per second).
- the range of speeds can be associated with a probability distribution that indicates a probability of a speed within the range of speeds of occurring.
- the set of regions can indicate portions of an environment that can be used to place the objects in a simulated environment.
- the initial scenario(s) 110 can indicate a scenario that includes a two-way, multi -lane driving surface associated with a speed limit of 35 miles per hour.
- the set of regions can exclude regions that would not include a two-way, multi-lane driving surface associated with a speed limit of 35 miles per hour such as, for example, parking lots.
- a parameterized scenario 122 can be used to cover the variations provided by the scenario parameter(s), the error model, the region data 118, etc.
- the scenario parameter(s) can include a vehicle traversing an environment with a velocity of 10, 11, or 12 meters per second (or any velocity) while approaching a junction.
- the set of regions can include an uncontrolled junction, a junction with a four-way stop, and a junction with a traffic light.
- the simulation component 124 can use the perception error model to introduce a perception error associated with the perception data determined by the vehicle.
- the parameterized scenario 122 can include more or fewer permutations as well as different types of scenario parameters, regions, and/or perception errors.
- the simulation component 124 can execute the parameterized scenario 122 as a set of simulation instructions and generate simulation data 126.
- the simulation component 124 can instantiate a vehicle controller in the simulated scenario.
- the simulation component 124 can execute multiple simulated scenarios simultaneously and/or in parallel.
- the simulation component 124 can determine an outcome for the parameterized scenario 122.
- the simulation component 124 can execute a variation of the parameterized scenario 122 for use in a simulation for testing and validation.
- the simulation component 124 can generate the simulation data 126 indicating how the autonomous vehicle controller performed (e.g., responded) and can compare the simulation data 126 to a predetermined outcome and/or determine if any predetermined rules/assertions were broken/triggered.
- the variation for simulation can be selected based on generalized spacing of a scenario parameter.
- a scenario parameter can be associated with a speed of a vehicle.
- the scenario parameter can be associated with a range of values of the vehicle.
- Variations for simulation can be selected based on generalized spacing to increase a coverage of the range of values (e.g., selecting the 25 th percentile, 50 th percentile, 75% percentile, etc.).
- variations can be randomly selected and/or variations can be randomly selected within standard deviations of the range of values.
- the predetermined rules/assertions can be based on the parameterized scenario 122 (e.g., traffic rules regarding crosswalks can be enabled based on a crosswalk scenario or traffic rules regarding crossing a lane marker can be disabled for a stalled vehicle scenario).
- the simulation component 124 can enable and disable rules/assertions dynamically as the simulation progresses. For example, as a simulated object approaches a school zone, rules/assertions related to school zones can be enabled and disabled as the simulated object departs from the school zone.
- the rules/assertions can include comfort metrics that relate to, for example, how quickly an object can accelerate given the simulated scenario.
- the rules may include, for example, following rules of the road, leaving a safety buffer between objects, etc.
- the simulation component 124 can determine that the autonomous vehicle controller succeeded. Based at least in part on determining that the autonomous vehicle controller performance was inconsistent with the predetermined outcome (that is, the autonomous vehicle controller did something that it wasn't supposed to do) and/or determining that a rule was broken or than an assertion was triggered, the simulation component 124 can determine that the autonomous vehicle controller failed.
- simulation data 126 can indicate how the autonomous vehicle controller responds to each variation of the parameterized scenario 122, as described above and determine a successful outcome or an unsuccessful outcome based at least in part on the simulation data 126.
- An analysis component 128 can be configured to determine degrees of a success or a failure.
- a rule can indicate that a vehicle controlled by an autonomous vehicle controller must stop within a threshold distance of an object.
- the simulation data 126 can indicate that in a first variation of the parameterized scenario 122, the simulated vehicle stopped in excess of 5 meters from the threshold distance.
- the simulation data 126 can indicate that the simulated vehicle stopped in excess of 10 meters from the threshold distance.
- the analysis component 128 can indicate that the simulated vehicle performed more successfully in the second variation compared to the simulated vehicle in the first variation.
- the analysis component 128 can determine an ordered list (e.g., ordered according to a relative success scale) that includes simulated vehicles and the associated variations of the parameterized scenario 122. Such variations may also be used to determine limitations of the various components of the system being simulated.
- an ordered list e.g., ordered according to a relative success scale
- the analysis component 128 can, based on the simulation data 126, determine additional variations of the parameterized scenario 122.
- the simulation data 126 output by the simulation component 124 can indicate variations of the parameterized scenario 122 associated with a success or a failure (which may be represented as a continuous likelihood).
- the analysis component 128 can determine additional variations based on the variations associated with a failure.
- a variation of the parameterized scenario 122 associated with a failure can represent a vehicle traveling on a driving surface at a speed of 15 meters per second and an animal crossing the driving surface at a distance of 20 meters in front of the vehicle.
- the analysis component 128 can determine additional variations of the scenario to determine additional simulation data 126 for analysis.
- the analysis component 128 can determine additional variations that include the vehicle traveling at 10 meters per second, 12.5 meters per second, 17.5 meters per second, 20 meters per second, etc. Additionally, the analysis component 128 can determine additional variations that include the animal crossing the driving surface at a distance of 15 meters, 17.5 meters, 22.5 meters, and 25 meters, etc.
- the additional variations can be input into the simulation component 124 to generate additional simulation data. Such additional variations may be determined based on, for example, perturbations to the scenario parameters for the scenario being run in simulation.
- the analysis component 128 can determine additional variations of the scenario by disabling a scenario parameter.
- a parameterized scenario can include a first scenario parameter that is associated with a speed of an object and a second scenario parameter that is associated with a position of the object.
- the parameterized scenario 122 can include a first range of values associated with the speed and a second range of values associated with the position.
- the simulation data 126 can indicate that some variations of the parameterized scenario 122 resulted in a successful outcome and some variations resulted in a failed outcome.
- the analysis component 128 can determine to disable the first scenario parameter (e.g., set a fixed value associated with the first scenario parameter) and vary the parameterized scenario 122 based on the second scenario parameter.
- the analysis component 128 can determine whether a scenario parameter and/or a value of a scenario parameter is associated with the successful outcomes or the failed outcomes.
- Such parameters may be disabled based on likelihoods of failure, randomly, or otherwise.
- simulation data 126 as a result of disabling all of the scenario parameters can indicate a problem with a planning component of an autonomous vehicle.
- the analysis component 128 can be used to perform sensitivity analysis. For example, the analysis component 128 can disable a scenario parameter and, based on the simulation data 126 generated by the simulation component 124, determine how disabling the scenario parameter affects the simulation data 126 (e.g., increase a success rate, decrease a success rate, have minimal impact on the success rate). In some instances, the analysis component 128 can disable scenario parameters individually to determine how disabling each scenario parameter affects the simulation data 126. The analysis component 128 can collect statistical data that indicates how individual scenario parameters affect the simulation data 126 over the course of many simulations.
- the analysis component 128 can be configured to disable a set of scenario parameters (e.g., disabling a night time environmental parameter and disabling a wet conditions environmental parameter).
- the analysis component can collect statistical data that indicates how sets of scenario parameters affect the simulation data 126. The statistical data can be used to determine scenario parameters that increase or decrease a likelihood of resulting is a successful simulation and can be used to identify subsystems of an autonomous vehicle that are associated with a scenario parameter as increasing or decreasing a success rate of the simulation data 126.
- the analysis component 128 can adjust a degree by which a scenario parameter is adjusted.
- a scenario parameter can indicate wet environment conditions (e.g., rainy conditions).
- the scenario parameter can be adjusted on a range (e.g., a quarter inch of rainfall, an inch of rainfall, etc.) ⁇
- the analysis component 128 can adjust a magnitude of the scenario parameter and perform the sensitivity analysis based on the magnitude of the scenario parameter to determine a threshold associated with the scenario parameter that can result in a successful or an unsuccessful outcome of the simulation.
- a binary search algorithm, a particle filter algorithm, and/or a Monte Carlo method can be used to determine the threshold, although other suitable algorithms are contemplated.
- the vehicle performance component 130 can, based on the simulation data 126 (and/or the additional simulation data based on the additional variations from the analysis component 128) and the types of failures, determine the vehicle performance data 132.
- the vehicle performance data 132 can indicate how a vehicle performs in an environment.
- the vehicle performance data 132 can indicate that a vehicle traveling at a speed of 15 meters per second has a stopping distance of 15 meters.
- the vehicle performance data can indicate safety metrics.
- the vehicle performance data 132 can indicate an event (e.g., a failure) and a cause of the event.
- such indication may be binary (failure or not), coarse (levels of failure, e.g., “critical”, “non-critical”, and “pass), or continuous (e.g., representing a probability of failure), though any other indication is contemplated.
- the data 134(1) can indicate a safety rating and similarly for data 134(2)- 134(4).
- the cause type 1 and the cause type 2 can indicate a fault such as a fault of the vehicle or a fault of an object (e.g., a bicyclist).
- the vehicle performance data 132 can then indicate a safety metric associated with the parameterized scenario 122.
- the vehicle performance component 130 can use a target metric and compare the vehicle performance data 132 with the target metric to determine if the safety metric meets or exceeds the target metric.
- the target metric can be based on standards and/or regulations associated with autonomous vehicles.
- the vehicle performance data 132 can be input into a filter component 136 to determine filtered data 138 based on the vehicle performance data 132.
- the filter component 136 can be used to determine filtered data 138 that identifies regions that do not meet a coverage threshold.
- the initial scenario(s) 110 can indicate a two-way, multi-lane driving surface associated with a speed limit of 35 miles per hour and the region data 118.
- the parameter component 112 can identify five regions of an environment that can be used to simulate the initial scenario(s) 110.
- the vehicle performance data 132 can indicate that the simulation data 126 is associated with three of the five regions.
- the simulation component 124 can simulate the scenarios and generate simulation data 126 based on executing the scenario in three of the five regions identified by the parameter component 112.
- the filter component 136 can, based on one or more filters, determine filtered data 138 that indicates that the remaining 2 of the 5 regions do not meet a coverage threshold (e.g., a minimum number of simulations associated with a region).
- the filter component 136 can determine, based on the vehicle performance data 132, filtered data 138 that indicates occurrences of an event.
- the simulation data 126 can include occurrences of events such as an emergency stop, a flat tire, an animal crossing a driving surface, and the like.
- the filter component 136 can, based on one or more filters, determine filtered data 138 that indicates occurrences of the emergency stop.
- the filtered data 138 can include portions of the simulation data 126 associated with the occurrences of the emergency stop such as, for example, a stopping distance associated with the emergency stop.
- FIG. 2 illustrates an example 200 of vehicle(s) 202, which can be similar to the vehicle generating vehicle data 104 described with reference to FIG. 1, generating the vehicle data 104 and transmitting the vehicle data 104 to the computing device(s) 204.
- the scenario editor component 108 can be configured to scan the input data 102 (e.g., the vehicle data 104 and/or the additional situational data 106) and identify one or more scenarios represented in the input data 102. As a non-limiting example, such scenarios may be determined based on, for example, clustering (e.g., using k-means or the like) parameterizations of the log data.
- the scenario editor component 108 can use scenario definition data 206 to identify the one or more scenarios represented in the input data 102.
- the scenario definition data 206 can identify features associated with a type of scenario.
- the scenario definition data 206 can identify a jaywalking scenario that includes features such as a pedestrian crossing a portion of a driving surface that is not associated with a crosswalk.
- the scenario editor component 108 can scan the input data 102 to identify portions of the input data that include the features of a pedestrian crossing a portion of a driving surface that is not associated with a crosswalk to determine jaywalking scenarios.
- such scenarios may further be manually entered and/or derived from third-party data (e.g., police reports, video clips generally available, and the like).
- the parameter component 112 can determine scenario parameter(s) that can indicate a value or a range of values associated with a parameter of an object in the scenario.
- the parameter component 112 can generate scenario data 208 as depicted in FIG. 2.
- the scenario data 208 can indicate a base scenario that includes a vehicle 210 traversing along a driving surface and an object 212 (which can be a different vehicle) traversing near the vehicle 210 in a same direction as the vehicle 210.
- the vehicle 210 and the object 212 can be approaching a junction with a crosswalk.
- the parameter component 112 can determine a scenario parameter that indicates a distance between the vehicle 210 and the object 212 as having a range of distances.
- the scenario Si can represent a first scenario of a set of scenarios having a first distance between the vehicle 210 and the object 212
- the scenario S2 can represent a second scenario of the set of scenarios having a second distance between the vehicle 210 and the object 212
- the scenario S N can represent an Nth scenario of the set of scenarios having an Nth distance between the vehicle 210 and the object 212.
- additional types of parameters also referred to as attributes
- scenario parameters can include, but are not limited to, a velocity of the object 212, an acceleration of the object 212, an x-position of the object 212 (e.g., a global position, local position, and/or a position with respect to any other frame of reference), a y-position of the object 212 (e.g., a size, a pose, a local position, a global position and/or a position with respect to any other frame of reference), a bounding box associated with the object 212 (e.g., extents (length, width, and/or height), yaw, pitch, roll, etc.), lighting states (e.g., brake light(s), blinker light(s), hazard light(s), headlight(s), reverse light(s), etc.), a wheel orientation of the object 212, map elements (e.g., a distance between the object 212 and a stop light, stop sign, speed bump, intersection, yield sign, and the like), a classification of the
- the parameter component 112 can determine a range of values associated with the scenario parameter as represented by the vehicle data 104 and/or the other input data. Therefore, each of the example scenario parameters identified above as well as other scenario parameters can be associated with a set or a range of values that can be used to generate a set of scenarios where the scenarios of the set of scenarios differ by one or more values of the scenario parameters.
- FIG. 3 illustrates an example 300 of generating error model data based at least in part on vehicle data and ground truth data.
- vehicle(s) 202 can generate vehicle data 104 and transmit the vehicle data 104 to an error model component 116.
- the error model component 116 can determine an error model that can indicate an error associated with a scenario parameter.
- the vehicle data 104 can be data associated with a subsystem of the vehicle(s) 202 such as a perception system, a planning system, a tracking system (also referred to as a tracker system), a prediction system, and the like.
- the vehicle data 104 can be associated with a perception system and the vehicle data 104 can include a bounding box associated with an object detected by the vehicle(s) 202 in an environment.
- the error model component 116 can receive ground truth data 302 which can be manually labeled and/or determined from other, validated, machine-learned components.
- the ground truth data 302 can include a validated bounding box that is associated with the object in the environment.
- the error model component 116 can determine an error associated with the subsystem of the vehicle(s) 202.
- the vehicle data 104 can include one or more characteristics (also referred to as parameters) associated with a detected entity and/or the environment in which the entity is positioned.
- characteristics associated with an entity can include, but are not limited to, an x-position (global position), a y-position (global position), a z-position (global position), an orientation, an entity type (e.g., a classification), a velocity of the entity, an extent of the entity (size), etc.
- Characteristics associated with the environment can include, but are not limited to, a presence of another entity in the environment, a state of another entity in the environment, a time of day, a day of a week, a season, a weather condition, an indication of darkness/light, etc. Therefore, the error can be associated with the other characteristics (e.g., environmental parameters).
- the error model component 116 can process a plurality of vehicle data 104 and a plurality of ground truth data 302 to determine the error model data 304.
- the error model data 304 can include the error calculated by the error model component 116 which can be represented as error 306(l)-(3). Additionally, the error model component 116 can determine a probability associated with the error 306(l)-(3) represented as probability 308(l)-(3) which can be associated with an environmental parameter to present error models 310(1)— (3).
- the vehicle data 104 can include a bounding box associated with an object at a distance of 50 meters from the vehicle(s) 202 in an environment that includes rainfall.
- the ground truth data 302 can provide the validated bounding box associated with the object.
- the error model component 116 can determine error model data 304 that determines that the error associated with the perception system of the vehicle(s) 202. The distance of 50 meters and the rainfall can be used as environmental parameters to determine which of error model of error models 310(1)— (3) to use. Once the error model is identified, the error model 310(1)— (3) can provide an error 306(l)-(3) based on the probability 308(l)-(3) where errors 306(l)-(3) associated with higher probabilities 308(l)-(3) are more likely to be selected than errors 306(l)-(3) associated with lower probabilities 308(l)-(3).
- FIG. 4 illustrates an example 400 of using error model data to perturb a simulation by providing at least one of an error or an uncertainty associated with a simulated environment.
- the error model data can include an error model that associates an error 306 with a probability 308 and the error model can be associated with environmental parameters.
- a simulation component 124 can use the error model data 304 to inject an error which can result in a perturbed parameterized scenario to perturb the simulation. Based on the injected error, simulation data can indicate how an autonomous controller responds to the injected error.
- the simulation component 124 can perturb the simulation by continuously injecting the error into the simulation.
- example 402 depicts a bounding box 404 associated with an object at a time to.
- the bounding box 404 can represent a detection of the object by a vehicle that includes an error such as an error in a size of the bounding box and/or a position of the bounding box.
- the simulation component 124 can use the bounding box 406 that represents the object and that includes a different error.
- the simulation component 124 can use a different error 306 based on the probability 308 associated with the error 306.
- the simulation component 124 can use the bounding box 408 that represents the object and that includes a different error.
- the simulation component 124 can perturb the simulation by injecting an uncertainty associated with a bounding box that represents an object in the environment.
- example 410 depicts a bounding box 412 associated with an object at a time to.
- the bounding box can include an uncertainty of 5% which can indicate an uncertainty of the object’s size and/or position by an amount of 5%.
- the uncertainly can persist with the object through time ti and t2 rather than injecting a different error at different simulation times as depicted in example 402.
- FIG. 5 illustrates an example 500 of vehicle(s) 202 generating vehicle data 104 and transmitting the vehicle data 104 to the computing device(s) 204.
- the error model component 116 can determine a perception error model that can indicate an error associated with a scenario parameter.
- the vehicle data 104 can include sensor data generated by a sensor of the vehicle(s) 202 and/or perception data generated by a perception system of the vehicle(s) 202.
- the perception error model can be determined by comparing the vehicle data 104 against ground truth data 302.
- the ground truth data 302 can be manually labeled and can be associated with the environment and can represent a known result.
- a deviation from the ground truth data 302 in the vehicle data 104 can be identified as an error in a sensor system and/or the perception system of the vehicle(s) 202.
- a perception system can identify an object as a bicyclist where the ground truth data 302 indicates that the object is a pedestrian.
- a sensor system can generate sensor data that represents an object as having a width of 2 meters where the ground truth data 302 indicates that the object has a width of 1.75 meters.
- the error model component 116 can determine a classification associated with the object represented in the vehicle data 104 and determine other objects of the same classification in the vehicle data 104 and/or other log data. Then the error model component 116 can determine a probability distribution associated with a range of errors of associated with the obj ect. Based on the comparison and the range of errors, the error model component 116 can determine the perception error model data 502.
- an environment 504 can include objects 506(1)— (3) represented as bounding boxes generated by a perception system.
- the perception error model data 502 can indicate scenario parameters as 508(1)— (3) and the error associated with the scenario parameters as 510(1)— (3).
- the error associated with object 508(1) can be visualized in environment 504 as a larger bounding box 512 that indicates an uncertainty regarding the dimensions of object 508(1).
- FIG. 6 illustrates an example 600 of the computing device(s) 204 generating simulation data 126 and determining vehicle performance data 132.
- the parameterized scenario component 120 can, based on the scenario parameter(s), the set of regions, and the perception error model, determine a parameterized scenario.
- the scenario parameter(s) can indicate the objects, positions associated with the objects, velocities associated with the objects, and the like in the parameterized scenario.
- the scenario parameter(s) can indicate ranges that indicate a range of values and/or probabilities associated with the scenario parameter(s).
- the set of regions can indicate portions of an environment that can be used to place the objects in a simulated environment.
- the perception error model can indicate an error associated with the scenario parameter(s).
- the parameterized scenario can be used by the simulation component 124 to simulate the variations of the parameterized scenario.
- the simulation component 124 can execute a variation of the parameterized scenario for use in a simulation for testing and validation.
- the simulation component 124 can generate the simulation data 126 indicating how the autonomous vehicle controller performed (e.g., responded) and can compare the simulation data 126 to a predetermined outcome and/or determine if any predetermined rules/assertions were broken/triggered.
- the simulation data 126 can indicate a number of simulations (e.g., simulation 1, simulation 2, etc.) and the result of the simulations (e.g., result 1, result 2).
- the result can indicate a pass or a fail based on rules/assertions that were broken/triggered.
- the simulation data 126 can indicate a probability of encountering the scenario.
- the simulation component 124 can simulate a scenario that includes a jay walking pedestrian.
- the input data can indicate that a vehicle encounters a jay walking pedestrian at a rate of 1 minute per 1 hour of driving. This can be used to determine a probability of encountering a particular simulation that is associated with a variation of the parameterized scenario.
- the simulation component 124 can identify variations of the parameterized scenario that have a low probability and execute simulations that correspond to those variations. This can allow testing and verification of the autonomous vehicle controller in more unique circumstances.
- the simulation component 124 can identify variations of the parameterized scenario for additional simulation based on the result.
- the result of a simulation can be a failure where a scenario parameter was associated with a vehicle velocity of 15 meters per second.
- the simulation component 124 can identify velocities that are near the 15 meters per second velocity to determine a threshold at which the simulation will pass which can further assist the development of safer vehicle controllers.
- the vehicle performance component 130 can generate the vehicle performance data 132.
- the data 134(1) can indicate a safety rating and similarly for data 134(2)- 134(4).
- the event type can indicate a that a cost has met or exceed a cost threshold although other event types are contemplated.
- a cost can include but is not limited to a reference cost, an obstacle cost, a lateral cost, a longitudinal cost, and/or the like.
- a reference cost can comprise a cost associated with a difference between a point (also referred to as a reference point) on the reference trajectory and a corresponding point (also referred to as a point or a target point) on the target traj ectory , whereby the difference represents one or more difference in a yaw, lateral offset, velocity, acceleration, curvature, curvature rate, and/or the like.
- decreasing a weight associated with a reference cost can reduce a penalty associated with the target trajectory being located a distance away from the reference trajectory, which can provide smoother transitions leading towards safer and/or more comfortable vehicle operations.
- an obstacle cost can comprise a cost associated with a distance between a point on the reference trajectory or the target trajectory and a point associated with an obstacle in the environment.
- the point associated with the obstacle can correspond to a point on a boundary of a drivable area or can correspond to a point associated with the obstacle in the environment.
- an obstacle in the environment can include, but is not limited to a static object (e.g., building, curb, sidewalk, lane marking, sign post, traffic light, tree, etc.) or a dynamic object (e.g., a vehicle, bicyclist, pedestrian, animal, etc.).
- a dynamic object can also be referred to as an agent.
- a static object or a dynamic object can be referred to generally as an object or an obstacle.
- a lateral cost can refer to a cost associated with steering inputs to the vehicle, such as maximum steering inputs relative to a velocity of the vehicle.
- a longitudinal cost can refer to a cost associated with a velocity and/or acceleration of the vehicle (e.g., maximum braking and/or acceleration). Such costs can be used to ensure that the vehicle is operating within feasible limits and/or comfort limits for passengers being ferried.
- the cause type 1 and the cause type 2 can indicate a fault such as a fault of the vehicle or a fault of an object (e.g., a bicyclist).
- the vehicle performance component 130 can use predetermined rules/assertions to determine the fault.
- a rule can indicate that when the vehicle is impacted by an obj ect in the rear of the vehicle, while a fault can be associated with the object.
- additional rules can be used such as indicating that the vehicle must be traversing the environment in a forward direction when impacted by an object in the rear.
- a cause type (e.g., cause type 1 and/or cause type 2) can be associated with a component of the autonomous vehicle controller.
- such causes may comprise a failure of a perception system, a prediction system, a planner system, a network latency, a torque/acceleration fault, and/or any other component or subcomponent of the vehicle.
- an analysis component can determine, based on the simulation data 126, to disable a scenario parameter (e.g., set a fixed value associated with the scenario parameter) and vary other scenario parameters.
- a scenario parameter e.g., set a fixed value associated with the scenario parameter
- the analysis component can determine scenario parameters that are associated with a successful or a failed outcome.
- the vehicle performance data 132 can then indicate a safety metric associated with the parameterized scenario.
- the analysis component can perform sensitivity analysis to determine cause of a failure. For example, the analysis component can disable scenario parameters individually to isolate one or more scenario parameters and determine how disabling the scenario parameters affects a response of an autonomous vehicle, capture statistical data associated with disabling the one or more scenario parameters, and capture the results as successful or failed outcomes.
- the statistical data can that indicate how sets of scenario parameters affect the outcomes and can be used to determine scenario parameters that increase or decrease a likelihood of resulting is a successful simulation and can be used to identify subsystems of an autonomous vehicle that are associated with a scenario parameter as increasing or decreasing a success rate of the simulation data.
- FIG. 7 depicts a block diagram of an example system 700 for implementing the techniques discussed herein.
- the system 700 can include a vehicle(s) 202.
- the vehicle(s) 202 is an autonomous vehicle; however, the vehicle(s) 202 can be any other type of vehicle (e.g., a driver- controlled vehicle that may provide an indication of whether it is safe to perform various maneuvers).
- the vehicle(s) 202 can include a computing device(s) 702, one or more sensor system(s) 704, one or more emitter(s) 706, one or more communication connection(s) 708 (also referred to as communication devices and/or modems), at least one direct connection 710 (e.g., for physically coupling with the vehicle(s) 202 to exchange data and/or to provide power), and one or more drive system(s) 712.
- the one or more sensor system(s) 704 can be configured to capture sensor data associated with an environment.
- the sensor system(s) 704 can include time-of-flight sensors, location sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), lidar sensors, radar sensors, sonar sensors, infrared sensors, cameras (e.g., RGB, IR, intensity, depth, etc.), microphone sensors, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), ultrasonic transducers, wheel encoders, etc.
- the sensor system(s) 704 can include multiple instances of each of these or other types of sensors.
- the time-of-flight sensors can include individual time- of-flight sensors located at the comers, front, back, sides, and/or top of the vehicle(s) 202.
- the camera sensors can include multiple cameras disposed at various locations about the exterior and/or interior of the vehicle(s) 202.
- the sensor system(s) 704 can provide input to the computing device(s) 702.
- the vehicle(s) 202 can also include one or more emitter(s) 706 for emitting light and/or sound.
- the one or more emitter(s) 706 in this example include interior audio and visual emitters to communicate with passengers of the vehicle(s) 202.
- interior emitters can include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and/or force feedback), mechanical actuators (e.g., seatbelt tensioners, seat positioners, headrest positioners, etc.), and the like.
- the one or more emitter(s) 706 in this example also include exterior emitters.
- the exterior emitters in this example include lights to signal a direction of travel or other indicator of vehicle action (e.g., indicator lights, signs, light arrays, etc.), and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) to audibly communicate with pedestrians or other nearby vehicles, one or more of which may comprise acoustic beam steering technology.
- the vehicle(s) 202 can also include one or more communication connection(s) 708 that enable communication between the vehicle(s) 202 and one or more other local or remote computing device(s) (e.g., a remote teleoperations computing device) or remote services.
- the communication connection(s) 708 can facilitate communication with other local computing device(s) on the vehicle(s) 202 and/or the drive system(s) 712. Also, the communication connection(s) 708 can allow the vehicle(s) 202 to communicate with other nearby computing device(s) (e.g., other nearby vehicles, traffic signals, etc.). [0096]
- the communications connection(s) 708 can include physical and/or logical interfaces for connecting the computing device(s) 702 to another computing device or one or more external network(s) 714 (e.g., the Internet).
- the communications connection(s) 708 can enable Wi-Fi-based communication such as via frequencies defined by the IEEE 802.11 standards, short range wireless frequencies such as Bluetooth, cellular communication (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), satellite communication, dedicated short-range communications (DSRC), or any suitable wired or wireless communications protocol that enables the respective computing device to interface with the other computing device(s).
- the communication connection(s) 708 may comprise the one or more modems as described in detail above.
- the vehicle(s) 202 can include one or more drive system(s) 712. In some examples, the vehicle(s) 202 can have a single drive system 712. In at least one example, if the vehicle(s) 202 has multiple drive systems 712, individual drive systems 712 can be positioned on opposite ends of the vehicle(s) 202 (e.g., the front and the rear, etc.). In at least one example, the drive system(s) 712 can include one or more sensor system(s) 704 to detect conditions of the drive system(s) 712 and/or the surroundings of the vehicle(s) 202.
- the sensor system(s) 704 can include one or more wheel encoders (e.g., rotary encoders) to sense rotation of the wheels of the drive systems, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) to measure orientation and acceleration of the drive system, cameras or other image sensors, ultrasonic sensors to acoustically detect objects in the surroundings of the drive system, lidar sensors, radar sensors, etc. Some sensors, such as the wheel encoders can be unique to the drive system(s) 712. In some cases, the sensor system(s) 704 on the drive system(s) 712 can overlap or supplement corresponding systems of the vehicle(s) 202 (e.g., sensor system(s) 704).
- wheel encoders e.g., rotary encoders
- inertial sensors e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.
- ultrasonic sensors to
- the drive system(s) 712 can include many of the vehicle systems, including a high voltage battery, a motor to propel the vehicle, an inverter to convert direct current from the battery into alternating current for use by other vehicle systems, a steering system including a steering motor and steering rack (which can be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and/or pneumatic components, a stability control system for distributing brake forces to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head/tail lights to illuminate an exterior surrounding of the vehicle), and one or more other systems (e.g., cooling system, safety systems, onboard charging system, other electrical components such as a DC/DC converter, a high voltage junction, ahigh voltage cable, charging system, charge port, etc.).
- a high voltage battery including a motor to propel the vehicle
- an inverter to convert direct current from the battery into alternating current for use by other vehicle systems
- a steering system including a steering motor and steering rack (which can
- the drive system(s) 712 can include a drive system controller which can receive and preprocess data from the sensor system(s) 704 and to control operation of the various vehicle systems.
- the drive system controller can include one or more processor(s) and memory communicatively coupled with the one or more processor(s).
- the memory can store one or more modules to perform various functionalities of the drive system(s) 712.
- the drive system(s) 712 also include one or more communication connection(s) that enable communication by the respective drive system with one or more other local or remote computing device(s).
- the computing device(s) 702 can include one or more processor(s) 516 and memory 518 communicatively coupled with the one or more processor(s) 716.
- the memory 718 of the computing device(s) 702 stores a localization component 720, a perception component 722, a prediction component 724, a planning component 726, and one or more system controller(s) 728.
- the localization component 720, the perception component 722, the prediction component 724, the planning component 726, and the one or more system controller(s) 728 can additionally, or alternatively, be accessible to the computing device(s) 702 (e.g., stored in a different component of vehicle(s) 202 and/or be accessible to the vehicle(s) 202 (e.g., stored remotely).
- the localization component 720 can include functionality to receive data from the sensor system(s) 704 to determine a position of the vehicle(s) 202.
- the localization component 720 can include and/or request/receive a three-dimensional map of an environment and can continuously determine a location of the autonomous vehicle within the map.
- the localization component 720 can use SLAM (simultaneous localization and mapping) or CLAMS (calibration, localization and mapping, simultaneously) to receive time-of-flight data, image data, lidar data, radar data, sonar data, IMU data, GPS data, wheel encoder data, or any combination thereof, and the like to accurately determine a location of the autonomous vehicle.
- the localization component 720 can provide data to various components of the vehicle(s) 202 to determine an initial position of an autonomous vehicle for generating a trajectory, as discussed herein.
- the perception component 722 can include functionality to perform object detection, segmentation, and/or classification.
- the perception component 722 can provide processed sensor data that indicates a presence of an entity that is proximate to the vehicle(s) 202 and/or a classification of the entity as an entity type (e.g., car, pedestrian, cyclist, building, tree, road surface, curb, sidewalk, unknown, etc.).
- the perception component 722 can provide processed sensor data that indicates one or more characteristics (also referred to as parameters) associated with a detected entity and/or the environment in which the entity is positioned.
- characteristics associated with an entity can include, but are not limited to, an x-position (global position), a y-position (global position), a z-position (global position), an orientation, an entity type (e.g., a classification), a velocity of the entity, an extent of the entity (size), etc.
- Characteristics associated with the environment can include, but are not limited to, a presence of another entity in the environment, a state of another entity in the environment, a time of day, a day of a week, a season, a weather condition, a geographic position, an indication of darkness/light, etc.
- the perception component 722 can include functionality to store perception data generated by the perception component 722. In some instances, the perception component 722 can determine a track corresponding to an object that has been classified as an object type. For purposes of illustration only, the perception component 722, using sensor system(s) 704 can capture one or more images of an environment. The sensor system(s) 704 can capture images of an environment that includes an object, such as a pedestrian. The pedestrian can be at a first position at a time T and at a second position at time T + t (e.g., movement during a span of time t after time T). In other words, the pedestrian can move during this time span from the first position to the second position. Such movement can, for example, be logged as stored perception data associated with the object.
- the stored perception data can, in some examples, include fused perception data captured by the vehicle.
- Fused perception data can include a fusion or other combination of sensor data from sensor system(s) 704, such as image sensors, lidar sensors, radar sensors, time-of-flight sensors, sonar sensors, global positioning system sensors, internal sensors, and/or any combination of these.
- the stored perception data can additionally or alternatively include classification data including semantic classifications of objects (e.g., pedestrians, vehicles, buildings, road surfaces, etc.) represented in the sensor data.
- the stored perception data can additionally or alternatively include a track data (collections of historical positions, orientations, sensor features, etc. associated with the object over time) corresponding to motion of objects classified as dynamic objects through the environment.
- the track data can include multiple tracks of multiple different objects over time. This track data can be mined to identify images of certain types of objects (e.g., pedestrians, animals, etc.) at times when the object is stationary (e.g., standing still) or moving (e.g., walking, running, etc.). In this example, the computing device determines a track corresponding to a pedestrian.
- objects e.g., pedestrians, animals, etc.
- the computing device determines a track corresponding to a pedestrian.
- the prediction component 724 can generate one or more probability maps representing prediction probabilities of possible locations of one or more objects in an environment. For example, the prediction component 724 can generate one or more probability maps for vehicles, pedestrians, animals, and the like within a threshold distance from the vehicle(s) 202. In some instances, the prediction component 724 can measure a track of an object and generate a discretized prediction probability map, a heat map, a probability distribution, a discretized probability distribution, and/or a trajectory for the object based on observed and predicted behavior. In some instances, the one or more probability maps can represent an intent of the one or more objects in the environment.
- the planning component 726 can determine a path for the vehicle(s) 202 to follow to traverse through an environment. For example, the planning component 726 can determine various routes and paths and various levels of detail. In some instances, the planning component 726 can determine a route to travel from a first location (e.g., a current location) to a second location (e.g., a target location). For the purpose of this discussion, a route can be a sequence of waypoints for traveling between two locations. As non-limiting examples, waypoints include streets, intersections, global positioning system (GPS) coordinates, etc. Further, the planning component 726 can generate an instruction for guiding the autonomous vehicle along at least a portion of the route from the first location to the second location.
- GPS global positioning system
- the planning component 726 can determine how to guide the autonomous vehicle from a first waypoint in the sequence of waypoints to a second waypoint in the sequence of waypoints.
- the instruction can be a path, or a portion of a path.
- multiple paths can be substantially simultaneously generated (i.e., within technical tolerances) in accordance with a receding horizon technique. A single path of the multiple paths in a receding data horizon having the highest confidence level may be selected to operate the vehicle.
- the planning component 726 can alternatively, or additionally, use data from the perception component 722 to determine a path for the vehicle(s) 202 to follow to traverse through an environment. For example, the planning component 726 can receive data from the perception component 722 regarding objects associated with an environment. Using this data, the planning component 726 can determine a route to travel from a first location (e.g., a current location) to a second location (e.g., a target location) to avoid objects in an environment. In at least some examples, such a planning component 726 may determine there is no such collision free path and, in turn, provide a path which brings vehicle(s) 202 to a safe stop avoiding all collisions and/or otherwise mitigating damage.
- a first location e.g., a current location
- a second location e.g., a target location
- the computing device(s) 702 can include one or more system controller(s) 728, which can be configured to control steering, propulsion, braking, safety, emitters, communication, and other systems of the vehicle(s) 202. These system controller(s) 728 can communicate with and/or control corresponding systems of the drive system(s) 712 and/or other components of the vehicle(s) 202, which may be configured to operate in accordance with a path provided from the planning component 726.
- system controller(s) 728 can be configured to control steering, propulsion, braking, safety, emitters, communication, and other systems of the vehicle(s) 202.
- These system controller(s) 728 can communicate with and/or control corresponding systems of the drive system(s) 712 and/or other components of the vehicle(s) 202, which may be configured to operate in accordance with a path provided from the planning component 726.
- the vehicle(s) 202 can connect to computing device(s) 204 via network(s) 514 and can include one or more processor(s) 730 and memory 732 communicatively coupled with the one or more processor(s) 730.
- the one or more processor(s) 730 can be similar to the processor(s) 716 and the memory 732 can be similar to the memory 718.
- the memory 732 of the computing device(s) 204 stores a scenario editor component 108, a parameter component 112, an error model component 116, a parameterized scenario component 120, a simulation component 124, an analysis component 128, and a vehicle performance component 130.
- the scenario editor component 108, the parameter component 112, the error model component 116, the parameterized scenario component 120, the simulation component 124, the analysis component 128, and the vehicle performance component 130 can additionally, or alternatively, be accessible to the computing device(s) 204 (e.g., stored in a different component of computing device(s) 204 and/or be accessible to the computing device(s) 204 (e.g., stored remotely).
- the scenario editor component 108, the parameter component 112, the error model component 116, the parameterized scenario component 120, the simulation component 124, the analysis component 128, and the vehicle performance component 130 can be substantially similar to the scenario editor component 108, the parameter component 112, the error model component 116, the parameterized scenario component 120, the simulation component 124, the analysis component 128, and the vehicle performance component 130 of FIG. 1.
- the processor(s) 716 of the computing device(s) 702 and the processor(s) 730 of the computing device(s) 204 can be any suitable processor capable of executing instructions to process data and perform operations as described herein.
- the processor(s) 716 and 730 can comprise one or more Central Processing Units (CPUs), Graphics Processing Units (GPUs), or any other device or portion of a device that processes electronic data to transform that electronic data into other electronic data that can be stored in registers and/or memory.
- integrated circuits e.g., ASICs, etc.
- gate arrays e.g., FPGAs, etc.
- other hardware devices can also be considered processors in so far as they are configured to implement encoded instructions.
- the memory 718 computing device(s) 702 and the memory 732 of the computing device(s) 204 are examples of non-transitory computer-readable media.
- the memory 718 and 732 can store an operating system and one or more software applications, instructions, programs, and/or data to implement the methods described herein and the functions attributed to the various systems.
- the memory 718 and 732 can be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), nonvolatile/Flash-type memory, or any other type of memory capable of storing information.
- SRAM static random access memory
- SDRAM synchronous dynamic RAM
- Flash-type memory any other type of memory capable of storing information.
- the architectures, systems, and individual elements described herein can include many other logical, programmatic, and physical components, of which those shown in the accompanying figures are merely examples that are related to the discussion herein.
- aspects of some or all of the components discussed herein can include any models, algorithms, and/or machine learning algorithms.
- the components in the memory 718 and 732 can be implemented as a neural network.
- FIG. 8 depicts an example process 800 for determining a safety metric associated with a vehicle controller. Some or all of the process 800 can be performed by one or more components in FIGS. 1-7, as described herein. For example, some or all of the process 800 can be performed by the computing device(s) 204, and/or computing device(s) 702.
- the process 800 can include receiving log data associated with operating in autonomous vehicle in an environment.
- the log data can be generated by a vehicle(s) capturing, at least, sensor data of an environment.
- the process 800 can include determining, based on the log data (or other data), a set of scenarios, a scenario of the set of scenarios comprising a scenario parameter associated with an aspect of the environment.
- a computing device can group similar scenarios represented in the log data. For example, scenarios can be grouped together using, for example, k-means clustering and/or evaluating a weighted distance (e.g., Euclidian) between parameters of the environment.
- the scenario parameter can represent an environmental parameter such as a night time environmental parameter or a wet conditions environmental parameter.
- the scenario parameter can be associated with a vehicle or an object (e.g., a pose, a speed, etc.).
- the process 800 can include determining an error model associated with a subsystem of the autonomous vehicle.
- An error model component can compare the vehicle data (e.g., log data) with the ground truth data to determine a difference between the vehicle data and the ground truth data.
- the vehicle data can represent an estimated value associated with an object in an environment such as an estimated position, an estimated orientation, an estimated extent, and the like and the ground truth data can represent an actual position, an actual orientation, or an actual extent of the object.
- the error model component can determine an error associated with a subsystem of a vehicle (e.g., a perception system, a tracking system, a prediction system, and the like).
- the process 800 can include determining, based on the scenario parameter and the error model, a parameterized scenario. These can be combined to create a parameterized scenario that can cover the variations provided by the scenario parameter(s) and/or the error model.
- the scenario parameter(s) can be randomly selected and combined to create the parameterized scenario.
- the scenario parameter(s) can be combined based on a probability of occurring concurrently.
- log data can indicate that 5% of driving experiences include a pedestrian encounter and the parameterized scenario component can include a pedestrian as a scenario parameter in 5% of the parameterized scenarios generated by the parameterized scenario component.
- the parameterized scenario component can verify a parameterized scenario to reduce unlikely or improbably combinations of scenario parameters.
- a vehicle would not be placed in a lake and a pedestrian would not be walking at a speed of 30 meters per second.
- a parameterized scenario may comprise ranges of distances, velocities, lighting conditions, weather conditions, etc. of a vehicle and a jaywalker in a particularly defined roadway having various Gaussian distributions (or other distributions) of error on the perception models, prediction models, etc. based at least in part on the scenario parameter(s).
- the process 800 can include perturbing the parameterized scenario by modifying, based at least in part on the error, at least one of the parameterized scenario, the scenario parameter, or a component of a simulated vehicle.
- an uncertainty can be associated with a scenario parameter.
- a position of an object can be associated with a 5% uncertainty causing an autonomous controller to traverse the environment while accounting for the 5% uncertainty.
- a simulator can determine an error to incorporate into the simulation from the error model as the simulator executes the simulation.
- the process 800 can include instantiating the simulated vehicle in the perturbed parametrized scenario.
- the simulator can use a simulated vehicle that can be associated with an autonomous controller and have the autonomous controller traverse the simulated environment. Instantiating the autonomous vehicle controller in the parameterized scenario and simulated the parameterized scenario can efficiently cover a wide range of variations of a scenario without requiring a manual enumeration of the variations.
- simulation data can indicate how the autonomous vehicle controller responded to the parameterized scenario and determine a successful outcome or an unsuccessful outcome based at least in part on the simulation data.
- the process can include receiving simulation data indicating how the simulated vehicles responds to the perturbed parameterized scenario.
- the result can indicate a pass (e.g., a successful outcome), a failure, and/or degrees of success or failure associated with the vehicle controller.
- the process can include determining, based on the simulation data, a safety metric associated with the vehicle controller. For example, each simulation can result in a successful or an unsuccessful outcome.
- a vehicle performance component can, based on the simulation data, determine vehicle performance data which can indicate how the vehicle performs in the environment. Based on sensitivity analysis, the vehicle performance data can indicate scenarios that where the result of the simulation was unsuccessful, a cause of the unsuccessful simulation, and/or bounds of the scenario parameters indicating values of a scenario parameter where the result of the simulation was successful. Therefore, the safety metric can indicate a passing/failing rate of the vehicle controller in a variety of simulated scenarios.
- FIG. 9 depicts a flow diagram of an example process for determining a statistical model associated with a subsystem of an autonomous vehicle. Some or all of the process 900 can be performed by one or more components in FIGS. 1-7, as described herein. For example, some or all of the process 900 can be performed by the computing device(s) 204, and/or computing device(s) 702.
- the process 900 can include receiving vehicle data (or other data) associated with a subsystem of an autonomous vehicle.
- vehicle data can include log data captured by a vehicle traveling through an environment.
- vehicle data can include control data (e.g., data used to control systems such as steering, braking, and the like) and/or sensor data (e.g., lidar data, radar data, and the like).
- the process 900 can include determining, based on the vehicle data, output data associated with the subsystem.
- the subsystem can be a perception system and the output data can be a bounding box associated with an object in the environment.
- the process 900 can include receiving ground truth data associated with the subsystem.
- the ground truth data can be manually labeled and/or determined from other, validated, machine-learned components.
- the ground truth data can include a validated bounding box that is associated with the object in the environment.
- the process 900 can include determining a difference between a first portion of the output data and a second portion of the ground truth data, the difference representing an error associated with the subsystem.
- the output data can include a bounding box associated with the object in the environment as detected by a perception system of a vehicle and the ground truth data can include a validated bounding box associated with the object.
- a difference between the two bounding boxes can indicate an error associated with the perception system of the vehicle.
- the bounding box of the output data can be larger than the validated bounding box indicating that the perception system detects that the object is larger than in the environment.
- the process 900 can include determining, based on the difference, a statistical model associated with the subsystem.
- FIG. 10 depicts an example 1000 of a plurality of regions of an environment that are associated with an error probability.
- an error model can indicate one or more parameters associated with an environment and the error model can indicate a probability and/or error distribution associated with the one or more parameters.
- An error can represent a difference between data that is associated with an environment generated by a vehicle and the actual environment (or ground truth data representing the environment). For example, an error can indicate a difference in a size, a position, and/or a velocity of an object in the environment, or whether the object was detected in the environment.
- the error can indicate a difference between a classification of an object (e.g., a pedestrian, a bicyclist, a vehicle, signage, a static obj ect, a dynamic obj ect, etc.) generated by a vehicle and an actual classification of the object.
- the error can indicate a difference between a predicted trajectory of an object and an actual trajectory of the object.
- the example 1000 illustrates probabilities associated with errors and how factors such as a distance between a sensor and a region in the environment or the type of object in the environment can affect such probabilities.
- a region 1002 can be associated with a portion of an environment that is within a line of sight of the vehicle 210. Based at least in part on, for example, a distance between the region 1002 and the vehicle 210, the error model can indicate a probability 1004(1) associated with an error 1006(1). For example, a relatively close distance between the region 1002 and the vehicle 210 can lower the probability of an error due to an increase in sensor data accuracy, precision, and/or density compared to the sensor data at a region of the environment that is relatively farther from the vehicle 210.
- environment conditions can affect a probability of errors.
- the probability 1004(1) of an error 1006(1) associated with an object that is perceived by the vehicle 210 but does not exist in the environment can be lower during ideal conditions (e.g., good lighting, clear weather, etc.) compared to a probability 1004(1) associated with the false positive error during environmental conditions such as darkness, rain, snow, etc.
- a region 1008 can be associated with a portion of the environment that is within a line of sight of the vehicle 210 and is associated with an object 212. Based at least in part on, for example, a distance between the region 1008 and the vehicle 210 and/or a classification associated with the object 212, the error model can indicate a probability 1004(2) associated with an error 1006(2).
- the vehicle 210 can, based on perception data, determine a size associated with the object 212.
- the error model can indicate that the probability 1004(2) of an error 1006(2) that the actual size of the object 212 being twice as large as what is indicated by the perception data is lower than the probability 1004(2) of an error 1006(2) that the actual size of the object 212 is fractionally larger than what is indicated by the perception data.
- a region 1010 can be associated with a portion of the environment that is occluded, or partially occluded, by the object 212. Based at least in part on, for example, an amount of the region 1010 that is occluded, the error model can indicate a probability 1004(3) associated with an error 1006(3).
- the probability 1004(3) of an error 1006(3) associated with an object that is not perceived (also referred to as omitted) by the vehicle 210 but does exist in the environment can be higher compared to the probability 1004(1) of the error 1006(1) associated with the false negative error at the region 1002 due to the occlusion, or partial occlusion, of the region 1010.
- a region 1012 can be associated with a portion of the environment that is within line of sight of the vehicle 210 and is associated with an object 1014. Based at least in part on, for example, a distance between the region 1012 and the vehicle 210 and/or a classification associated with the object 1014, the error model can indicate a probability 1004(4) associated with an error 1006(4).
- the vehicle 210 can, based on perception data, determine a pose associated with the object 1014.
- the error model can indicate that the probability 1004(4) of an error 1006(4) that the actual pose of the object 1014 is different than the pose that is indicated by the perception data.
- the probability 1004(4) of the error 1006(4) associated with the region 1012 can be higher than the probability 1004(2) of the error 1006(2) due to an increase in distance between the region 1012 and the vehicle 210 compared to the distance between the region 1008 and the vehicle 210.
- the errors 1006(4) and 1006(2) may vary based on the classifications associated with the objects 212 and 1014. That is, the error models may be conditioned based on classification, distance, environmental factors, and the like.
- the error model indicating the probabilities 1004(1), 1004(2), 1004(3), and 1004(4) and the errors 1006(1), 1006(2), 1006(3), and 1006(4) can be generated based at least in part on vehicle data 104 generated by vehicle(s) 202.
- vehicle(s) 202 can generate vehicle data 104 that is associated with an environment and a time.
- the vehicle data 104 can include sensor data, perception data, prediction data, and the like.
- an error model component 116 can receive the vehicle data 104 and ground truth data 302 to determine a difference between the vehicle data 104 and the ground truth data 302. The difference can indicate an error of the vehicle data 104.
- the error can be associated with a difference in a classification of an object, a size of an object, a position of an object, a velocity of an object, and/or a trajectory of an object in the environment.
- the difference can be associated with the vehicle data 104 indicating that an object exists in the environment when it does not actually exist (e.g., a false positive error) and, in some instances, the difference can be associated with the vehicle data 104 indicating that an object does not exist (e.g., that is omitted) in the environment when it does actually exist (e.g., a false negative error).
- the vehicle data 104 can include data associated with a plurality of environments and the error model component 116 can determine a frequency (also referred to as an occurrence frequency) associated with one or more errors.
- a frequency also referred to as an occurrence frequency
- the error model can indicate that the probability of the vehicle data 104 representing a height of a curb as taller than the actual curb in the environment by 2 cm is higher than the probability of the vehicle data 104 representing the height of the curb as taller than the actual curb by 1 meter and the difference and the probability can be represented as a distribution in the error model.
- various environmental parameters such as, but not limited to, vehicle velocity, position, object position, velocity, object classification, weather, time of day, etc.
- vehicle velocity, position, object position, velocity, object classification, weather, time of day, etc. may be used in conjunction with the error in order to cluster the measurements into various categories (e.g., using k-means, decision trees, other suitable clustering algorithms, etc.). Measurements in the respective clusters can then be aggregated to determine error distributions associated with the clusters.
- the vehicle may select the cluster having the closest environmental parameters as currently observed and, in turn, use the associated error model.
- a machine-learned model can be used to determine a distribution fit.
- the difference between the vehicle data 104 and the ground truth data 302 can represent discrete points.
- a machine-learned model can determine, based on the discrete points, a distribution by fitting one or more distributions to the discrete points.
- the distribution can be a discrete distribution (e.g., a logarithmic distribution, a Poisson distribution, etc.), a Bernoulli distribution, a continuous distribution (e.g., a generalized normal distribution, a Gaussian distribution, etc.), a mixed discrete/continuous distribution, a joint distribution, and the like.
- a histogram (or the like) of measured data associated with a particular cluster may be stored in addition to/altematively to determining a best fit of the distribution.
- clustering algorithms can be used to cluster the vehicle data 104.
- the vehicle data 104 can be clustered based on a classification associated with an object, a time of day, an amount of visible light that is detected by the vehicle(s) 202, a weather condition (e.g., sunshine, overcast, precipitation, fog, and the like), a velocity of the vehicle(s) 202, a pose of the vehicle(s) 202, and/or a region of an environment.
- Clustering algorithms can include, for example, class specific clustering, k-means algorithms, expectation-maximization algorithms, multivariate normal distributions, and the like.
- the clustered vehicle data 104 can be used to determine distributions associated with a vehicle data cluster of the clustered vehicle data 104.
- class specific clustering can include clustering objects represented in the vehicle data 104 by classification (e.g., pedestrian, vehicle, etc.) and a distribution associated with the pedestrian cluster can be used as a pedestrian error model and a distribution associated with the vehicle cluster can be used as a vehicle error model.
- FIG. 11 illustrates an example 1100 of vehicle data and the environment represented by the vehicle data and differences between the vehicle data and the environment.
- the example 1100 illustrates a matrix of situations: true positive errors, false negative errors, false positive errors, and true negative situations (no error), each discussed below and throughout this disclosure.
- Example 1102 illustrates vehicle data 1104(1) that is associated with environment 1106(1).
- the vehicle data 1104(1) can include perception data 1108 which can be associated with an object (e.g., object 1110) that is perceived by a vehicle in an environment 1106(1).
- the perception data 1108 can misrepresent attribute value(s) associated with the object 1110 such as a position, classification, size, etc. of the associated object 1110. Therefore, while this can be treated as a true positive scenario, other differences between the vehicle data 1104(1) and the environment 1106(1) can occur.
- Example 1112 illustrates vehicle data 1104(2) that is associated with environment 1106(2). As illustrated in example 1112, the vehicle data 1104(2) can indicate that a vehicle does not perceive any objects in the environment but that an object 1114 does exist in the environment 1106(2). This can be treated as a false negative scenario.
- Example 1116 illustrates vehicle data 1104(3) that is associated with the environment 1106(3).
- the vehicle data 1104(3) can include perception data 1118 which can be associated with an object that is perceived by a vehicle but does not exist as illustrated by environment 1106(3). This can be treated as a false positive scenario.
- Example 1120 illustrates vehicle data 1104(4) that is associated with the environment 1106(4).
- the vehicle data 1104(4) does not represent any objects and, likewise, the environment 1106(4) does not include any objects. Therefore, this can be treated as a true negative scenario.
- an error model can be conditioned by distance, environmental factors, and the like.
- a false positive error may indicate, for example, a probability that a perception system determines an object to be present in a region of an environment when an object does not exist in that region of the environment.
- an error model can be conditioned by distance, environment factors, attributes of an object, and the like.
- a false negative error may indicate, for example, a probability that a perception system determines an obj ect not to be present in a region of an environment when an obj ect does, in fact, exist in that region of the environment.
- a false negative error may be based at least in part on a classification of the true object, a size of the true object, a distance between a sensor capturing data and the region of the environment associated with the object, an amount of occlusion of the true object, and the like.
- an error model can include distributions that correspond to the examples 1102, 1112, 1116, and/or 1120 (e.g., true positive, false negative, false positive, and/or true negative scenarios).
- An autonomous vehicle controller can traverse a simulated environment and a simulation component 124 can, based at least in part on the error model, determine a probability of a true negative, a false positive, a false negative, and/or a true positive associated with a region of the simulated environment.
- the simulation component 124 can determine perturbed simulation data to perturb the simulated environment by creating a false positive scenario, a false negative scenario, and/or a true positive scenario and determine a response of the simulated vehicle controller (also referred to as a simulated autonomous vehicle controller) to the false positive scenario, the false negative scenario, and/or the true positive scenario.
- a false positive scenario also referred to as a false negative scenario
- a true positive scenario determines a response of the simulated vehicle controller (also referred to as a simulated autonomous vehicle controller) to the false positive scenario, the false negative scenario, and/or the true positive scenario.
- the simulation component 124 can select an error model based at least in part on a distance between the simulated vehicle controller and a simulated object and/or any additional environmental parameters as discussed herein. For example, when a simulated object is represented as 5 meters away from the simulated vehicle controller at a first time, the simulation component 124 can determine an error model representative of errors encountered at such a distance. The simulation can represent the simulated obj ect at a second distance (e.g., 10 meters) away from the simulated vehicle controller at a second time after the first time. The simulation component 124 can select an error model based on such a distance and can determine additional perturbed scenario data that represents the simulated object in the simulated environment at a second time. Therefore, the simulation component 124 can use the error model to continuously (or intermittingly) perturb the simulated environment as the simulation executes.
- the error distribution of the error model can be based, at least in part, on the elapsed simulation time.
- a simulated vehicle controller can perceive a simulated object at a first time and perceive the simulated object at a second time.
- the error model can indicate that a probability of an error associated with the perception of the simulated object at the first time can be higher than the probability of an error associated with the perception of the simulated object at the second time.
- the first time can represent a time in which a simulated vehicle controller first detects a simulated object. That is, an error model may vary based at least in part on a duration associated with a simulated object being tracked by the simulated vehicle controller.
- the error model can be used during the operation of an autonomous vehicle in an environment that is not simulated.
- an autonomous vehicle can traverse an environment and capture sensor data of the environment.
- a perception system of the autonomous vehicle can generate perception data that indicates, for example, objects in the environment.
- the vehicle can include an error model component and can use the error model component to determine a probability of an error associated with the perception data.
- the autonomous vehicle can perceive an object in an environment such as a parking curb in a parking lot.
- the autonomous vehicle can receive a probability of an error associated with a location of the parking curb.
- the error model can indicate that there is a 5% probability that the parking curb is within 0.1 meters of a lateral direction from the location that the autonomous vehicle perceived the parking curb.
- the probability can be based on characteristics such as x-position (global position), a y-position (global position), a z-position (global position), an orientation, an object type (e.g., a classification), a velocity of the object, an extent of the object (size), a presence of another object in the environment, a state of another object in the environment, a time of day, a day of a week, a season, a weather condition, an indication of darkness/light, etc.
- a planning system of the autonomous vehicle can more safely traverse the environment by accounting for the probability of the perception error.
- the planning system can use a probability threshold and determine a trajectory that avoids a location of an object that meets or exceeds a probability threshold.
- the planning system can use a probability threshold of 1% and traverse around locations of objects that have probabilities of 1% or greater.
- the planning system can use probability thresholds associated with a classification of the object.
- the planning system can have a lower probability threshold associated with a pedestrian classification compared to a probability threshold associated with debris on a driving surface.
- FIG. 12 depicts a flow diagram of an example process for determining an error model and determining perturbed simulation data.
- the process 1200 can include receiving vehicle data indicative of a perceived state of an object.
- the vehicle data can include log data captured by a vehicle traveling through an environment.
- the vehicle data can include control data (e.g., data used to control systems such a steering, braking, and the like), sensor data (e.g., lidar data, radar data, and the like), and/or an output from a subsystem of the vehicle (e.g., perception data (e.g., classification data, bounding boxes, x-position (global position), a y-position (global position), a z-position (global position), an orientation, an entity type (e.g., a classification), a velocity of the entity, an extent of the entity, prediction data, and the like).
- perception data e.g., classification data, bounding boxes, x-position (global position), a y-position (global position), a z-position (global position)
- an entity type e.g., a classification
- the process 1200 can include receiving, based at least in part on the vehicle data, ground truth data associated with the object.
- the ground truth data can be manually labeled and/or determined from other, validated, machine-learned components.
- the ground truth data can include a validated bounding box that is associated with the object in the environment.
- the process 1200 can include determining a difference between the vehicle data and the ground truth data. If a difference does not exist, the process 1200 can return to operation 1202. If a difference does exist, the process can proceed to operation 1208.
- the process 1200 can include determining, based at least in part on the vehicle data and the ground truth data, an error. As discussed above, a difference between the vehicle data and the ground truth data can represent an error in the vehicle data.
- the error can be associated with a probability and a distribution of errors and probabilities can be included in the error model.
- the process 1200 can include determining, based at least in part on the vehicle data, a plurality of parameters.
- the vehicle data can include data such as control data (e.g., data used to control systems such a steering, braking, and the like), sensor data (e.g., lidar data, radar data, and the like), and/or an output from a subsystem of the vehicle (e.g., perception data (e.g., classification data, bounding boxes, x-position (global position), a y-position (global position), a z-position (global position), an orientation, an entity type (e.g., a classification), a velocity of the entity, an extent of the entity, prediction data, and the like).
- control data e.g., data used to control systems such a steering, braking, and the like
- sensor data e.g., lidar data, radar data, and the like
- an output from a subsystem of the vehicle e.g., perception data (e.g.,
- the vehicle data can include characteristics associated with the environment such as, but not limited to, presence of another entity in the environment, a state of another entity in the environment, a time of day, a day of a week, a season, a weather condition, an indication of darkness/light, etc.
- the process 1200 can include clustering, based at least in part on the plurality of parameters and the error, at least a portion of the vehicle data.
- the vehicle data can be clustered based on a classification of an object represented in the vehicle data, a type of day, an amount of visible light that is detected by a vehicle, a weather condition, a velocity of the vehicle, a pose of the vehicle, and/or a region of an environment.
- the distribution of errors and probabilities can be associated with a cluster of the clustered vehicle data.
- the process 1200 can include determining, based at least in part on the portion of the vehicle data, an error model.
- a machine-learned model can be used to determine a distribution fit to fit a distribution to the distribution of errors and probabilities represented in the vehicle data.
- a fit metric can be used to determine the fit of a distribution to the distribution of errors and probabilities.
- a negative log-likelihood function or a log-likelihood function can be used to determine the fit metric.
- a distribution against non-clustered vehicle data can produce a negative log-likelihood value that is higher than a distribution against vehicle data that is clustered using multivariate expectation-maximization, where a higher negative log-likelihood value indicates a poorer fit of the distribution.
- the operation 1214 can include determining, for “empty” region(s) of an environment, an error model associated with a false positive at that location. In some examples, the operation 1214 can include determining, for “occupied” region(s) of an environment, an error model associated with a false negative at that location. For example, the operation 1214 can include determining an error model based on distance, environmental factors, and the like.
- a system comprising: one or more processors; and one or more computer-readable media storing computer-executable instructions that, when executed, cause the system to perform operations comprising: receiving log data associated with operating an autonomous vehicle in an environment; determining, based at least in part on the log data, a set of scenarios, a scenario of the set of scenarios comprising a scenario parameter associated with an aspect of the environment; determining an error model associated with a subsystem of the autonomous vehicle; determining, based at least in part on the scenario parameter and the error model, a parameterized scenario; perturbing the parameterized scenario by adding an error to at least one of the scenario parameter or a component of a simulated vehicle to be instantiated in a perturbed parameterized scenario, the simulated vehicle controlled by a vehicle controller; instantiating the simulated vehicle in the perturbed parameterized scenario; receiving simulation data indicating how the simulated vehicle responds to the perturbed parameterized scenario; and determining, based at least in part on the simulation data, a safety metric associated with
- determining the set of scenarios comprises: clustering the log data to determine a first set of clusters, wherein an individual cluster of the first set of clusters is associated with an individual scenario; determining, based at least in part on the first set of clusters, a probability associated with the individual cluster; and determining, based at least in part on a probability threshold and the first set of clusters, a second set of clusters.
- determining the error model comprises: receiving ground truth data associated with the environment; determining, based at least in part on comparing the ground truth data to the log data, an error; and determining, based at least in part on the error, an error distribution; wherein the error model comprises the error distribution.
- D The system of paragraph A, wherein the parameterized scenario is a first parameterized scenario, the perturbed parameterized scenario is a first perturbed parameterized scenario, and the simulation data is first simulation data, the operations further comprising: determining, based on the first simulation data, a second parameterized scenario comprising at least one of a first subset of the scenario parameter or a second subset of the error model; perturbing the second parameterized scenario as a second perturbed parameterized scenario; instantiating the simulated vehicle in the second perturbed parameterized scenario; receiving second simulation data; and updating, based at least in part on the second simulation data, the safety metric.
- E A method comprising: determining a scenario comprising a scenario parameter describing a portion of an environment; receiving an error model associated with a subsystem of a vehicle; determining, based at least in part on the scenario, the scenario parameter, and the error model, a parameterized scenario; perturbing the parameterized scenario as a perturbed parameterized scenario; receiving simulation data indicating how the subsystem of the vehicle responds to the perturbed parameterized scenario; and determining, based at least in part on the simulation data, a safety metric associated with the subsystem of the vehicle.
- determining the scenario comprises : receiving log data associated with an autonomous vehicle; clustering the log data to determine a first set of clusters, wherein an individual cluster of the first set of clusters is associated with the scenario; determining, based at least in part on the first set of clusters, a probability associated with the individual cluster; and determining that the probability meets or exceeds a probability threshold.
- K The method of paragraph E, wherein the safety metric indicates a probability of meeting or exceeding a cost threshold.
- L The method of paragraph E, wherein the portion is a first portion, the method further comprising: receiving map data, wherein a second portion of the map data is associated with the first portion of the environment; and determining that the second portion of the map data is associated with a scenario associated with a probability that meets or exceeds a threshold probability associated with the scenario parameter.
- M A non-transitory computer-readable medium storing instructions executable by a processor, wherein the instructions, when executed, cause the processor to perform operations comprising: determining a scenario comprising a scenario parameter describing a portion of an environment; one or more of receiving or determining an error model associated with a subsystem of a vehicle; determining, based at least in part on the scenario, the scenario parameter, and the error model, a parameterized scenario; perturbing the parameterized scenario as a perturbed parameterized scenario; receiving simulation data indicating how the subsystem of the vehicle responds to the perturbed parameterized scenario; and determining, based at least in part on the simulation data, a safety metric associated with the subsystem of the vehicle.
- N The non-transitory computer-readable medium of paragraph M, wherein the scenario parameter is associated with at least one of an object size, an object velocity, an object pose, an object density, a vehicle velocity, a vehicle trajectory.
- O The non-transitory computer-readable medium of paragraph M, wherein determining the scenario comprises: receiving log data associated with an autonomous vehicle; clustering the log data to determine a first set of clusters, wherein an individual cluster of the first set of clusters is associated with the scenario; determining, based at least in part on the first set of clusters, a probability associated with the individual cluster; and determining that the probability meets or exceeds a probability threshold.
- R The non-transitory computer-readable medium of paragraph Q, the operations further comprising: disabling at least a first portion of one of the scenario parameter or the error model; and associating the second simulation data with at least a second portion of one of the scenario parameter or the error model that is not disabled.
- S The non-transitory computer-readable medium of paragraph M, wherein the safety metric indicates a probability of meeting or exceeding a cost threshold.
- T The non-transitory computer-readable medium of paragraph M, wherein the error model is associated with one or more of a perception system of the vehicle, a prediction system of the vehicle, or a planner system of the vehicle.
- a system comprising: one or more processors; and one or more computer-readable media storing computer-executable instructions that, when executed, cause the system to perform operations comprising: receiving vehicle data; inputting at least a first portion of the vehicle data into a subsystem of an autonomous vehicle, the subsystem associated with at least one of a perception system, a planning system, a tracking system, or a prediction system; determining, based at least in part on a second portion of the vehicle data, an environmental parameter; receiving, from the subsystem, an estimated value; receiving ground truth data associated with the subsystem; determining a difference between the estimated value and the ground truth data, the difference representing an error associated with the subsystem; and determining, based at least in part on the difference, a statistical model associated with the subsystem indicating a probability of the error, the probability associated with the environmental parameter.
- V The system of paragraph U, wherein the vehicle data comprises sensor data from a sensor on the autonomous vehicle, wherein the environmental parameter comprises one or more of a speed of the autonomous vehicle or a weather condition, and wherein the subsystem is a perception subsystem, the estimated value is one or more of an estimated position, estimated orientation, or estimated extent of an object represented in the vehicle data, and the ground truth data represents an actual position, actual orientation, or actual extents of the object.
- the vehicle data comprises sensor data from a sensor on the autonomous vehicle
- the environmental parameter comprises one or more of a speed of the autonomous vehicle or a weather condition
- the subsystem is a perception subsystem
- the estimated value is one or more of an estimated position, estimated orientation, or estimated extent of an object represented in the vehicle data
- the ground truth data represents an actual position, actual orientation, or actual extents of the object.
- determining the statistical model comprises: determining, based at least in part on the vehicle data, a first frequency associated with the environmental parameter and a second frequency associated with the difference; and determining, based at least in part on the first frequency and the second frequency, the probability.
- X The system of paragraph U, the operations further comprising: determining, based at least in part on simulated vehicle data, a simulated environmental parameter; determining the simulated environmental parameter corresponds to the environmental parameter; determining, based at least in part on the simulated vehicle data and the subsystem, a simulated estimated value; and perturbing, based at least in part on the probability, the simulated estimated value by altering a portion of a corresponding simulated scenario based at least in part on the error.
- Y A method comprising: receiving data associated with a vehicle; determining, based at least in part on a first portion of the data, an environmental parameter; determining, based at least in part on a second portion of the data, output data associated with a system of the vehicle; receiving ground truth data associated with the system and the data; determining a difference between the output data and a the ground truth data, the difference representing an error associated with the system; and determining, based at least in part on the difference, a statistical model associated with the system indicating a probability of the error, the probability associated with the environmental parameter.
- Z The method of paragraph Y, wherein determining the statistical model comprises: determining, based at least in part on the data, a frequency associated with the error.
- AA The method of paragraph Y, wherein the environmental parameter comprises one or more of a speed of the vehicle, a weather condition, a geographic position of the vehicle, or a time of day.
- AC The method of paragraph Y, wherein the system is a perception system, the output data comprises a first bounding box associated with an object, the ground truth data comprises a second bounding box associated with the object, and wherein determining the difference comprises determining the difference between at least one of: a first extent of the first bounding box and a second extent of the second bounding box; or a first pose of the first bounding box and a second pose of the second bounding box.
- AD The method of paragraph Y, wherein the system is a tracker system, the output data comprises a planned trajectory data of the vehicle, the ground truth data comprises a measured trajectory of the vehicle, and wherein determining the difference comprises determining the difference between the planned trajectory data and the measured trajectory.
- AE The method of paragraph Y, wherein the system is associated with a prediction system, the data comprises a predicted trajectory of an object in an environment, the ground truth data comprises an observed trajectory of the object, and wherein determining the difference comprises determining the difference between the predicted trajectory and the observed trajectory.
- AF The method of paragraph Y, wherein the data is first data, the environmental parameter is a first environmental parameter, the difference is a first difference, the error is a first error, and the probability is a first probability, the method further comprising: receiving second data associated with the system of the vehicle; determining, based at least in part on the second data, a second environmental parameter; determining a second difference between a third portion of the output data and a fourth portion of the ground truth data, the second difference representing a second error associated with the system; and updating the statistical model associated with the system, the statistical model indicating a second probability of the second error, the second probability associated with the second environmental parameter.
- a non-transitory computer-readable medium storing instructions executable by a processor, wherein the instructions, when executed, cause the processor to perform operations comprising: receiving data; determining, based at least in part on the data, an environmental parameter; determining, based at least in part on the data and a system of a vehicle, output data; receiving ground truth data associated the system and the data; determining a difference between a first portion of the output data and a second portion of the ground truth data, the difference representing an error associated with the system; determining, based at least in part on the difference, a statistical model associated with the system indicating a probability of the error; and associating the probability with the environmental parameter.
- AH The non-transitory computer-readable medium of paragraph AG, wherein determining the statistical model comprises: determining, based at least in part on the data, a frequency associated with the difference.
- AI The non-transitory computer-readable medium of paragraph AG, wherein the environmental parameter comprises one or more of a speed of the vehicle, a weather condition, or a time of day.
- AJ The non-transitory computer-readable medium of paragraph AG, the operations further comprising: generating a simulation comprising a simulated vehicle; receiving simulated data; determining a simulated environmental parameter corresponds to the environmental parameter; inputting at least a portion of the simulated data into the system; receiving, from the system, simulated output data; and altering, based at least in part on a request and the probability and the error, the simulated output data.
- AK The non-transitory computer-readable medium of paragraph AG, wherein the system is a perception system, the data comprises a first bounding box associated with an object, the ground truth data comprises a second bounding box associated with the object, and wherein determining the difference comprises determining the difference between at least one of: a first extent of the first bounding box and a second extent of the second bounding box; or a first pose of the first bounding box and a second pose of the second bounding box.
- AL The non-transitory computer-readable medium of paragraph AG, wherein the system is a tracker system, the data comprises a planned trajectory data of the vehicle, the ground truth data comprises a measured trajectory of the vehicle, and wherein determining the difference comprises determining the difference between the planned trajectory data and the measured trajectory.
- AM The non-transitory computer-readable medium of paragraph AG, wherein the system is associated with a prediction system, the data comprises a predicted trajectory of an object in an environment, the ground truth data comprises an observed trajectory of the object, and wherein determining the difference comprises determining the difference between the predicted traj ectory and the observed traj ectory .
- AN The non-transitory computer-readable medium of paragraph AG, wherein the data is first data, the environmental parameter is a first environmental parameter, the difference is a first difference, the error is a first error, and the probability is a first probability, the operations further comprising: receiving second data associated with the system of the vehicle; determining, based at least in part on the second data, a second environmental parameter; determining a second difference between a third portion of the output data and a fourth portion of the ground truth data, the second difference representing a second error associated with the system; and updating the statistical model associated with the system, the statistical model indicating a second probability of the second error, the second probability associated with the second environmental parameter.
- AO A system comprising: one or more processors; and one or more non- transitory computer-readable media storing computer-executable instructions that, when executed, cause the system to perform operations comprising: receiving vehicle data from a vehicle, the vehicle data associated with a state of an object; receiving ground truth data associated with the object; determining, based at least in part on the vehicle data and the ground truth data, an error; determining, based at least in part on the vehicle data, a plurality of parameters; clustering, as a plurality of clusters and based at least in part on the plurality of parameters and the error, at least a portion of the vehicle data; and determining, based at least in part on the portion of the vehicle data associated with a cluster of the plurality of clusters, an error model.
- AP The system of paragraph AO, wherein the vehicle data is based at least in part on sensor data from a sensor associated with the vehicle.
- AQ The system of paragraph AO, wherein the plurality of parameters is associated with at least two or more of a weather condition, a first time of day, a second time of year, a distance to the object, a classification of the object, a size of the object, a velocity of the object, a position of the object, or an orientation of the object.
- AR The system of paragraph AO, wherein the error is a first error, the operations further comprising: receiving perception data; determining, based at least in part on the perception data and the error model, a second error associated with the perception data; and controlling, based at least in part on the perception data and the second error, the vehicle.
- AS A method comprising: receiving vehicle data from a vehicle, the vehicle data associated with a state of an object; receiving, based at least in part on the vehicle data, ground truth data associated with the object; determining, based at least in part on the vehicle data and the ground truth data, an error; determining, based at least in part on the vehicle data, a parameter; clustering, as a plurality of clusters and based at least in part on the parameter and the error, a portion of the vehicle data; and determining, based at least in part on the portion of the vehicle data associated with a cluster of the plurality of clusters, an error model.
- AV The method of paragraph AS, further comprising: determining classification data identifying a classification of the object; determining object data identifying an object parameter of the object; and determining an error distribution associated with at least one of a first cluster of the plurality of clusters or a second cluster of the plurality of clusters, the first cluster associated with the classification of the object and the second cluster associated with the object parameter of the object; wherein the error model comprises the error distribution associated with at least one of a true positive error or a false positive error.
- AX The method of paragraph AW, wherein: the error is a first error; the simulation data comprises a classification associated with a simulated object represented in the simulated environment; and the perturbed simulation data indicates a second error associated with at least one of a position of the object, an orientation of the object, an extent of the object, or a velocity of the object.
- AY The method of paragraph AS further comprising: determining, at a first time, a first error distribution associated with a false negative error; determining, at a second time after the first time, a second error distribution associated with a true positive error; and determining, at a third time after the second time, a third error distribution associated with a false positive error; wherein the error model comprises the first error distribution, the second error distribution, and the third error distribution.
- a Z The method of paragraph AS, further comprising: determining a cost associated with fitting the vehicle data to the error model; and determining, based at least in part on the cost, the error model.
- BA One or more non-transitory computer-readable media storing instructions executable by one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising: receiving vehicle data from a vehicle, the vehicle data associated with a state of an object; receiving ground truth data associated with the object; determining, based at least in part on the vehicle data and the ground truth data, an error; determining, based at least in part on the vehicle data, a parameter; clustering, as a plurality of clusters and based at least in part on the parameter, a portion of the vehicle data; and determining, based at least in part on the portion of the vehicle data associated with a cluster of the plurality of clusters, an error model.
- BB The one or more non-transitory computer-readable media of paragraph BA, wherein the state of the object comprises at least one of a size of the object, a location of the object, an orientation of the object, a velocity of the object, or a position of the object.
- BC The one or more non-transitory computer-readable media of paragraph BA, wherein the error model comprises an error distribution, the operations further comprising: determining, based at least in part on the vehicle data and the ground truth data, an occurrence frequency associated with the error; and determining, based at least in part on the occurrence frequency, the error distribution.
- BD The one or more non-transitory computer-readable media of paragraph BA, the operations further comprising: determining classification data identifying a classification of the object; determining object data identifying an object parameter of the object; and determining an error distribution associated with at least one of a first cluster of the plurality of clusters or a second cluster of the plurality of clusters, the first cluster associated with the classification of the object and the second cluster associated with the object parameter of the object; wherein the error model comprises the error distribution associated with at least one of a true positive error or a false positive error.
- BE The one or more non-transitory computer-readable media of paragraph BA, wherein the error model is one of a plurality of error models, the operations further comprising: receiving simulation data associated with a simulated vehicle controller in a simulated environment; determining, based at least in part on the error model and the simulation data, perturbed simulation data; sending the perturbed simulation data to the simulated vehicle controller in the simulated environment; and determining, based at least in part on the perturbed simulation data, a response indicating how the simulated vehicle controller responds to the perturbed simulation data.
- [0218] BF The one or more non-transitory computer-readable media of paragraph BE, wherein: the error is a first error; the simulation data comprises a classification associated with a simulated object represented in the simulated environment; and the perturbed simulation data indicates a second error associated with at least one of a position of the object, an orientation of the object, an extent of the object, or a velocity of the object.
- BG The one or more non-transitory computer-readable media of paragraph BA, the operations further comprising: determining, at a first time, a first error distribution associated with a false negative error; determining, at a second time after the first time, a second error distribution associated with a true positive error; and determining, at a third time after the second time, a third error distribution associated with a false positive error; wherein the error model comprises the first error distribution, the second error distribution, and the third error distribution.
- BH The one or more non-transitory computer-readable media of paragraph BA, the operations further comprising: determining a cost associated with fitting the vehicle data to the error model; and determining, based at least in part on the cost, the error model.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Quality & Reliability (AREA)
- Geometry (AREA)
- Evolutionary Computation (AREA)
- Multimedia (AREA)
- Aviation & Aerospace Engineering (AREA)
- Business, Economics & Management (AREA)
- Educational Administration (AREA)
- Educational Technology (AREA)
- Traffic Control Systems (AREA)
- Control Of Driving Devices And Active Controlling Of Vehicle (AREA)
Abstract
Description
Claims
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/708,019 US11734473B2 (en) | 2019-09-27 | 2019-12-09 | Perception error models |
PCT/US2020/062602 WO2021118822A1 (en) | 2019-12-09 | 2020-11-30 | Perception error models |
Publications (2)
Publication Number | Publication Date |
---|---|
EP4073782A1 true EP4073782A1 (en) | 2022-10-19 |
EP4073782A4 EP4073782A4 (en) | 2024-01-17 |
Family
ID=76330731
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP20900042.1A Pending EP4073782A4 (en) | 2019-12-09 | 2020-11-30 | Perception error models |
Country Status (4)
Country | Link |
---|---|
EP (1) | EP4073782A4 (en) |
JP (1) | JP2023504506A (en) |
CN (1) | CN114787894A (en) |
WO (1) | WO2021118822A1 (en) |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2020230323A1 (en) * | 2019-05-16 | 2020-11-19 | 日本電信電話株式会社 | Abnormality detection device, method, system, and program |
DE102022120754A1 (en) * | 2022-08-17 | 2024-02-22 | Dspace Gmbh | Method and system for running a virtual test |
Family Cites Families (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10346564B2 (en) | 2016-03-30 | 2019-07-09 | Toyota Jidosha Kabushiki Kaisha | Dynamic virtual object generation for testing autonomous vehicles in simulated driving scenarios |
US10481044B2 (en) | 2017-05-18 | 2019-11-19 | TuSimple | Perception simulation for improved autonomous vehicle control |
US10877476B2 (en) * | 2017-11-30 | 2020-12-29 | Tusimple, Inc. | Autonomous vehicle simulation system for analyzing motion planners |
US20190179979A1 (en) * | 2017-12-13 | 2019-06-13 | Uber Technologies, Inc. | Simulated Sensor Testing |
US10275689B1 (en) * | 2017-12-21 | 2019-04-30 | Luminar Technologies, Inc. | Object identification and labeling tool for training autonomous vehicle controllers |
-
2020
- 2020-11-30 WO PCT/US2020/062602 patent/WO2021118822A1/en unknown
- 2020-11-30 JP JP2022533057A patent/JP2023504506A/en active Pending
- 2020-11-30 CN CN202080084729.5A patent/CN114787894A/en active Pending
- 2020-11-30 EP EP20900042.1A patent/EP4073782A4/en active Pending
Also Published As
Publication number | Publication date |
---|---|
CN114787894A (en) | 2022-07-22 |
EP4073782A4 (en) | 2024-01-17 |
JP2023504506A (en) | 2023-02-03 |
WO2021118822A1 (en) | 2021-06-17 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11734473B2 (en) | Perception error models | |
US11625513B2 (en) | Safety analysis framework | |
US11351995B2 (en) | Error modeling framework | |
US11574089B2 (en) | Synthetic scenario generator based on attributes | |
US11568100B2 (en) | Synthetic scenario simulator based on events | |
US11150660B1 (en) | Scenario editor and simulator | |
CN113165652B (en) | Verifying predicted trajectories using a mesh-based approach | |
US20210339741A1 (en) | Constraining vehicle operation based on uncertainty in perception and/or prediction | |
WO2021041121A1 (en) | Yield behavior modeling and prediction | |
WO2020097011A2 (en) | Vehicle trajectory modification for following | |
US11628850B2 (en) | System for generating generalized simulation scenarios | |
WO2021061488A1 (en) | Safety analysis framework | |
US11526721B1 (en) | Synthetic scenario generator using distance-biased confidences for sensor data | |
US11648939B2 (en) | Collision monitoring using system data | |
US11697412B2 (en) | Collision monitoring using statistic models | |
US11415997B1 (en) | Autonomous driving simulations based on virtual simulation log data | |
US11577741B1 (en) | Systems and methods for testing collision avoidance systems | |
CN114077541A (en) | Method and system for validating automatic control software for an autonomous vehicle | |
US11810365B1 (en) | Perception error modeling | |
US11648962B1 (en) | Safety metric prediction | |
US20230150549A1 (en) | Hybrid log simulated driving | |
WO2020264276A1 (en) | Synthetic scenario generator based on attributes | |
EP4073782A1 (en) | Perception error models | |
US11767030B1 (en) | Scenario simulation execution within a truncated parameter space | |
US11814070B1 (en) | Simulated driving error models |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20220420 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
DAV | Request for validation of the european patent (deleted) | ||
DAX | Request for extension of the european patent (deleted) | ||
A4 | Supplementary search report drawn up and despatched |
Effective date: 20231215 |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G06V 20/56 20220101ALI20231211BHEP Ipc: G06F 30/20 20200101ALI20231211BHEP Ipc: G06F 11/36 20060101ALI20231211BHEP Ipc: B60W 30/095 20120101ALI20231211BHEP Ipc: G09B 9/042 20060101AFI20231211BHEP |