EP3710980A1 - Systèmes et procédés de détection de limites de voie pour véhicules autonomes - Google Patents

Systèmes et procédés de détection de limites de voie pour véhicules autonomes

Info

Publication number
EP3710980A1
EP3710980A1 EP18836692.6A EP18836692A EP3710980A1 EP 3710980 A1 EP3710980 A1 EP 3710980A1 EP 18836692 A EP18836692 A EP 18836692A EP 3710980 A1 EP3710980 A1 EP 3710980A1
Authority
EP
European Patent Office
Prior art keywords
autonomous vehicle
surrounding environment
computing system
vehicle
machine
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP18836692.6A
Other languages
German (de)
English (en)
Inventor
Min Bai
Gellert Sandor Mattyus
Namdar Homayounfar
Shenlong WANG
Shrinidhi Kowshika LAKSHMIKANTH
Raquel URTASUN
Wei-Chiu Ma
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Aurora Operations Inc
Original Assignee
Uatc LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US16/122,413 external-priority patent/US11080537B2/en
Priority claimed from US16/122,267 external-priority patent/US10803325B2/en
Application filed by Uatc LLC filed Critical Uatc LLC
Publication of EP3710980A1 publication Critical patent/EP3710980A1/fr
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/93—Lidar systems specially adapted for specific applications for anti-collision purposes
    • G01S17/931—Lidar systems specially adapted for specific applications for anti-collision purposes of land vehicles
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
    • G06F18/24133—Distances to prototypes
    • G06F18/24137—Distances to cluster centroïds
    • G06F18/2414—Smoothing the distance, e.g. radial basis function networks [RBFN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/64—Three-dimensional objects
    • G06V20/647—Three-dimensional objects by matching two-dimensional images to three-dimensional objects

Definitions

  • the present disclosure relates generally to detecting lane boundaries within the surrounding environment of an autonomous vehicle.
  • An autonomous vehicle can be capable of sensing its environment and navigating with little to no human input.
  • an autonomous vehicle can observe its surrounding environment using a variety of sensors and can attempt to comprehend the environment by performing various processing techniques on data collected by the sensors. Given knowledge of its surrounding environment, the autonomous vehicle can navigate through such surrounding environment.
  • the computing system includes one or more processors and one or more tangible, non- transitory, computer readable media that collectively store instructions that when executed by the one or more processors cause the computing system to perform operations.
  • the operations include obtaining rasterized LIDAR data associated with a surrounding environment of an autonomous vehicle.
  • the rasterized LIDAR data includes LIDAR image data that is rasterized from a LIDAR point cloud.
  • the operations include accessing data indicative of a machine-learned lane boundary detection model.
  • the operations include inputting the rasterized LIDAR data associated with the surrounding environment of the autonomous vehicle into the machine-learned lane boundary detection model.
  • the operations include obtaining an output from the machine-learned lane boundary detection model. The output is indicative of one or more lane boundaries within the surrounding environment of the autonomous vehicle.
  • the method includes obtaining, by a computing system including one or more computing devices, LIDAR data associated with a surrounding environment of an autonomous vehicle.
  • the LIDAR data is indicative of a LIDAR point cloud associated with the surrounding environment of the autonomous vehicle.
  • the method includes generating, by the computing system, rasterized LIDAR data based at least in part on the LIDAR data associated with the surrounding environment of the autonomous vehicle.
  • the method includes determining, by the computing system, one or more lane boundaries within the surrounding environment of the autonomous vehicle based at least in part on the rasterized LIDAR data and a machine-learned lane boundary detection model.
  • the autonomous vehicle includes a machine-learned lane boundary detection model, one or more processors, and one or more tangible, non-transitory, computer readable media that collectively store instructions that when executed by the one or more processors cause the autonomous vehicle to perform operations.
  • the operations include obtaining rasterized LIDAR data associated with a surrounding environment of an autonomous vehicle.
  • the rasterized LIDAR data includes LIDAR image data that is rasterized from a LIDAR point cloud.
  • the operations include inputting the rasterized LIDAR data associated with the surrounding environment of the autonomous vehicle into the machine-learned lane boundary detection model.
  • the operations include obtaining an output from the machine-learned lane boundary detection model. The output is indicative of one or more lane boundaries within the surrounding environment of the autonomous vehicle.
  • the computing system includes one or more processors and one or more tangible, non- transitory, computer readable media that collectively store instructions that when executed by the one or more processors cause the computing system to perform operations.
  • the operations include obtaining a first type of sensor data associated with a surrounding environment of an autonomous vehicle and a second type of sensor data associated with the surrounding environment of the autonomous vehicle.
  • the operations include generating overhead image data indicative of at least a portion of the surrounding environment of the autonomous vehicle based at least in part on the first type of sensor data and the second type of sensor data.
  • the operations include accessing data indicative of a machine-learned lane boundary detection model.
  • the operations include inputting the overhead image data indicative of at least the portion of the surrounding environment of the autonomous vehicle into the machine-learned lane boundary detection model.
  • the operations include obtaining an output from the machine-learned lane boundary detection model. The output is indicative of one or more lane boundaries within the surrounding environment of the autonomous vehicle.
  • the method includes obtaining, by a computing system including one or more computing devices, a first type of sensor data associated with a surrounding environment of an autonomous vehicle.
  • the method includes determining, by the computing system, a multi-dimensional feature volume of the first type of sensor data based at least in part on a first machine-learned model and the first type of sensor data.
  • the method includes obtaining, by the computing system, a second type of sensor data associated with the surrounding environment of the autonomous vehicle.
  • the method includes determining, by the computing system, a ground height estimation associated with the surrounding environment of the autonomous vehicle based at least in part on a second machine-learned model and the second type of sensor data.
  • the method includes generating, by the computing system, overhead image data indicative of at least a portion of the surrounding environment of the autonomous vehicle based at least in part on the multi- dimensional feature volume and the ground height estimation.
  • the method includes determining, by the computing system, one or more lane boundaries within the surrounding environment of the autonomous vehicle based at least in part on the overhead image data and a third machine-learned model.
  • the autonomous vehicle includes one or more processors and one or more tangible, non-transitory, computer readable media that collectively store instructions that when executed by the one or more processors cause the autonomous vehicle to perform operations.
  • the operations include obtaining camera image data associated with a surrounding environment of the autonomous vehicle and LIDAR data associated with the surrounding environment of the autonomous vehicle.
  • the operations include generating overhead image data indicative of at least a portion of the surrounding environment of the autonomous vehicle based at least in part on the camera image data and the LIDAR data.
  • the operations include determining one or more lane boundaries within the surrounding environment of the autonomous vehicle based at least in part on the overhead image data indicative of at least the portion of the surrounding environment of the autonomous vehicle and a machine-learned lane boundary detection model.
  • FIG. 1 depicts an example system overview according to example embodiments of the present disclosure
  • FIG. 2 depicts an example environment of an autonomous vehicle according to example embodiments of the present disclosure
  • FIG. 3 depicts a diagram of an example computing system with a model architecture according to example embodiments of the present disclosure
  • FIGS. 4A-B depict diagrams of example sensor data according to example embodiments of the present disclosure
  • FIG. 5 depicts an example model output according to example embodiments of the present disclosure
  • FIG. 6 depicts a flow diagram of an example method for detecting lane boundaries according to example embodiments of the present disclosure
  • FIG. 7 depicts a flow diagram of another example method for detecting lane boundaries according to example embodiments of the present disclosure.
  • FIG. 8 depicts example system components according to example embodiments of the present disclosure.
  • lane boundaries can include, for example, lane markings and/or other indicia associated with a travel lane and/or travel way (e.g., the boundaries thereof).
  • lane markings can include, for example, lane markings and/or other indicia associated with a travel lane and/or travel way (e.g., the boundaries thereof).
  • accurate detections of lane boundaries can help guide an autonomous vehicle’s detection and tracking of other traffic participants as well as the planning of its motion.
  • an autonomous vehicle can obtain camera image data and Light Detection and Ranging (LIDAR) data via its onboard cameras and LIDAR system.
  • This sensor data can depict at least a portion of the vehicle’s
  • the autonomous vehicle can pre-process the camera image data and the LIDAR data to generate input data to be used with a machine-learned lane boundary detection model (e.g., a convolutional neural network). For instance, the autonomous vehicle can pre-process the camera image data and the LIDAR data (e.g., using one or more other machine-learned models) to generate overhead image data that is indicative of an overhead, bird’s eye view of a portion of the vehicle’s surrounding environment.
  • the overhead image data can be feed into the machine-learned lane boundary detection model, which can be trained to accurately detect one or more lane boundaries within the surrounding environment based on the overhead image data.
  • the LIDAR data can be inputted into the machine-learned lane boundary detection model for accurate lane boundary detection.
  • An autonomous vehicle can utilize the detected lane boundaries to improve its performance of various actions such as, for example, localizing itself within its surroundings, perceiving object(s) (e.g., nearby vehicles, bicycles, pedestrians, etc.), predicting the motion of such object(s), and/or planning vehicle motion.
  • an autonomous vehicle can be a ground-based autonomous vehicle (e.g., car, truck, bus, etc.) or another type of vehicle (e.g., aerial vehicle) that can operate with minimal and/or no interaction from a human operator.
  • An autonomous vehicle can include a vehicle computing system located onboard the autonomous vehicle to help control the autonomous vehicle.
  • the vehicle computing system can be located onboard the autonomous vehicle, in that the vehicle computing system can be located on or within the autonomous vehicle.
  • the vehicle computing system can include one or more sensors (e.g., cameras, Light Detection and Ranging (LIDAR), Radio Detection and Ranging (RADAR), etc.), an autonomy computing system (e.g., for determining autonomous navigation), one or more vehicle control systems (e.g., for controlling braking, steering, powertrain, etc.), and/or other systems.
  • the vehicle computing system can obtain sensor data from sensor(s) onboard the vehicle (e.g., cameras, LIDAR, RADAR, etc.), attempt to comprehend the vehicle’s surrounding environment by performing various processing techniques on the sensor data, and generate an appropriate motion plan through the vehicle’s surrounding environment.
  • the vehicle’s computing system can be configured to utilize various types of sensor data to detect one or more lane boundaries.
  • the vehicle computing system can obtain a first type of sensor data and a second type of sensor data.
  • the first type of sensor data can include camera image data associated with the surrounding environment for the autonomous vehicle.
  • the camera image data can be, for example, monocular, 3 -channel RGB image data (e.g., with 960 x 300 resolution, etc.).
  • the camera image data can depict at least a portion of the surrounding environment of the autonomous vehicle (e.g., a portion that is expected to include lane boundaries, etc.).
  • the second type of sensor data can include LIDAR data associated with the surrounding environment of the autonomous vehicle.
  • the LIDAR data can be indicative of a LIDAR point cloud associated with the surrounding environment of the autonomous vehicle (e.g., created by LIDAR sweep(s) of the vehicle’s LIDAR system) that has been rasterized.
  • the vehicle computing system can rasterize the LIDAR point cloud into a two-dimensional overhead view image (e.g., with a resolution of 960 x 960 at a 5cm per pixel resolution).
  • the overhead view image can depict at least a portion of the surrounding environment (e.g., a 48m by 48m area with the vehicle at the center bottom of the image).
  • the rasterized LIDAR data e.g., the overhead view image
  • Each spatial location can contain a plurality of channels.
  • each spatial location can include a first channel representing the height of at least one pixel (e.g., the highest pixel in a discretization bin), a second channel representing a brightness level of the least one pixel (e.g., in the discretization bin), and a third channel representing a total number of points (e.g., within the discretization bin).
  • the vehicle computing system can obtain other types of data.
  • the vehicle computing system can obtain Radio Detection and Ranging (RADAR) data, motion sensor data (e.g., data from inertial measurement unit(s), wheel odometry readings, etc.), and/or other types of sensor data.
  • RADAR Radio Detection and Ranging
  • motion sensor data e.g., data from inertial measurement unit(s), wheel odometry readings, etc.
  • sensor data can be acquired via one or more of the sensors onboard the autonomous vehicle.
  • the vehicle computing system can pre-process the sensor data in order help improve the accuracy of its lane detection analysis. For instance, the vehicle computing system can generate overhead image data indicative of at least a portion of the surrounding environment of the autonomous vehicle based at least in part on the first type of sensor data (e.g., camera image data) and the second type of sensor data (e.g., LIDAR data). To do so, the vehicle computing system can include, employ, and/or otherwise leverage one or more machine-learned model(s).
  • the machine-learned models can be or can otherwise include one or more various model(s) such as, for example, neural networks (e.g., deep neural networks), and/or other multi-layer non-linear models.
  • the machine-learned models can include convolutional neural networks and/or other forms of neural networks.
  • the vehicle computing system can utilize a first machine-learned model to transform the camera image data.
  • the first machine-learned model can be a machine-learned feature projection model (e.g., a convolutional neural network) that is trained to generate a multi-dimensional feature volume based at least in part on a set of sensor data.
  • the vehicle computing system can access the machine-learned feature projection model (e.g., from a local memory onboard the vehicle, from a memory that is remote from the vehicle, etc.).
  • the vehicle computing system can input the camera image data into the machine-learned feature projection model and obtain an output that is indicative a multi-dimensional feature volume of the camera image data.
  • the machine-learned feature projection model can project the camera image data into a multi-dimensional feature volume (e.g., a sixteen-dimensional feature volume).
  • the vehicle computing system can utilize a second machine-learned model to estimate a ground height of the surrounding environment.
  • the second machine-learned model can be a machine-learned ground height estimation model that is trained to determine a ground height estimation of the surrounding environment based at least in part on a set of sensor data.
  • the vehicle computing system can access the machine-learned ground height estimation model (e.g., from a local memory, from a remote memory, etc.).
  • the vehicle computing system can input the LIDAR data into the machine-learned ground height estimation model and obtain an output that is indicative of the ground height estimation.
  • the machine-learned ground height estimation model can predict the ground height in an overhead view from the rasterized LIDAR data.
  • the vehicle computing system can project the multi-dimensional feature volume of the camera image data onto the ground height estimation (determined using the LIDAR data) to generate the overhead image data.
  • the vehicle computing system can detect one or more lane boundaries within the surrounding environment of the autonomous vehicle based at least in part on the overhead image data and a third machine-learned model.
  • the vehicle computing system can include, employ, and/or otherwise leverage a machine-learned lane boundary detection model (e.g., a convolutional neural network).
  • a machine-learned lane boundary detection model e.g., a convolutional neural network
  • the machine-learned lane boundary detection model can be or otherwise include an object segmentation neural network.
  • a pre-trained object segmentation neural network can be modified to add a plurality of parallel pooling operations (e.g., three pooling operations, etc.) at the end of a decoder (e.g., the 5-th decoder, etc.) with various kernel and stride sizes (e.g., of 2, 4, 8, etc.).
  • the results of these pooling operations can be concatenated with the output of the decoder (e.g., the 5 th decoder) and passed through the decoder modules.
  • This can allow the machine-learned lane boundary detection model to better extrapolate the one or more lane boundaries of further away regions.
  • One example existing object segmentation neural network is LinkNet (see A. Chaurasia and E. Culurciello. Linknet: Exploiting encoder representations for efficient semantic segmentation. arXiv preprint arXiv: 1707.03718, 2017).
  • the machine-learned lane boundary detection model can be trained to identify one or more lane boundaries within the surrounding environment of the autonomous vehicle based at least in part on a set of input data.
  • the input data can include, for example, the overhead image data.
  • the vehicle computing system can access data indicative of the machine-learned lane boundary detection model and input the overhead image data into the machine-learned lane boundary detection model.
  • the vehicle computing system can obtain, from the machine-learned lane boundary detection model, an output indicative of the one or more lane boundaries within the surrounding environment of the autonomous vehicle.
  • the output can be indicative of the position of the lane boundaries within a reference frame associated with the surrounding environment.
  • the vehicle computing system can provide additional input data into the machine-learned lane boundary detection model for the detection of the lane boundaries.
  • the vehicle computing system also input the LIDAR data into the machine-learned lane boundary detection model.
  • the re-projected image data can be concatenated with the rasterized LIDAR data and provided as input into the machine-learned lane boundary detection model (e.g., a modified object segmentation neural network).
  • the vehicle computing system can also input other types of sensor data into the machine-learned lane boundary detection model.
  • the vehicle computing system can provide motion sensor data (e.g., data from inertial measurement unit(s), wheel odometry readings, etc.) into the machine- learned lane boundary detection model.
  • motion sensor data e.g., data from inertial measurement unit(s), wheel odometry readings, etc.
  • Such sensor aggregation can allow the machine- learned lane boundary detection model to aggregate the lane detection outputs over time, resulting in more temporally coherent detections.
  • the vehicle computing system can be configured to detect one or more lane boundaries within the surrounding environment of the autonomous vehicle based on the LIDAR data as the sole input. For example, the vehicle computing system can obtain the rasterized LIDAR data associated with a surrounding environment of an autonomous vehicle. The vehicle computing system can access data indicative of the machine-learned lane boundary detection model (e.g., from a local and/or remote memory). The vehicle computing system can input the rasterized LIDAR data into the machine-learned lane boundary detection model and obtain an output that is indicative of one or more lane boundaries within the surrounding environment of the autonomous vehicle.
  • the machine-learned lane boundary detection model e.g., from a local and/or remote memory
  • the vehicle computing system can utilize the detected lane boundaries to perform one or more vehicle actions. For instance, the vehicle computing system can initiate the performance of one or more vehicle actions by the autonomous vehicle based at least in part on the one or more lane boundaries (e.g., via the transmission of control signal(s)). For example, the vehicle computing system can provide data indicative of the one or more lane boundaries (e.g., their associated position within the surrounding environment) for storage in a memory onboard and/or off-board of the vehicle. In another example, the vehicle computing system (e.g., a positioning system) can determine a location of the autonomous vehicle (e.g., within a travel lane on a highway) based at least in part on the one or more lane boundaries.
  • a positioning system can determine a location of the autonomous vehicle (e.g., within a travel lane on a highway) based at least in part on the one or more lane boundaries.
  • the vehicle computing system (e.g., a perception system) can perceive an object within the surrounding environment based at least in part on the one or more lane boundaries.
  • the lane boundaries can help the vehicle computing system determine that an object is more likely a vehicle than any other type of object because a vehicle is more likely (e.g., than a bicycle, pedestrian, etc.) to be within the travel lane on a highway.
  • the vehicle computing system e.g., a prediction system
  • the vehicle computing system can predict that another vehicle is more likely to travel in a manner such that the other vehicle will stay between the lane boundaries. Additionally, or alternatively, the vehicle computing system (e.g., a motion planning system) can plan a motion of the autonomous vehicle based at least in part on the one or more lane boundaries. For example, the vehicle computing system can generate a motion plan by which the autonomous vehicle is to travel between the lane boundaries (e.g., in a travel lane), queue for another object within the travel lane, pass an object outside of the travel lane, etc.
  • a motion planning system can plan a motion of the autonomous vehicle based at least in part on the one or more lane boundaries. For example, the vehicle computing system can generate a motion plan by which the autonomous vehicle is to travel between the lane boundaries (e.g., in a travel lane), queue for another object within the travel lane, pass an object outside of the travel lane, etc.
  • the present disclosure allows an autonomous vehicle to more accurately detect lane boundaries by using a series of improved models (e.g., neural network models, etc.) capable of leveraging combinations and subsets of sensor data, as well as temporal image sequences to produce centimeter level accurate lane predictions.
  • a series of improved models e.g., neural network models, etc.
  • Such an approach can allow for improved lane detection across a variety of scenarios including, for example, a highway scenario.
  • the systems and methods of the present disclosure provide a lane detection formulation that is robust to noise and encapsulates rich information at the pixel level.
  • the detection models of the present disclosure allow for accurate reasoning about lane boundary positions even when they are not immediately evident from the sensors. This also provides for more accurate detection at longer distance.
  • Such an approach can provide more reliable, flexible, and scalable solution than models with handcrafted rules, especially in less ideal scenarios like heavy occlusion.
  • the systems and methods of the present disclosure also provide an improvement to vehicle computing technology, such as autonomous vehicle computing technology.
  • vehicle computing technology such as autonomous vehicle computing technology.
  • the systems and methods allow the vehicle technology to leverage the sensor data acquired by the autonomous vehicle to more accurately detect lane boundaries within the vehicle’s surroundings.
  • a computing system e.g., a vehicle computing system
  • can obtain a first type of sensor data e.g., camera image data
  • a second type of sensor data e.g., LIDAR data
  • the computing system can generate an overhead image data indicative of at least a portion of the surrounding environment of the autonomous vehicle based at least in part on the first and second types of sensor data.
  • the computing system can access a machine- learned lane boundary detection model and input the overhead image data into the machine- learned lane boundary detection model.
  • the vehicle computing system can obtain an output (from the machine-learned lane boundary detection model) that is indicative of one or more lane boundaries within the surrounding environment of the autonomous vehicle.
  • the vehicle computing system can generate a more accurate representation of the surrounding environment (e.g., due to the inclusion of the ground height estimation).
  • the systems and methods of the present disclose can combine the strength of two distinct types of sensor data (e.g., camera’s dense observations as well as LiDAR’s unambiguous 3D measurements) to help provide a significant improvement (e.g., 18-45% increase in performance of lane detection) over other lane detection approaches.
  • the use of overhead image data with the machine-learned lane boundary detection model can lead to more reliable lane boundary detection as the machine- learned lane boundary detection model can be configured to more precisely reason about the overhead image data.
  • the machine-learned lane boundary detection model utilizes only the rasterized LIDAR data, the machine-learned lane boundary detection model can provide a more reliable detection of the lane boundaries within the vehicle’s surrounding environment.
  • the improvement of lane boundary detection results in an improvement in the vehicle autonomy operations (e.g., perception, prediction, motion planning, other autonomous navigation functions, etc.) as well as the safety of the vehicle and others.
  • the present disclosure is discussed with particular reference to autonomous vehicles and lane boundaries, the systems and methods described herein are applicable to the use of machine-learned models for other purposes.
  • the techniques described herein can be implemented and utilized by other computing systems such as, for example, user devices, robotic systems, non-autonomous vehicle systems, etc. (e.g., to detect other types of markings).
  • the present disclosure is discussed with particular reference to certain networks, the systems and methods described herein can also be used in conjunction with many different forms of machine-learned models in addition or alternatively to those described herein.
  • the reference to implementations of the present disclosure with respect to an autonomous vehicle is meant to be presented by way of example and is not meant to be limiting.
  • FIG. 1 illustrates an example system 100 according to example embodiments of the present disclosure.
  • the system 100 can include a vehicle computing system 105 associated with a vehicle 110.
  • the system 100 can include an operations computing system 115 that is remote from the vehicle 110.
  • the vehicle 110 can be associated with an entity (e.g., a service provider, owner, manager).
  • entity can be one that offers one or more vehicle service(s) to a plurality of users via a fleet of vehicles that includes, for example, the vehicle 110.
  • the entity can be associated with only vehicle 110 (e.g., a sole owner, manager).
  • the operations computing system 115 can be associated with the entity.
  • the vehicle 110 can be configured to provide one or more vehicle services to one or more users 120.
  • the vehicle service(s) can include transportation services (e.g., rideshare services in which user rides in the vehicle 110 to be transported), courier services, delivery services, and/or other types of services.
  • the vehicle service(s) can be offered to the users 120 by the entity, for example, via a software application (e.g., a mobile phone software application).
  • the entity can utilize the operations computing system 115 to coordinate and/or manage the vehicle 110 (and its associated fleet, if any) to provide the vehicle services to a user 120.
  • the operations computing system 115 can include one or more computing devices that are remote from the vehicle 110 (e.g., located off-board the vehicle 110).
  • computing device(s) can be components of a cloud-based server system and/or other type of computing system that can communicate with the vehicle computing system 105 of the vehicle 110 (and/or a user device).
  • the computing device(s) of the operations computing system 115 can include various components for performing various operations and functions.
  • the computing device(s) can include one or more processor(s) and one or more tangible, non-transitory, computer readable media (e.g., memory devices, etc.).
  • the one or more tangible, non-transitory, computer readable media can store instructions that when executed by the one or more processor(s) cause the operations computing system 115 (e.g., the one or more processors, etc.) to perform operations and functions, such as providing data to and/or obtaining data from the vehicle 110, for managing a fleet of vehicles (that includes the vehicle 110), etc.
  • the operations computing system 115 e.g., the one or more processors, etc.
  • the vehicle 110 incorporating the vehicle computing system 105 can be various types of vehicles.
  • the vehicle 110 can be a ground-based autonomous vehicle such as an autonomous truck, autonomous car, autonomous bus, etc.
  • the vehicle 110 can be an air-based autonomous vehicle (e.g., airplane, helicopter, or other aircraft) or other types of vehicles (e.g., watercraft, etc.).
  • the vehicle 110 can be an autonomous vehicle that can drive, navigate, operate, etc. with minimal and/or no interaction from a human operator (e.g., driver).
  • a human operator can be omitted from the vehicle 110 (and/or also omitted from remote control of the vehicle 110).
  • a human operator can be included in the vehicle 110.
  • the vehicle 110 can be a non-autonomous vehicle (e.g., ground-based, air-based, water-based, other vehicles, etc.).
  • the vehicle 110 can be configured to operate in a plurality of operating modes.
  • the vehicle 110 can be configured to operate in a fully autonomous (e.g., self-driving) operating mode in which the vehicle 110 is controllable without user input (e.g., can drive and navigate with no input from a human operator present in the vehicle 110 and/or remote from the vehicle 110).
  • the vehicle 110 can operate in a semi-autonomous operating mode in which the vehicle 110 can operate with some input from a human operator present in the vehicle 110 (and/or a human operator that is remote from the vehicle 110).
  • the vehicle 110 can enter into a manual operating mode in which the vehicle 110 is fully controllable by a human operator (e.g., human driver, pilot, etc.) and can be prohibited and/or disabled (e.g., temporary, permanently, etc.) from performing autonomous navigation (e.g., autonomous driving).
  • a human operator e.g., human driver, pilot, etc.
  • autonomous navigation e.g., autonomous driving
  • the vehicle 110 can implement vehicle operating assistance technology (e.g., collision mitigation system, power assist steering, etc.) while in the manual operating mode to help assist the human operator of the vehicle 110.
  • vehicle operating assistance technology e.g., collision mitigation system, power assist steering, etc.
  • the operating modes of the vehicle 110 can be stored in a memory onboard the vehicle 110.
  • the operating modes can be defined by an operating mode data structure (e.g., rule, list, table, etc.) that indicates one or more operating parameters for the vehicle 110, while in the particular operating mode.
  • an operating mode data structure can indicate that the vehicle 110 is to autonomously plan its motion when in the fully autonomous operating mode.
  • the vehicle computing system 105 can access the memory when implementing an operating mode.
  • the operating mode of the vehicle 110 can be adjusted in a variety of manners.
  • the operating mode of the vehicle 110 can be selected remotely, off-board the vehicle 110.
  • an entity associated with the vehicle 110 e.g., a service provider
  • the operations computing system 115 can send data to the vehicle 110 instructing the vehicle 110 to enter into, exit from, maintain, etc. an operating mode.
  • the operations computing system 115 can send data to the vehicle 110 instructing the vehicle 110 to enter into the fully autonomous operating mode.
  • the operating mode of the vehicle 110 can be set onboard and/or near the vehicle 110.
  • the vehicle computing system 105 can automatically determine when and where the vehicle 110 is to enter, change, maintain, etc.
  • the operating mode of the vehicle 110 can be manually selected via one or more interfaces located onboard the vehicle 110 (e.g., key switch, button, etc.) and/or associated with a computing device proximate to the vehicle 110 (e.g., a tablet operated by authorized personnel located near the vehicle 110).
  • the operating mode of the vehicle 110 can be adjusted by manipulating a series of interfaces in a particular order to cause the vehicle 110 to enter into a particular operating mode.
  • the vehicle computing system 105 can include one or more computing devices located onboard the vehicle 110.
  • the computing device(s) can be located on and/or within the vehicle 110.
  • the computing device(s) can include various components for performing various operations and functions.
  • the computing device(s) can include one or more processors and one or more tangible, non-transitory, computer readable media (e.g., memory devices, etc.).
  • the one or more tangible, non-transitory, computer readable media can store instructions that when executed by the one or more processors cause the vehicle 110 (e.g., its computing system, one or more processors, etc.) to perform operations and functions, such as those described herein for determining object motion, controlling vehicle motion, etc.
  • the vehicle 110 can include a communications system 125 configured to allow the vehicle computing system 105 (and its computing device(s)) to communicate with other computing devices.
  • the vehicle computing system 105 can use the communications system 125 to communicate with the operations computing system 115 and/or one or more other computing device(s) over one or more networks (e.g., via one or more wireless signal connections).
  • the communications system 125 can allow communication among one or more of the system(s) on-board the vehicle 110.
  • the communications system 125 can include any suitable components for interfacing with one or more network(s), including, for example, transmitters, receivers, ports, controllers, antennas, and/or other suitable components that can help facilitate communication.
  • the vehicle 110 can include one or more vehicle sensors 130, an autonomy computing system 135, one or more vehicle control systems 140, and other systems, as described herein.
  • One or more of these systems can be configured to
  • the communication channel can include one or more data buses (e.g., controller area network (CAN)), on-board diagnostics connector (e.g., OBD-II), and/or a combination of wired and/or wireless communication links.
  • the onboard systems can send and/or receive data, messages, signals, etc. amongst one another via the communication channel.
  • the vehicle sensor(s) 130 can be configured to acquire sensor data 145. This can include sensor data associated with the surrounding environment of the vehicle 110. For instance, the sensor data 145 can acquire image and/or other data within a field of view of one or more of the vehicle sensor(s) 130.
  • the vehicle sensor(s) 130 can include a Light Detection and Ranging (LIDAR) system, a Radio Detection and Ranging (RADAR) system, one or more cameras (e.g., visible spectrum cameras, infrared cameras, etc.), motion sensors, and/or other types of imaging capture devices and/or sensors.
  • LIDAR Light Detection and Ranging
  • RADAR Radio Detection and Ranging
  • the sensor data 145 can include image data, radar data, LIDAR data, and/or other data acquired by the vehicle sensor(s) 130.
  • the vehicle 110 can also include other sensors configured to acquire data associated with the vehicle 110.
  • the vehicle can include inertial measurement unit(s), wheel odometry devices, and/or other sensors that can acquire data indicative of a past, present, and/or future state of the vehicle 110.
  • the sensor data 145 can be indicative of one or more objects within the surrounding environment of the vehicle 110.
  • the object(s) can include, for example, vehicles, pedestrians, bicycles, and/or other objects.
  • the object(s) can be located in front of, to the rear of, to the side of the vehicle 110, etc.
  • the sensor data 145 can be indicative of locations associated with the object(s) within the surrounding environment of the vehicle 110 at one or more times.
  • the vehicle sensor(s) 130 can provide the sensor data 145 to the autonomy computing system 135.
  • the autonomy computing system 135 can retrieve or otherwise obtain map data 150.
  • the map data 150 can provide detailed information about the surrounding environment of the vehicle 110.
  • the map data 150 can provide information regarding: the identity and location of different roadways, road segments, buildings, or other items or objects (e.g., lampposts, crosswalks, curbing, etc.); the location and directions of traffic lanes (e.g., the location and direction of a parking lane, a turning lane, a bicycle lane, or other lanes within a particular roadway or other travel way and/or one or more boundary markings associated therewith); traffic control data (e.g., the location and instructions of signage, traffic lights, or other traffic control devices); the location of obstructions (e.g., roadwork, accidents, etc.); data indicative of events (e.g., scheduled concerts, parades, etc.); and/or any other map data that provides information that assists the vehicle 110 in comprehending and perceiving its surrounding environment and its relationship thereto.
  • traffic lanes e.g., the
  • the vehicle 110 can include a positioning system 155.
  • the positioning system 155 can determine a current position of the vehicle 110.
  • the positioning system 155 can be any device or circuitry for analyzing the position of the vehicle 110.
  • the positioning system 155 can determine position by using one or more of inertial sensors (e.g., inertial measurement unit(s), etc.), a satellite positioning system, based on IP address, by using triangulation and/or proximity to network access points or other network components (e.g., cellular towers, WiFi access points, etc.) and/or other suitable techniques.
  • the position of the vehicle 110 can be used by various systems of the vehicle computing system 105 and/or provided to a remote computing device (e.g., of the operations computing system 115).
  • the map data 150 can provide the vehicle 110 relative positions of the surrounding environment of the vehicle 104.
  • the vehicle 110 can identify its position within the surrounding environment (e.g., across six axes) based at least in part on the data described herein.
  • the vehicle 110 can process the vehicle sensor data 145 (e.g., LIDAR data, camera data) to match it to a map of the surrounding environment to get an
  • the autonomy computing system 135 can include a perception system 160, a prediction system 165, a motion planning system 170, and/or other systems that cooperate to perceive the surrounding environment of the vehicle 110 and determine a motion plan for controlling the motion of the vehicle 110 accordingly.
  • the autonomy computing system 135 can obtain the sensor data 145 from the vehicle sensor(s) 130, process the sensor data 145 (and/or other data) to perceive its surrounding environment, predict the motion of objects within the surrounding environment, and generate an appropriate motion plan through such surrounding environment.
  • the autonomy computing system 135 can control the one or more vehicle control systems 140 to operate the vehicle 110 according to the motion plan.
  • the vehicle computing system 105 can identify one or more objects that are proximate to the vehicle 110 based at least in part on the sensor data 130 and/or the map data 150.
  • the vehicle computing system 105 e.g., the perception system 160
  • the vehicle computing system 105 can process the sensor data 145, the map data 150, etc. to obtain perception data 175.
  • the vehicle computing system 105 can generate perception data 175 that is indicative of one or more states (e.g., current and/or past state(s)) of a plurality of objects that are within a surrounding environment of the vehicle 110.
  • the perception data 175 for each object can describe (e.g., for a given time, time period) an estimate of the object’s: current and/or past location (also referred to as position); current and/or past speed/velocity; current and/or past acceleration; current and/or past heading; current and/or past orientation; size/footprint (e.g., as represented by a bounding shape); class (e.g., pedestrian class vs. vehicle class vs. bicycle class), the uncertainties associated therewith, and/or other state information.
  • the perception system 160 can provide the perception data 175 to the prediction system 165 (and/or the motion planning system 170).
  • the prediction system 165 can be configured to predict a motion of the object(s) within the surrounding environment of the vehicle 110. For instance, the prediction system 165 can generate prediction data 180 associated with such object(s). The prediction data 180 can be indicative of one or more predicted future locations of each respective object. For example, the prediction system 180 can determine a predicted motion trajectory along which a respective object is predicted to travel over time. A predicted motion trajectory can be indicative of a path that the object is predicted to traverse and an associated timing with which the object is predicted to travel along the path. The predicted path can include and/or be made up of a plurality of way points. In some implementations, the prediction data 180 can be indicative of the speed and/or acceleration at which the respective object is predicted to travel along its associated predicted motion trajectory. The predictions system 165 can output the prediction data 180 (e.g., indicative of one or more of the predicted motion trajectories) to the motion planning system 170.
  • the predictions system 165 can output the prediction data 180 (e.g., indicative of one or more of the predicted motion tra
  • the vehicle computing system 105 can determine a motion plan 185 for the vehicle 110 based at least in part on the perception data 175, the prediction data 180, and/or other data.
  • a motion plan 185 can include vehicle actions (e.g., planned vehicle trajectories, speed(s), acceleration(s), other actions, etc.) with respect to one or more of the objects within the surrounding environment of the vehicle 110 as well as the objects’ predicted movements.
  • the motion planning system 170 can implement an optimization algorithm, model, etc.
  • the motion planning system 170 can determine that the vehicle 110 can perform a certain action (e.g., pass an object) without increasing the potential risk to the vehicle 110 and/or violating any traffic laws (e.g., speed limits, lane boundaries, signage, etc.). For instance, the motion planning system 170 can evaluate one or more of the predicted motion trajectories of one or more objects during its cost data analysis as it determines an optimized vehicle trajectory through the surrounding environment. The motion planning system 185 can generate cost data associated with such trajectories.
  • a certain action e.g., pass an object
  • traffic laws e.g., speed limits, lane boundaries, signage, etc.
  • the motion planning system 185 can generate cost data associated with such trajectories.
  • one or more of the predicted motion trajectories may not ultimately change the motion of the vehicle 110 (e.g., due to an overriding factor such as a jaywalking pedestrian).
  • the motion plan 185 may define the vehicle’s motion such that the vehicle 110 avoids the object(s), reduces speed to give more leeway to one or more of the object(s), proceeds cautiously, performs a stopping action, etc.
  • the motion planning system 170 can be configured to continuously update the vehicle’s motion plan 185 and a corresponding planned vehicle motion trajectory. For example, in some implementations, the motion planning system 170 can generate new motion plan(s) 185 for the vehicle 110 (e.g., multiple times per second). Each new motion plan can describe a motion of the vehicle 110 over the next planning period (e.g., next several seconds). Moreover, a new motion plan may include a new planned vehicle motion trajectory. Thus, in some implementations, the motion planning system 170 can continuously operate to revise or otherwise generate a short-term motion plan based on the currently available data. Once the optimization planner has identified the optimal motion plan (or some other iterative break occurs), the optimal motion plan (and the planned motion trajectory) can be selected and executed by the vehicle 110.
  • the optimization planner Once the optimization planner has identified the optimal motion plan (or some other iterative break occurs), the optimal motion plan (and the planned motion trajectory) can be selected and executed by the vehicle 110.
  • the vehicle computing system 105 can cause the vehicle 110 to initiate a motion control in accordance with at least a portion of the motion plan 185.
  • the motion plan 185 can be provided to the vehicle control system(s) 140 of the vehicle 110.
  • the vehicle control system(s) 140 can be associated with a vehicle controller (e.g., including a vehicle interface) that is configured to implement the motion plan 185.
  • the vehicle controller can, for example, translate the motion plan into instructions for the appropriate vehicle control component (e.g., acceleration control, brake control, steering control, etc.).
  • the vehicle controller can translate a determined motion plan 185 into instructions to adjust the steering of the vehicle 110“X” degrees, apply a certain magnitude of braking force, etc.
  • the vehicle controller (e.g., the vehicle interface) can help facilitate the responsible vehicle control (e.g., braking control system, steering control system, acceleration control system, etc.) to execute the instructions and implement the motion plan 185 (e.g., by sending control signal(s), making the translated plan available, etc.). This can allow the vehicle 110 to autonomously travel within the vehicle’s surrounding environment.
  • the responsible vehicle control e.g., braking control system, steering control system, acceleration control system, etc.
  • the motion plan 185 e.g., by sending control signal(s), making the translated plan available, etc.
  • FIG. 2 depicts an example environment 200 of the vehicle 110 according to example embodiments of the present disclosure.
  • the surrounding environment 200 of the vehicle 110 can be, for example, a highway environment, an urban environment, a residential environment, a rural environment, and/or other types of environments.
  • the surrounding environment 200 can include one or more objects such as an object 202 (e.g., another vehicle, etc.).
  • the surrounding environment 200 can include one or more lane boundaries 204A-C.
  • the lane boundaries 204A-C can include, for example, lane markings and/or other indicia associated with a travel lane and/or travel way (e.g., the boundaries thereof).
  • the one or more lane boundaries 204 A-C can be located within a highway on which the vehicle 110 is located.
  • the vehicle 110 can be configured to reliably detect the one or more lane boundaries 204A-C using the systems and methods of the present disclosure.
  • FIG. 3 depicts a diagram of an example computing system 300 that is configured to detect the one or more lane boundaries 204A-C of the vehicle’s surrounding environment.
  • the computing system 300 can be located onboard the vehicle 110 (e.g., as at least a portion of the vehicle computing system 105) and/or can be located off-board the vehicle 110 (e.g., remote from the vehicle computing system 105).
  • the computing system 300 can include one or more computing devices.
  • the computing devices can implement the model architecture 302 to detect lane boundaries.
  • the computing system 300 can include one or more processors and one or more tangible, non-transitory, computer readable media that collectively store instructions that when executed by the one or more processors cause the computing system to perform the operations described herein for detecting lane boundaries 204A-C within the surrounding environment 200 of the vehicle 110 (e.g., the processing pipeline operations of the model architecture 302).
  • the computing system 300 can be configured to utilize various types of sensor data to detect one or more lane boundaries. For instance, the computing system 300 can obtain a first type of sensor data 304 A associated with the surrounding environment 200 of the vehicle 110 and a second type of sensor data 304B associated with the surrounding environment 200 of the vehicle 110. In the event the computing system 300 is located remotely from vehicle(s), the computing system 300 can obtain the sensor data via one or more wireless networks.
  • the first type of sensor data 304 A can include, for example, camera image data associated with the surrounding environment 200 of the vehicle 110.
  • the camera image data can be, for example, monocular, 3 -channel RGB image data (e.g., with 960 x 300 resolution, etc.).
  • the camera image data can depict at least a portion of the surrounding environment 200 of the vehicle 110 (e.g., a portion that is expected to include lane boundaries).
  • the camera image data can include, for example, the bottom half of a camera image where one or more lane boundaries 204A-C are expected to be present.
  • Example camera image data 400A is shown in FIG. 4 A.
  • the second type of sensor data 304B can include LIDAR data associated with the surrounding environment 200 of the vehicle 110.
  • the LIDAR data can be indicative of a LIDAR point cloud associated with the surrounding environment 200 of the vehicle 110 (e.g., created by LIDAR sweep(s) of the vehicle’s LIDAR system) that has been rasterized.
  • the computing system 300 can obtain the rasterized LIDAR data associated with a surrounding environment 200 of the vehicle 110 (e.g., including LIDAR image data that is rasterized from a LIDAR point cloud).
  • the rasterized LIDAR data can be indicative of a two-dimensional overhead view depicting at least a portion of the surrounding environment 200.
  • the computing system 300 can rasterize the LIDAR point cloud into a two-dimensional overhead view image (e.g., with a resolution of 960 x 960 at a 5cm per pixel resolution).
  • the overhead view image can depict at least a portion of the surrounding environment 200 (e.g., a 48m by 48m area with the vehicle 110 at the center bottom of the image).
  • the rasterized LIDAR data e.g., the overhead view image
  • Each of the one or more spatial locations can include a plurality of channels.
  • each of the one or more spatial locations can include a first channel representing a height of at least one pixel (e.g., the highest pixel in a discretization bin), a second channel representing a brightness level of the least one pixel (e.g., in the discretization bin), and a third channel representing a total number of points (e.g., within the discretization bin).
  • Example LIDAR data 400B is shown in FIG. 4B.
  • the computing system 300 can obtain a third type of sensor data 304C associated with the vehicle 110.
  • the computing system 300 can obtain RADAR data, motion sensor data (e.g., data from inertial measurement unit(s), wheel odometry readings, etc.), and/or other types of sensor data.
  • the third type of sensor data 304C can be acquired, for example, via one or more of the sensors 130 onboard the vehicle 110.
  • the computing system 300 can pre-process the first type of sensor data 304 A and second type of sensor data 304B to improve the accuracy of its lane detection analysis.
  • the use of both of these types of sensor data can be shown in path 1 and path 2 through the model architecture 302.
  • the computing system 300 can generate overhead image data indicative of at least a portion of the surrounding environment 200 of the vehicle 110 based at least in part on the first type of sensor data 304A (e.g., the camera image data) and the second type of sensor data 304B (e.g., the LIDAR data).
  • the computing system 300 can include, employ, and/or otherwise leverage one or more machine-learned model(s).
  • the machine-learned models can be or can otherwise include one or more various model(s) such as, for example, neural networks (e.g., deep neural networks), and/or other multi-layer non linear models.
  • neural networks e.g., deep neural networks
  • Neural networks can include feed-forward neural networks (e.g.,
  • convolutional neural networks e.g., convolutional neural networks
  • recurrent neural networks e.g., long short-term memory recurrent neural networks
  • supervised training techniques can be performed to train the models to transform, process, and/or utilize sensor data, as described herein.
  • the computing system 300 can utilize a first machine-learned model 306 A to transform the first type of sensor data 304 A (e.g., the camera image data).
  • the first machine-learned model 306 A can be a machine-learned feature projection model (e.g., a convolutional neural network) that is trained to generate a multi dimensional feature volume based at least in part on a set of sensor data.
  • the computing system 300 can access data indicative of the first machine-learned model 306A (e.g., the machine-learned feature projection model) from a local memory onboard the vehicle 110, from a memory that is remote from the vehicle 110, etc.
  • the computing system 300 can input the first type of sensor data 304 A (e.g., the camera image data) associated with the surrounding environment 200 of the vehicle 110 into the first machine-learned model 306A (e.g., the machine-learned feature projection model).
  • the computing system 300 can obtain an output 308.
  • the output 308 can be indicative of a multi-dimensional feature volume of the first type of sensor data 304A (e.g., the camera image data).
  • the machine-learned feature projection model can project the camera image data into a multi dimensional feature volume (e.g., a sixteen-dimensional feature volume).
  • the computing system 300 can determine a ground height estimation associated with the surrounding environment 200 of the vehicle 110 based at least in part on the second type of sensor data 304B (e.g. the LIDAR data).
  • a camera sensor can be calibrated and fixed to the vehicle.
  • the computing system 300 can utilize a second machine-learned model 306B to determine a ground height estimation of the surrounding environment 200 of the vehicle 110.
  • the second machine-learned model 306B can be a machine-learned ground height estimation model that is trained to determine a ground height estimation of a surrounding environment based at least in part on a set of sensor data (e.g., rasterized LIDAR image data).
  • the ground height estimation can be indicative of, for example, a dense ground elevation.
  • the second machine-learned model 306B (e.g., the machine-learned ground height estimation model) can include a multi-scale encoder decoder-style network.
  • the second machine-learned model 306B (e.g., the machine-learned ground height estimation model) can include five sequential encoder stages and five sequential decoder stages containing one residual block for each stage.
  • the second machine-learned model 306B (e.g., the machine-learned ground height estimation model) can include additive skip connection links between each pair of encoder/decoder stages.
  • the feature dimensions of the second machine-learned model 306B (e.g., the machine-learned ground height estimation model) can be reduced (e.g., to one half, etc.) at every layer of the network.
  • the second machine-learned model 306B (e.g., the machine-learned ground height estimation model) can include a fast convolutional neural network (CNN) trained to determine the ground height estimation from the sensor data.
  • CNN fast convolutional neural network
  • a pre-existing fast convolutional neural net (see, e.g., K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR, 2016) can be modified for this purpose.
  • feature dimensions can be reduced (e.g., by a factor of 2 for scales 1 and 2, and a factor of 4 for scales 3 to 5, etc.) and convolutional block(s) can be removed (e.g., one convolutional block from each of scale 2, 3, and 5, etc.).
  • convolutional block(s) can be removed (e.g., one convolutional block from each of scale 2, 3, and 5, etc.).
  • This can be followed by large receptive field average pooling.
  • three additional feature volumes can be produced using pooling (e.g., with receptive field and stride sizes of 10, 25, and 60, etc.) before concatenating feature volume(s) with an original output (e.g., of scale 5). This can give the network the capability to propagate information about the ground height from regions with high data density (e.g., LiDAR point density) to regions with low density.
  • data density e.g., LiDAR point density
  • a feature volume can undergo additional bottleneck residual blocks (e.g., 3 additional bottleneck residual blocks, etc.), before being upsampled using interleaved transposed convolutions and basic residual blocks.
  • the result can include, for example, a dense ground height image in bird’s eye view (e.g., of size 960x960 corresponding to a region of 48 x 48m at 5cm per pixel).
  • the computing system 300 can access data indicative of the second machine- learned model 306B (e.g., the machine-learned ground height estimation model) from a memory onboard the vehicle 110, from a memory that is remote from the vehicle 110, etc.
  • the second machine- learned model 306B e.g., the machine-learned ground height estimation model
  • the computing system 300 can input the second type of sensor data 304B (e.g., the LIDAR data) associated with the surrounding environment of the vehicle 110 into the machine- learned ground height estimation model.
  • the computing system 300 can obtain an output 310 from the machine-learned ground height estimation model. Such an output 310 can be indicative of a ground height estimation of the surrounding environment 200.
  • the computing system 300 can generate overhead image data 312 indicative of at least the portion of the surrounding environment 200 of the vehicle 110 based at least in part on the ground height estimation and the first type of sensor data 304A (e.g., the camera image data). For instance, computing system 300 can generate the overhead image data 312 based at least in part on the multi-dimensional feature volume associated with the first type of sensor data 304A (e.g., the camera image data) and the ground height estimation (e.g., indicated in output 310). For increased processing speed, the first type of sensor data 304A (e.g., the camera image data) can be downsampled (e.g., from 1920 x 1200 to 1024 x 640).
  • a neural network can be applied to extract features directly from the first type of sensor data 304A (e.g., the camera image data) and warp those features into the top-down view. If such a network is denoted F c , the re-projected features can be / // , f(If(I o ),I'z(Ii) ().
  • the computing system 300 can project the multi-dimensional feature volume of the camera image data (e.g., the output 308 of the first machine-learned model 306 A) onto the ground height estimation (determined using the LIDAR data) to generate the overhead image data 312. This can be concatenated with the multi-channel LIDAR image data.
  • the computing system 300 can detect one or more lane boundaries within the surrounding environment of the vehicle 110 based at least in part on the overhead image data 312 indicative of at least the portion of the surrounding environment 200 of the vehicle 110 and a third machine-learned model 306C.
  • the third machine-learned model 306C can be a machine-learned lane boundary detection model.
  • the third machine-learned model 306C (e.g., the machine-learned lane boundary detection model) can include a convolutional neural network.
  • the third machine-learned model 306C (e.g., the machine-learned lane boundary detection model) can be or otherwise include an object segmentation neural network.
  • a pre-trained object segmentation neural network can be modified to add a plurality of parallel pooling operations (e.g., three pooling operations, etc.) at the end of a decoder (e.g., the 5-th decoder, etc.) with various kernel and stride sizes (e.g., of 2, 4, 8, etc.).
  • the results of these pooling operations can be concatenated with the output of the decoder (e.g., the 5 th decoder) and passed through the decoder modules.
  • This can allow the third machine-learned model 306C (e.g., the machine-learned lane boundary detection model) to better extrapolate the one or more lane boundaries of further away regions.
  • this convolutional neural network can be similar to that used in the ground height estimation (e.g., with the exception that the feature dimension of scales 3 to 5 are reduced by a factor of 2 relative to the preconfigured model, with no blocks removed).
  • the scales 1 to 3 at the input can be duplicated without weight sharing such that the separate types of sensor data are passed into separate inputs.
  • the convolutional neural network can be utilized to leverage both the first sensor data (e.g., the re-projected camera image) and the second sensor data (e.g., the LiDAR data) to produce pixel-wise lane detection results.
  • the two streams of data can be concatenated (e.g., at scale 4) and the output of this network can be distance transform estimates.
  • the third machine-learned model 306C (e.g., the machine-learned lane boundary detection model) can be trained to identify one or more lane boundaries within the
  • the computing system 300 can access data indicative of third machine-learned model 306C (e.g., the machine-learned lane boundary detection model) from a memory that is onboard the vehicle 110 and/or off-board the vehicle 110.
  • the computing system 300 can input the overhead image data 312 indicative of at least the portion of the surrounding environment 200 of the vehicle 110 into the third machine-learned model 306C (e.g., the machine-learned lane boundary detection model).
  • the computing system 300 can obtain an output 314 from the third machine-learned model 306C (e.g., the machine-learned lane boundary detection model).
  • the output 314 can be indicative of the one or more lane boundaries 204A-C within the surrounding environment 200 of the vehicle 110.
  • the output can be indicative of the position of the lane boundaries 204A-C within a reference frame associated with the surrounding environment 200.
  • a visual representation 500 of an example output from the third machine-learned model 306C (e.g., the machine-learned lane boundary detection model) is depicted in FIG. 5.
  • the output 314 of the third machine-learned model 306C (e.g., the machine- learned lane boundary detection model) can be phrased as the minimum Euclidean distance to the nearest lane boundary at each pixel.
  • This can allow the third machine-learned model 306C (e.g., the machine-learned lane boundary detection model) to reason about the relationship of each pixel and the one or more lane boundaries, providing an output that encapsulates richer information than class labels.
  • the loss can be more forgiving to small offsets between the prediction and ground truth, as well as to noise in the ground truth.
  • the computing system 300 can provide additional input data into the third machine-learned model 306C (e.g., the machine-learned lane boundary detection model) for the detection of the lane boundaries 204A-C.
  • the computing system 300 also input the second type of sensor data 304B (e.g., LIDAR data) associated with the surrounding environment 200 of the vehicle 110 into the third machine-learned model 306C (e.g., the machine-learned lane boundary detection model). This is shown in path 3 of FIG. 3.
  • the re-projected image data can be concatenated with the rasterized LIDAR data and provided as input into the machine-learned lane boundary detection model (e.g., a modified object segmentation neural network).
  • the computing system 300 can input other types of sensor data into the third machine-learned model 306C (e.g., the machine-learned lane boundary detection model) for the detection of the lane boundaries 204 A-C.
  • the computing system 300 can input a third type of sensor data 304C into the third machine- learned model 306C (e.g., the machine-learned lane boundary detection model).
  • the third type of sensor data 304C can include, for example, motion sensor data from inertial measurement unit(s), wheel odometry readings, etc.
  • Such data can be provided (e.g., input into, sent to, transferred to, transmitted to, made available to, etc.) to the third machine-learned model 306C (e.g., the machine-learned lane boundary detection model).
  • This can allow the machine-learned lane boundary detection model to aggregate the lane detection outputs over time, resulting in more temporally coherent detections.
  • the aggregated output A u for each pixel i at time t can be calculated as mean of each observations for a give location.
  • the transformation to the previous frame by the E t matrix can be calculated and ortho projected (e.g., to remove the z coordinate). This can result in a 2D-2D assignment which can be used as warping denoted, for example, as ⁇ />(A t -u, E t ).
  • the computing system 300 can be configured to detect one or more lane boundaries within the surrounding environment of the vehicle 110 based on the second type of sensor data 304B as the sole input (e.g., only implementing path 3). For example, the computing system 300 can obtain rasterized LIDAR data associated with a surrounding environment of a vehicle 110. As described herein, the rasterized LIDAR data can include LIDAR image data that is rasterized from a LIDAR point cloud. The computing system 300 can access data indicative of the third machine-learned model 306C (e.g., the machine-learned lane boundary detection model).
  • the third machine-learned model 306C e.g., the machine-learned lane boundary detection model
  • the computing system 300 can input the rasterized LIDAR data associated with the surrounding environment 200 of the vehicle 110 into the third machine-learned model 306C (e.g., the machine-learned lane boundary detection model).
  • the computing system 300 can obtain an output 314 from the third machine-learned model 306C (e.g., the machine-learned lane boundary detection model). As described herein, such an output 314 can be indicative of one or more lane boundaries 204 A- C within the surrounding environment 200 of the vehicle 110 (e.g., as shown in FIG. 5).
  • the computing system 300 can be configured to detect the one or more lane boundaries 204A-C within the surrounding environment 200 of the vehicle 110 based on the first type of sensor data 304A (e.g., two-dimensional camera data) as the sole input (e.g., only implementing path 1). To do so, the computing system 300 can utilize a modified convolutional neural network. For example, an object classification network into a pixel-wise prediction network by removing the fully connected layers. The network’s effective receptive field can be expanded (e.g., by adding 15 x and 60 x pooling at the output of scale5). These can then be bilinearly upsampled and concatenated with an original scale5 output.
  • a modified convolutional neural network For example, an object classification network into a pixel-wise prediction network by removing the fully connected layers. The network’s effective receptive field can be expanded (e.g., by adding 15 x and 60 x pooling at the output of scale5). These can then be bilinearly up
  • a feature volume can be upsampled to match the input dimensions by interleaving a set of 2 x learned upsampling filters and corresponding residual blocks. From this model, the computing system 300 can obtain an output indicative of the one or more lane boundaries 204A-C within the surrounding environment 200.
  • the vehicle 110 can utilize the detected lane boundaries 204A-C to perform one or more vehicle actions.
  • the vehicle computing system 105 can obtain data indicative of the one or more lane boundaries from the computing system 300, which can be a portion of the vehicle computing system 105 and/or remote therefrom.
  • the vehicle computing system 105 can initiate the performance of one or more vehicle actions by the vehicle 110 based at least in part on the one or more lane boundaries 204 A-C (e.g., via the transmission of one or more signal(s)).
  • the vehicle computing system 105 (e.g., the computing system 300 implemented therein) can provide the output indicative of the one or more lane boundaries 204A-C (e.g., their associated position within the surrounding environment 200) for storage in a memory onboard the vehicle 110 and/or off-board of the vehicle 110.
  • the vehicle computing system 105 e.g., a positioning system 155
  • the vehicle computing system 105 can determine a location of the vehicle 110 (e.g., within a travel lane on a highway) based at least in part on the one or more lane boundaries 204B-C.
  • the vehicle computing system 105 can perceive an object 202 within the surrounding environment 200 based at least in part on the one or more lane boundaries 204 A-C.
  • the lane boundaries 204 A-C can help the vehicle computing system 105 determine that the object 202 is more likely another vehicle because a vehicle is more likely to be located within the particular surrounding environment 200 (e.g., within the travel lane on a highway) than any other type of object (e.g., pedestrian, bicycle, etc.).
  • the vehicle computing system 105 (e.g., a prediction system 180) can predict a motion trajectory of an object 202 within the surrounding environment 200 of the vehicle 110 based at least in part on the one or more lane boundaries 204A-C.
  • the vehicle computing system 105 can predict that the object 202 (e.g., another vehicle) is more likely to travel in a manner such that the object 202 will stay between the lane boundaries 204 A-B.
  • the vehicle computing system 105 e.g., a motion planning system 185) can plan a motion of the vehicle 110 based at least in part on the one or more lane boundaries 204A-C.
  • the vehicle computing system 105 can generate a motion plan 185 by which the vehicle 110 is to travel between the lane boundaries 204A-B (e.g., in a travel lane), queue for another object within the travel lane, pass an object 202 outside of the travel lane, etc.
  • the systems and methods described herein are applicable to the use of machine-learned models for other purposes.
  • the techniques described herein can be implemented and utilized by other computing systems such as, for example, user devices, robotic systems, non-autonomous vehicle systems, etc. (e.g., to detect markings).
  • the present disclosure is discussed with particular reference to certain networks, the systems and methods described herein can also be used in conjunction with many different forms of machine-learned models in addition or alternatively to those described herein.
  • FIG. 6 depicts a flow diagram of an example method 600 of detecting lane boundaries or other markings according to example embodiments of the present disclosure.
  • One or more portion(s) of the method 600 can be implemented by a computing system that includes one or more computing devices such as, for example, the computing systems described with reference to FIGS. 1, 3, and/or 8 and/or other computing systems. Each respective portion of the method 600 can be performed by any (or any combination) of one or more computing devices.
  • one or more portion(s) of the method 600 can be implemented as an algorithm on the hardware components of the device(s) described herein (e.g., as in FIGS.
  • FIG. 6 depicts elements performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that the elements of any of the methods discussed herein can be adapted, rearranged, expanded, omitted, combined, and/or modified in various ways without deviating from the scope of the present disclosure.
  • the method 600 can include obtaining a first type of sensor data associated with a surrounding environment of a vehicle.
  • the computing system 300 can obtain a first type of sensor data 304 A associated with a surrounding environment 200 of a vehicle 110 (e.g., an autonomous vehicle).
  • the first type of sensor data 304A can be acquired via one or more sensors 130 onboard the vehicle 110 (e.g., onboard camera(s), etc.).
  • the first type of sensor data 304 A can include camera image data, as described herein.
  • the computing system 300 can obtain camera image data associated with the surrounding environment 200 of the vehicle 110 (e.g., via the sensor(s) 130, from a memory, etc.).
  • the first type of sensor data can be associated with a surrounding environment of a computing system other than that associated with a vehicle (e.g., the surrounding environment of a user device).
  • the method 600 can include determining a multi-dimensional feature volume for the first type of sensor data.
  • the computing system 300 can determine a multi-dimensional feature volume of the first type of sensor data 304 A based at least in part on a first machine-learned model 306 A and the first type of sensor data 304 A.
  • the first machine-learned model 306 A can be a machine-learned feature projection model that includes a convolutional neural network.
  • the machine-learned feature projection model can be configured to generate a multi-dimensional feature volume based at least in part on the first type of sensor data 304B (e.g., the camera image data).
  • the computing system 300 can access a machine-learned feature projection model.
  • the computing system 300 can input the camera image data associated with the surrounding environment 200 (e.g., of the vehicle 110) into the machine-learned feature projection model.
  • the machine-learned feature projection model can be trained to process the camera image data to generate a multi-dimensional feature volume of the camera image data (e.g., a sixteen dimension feature volume).
  • the computing system 300 can obtain an output 308 that is indicative of a multi-dimensional feature volume of the camera image data from the machine- learned feature projection model.
  • the method 600 can include obtaining a second type of sensor data associated with the surrounding environment of the vehicle.
  • the computing system 300 can obtain a second type of sensor data 304 A associated with the surrounding environment 200 of the vehicle 110.
  • the second type of sensor data 304B can be acquired via one or more sensors 130 onboard the vehicle 110 (e.g., LIDAR system, etc.).
  • the second type of sensor data 304B can include rasterized LIDAR data, as described herein.
  • the computing system 300 can obtain LIDAR data associated with the surrounding environment 200 of the vehicle 110 (e.g., via the sensor(s) 130, from a memory, etc.).
  • the second type of sensor data 304B (e.g., the LIDAR point clouds) can be brought from a plurality of consecutive frames (e.g., five, etc.) to a common reference frame by correcting for vehicle motion.
  • the second type of sensor data 304B (e.g., LIDAR point clouds) can be rasterized to a 3-channel image in Bird’s Eye View (BEV) encoding both the intensity and height of the highest point as well as the height of the lowest point in each discretization bin.
  • the second type of sensor data can be associated with a surrounding environment of a computing system other than that associated with a vehicle (e.g., the surrounding environment of a user device).
  • the method 600 can include determining a ground height estimation associated with the surrounding environment.
  • the computing system 300 can determine a ground height estimation associated with the surrounding environment 200 (e.g., of the vehicle 110) based at least in part on a second machine-learned model 306B and the second type of sensor data 304B.
  • the second machine-learned model 306B can be machine- learned ground height estimation model that includes a convolutional neural network.
  • the machine-learned ground height estimation model can be configured to estimate the ground height of at least a portion of the surrounding environment 300 (e.g., the portion depicted in the sensor data) based at least in part on the second type of sensor data 304B (e.g., the LIDAR data).
  • the computing system 300 can access the machine-learned ground height estimation model.
  • the computing system 300 can input LIDAR data associated with the surrounding environment 200 (e.g., of the vehicle 110) into the machine- learned ground height estimation model.
  • the machine-learned ground height estimation model can process the LIDAR data to determine the ground height estimation.
  • the computing system 300 can obtain an output 310 that is indicative the ground height estimation from the machine-learned ground height estimation model.
  • the method 600 can include generating overhead image data indicative of at least a portion of the surrounding environment.
  • the computing system 300 can generate overhead image data 312 indicative of at least a portion of the surrounding environment 200 (e.g., of the vehicle 110) based at least in part on the multi-dimensional feature volume and the ground height estimation.
  • the computing system 300 can project the multi-dimensional feature volume of the camera image data onto the ground height estimation to generate the overhead image data 312. In some implementations, this can be concatenated with multi-channel LIDAR image data and fed into the machine-learned lane boundary detection model, as described further herein.
  • the camera image can be projected to the ground using a differentiable warping function.
  • the method 600 can include obtaining a third type of sensor data.
  • the computing system 300 can obtain a third type of sensor data 304C associated with the vehicle 110 (and/or other computing system).
  • the third type of sensor data 304C can include, for example, data associated with one or more motion sensors (e.g., vehicle IMUs) and/or other data indicative of a state of the vehicle 110 (e.g., wheel odometry readings) and/or computing system.
  • the method 600 can include determining one or more lane boundaries and/or other markings within the surrounding environment.
  • the computing system 300 can determine one or more lane boundaries 204A-C and/or other markings within the surrounding environment 200 of the vehicle 110 (and/or other computing system) based at least in part on the overhead image data 312 and a third machine-learned model 306C.
  • the third machine-learned model 306C can be a machine- learned lane boundary detection model that includes a convolutional neural network.
  • the machine-learned lane boundary detection model can be configured to determine the presence and/or location of one or more lane boundaries 204A-C of the surrounding environment 300 based at least in part on overhead image data 312 and/or other data.
  • the computing system 300 can access the machine-learned lane boundary detection model.
  • the computing system 300 can input the overhead image data 312 into the machine-learned lane boundary detection model.
  • the computing system 300 can utilize the second type of sensor data 304A (e.g., the LIDAR data) as input data for the machine-learned lane boundary detection model.
  • the computing system 300 can utilize the third type of sensor data 304C (e.g., IMU data, wheel odemetry data, other vehicle data, etc.) as input data for the machine-learned lane boundary detection model.
  • the machine-learned lane boundary detection model can process the input data to determine the presence and location of the lane boundaries 204A-C within the surrounding environment 200 based at least in part on the overhead image data 312 and/or other data (e.g.,. LIDAR data, the third type of sensor data 304C, etc.).
  • the computing system 300 can obtain an output 314 from the machine-learned lane boundary detection model.
  • the output 314 can be indicative of the location of one or more of the lane boundaries 204A-C within the surrounding environment 200 of the vehicle 110.
  • the machine-learned lane boundary detection model or a similar model can be trained to detect other types of markings within the surrounding environment (e.g., of another type of computing system).
  • the method 600 can include initiating one or more vehicle actions.
  • the vehicle computing system 105 (which can include computing system 300 and/or communicate therewith) can initiate a performance of one or more vehicle actions by the vehicle 110 based at least in part on the one or more lane boundaries 204A-C.
  • the vehicle action(s) can include planning a motion of the vehicle 110 based at least in part on the one or more lane boundaries 204A-C.
  • the vehicle action(s) can include perceiving an object 202 within the surrounding environment 200 of the vehicle 110 based at least in part on the one or more lane boundaries 204A-C.
  • the vehicle action(s) can include predicting a motion trajectory of an object 202 within the surrounding environment 200 of the vehicle 110 based at least in part on the one or more lane boundaries 204A-C.
  • the vehicle computing system 105 can determine a location of the vehicle 110 (e.g., within the surrounding environment 200) based at least in part on the one or more lane boundaries 204A-C.
  • the computing system 300 can provide (e.g., for storage in a memory onboard the vehicle 110, etc.) the output 314 indicative of the one or more lane boundaries 204A-C within the surrounding environment 200 of the vehicle 110.
  • the method can include initiating actions associated with the computing system (e.g., localizing the user device based on detected markings, etc.).
  • FIG. 7 depicts a flow diagram of another example method 700 of detecting lane boundaries and/or other types of markings according to example embodiments of the present disclosure according to example embodiments of the present disclosure.
  • One or more portion(s) of the method 700 can be implemented by a computing system that includes one or more computing devices such as, for example, the computing systems described with reference to FIGS. 1, 3, and/or 8 and/or other computing systems. Each respective portion of the method 700 can be performed by any (or any combination) of one or more computing devices.
  • one or more portion(s) of the method 700 can be implemented as an algorithm on the hardware components of the device(s) described herein (e.g., as in FIGS. 1, 3, and 8), and/or other types of computing systems (e.g., user devices, robotic systems, other systems/devices, etc.) for example, to detect lane boundaries and/or other markings.
  • the method 700 can include obtaining LIDAR data associated with a surrounding environment of a vehicle.
  • the computing system 300 can obtain LIDAR data associated with a surrounding environment 200 of a vehicle 110.
  • the LIDAR data can be obtained via a LIDAR system that is mounted to a vehicle.
  • the LIDAR data can be indicative of a LIDAR point cloud associated with the surrounding environment 200 of the vehicle 110.
  • the LIDAR data can be associated with the surrounding environment of another type of system such as, for example, a computing system of a robot, user device, etc.
  • the method 700 can include generating rasterized LIDAR data.
  • the computing system 300 can generate rasterized LIDAR data based at least in part on the LIDAR data associated with the surrounding environment 200 (e.g., of the vehicle 110).
  • the computing system 300 can rasterize the LIDAR point cloud associated with the surrounding environment 200 (e.g., of the vehicle 110) into a two-dimensional overhead view image depicting at least the portion of the surrounding environment 200.
  • the computing system 300 can determine one or more lane boundaries 204 A-C and/or other markings within the surrounding environment 200 (e.g., of the vehicle 110, other computing system) based at least in part on the rasterized LIDAR data and a machine-learned model (e.g., machine-learned lane boundary detection model). For instance, at (706), the method 700 can include accessing a machine-learned lane boundary detection model. For instance, the computing system 300 can access data indicative of a machine-learned lane boundary detection model (e.g., from a local memory, remote memory, etc.). As described herein, the machine-learned lane boundary detection model can include a convolutional neural network.
  • the machine-learned lane boundary detection model can be trained to determine the presence and/or location of one or more lane boundaries 204A-C of the environment 300 based at least in part on the rasterized LIDAR data.
  • the machine-learned lane boundary detection model or a similar model can be trained to detect other types of markings within the surrounding environment (e.g., of another type of computing system).
  • the computing system 300 can input the rasterized LIDAR data associated with the surrounding environment 200 (e.g., of the vehicle 110, other computing system) into the machine-learned model (e.g., machine-learned lane boundary detection model).
  • the machine-learned model e.g., machine-learned lane boundary detection model
  • the method 700 can include inputting overhead image data into the machine-learned model (e.g., machine-learned lane boundary detection model).
  • the computing system 300 can generate overhead image data 312 indicative of at least a portion of the surrounding environment 200 (e.g., of the vehicle 110, other computing system) based at least in part on the rasterized LIDAR data and camera image data.
  • the computing system 300 can input the overhead image data 312 indicative of at least a portion of the surrounding environment 200 (e.g., of the vehicle 110, other computing system) into the machine-learned model (e.g., lane boundary detection model), as additional input data.
  • other data e.g., IMU data, etc.
  • the method 700 can include obtaining an output from the machine- learned mode (e.g., lane boundary detection model).
  • the computing system 300 can obtain an output from the machine-learned model (e.g., lane boundary detection model).
  • the output can be based at least in part on the rasterized LIDAR data and/or the overhead image data.
  • the output can be indicative of the one or more lane boundaries 204A-C within the surrounding environment 200 (e.g., of the vehicle 110, other computing system).
  • the method 700 can include initiating one or more vehicle actions.
  • the vehicle computing system 105 which can include and/or otherwise
  • the vehicle action(s) can include a variety of actions such as, for example, determining a location of the vehicle 110 within a travel lane based at least in part on the one or more detected lane boundaries 204A-C, planning a motion of the vehicle 110 based at least in part on the one or more detected lane boundaries 204 A-C, perceiving an object 202 within the surrounding environment 200 of the vehicle 110 based at least in part on the one or more detected lane boundaries 204 A-C, predicting a motion trajectory of an object 202 within the surrounding environment 200 of the vehicle 110 based at least in part on the one or more detected lane boundaries 204A-C, etc.
  • the method can include initiating actions associated with the computing system (e.g., localizing the user device based on detected markings, etc.).
  • FIG. 8 depicts example system components of an example system 800 according to example embodiments of the present disclosure.
  • the example system 800 can include the computing system 300 and a machine learning computing system 830 that are
  • the computing system 300 can be implemented onboard a vehicle (e.g., as a portion of the vehicle computing system 105) and/or can be remote from a vehicle (e.g., as portion of an operations computing system 115). In either case, a vehicle computing system 105 can utilize the operations and model(s) of the computing system 300 (e.g., locally, via wireless network communication, etc.).
  • the computing system 300 can include one or more computing device(s) 801.
  • the computing device(s) 801 of the computing system 300 can include processor(s) 802 and a memory 804.
  • the one or more processors 802 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected.
  • the memory 804 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, one or more memory devices, flash memory devices, etc., and combinations thereof.
  • the memory 804 can store information that can be obtained by the one or more processors 802.
  • the memory 804 e.g., one or more non-transitory computer- readable storage mediums, memory devices
  • the memory 804 can include computer-readable instructions 806 that can be executed by the one or more processors 802.
  • the instructions 806 can be software written in any suitable programming language or can be implemented in hardware. Additionally, or alternatively, the instructions 806 can be executed in logically and/or virtually separate threads on processor(s) 802.
  • the memory 804 can store instructions 806 that when executed by the one or more processors 802 cause the one or more processors 802 (the computing system 300) to perform operations such as any of the operations and functions of the computing system 300 and/or for which the computing system 300 is configured, as described herein, the operations for detecting lane boundaries (e.g., one or more portions of methods 600 and 700), and/or any other operations and functions for the computing system 300, as described herein.
  • the memory 804 can store data 808 that can be obtained (e.g., received, accessed, written, manipulated, generated, created, stored, etc.).
  • the data 608 can include, for instance, sensor data, processed sensor data, data indicative of feature volume(s), data indicative of ground height estimation(s), overhead image data, data indicative of machine-learned model(s), input data, output data, and/or other data/information described herein.
  • the computing device(s) 801 can obtain data from one or more memories that are remote from the computing system 300.
  • the computing device(s) 801 can also include a communication interface 809 used to communicate with one or more other system(s) (e.g., other systems onboard and/or remote from a vehicle, the other systems of FIG. 8, etc.).
  • the communication interface 809 can include any circuits, components, software, etc. for communicating via one or more networks (e.g., 880).
  • the communication interface 809 can include, for example, one or more of a communications controller, receiver, transceiver, transmitter, port, conductors, software and/or hardware for communicating data/information.
  • the computing system 300 can store or include one or more machine-learned models 840.
  • the machine- learned models 840 can be or can otherwise include various machine-learned models such as, for example, neural networks (e.g., deep neural networks), support vector machines, decision trees, ensemble models, k-nearest neighbors models, Bayesian networks, or other types of models including linear models and/or non-linear models.
  • Example neural networks include feed-forward neural networks (e.g., convolutional neural networks, etc.), recurrent neural networks (e.g., long short-term memory recurrent neural networks, etc.), and/or other forms of neural networks.
  • the machine-learned models 840 can include the machine-learned models 306A-C and/or other model(s), as described herein.
  • the computing system 300 can receive the one or more machine-learned models 840 from the machine learning computing system 830 over the network(s) 880 and can store the one or more machine-learned models 840 in the memory 804 of the computing system 300.
  • the computing system 300 can use or otherwise implement the one or more machine-learned models 840 (e.g., by processor(s) 802).
  • the computing system 300 can implement the machine learned model(s) 840 to generate feature volumes, determine ground height estimations, generate overhead image data, and/or detect lane boundaries, as described herein.
  • the machine learning computing system 830 can include one or more processors 832 and a memory 834.
  • the one or more processors 832 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a
  • the microcontroller, etc. can be one processor or a plurality of processors that are operatively connected.
  • the memory 834 can include one or more non-transitory computer- readable storage media, such as RAM, ROM, EEPROM, EPROM, one or more memory devices, flash memory devices, etc., and combinations thereof.
  • the memory 834 can store information that can be accessed by the one or more processors 832.
  • the memory 834 e.g., one or more non-transitory computer- readable storage mediums, memory devices
  • data 836 can be obtained (e.g., generated, retrieved, received, accessed, written, manipulated, created, stored, etc.).
  • the machine learning computing system 830 can obtain data from one or more memories that are remote from the machine learning computing system 830.
  • the memory 834 can also store computer-readable instructions 838 that can be executed by the one or more processors 832.
  • the instructions 838 can be software written in any suitable programming language or can be implemented in hardware.
  • the instructions 838 can be executed in logically and/or virtually separate threads on processor(s) 832.
  • the memory 834 can store the instructions 838 that when executed by the one or more processors 832 cause the one or more processors 832 to perform operations.
  • the machine learning computing system 830 can include a communication system 839, including devices and/or functions similar to that described with respect to the computing system 300.
  • the machine learning computing system 830 can include one or more server computing devices. If the machine learning computing system 830 includes multiple server computing devices, such server computing devices can operate according to various computing architectures, including, for example, sequential computing architectures, parallel computing architectures, or some combination thereof.
  • the machine learning computing system 830 can include one or more machine-learned models 850.
  • the machine-learned models 850 can be or can otherwise include various machine-learned models such as, for example, neural networks (e.g., deep neural networks), support vector machines, decision trees, ensemble models, k-nearest neighbors models, Bayesian networks, or other types of models including linear models and/or non-linear models.
  • Example neural networks include feed-forward neural networks (e.g., convolutional neural networks), recurrent neural networks (e.g., long short-term memory recurrent neural networks, etc.), and/or other forms of neural networks.
  • the machine-learned models 850 can be similar to and/or the same as the machine-learned models 840, 306A-C.
  • the machine learning computing system 830 can communicate with the computing system 300 according to a client-server relationship.
  • the machine learning computing system 830 can implement the machine-learned models 850 to provide a web service to the computing system 300 (e.g., including on a vehicle,
  • the web service can provide machine-learned models to an entity associated with a vehicle; such that the entity can implement the machine-learned model (e.g., to detect lane boundaries within a surrounding environment of a vehicle, etc.).
  • machine-learned models 850 can be located and used at the computing system 300 (e.g., on the vehicle, at the operations computing system, etc.) and/or the machine-learned models 850 can be located and used at the machine learning computing system 830.
  • the machine learning computing system 830 and/or the computing system 300 can train the machine-learned models 840 and/or 850 through use of a model trainer 860.
  • the model trainer 860 can train the machine-learned models 840 and/or 850 using one or more training or learning algorithms.
  • One example training technique is backwards propagation of errors.
  • the model trainer 860 can perform supervised training techniques using a set of labeled training data.
  • the model trainer 860 can perform unsupervised training techniques using a set of unlabeled training data.
  • the model trainer 860 can perform a number of generalization techniques to improve the generalization capability of the models being trained.
  • Generalization techniques include weight decays, dropouts, or other techniques.
  • the model trainer 860 can train a machine-learned model 840 and/or 850 based on a set of training data 862.
  • the training data 662 can include, for example, ground truth data (e.g., sensor data, lane graph, etc.) with annotated lane boundaries (e.g., in both the camera view and overhead view).
  • ground truth data e.g., sensor data, lane graph, etc.
  • high-definition mapping data can provide dense ground height ground truth for each example.
  • the annotated lane boundaries may not align perfectly with actual lane boundaries, and the models can still be trained to produce information-rich lane detections.
  • the parameters Q of the models can be optimized by minimizing a combination of the lane detection loss and the ground height estimation loss:
  • the lane detection loss li ane can be defined by:
  • d Pgt and d p pml are the ground truth and predicted distance transform values for pixel p, respectively.
  • T is the threshold used to cap the distance transform values to a range between [0, T], as it may be unnecessary for a convolutional neural network to produce exact distance transform values for regions far away from any lane boundary. Additionally, this is inverted such that the maximum predicted value T occurs at the lane boundary, and linearly decreases outward to 0. This removes the need for the network to predict a sharp drop-off in distance transform values at the thresholding boundary from T to 0.
  • T can be set to 30, which corresponds to l.5m in an overhead view.
  • the machine-learned lane boundary detection model (and sensor data aggregation models) can be trained by simultaneously minimizing the above distance transform loss as well as an additional ground height estimation loss, which is also formulated as a mean squared error: ! K i ⁇ b] - p.pred
  • the two losses can be combined additively, with a multiplier of 100 for l gnd (&).
  • the models described herein can be trained using an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. This can include, for example, a learning rate of le-4 and a weight decay of le-4 for 35000 epochs with a batch size of 5.
  • a distance transform value threshold can be selected based on the type of training scene.
  • the training data 862 can be taken from the same vehicle as that which utilizes that model 840/850. In this way, the models 840/850 can be trained to determine outputs in a manner that is tailored to that particular vehicle.
  • the training data 862 can be taken from one or more different vehicles than that which is utilizing that model 840/850.
  • the model trainer 860 can be implemented in hardware, firmware, and/or software controlling one or more processors.
  • the network(s) 880 can be any type of network or combination of networks that allows for communication between devices.
  • the network(s) 880 can include one or more of a local area network, wide area network, the Internet, secure network, cellular network, mesh network, peer-to-peer communication link and/or some combination thereof and can include any number of wired or wireless links.
  • Communication over the network(s) 880 can be accomplished, for instance, via a network interface using any type of protocol, protection scheme, encoding, format, packaging, etc.
  • FIG. 8 illustrates one example system 800 that can be used to implement the present disclosure.
  • the computing system 300 can include the model trainer 860 and the training dataset 862.
  • the machine-learned models 840 can be both trained and used locally at the computing system 300 (e.g., at a vehicle).
  • Computing tasks discussed herein as being performed at computing device(s) remote from the vehicle can instead be performed at the vehicle (e.g., via the vehicle computing system), or vice versa.
  • Such configurations can be implemented without deviating from the scope of the present disclosure.
  • the use of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and
  • Computer-implemented operations can be performed on a single component or across multiple components.
  • Computer-implemented tasks and/or operations can be performed sequentially or in parallel.
  • Data and instructions can be stored in a single memory device or across multiple memory devices.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Remote Sensing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Electromagnetism (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Multimedia (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Traffic Control Systems (AREA)

Abstract

L'invention concerne des systèmes et des procédés destinés à faciliter la communication avec des véhicules autonomes. Dans un mode de réalisation donné à titre d'exemple, un système informatique peut obtenir des données LIDAR rastérisées associées à un environnement ambiant d'un véhicule autonome. Les données LIDAR rastérisées peuvent comprendre des données d'image LIDAR qui sont rastérisées à partir d'un nuage de points LIDAR. Le système informatique peut accéder à des données indicatives d'un modèle de détection de limites de voie appris par machine. Le système informatique peut entrer les données LIDAR rastérisées, associées à l'environnement ambiant du véhicule autonome, dans le modèle de détection de limites de voie appris par machine. Le système informatique peut obtenir une sortie à partir du modèle de détection de limites de voie appris par machine. Cette sortie peut indiquer une ou plusieurs limites de voie se situant dans l'environnement ambiant du véhicule autonome.
EP18836692.6A 2017-11-15 2018-11-15 Systèmes et procédés de détection de limites de voie pour véhicules autonomes Pending EP3710980A1 (fr)

Applications Claiming Priority (5)

Application Number Priority Date Filing Date Title
US201762586725P 2017-11-15 2017-11-15
US201762586741P 2017-11-15 2017-11-15
US16/122,413 US11080537B2 (en) 2017-11-15 2018-09-05 Autonomous vehicle lane boundary detection systems and methods
US16/122,267 US10803325B2 (en) 2017-11-15 2018-09-05 Autonomous vehicle lane boundary detection systems and methods
PCT/US2018/061216 WO2019099622A1 (fr) 2017-11-15 2018-11-15 Systèmes et procédés de détection de limites de voie pour véhicules autonomes

Publications (1)

Publication Number Publication Date
EP3710980A1 true EP3710980A1 (fr) 2020-09-23

Family

ID=66539952

Family Applications (1)

Application Number Title Priority Date Filing Date
EP18836692.6A Pending EP3710980A1 (fr) 2017-11-15 2018-11-15 Systèmes et procédés de détection de limites de voie pour véhicules autonomes

Country Status (2)

Country Link
EP (1) EP3710980A1 (fr)
WO (1) WO2019099622A1 (fr)

Families Citing this family (33)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10678244B2 (en) 2017-03-23 2020-06-09 Tesla, Inc. Data synthesis for autonomous control systems
US11893393B2 (en) 2017-07-24 2024-02-06 Tesla, Inc. Computational array microprocessor system with hardware arbiter managing memory requests
US10671349B2 (en) 2017-07-24 2020-06-02 Tesla, Inc. Accelerated mathematical engine
US11157441B2 (en) 2017-07-24 2021-10-26 Tesla, Inc. Computational array microprocessor system using non-consecutive data formatting
US11409692B2 (en) 2017-07-24 2022-08-09 Tesla, Inc. Vector computational unit
US11561791B2 (en) 2018-02-01 2023-01-24 Tesla, Inc. Vector computational unit receiving data elements in parallel from a last row of a computational array
US11215999B2 (en) 2018-06-20 2022-01-04 Tesla, Inc. Data pipeline and deep learning system for autonomous driving
US11361457B2 (en) 2018-07-20 2022-06-14 Tesla, Inc. Annotation cross-labeling for autonomous control systems
US11636333B2 (en) 2018-07-26 2023-04-25 Tesla, Inc. Optimizing neural network structures for embedded systems
US11562231B2 (en) 2018-09-03 2023-01-24 Tesla, Inc. Neural networks for embedded devices
SG11202103493QA (en) 2018-10-11 2021-05-28 Tesla Inc Systems and methods for training machine models with augmented data
US11196678B2 (en) 2018-10-25 2021-12-07 Tesla, Inc. QOS manager for system on a chip communications
US11816585B2 (en) 2018-12-03 2023-11-14 Tesla, Inc. Machine learning models operating at different frequencies for autonomous vehicles
US11537811B2 (en) 2018-12-04 2022-12-27 Tesla, Inc. Enhanced object detection for autonomous vehicles based on field view
US11610117B2 (en) 2018-12-27 2023-03-21 Tesla, Inc. System and method for adapting a neural network model on a hardware platform
US11150664B2 (en) 2019-02-01 2021-10-19 Tesla, Inc. Predicting three-dimensional features for autonomous driving
US10997461B2 (en) 2019-02-01 2021-05-04 Tesla, Inc. Generating ground truth for machine learning from time series elements
US11567514B2 (en) 2019-02-11 2023-01-31 Tesla, Inc. Autonomous and user controlled vehicle summon to a target
US10956755B2 (en) 2019-02-19 2021-03-23 Tesla, Inc. Estimating object properties using visual image data
US11718324B2 (en) 2019-04-11 2023-08-08 Isee, Inc. Instance segmentation imaging system
US10937178B1 (en) * 2019-05-09 2021-03-02 Zoox, Inc. Image-based depth data and bounding boxes
US11087494B1 (en) 2019-05-09 2021-08-10 Zoox, Inc. Image-based depth data and localization
US10984543B1 (en) 2019-05-09 2021-04-20 Zoox, Inc. Image-based depth data and relative depth data
CN111986243A (zh) * 2019-05-24 2020-11-24 北京京东尚科信息技术有限公司 路肩提取方法及装置、电子设备和计算机可读介质
CN111291676B (zh) * 2020-02-05 2020-12-11 清华大学 一种基于激光雷达点云和相机图像融合的车道线检测方法及装置和芯片
US11620831B2 (en) 2020-04-29 2023-04-04 Toyota Research Institute, Inc. Register sets of low-level features without data association
US11845428B2 (en) * 2021-07-13 2023-12-19 Canoo Technologies Inc. System and method for lane departure warning with ego motion and vision
US12017661B2 (en) 2021-07-13 2024-06-25 Canoo Technologies Inc. System and method in vehicle path prediction based on full nonlinear kinematics
US11891059B2 (en) 2021-07-13 2024-02-06 Canoo Technologies Inc. System and methods of integrating vehicle kinematics and dynamics for lateral control feature at autonomous driving
US11891060B2 (en) 2021-07-13 2024-02-06 Canoo Technologies Inc. System and method in lane departure warning with full nonlinear kinematics and curvature
US11908200B2 (en) 2021-07-13 2024-02-20 Canoo Technologies Inc. System and method in the prediction of target vehicle behavior based on image frame and normalization
US11840147B2 (en) 2021-07-13 2023-12-12 Canoo Technologies Inc. System and method in data-driven vehicle dynamic modeling for path-planning and control
WO2023002093A1 (fr) * 2021-07-23 2023-01-26 Sensible 4 Oy Systèmes et procédés pour déterminer la possibilité de traverser une route à l'aide de données en temps réel et d'un modèle entrainé

Also Published As

Publication number Publication date
WO2019099622A1 (fr) 2019-05-23

Similar Documents

Publication Publication Date Title
US11972606B2 (en) Autonomous vehicle lane boundary detection systems and methods
US10803325B2 (en) Autonomous vehicle lane boundary detection systems and methods
EP3710980A1 (fr) Systèmes et procédés de détection de limites de voie pour véhicules autonomes
US11835951B2 (en) Object motion prediction and autonomous vehicle control
US10656657B2 (en) Object motion prediction and autonomous vehicle control
US11794785B2 (en) Multi-task machine-learned models for object intention determination in autonomous driving
US11217012B2 (en) System and method for identifying travel way features for autonomous vehicle motion control
US11755018B2 (en) End-to-end interpretable motion planner for autonomous vehicles
US20190147255A1 (en) Systems and Methods for Generating Sparse Geographic Data for Autonomous Vehicles
US10768628B2 (en) Systems and methods for object detection at various ranges using multiple range imagery
US20230229889A1 (en) Systems and Methods for Generating Motion Forecast Data for Actors with Respect to an Autonomous Vehicle and Training a Machine Learned Model for the Same
US10859384B2 (en) Lightweight vehicle localization systems and methods
US11691650B2 (en) Systems and methods for generating motion forecast data for a plurality of actors with respect to an autonomous vehicle
US20210276587A1 (en) Systems and Methods for Autonomous Vehicle Systems Simulation
US11780472B2 (en) Systems and methods for generating motion forecast data for a plurality of actors with respect to an autonomous vehicle
US11507090B2 (en) Systems and methods for vehicle motion control with interactive object annotation
WO2021178513A1 (fr) Systèmes et procédés d'intégration de données radar pour une détection d'objet améliorée dans des véhicules autonomes
US11820397B2 (en) Localization with diverse dataset for autonomous vehicles

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20200528

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20221026

RAP3 Party data changed (applicant data changed or rights of an application transferred)

Owner name: UATC, LLC

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: AURORA OPERATIONS, INC.