WO2019122952A1 - Procédé et système pour planification de mouvement personnalisée dans des véhicules à conduite autonome - Google Patents
Procédé et système pour planification de mouvement personnalisée dans des véhicules à conduite autonome Download PDFInfo
- Publication number
- WO2019122952A1 WO2019122952A1 PCT/IB2017/058085 IB2017058085W WO2019122952A1 WO 2019122952 A1 WO2019122952 A1 WO 2019122952A1 IB 2017058085 W IB2017058085 W IB 2017058085W WO 2019122952 A1 WO2019122952 A1 WO 2019122952A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- passenger
- vehicle
- motion
- planning
- autonomous driving
- Prior art date
Links
- 230000033001 locomotion Effects 0.000 title claims abstract description 249
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000006243 chemical reaction Methods 0.000 claims description 82
- 230000000007 visual effect Effects 0.000 claims description 36
- 238000003860 storage Methods 0.000 claims description 21
- 238000004891 communication Methods 0.000 claims description 15
- 238000011065 in-situ storage Methods 0.000 claims description 8
- 230000002996 emotional effect Effects 0.000 claims description 5
- 239000012190 activator Substances 0.000 claims description 3
- 230000003213 activating effect Effects 0.000 claims 1
- 230000006399 behavior Effects 0.000 description 102
- 238000012549 training Methods 0.000 description 82
- 238000001514 detection method Methods 0.000 description 35
- 238000010586 diagram Methods 0.000 description 28
- 230000008569 process Effects 0.000 description 27
- 230000007246 mechanism Effects 0.000 description 23
- 238000013459 approach Methods 0.000 description 16
- 230000004313 glare Effects 0.000 description 14
- 230000003044 adaptive effect Effects 0.000 description 13
- 230000008859 change Effects 0.000 description 13
- 230000001960 triggered effect Effects 0.000 description 13
- 230000004927 fusion Effects 0.000 description 9
- 238000012545 processing Methods 0.000 description 8
- 230000000694 effects Effects 0.000 description 7
- 230000015654 memory Effects 0.000 description 7
- 230000003287 optical effect Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 239000000284 extract Substances 0.000 description 4
- 230000036541 health Effects 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 230000003068 static effect Effects 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 238000010276 construction Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000007781 pre-processing Methods 0.000 description 3
- 208000019901 Anxiety disease Diseases 0.000 description 2
- 241001282135 Poromitra oscitans Species 0.000 description 2
- 206010048232 Yawning Diseases 0.000 description 2
- 230000036506 anxiety Effects 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 239000000446 fuel Substances 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000003340 mental effect Effects 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000001556 precipitation Methods 0.000 description 2
- 230000002787 reinforcement Effects 0.000 description 2
- 230000008439 repair process Effects 0.000 description 2
- 230000011664 signaling Effects 0.000 description 2
- RYGMFSIKBFXOCR-UHFFFAOYSA-N Copper Chemical compound [Cu] RYGMFSIKBFXOCR-UHFFFAOYSA-N 0.000 description 1
- 206010012289 Dementia Diseases 0.000 description 1
- 206010038743 Restlessness Diseases 0.000 description 1
- 206010041235 Snoring Diseases 0.000 description 1
- 206010041349 Somnolence Diseases 0.000 description 1
- 206010047700 Vomiting Diseases 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000003542 behavioural effect Effects 0.000 description 1
- 230000033228 biological regulation Effects 0.000 description 1
- 230000001276 controlling effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000008451 emotion Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000004387 environmental modeling Methods 0.000 description 1
- 230000004438 eyesight Effects 0.000 description 1
- 230000008921 facial expression Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000012804 iterative process Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000006996 mental state Effects 0.000 description 1
- 239000002184 metal Substances 0.000 description 1
- 229910052751 metal Inorganic materials 0.000 description 1
- 230000007935 neutral effect Effects 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 239000011800 void material Substances 0.000 description 1
- 230000008673 vomiting Effects 0.000 description 1
- 210000000707 wrist Anatomy 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60L—PROPULSION OF ELECTRICALLY-PROPELLED VEHICLES; SUPPLYING ELECTRIC POWER FOR AUXILIARY EQUIPMENT OF ELECTRICALLY-PROPELLED VEHICLES; ELECTRODYNAMIC BRAKE SYSTEMS FOR VEHICLES IN GENERAL; MAGNETIC SUSPENSION OR LEVITATION FOR VEHICLES; MONITORING OPERATING VARIABLES OF ELECTRICALLY-PROPELLED VEHICLES; ELECTRIC SAFETY DEVICES FOR ELECTRICALLY-PROPELLED VEHICLES
- B60L15/00—Methods, circuits, or devices for controlling the traction-motor speed of electrically-propelled vehicles
- B60L15/32—Control or regulation of multiple-unit electrically-propelled vehicles
- B60L15/38—Control or regulation of multiple-unit electrically-propelled vehicles with automatic control
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
- B60W30/08—Active safety systems predicting or avoiding probable or impending collision or attempting to minimise its consequences
- B60W30/095—Predicting travel path or likelihood of collision
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
- B60W30/10—Path keeping
- B60W30/12—Lane keeping
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
- B60W30/18—Propelling the vehicle
- B60W30/18009—Propelling the vehicle related to particular drive situations
- B60W30/18145—Cornering
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
- B60W30/18—Propelling the vehicle
- B60W30/18009—Propelling the vehicle related to particular drive situations
- B60W30/18154—Approaching an intersection
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
- B60W30/18—Propelling the vehicle
- B60W30/18009—Propelling the vehicle related to particular drive situations
- B60W30/18163—Lane change; Overtaking manoeuvres
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/26—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
- G01C21/34—Route searching; Route guidance
- G01C21/3407—Route searching; Route guidance specially adapted for specific applications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/26—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
- G01C21/34—Route searching; Route guidance
- G01C21/3453—Special cost functions, i.e. other than distance or default speed limit of road segments
- G01C21/3484—Personalized, e.g. from learned user behaviour or user-defined profiles
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2510/00—Input parameters relating to a particular sub-units
- B60W2510/06—Combustion engines, Gas turbines
- B60W2510/0638—Engine speed
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2510/00—Input parameters relating to a particular sub-units
- B60W2510/08—Electric propulsion units
- B60W2510/081—Speed
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2520/00—Input parameters relating to overall vehicle dynamics
- B60W2520/06—Direction of travel
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2520/00—Input parameters relating to overall vehicle dynamics
- B60W2520/10—Longitudinal speed
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2530/00—Input parameters relating to vehicle conditions or values, not covered by groups B60W2510/00 or B60W2520/00
- B60W2530/20—Tyre data
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2540/00—Input parameters relating to occupants
- B60W2540/22—Psychological state; Stress level or workload
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2540/00—Input parameters relating to occupants
- B60W2540/26—Incapacity
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2540/00—Input parameters relating to occupants
- B60W2540/30—Driving style
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2552/00—Input parameters relating to infrastructure
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2552/00—Input parameters relating to infrastructure
- B60W2552/05—Type of road, e.g. motorways, local streets, paved or unpaved roads
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2552/00—Input parameters relating to infrastructure
- B60W2552/15—Road slope, i.e. the inclination of a road segment in the longitudinal direction
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2552/00—Input parameters relating to infrastructure
- B60W2552/20—Road profile, i.e. the change in elevation or curvature of a plurality of continuous road segments
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2552/00—Input parameters relating to infrastructure
- B60W2552/30—Road curve radius
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2554/00—Input parameters relating to objects
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2555/00—Input parameters relating to exterior conditions, not covered by groups B60W2552/00, B60W2554/00
- B60W2555/20—Ambient conditions, e.g. wind or rain
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2555/00—Input parameters relating to exterior conditions, not covered by groups B60W2552/00, B60W2554/00
- B60W2555/60—Traffic rules, e.g. speed limits or right of way
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2556/00—Input parameters relating to data
- B60W2556/45—External transmission of data to or from the vehicle
- B60W2556/50—External transmission of data to or from the vehicle of positioning data, e.g. GPS [Global Positioning System] data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N7/00—Computing arrangements based on specific mathematical models
- G06N7/01—Probabilistic graphical models, e.g. probabilistic networks
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T90/00—Enabling technologies or technologies with a potential or indirect contribution to GHG emissions mitigation
- Y02T90/10—Technologies relating to charging of electric vehicles
- Y02T90/16—Information or communication technologies improving the operation of electric vehicles
Definitions
- the present teaching generally relates to autonomous driving. More specifically, the present teaching relates to planning and control in autonomous driving.
- an autonomous driving module 110 includes a planning module 120 and a vehicle control module 130.
- Planning may include, as shown in Fig. 2, route planning, motion planning, or behavior planning.
- Route planning refers to the effort to plan a route from a source to a destination based on certain considerations.
- Motion planning may generally refer to the effort of planning the movement of a vehicle to achieve certain effect.
- the movement of the vehicle may be planned in a way that complies with the traffic regulations or safety. Motion planning is then to determine what movement the vehicle needs to make to achieve that.
- Behavior planning generally refers to the effort to plan how the vehicle should behave in different situations, e.g., the vehicle behavior while crossing an intersection, the vehicle behavior in staying within or following a lane, or the vehicle behavior in making a turn. For instance, in terms of overtaking a slow moving front vehicle, certain vehicle behavior may be planned.
- Behavior planning and motion planning may be related. For example, the planned vehicle behavior may need to be translated into motion in order to implement the behavior.
- Vehicle control 130 as shown in Fig. 1 may involve various aspects of control. This is illustrated in Fig. 3, which shows that vehicle control may involve, e.g., roadway specific control, motion specific control, mass specific control, geometry specific control, aerodynamic specific control, and tire specific control.
- Surrounding information 100 in Fig. 1 may be used for vehicle planning.
- surrounding information 100 includes, e.g., current location of the vehicle, intended destination, and/or traffic information.
- the conventional planning module 120 may devise, e.g., a plan for a route from the current location to the destination.
- Known criteria used in route planning may include, e.g., shortest distance, shortest time, use of highways, use of local roads, traffic, etc. Such criteria may be applied based on known information such as the distance of each road segment, known traffic patterns associated with roads, etc.
- the planning module 120 may also perform motion planning, which is traditionally based on, e.g., rapidly exploring random trees (RRT) for state space or Markov Decision Process (MDP) for environmental modeling.
- the planning module 120 may generate, based on the planned route/motion, planning data to be fed to the vehicle control module 130 so that the vehicle control module 130 can proceed to control the vehicle in a way as planned.
- the vehicle control module 130 may then generate control signals 140 which may be sent to different parts of the vehicle to implement the planned vehicle movement.
- Vehicle control is traditionally exercised based on generic vehicle kinematic models and/or different types of feedback controllers.
- Each human driver generally operates or controls a vehicle differently with diverse preferences. Human drivers also operate vehicles adaptively based on real time situations, which may arise out of the present conditions of the vehicle itself, the extrinsic environment conditions that serve to limit the ability of the vehicle to operate, and/or the reaction or response to the current vehicle movement from passengers in the vehicle. For example, with children in the vehicle, a human driver may elect, for safety, to avoid (route planning) a route that is curvy on a snowy day. A human driver may drive in different manners when different passengers are riding in the vehicle to ensure comfort of the passenger. Although a human driver generally controls a vehicle by following a lane by staying roughly in the middle of the lane, the behavior may change when faced with a right turn.
- the same human driver may curve to the right side of the lane when the vehicle is approaching the point of the right turn.
- different human drivers may curve to the right in different ways.
- lane changing behavior may also differ with respect to different vehicles in different surrounding situations. The existing technologies do not address those issues, let alone providing solutions.
- the present teaching relates to methods, systems, and programming for developing a virtual agent that can have a dialog with a user.
- a method for motion planning for an autonomous driving vehicle by which self-aware capability parameters are instantiated with respect to a current location of the autonomous driving vehicle and are to be used to estimate the operational capability of the autonomous driving vehicle with respect to the current location.
- a preference of a passenger present in the autonomous driving vehicle is estimated with respect to vehicle motion.
- the motion of the autonomous driving vehicle is planned automatically based on the self-aware capability parameters and the preference of the passenger.
- a system for automatic motion planning for an autonomous driving vehicle comprises a self-aware capability analyzer, a passenger observation analyzer, and a passenger motion adapter.
- the self-aware capability analyzer is configured for obtaining self-aware capability parameters instantiated with respect to a current location of the autonomous driving vehicle and are to be used to estimate the operational capability of the autonomous driving vehicle with respect to the current location.
- the passenger observation analyzer is configured for estimating a preference of a passenger present in the autonomous driving vehicle with respect to vehicle motion.
- the passenger motion adapter is configured for planning a motion of the autonomous driving vehicle based on the self- aware capability parameters and the preference of the passenger.
- a software product in accord with this concept, includes at least one machine-readable non-transitory medium and information carried by the medium.
- the information carried by the medium may be executable program code data, parameters in association with the executable program code, and/or information related to a user, a request, content, or information related to a social group, etc.
- machine readable non-transitory medium wherein the medium has information recorded thereon for motion planning for an autonomous driving vehicle, wherein the information, when read by the machine, causes the machine to perform various operational steps.
- Self-aware capability parameters are obtained that are instantiated with respect to a current location of the autonomous driving vehicle and are to be used to estimate the operational capability of the autonomous driving vehicle with respect to the current location.
- a preference of a passenger present in the autonomous driving vehicle is estimated with respect to vehicle motion.
- the motion of the autonomous driving vehicle is planned automatically based on the self-aware capability parameters and the preference of the passenger.
- Fig. 1 shows some essential modules in autonomous driving
- FIG. 2 illustrates exemplary types of planning in autonomous driving
- FIG. 3 illustrates commonly known types of vehicle control
- FIG. 4 A depicts an autonomous driving vehicle with a planning module and a vehicle control module, according to an embodiment of the present teaching
- Fig. 4B illustrates exemplary types of real time data, according to an embodiment of the present teaching
- Fig. 5 depicts an exemplary high level system diagram of a planning module, according to an embodiment of the present teaching
- FIG. 6 A illustrates exemplary ways to realizing a safe-aware capability model, according to an embodiment of the present teaching
- Fig. 6B illustrates an exemplary construct of a self-aware capability model with parameters, according to an embodiment of the present teaching
- Fig. 6C illustrates exemplary types of intrinsic vehicle capability parameters, according to an embodiment of the present teaching
- Fig. 6D illustrates exemplary types of extrinsic capability parameters, according to an embodiment of the present teaching
- Fig. 7 depicts an exemplary high level system diagram of a mechanism for generating self-aware capability parameters to be considered for planning, according to an embodiment of the present teaching
- Fig. 8 depicts an exemplary high level system diagram of a self-aware capability parameter generator, according to an embodiment of the present teaching
- FIG. 9 is a flowchart of an exemplary process for generating self-aware capability parameters, according to an embodiment of the present teaching
- Fig. 10 depicts an exemplary high level system diagram of a route planning module, according to an embodiment of the present teaching
- FIG. 11 is a flowchart of an exemplary process for route planning, according to an embodiment of the present teaching
- Fig. 12 depicts an exemplary high level system diagram of a global route planner, according to an embodiment of the present teaching
- Fig. 13 is a flowchart of an exemplary process for a global route planner, according to an embodiment of the present teaching
- Fig. 14A depicts an exemplary high level system diagram of a motion planning module, according to an embodiment of the present teaching
- Fig. 14B illustrates exemplary types of passenger models, according to an embodiment of the present teaching
- Fig. 14C illustrates exemplary types of user reactions to be observed for motion planning, according to an embodiment of the present teaching
- Fig. 15 depicts an exemplary high level system diagram of a passenger observation analyzer, according to an embodiment of the present teaching
- Fig. 16 is a flowchart of an exemplary process of a passenger observation analyzer, according to an embodiment of the present teaching
- FIG. 17 is a flowchart of an exemplary process for a motion planning module, according to an embodiment of the present teaching
- Fig. 18 depicts an exemplary high level system diagram of a model training mechanism for generating different models for motion planning, according to an embodiment of the present teaching
- Fig. 19 illustrates different types of reactions to be observed and their roles in model training, according to an embodiment of the present teaching
- Fig. 20A illustrates exemplary types of lane related planning, according to an embodiment of the present teaching
- FIG. 20B illustrates exemplary types of behavior related to lane following, according to an embodiment of the present teaching
- FIG. 20C illustrates exemplary types of behavior related to lane changing, according to an embodiment of the present teaching
- Fig. 21 depicts an exemplary high level system diagram of a lane planning module, according to an embodiment of the present teaching
- Fig. 22 is a flowchart of an exemplary process for a lane planning module, according to an embodiment of the present teaching
- Fig. 23A illustrates the traditional approach to generate vehicle control signals based on a vehicle kinematic model
- Fig. 23B depicts a high level system diagram of a vehicle control module that enables human-like vehicle control, according to an embodiment of the present teaching
- Fig. 23 C depicts a high level system diagram of a vehicle control module that enables personalized human-like vehicle control, according to an embodiment of the present teaching
- Fig. 24 depicts an exemplary high level system diagram of a human-like vehicle control unit, according to an embodiment of the present teaching
- Fig. 25 is a flow chart of an exemplary process of a human-like vehicle control unit, according to an embodiment of the present teaching
- Fig. 26 depicts an exemplary high level system diagram of a human-like vehicle control model generator, according to an embodiment of the present teaching
- Fig. 27 is a flowchart of an exemplary process of a human-like vehicle control model generator, according to an embodiment of the present teaching
- Fig. 28 depicts an exemplary high level system diagram of a human-like vehicle control signal generator, according to an embodiment of the present teaching
- Fig. 29 is a flowchart of an exemplary process of a human-like vehicle control signal generator, according to an embodiment of the present teaching
- Fig. 30 depicts the architecture of a mobile device which can be used to implement a specialized system incorporating the present teaching.
- Fig. 31 depicts the architecture of a computer which can be used to implement a specialized system incorporating the present teaching.
- the present disclosure generally relates to systems, methods, medium, and other implementations for planning and control of route/vehicle behaviors in a self-capability aware, human-like, and personalized manner that is adaptive to real time situations.
- Fig. 4A shows an autonomous vehicle with a vehicle planning/control mechanism 410, according to an embodiment of the present teaching.
- the autonomous vehicle planning/control mechanism 410 includes a planning module 440 and a vehicle control module 450. Both modules take various types of information as input in order to achieve operations that are self-capability aware, human-like, personalized and adaptive to real time situations.
- both the planning module 440 and the vehicle control module 450 receive historical manual driving data 430 in order to learn human like ways to handle the vehicle in different situations.
- These modules also receive real time data 480 in order to be aware of the dynamic situations surround the vehicle in order to adapt the operations accordingly. Furthermore, the planning module 440 accesses a self-aware capability model 490 which characterizes what limits the operational ability of the vehicle given the situation the vehicle is currently in.
- Real time data 480 may include various types of information useful or relevant for planning and control of the vehicle.
- Fig. 4B illustrates exemplary types of real time data, according to an embodiment of the present invention.
- exemplary real time data may include vehicle related data, time related data, passenger related data, weather related data, ... , and data related to the nearby roads.
- Vehicle related data may include, e.g., the motion state, position, or conditions of the vehicle at the time.
- the motion state of a vehicle may involve, e.g., its current speed and driving direction.
- the real time position information may include, e.g., the current latitude, longitude, and altitude of the vehicle.
- the real time conditions of the vehicle may include the functional state of the vehicle such as whether the vehicle is currently in a full or partial functional state or specific parameters under which different components of the vehicle are operating, etc.
- Real time data related to time may generally include current date, time, or month.
- Passenger related data may include various characteristics related to the passenger of the vehicle such as passenger reaction cues, which may include visual, acoustic, or behavior cues observed from the passenger, or conditions of the passenger such as mental state, physical state, or functional state of the passenger. The conditions of the passenger may be inferred based on the cues observed from the passenger reaction cues.
- Weather related data may include the weather of the locale where the vehicle is currently situated.
- the road related data may include information about the physical condition of the nearby road(s), e.g., wetness, steepness, or curviness of the road, or the local traffic condition such as congestion along the road.
- Fig. 5 depicts an exemplary high level system diagram of the planning module 440, according to an embodiment of the present teaching.
- this exemplary is a high level system diagram of the planning module 440, according to an embodiment of the present teaching.
- the planning includes, but not limited to, route planning, motion planning, and the planning of lane related behavior, including lane following, lane changing, etc.
- the planning module 440 comprises a route planning module 550, a motion planning module 560, and a lane planning module 570.
- Each of the modules aims at operating in a self-capability aware, human-like, and personalized manner.
- Each of the modules 550, 560, and 570 takes, in addition to the surrounding information 420, recorded human driving data 430, real time data 480, and the self-aware capability model 490 as inputs and generates their respective outputs to be used by the vehicle control module 450 to convert into the vehicle control signals 470 to control the vehicle.
- the route planning module 550 generates the planned route information 520 as its output
- the motion planning module 560 generates planned motion 530 as its output
- the lane planning module 570 generates planned lane control information 540 as its output.
- Each of the planning modules may be triggered via some triggering signal.
- the route planning module 550 may be activated via a route planning trigger signal; the motion planning module 560 may be activated upon receiving a motion planning trigger signal; while the lane planning module 570 may start to operate when a lane planning trigger signal is received.
- a trigger signal may be manually provided (by, e.g., a driver or a passenger) or automatically generated based on, e.g., certain configuration or certain event.
- a driver may manually activate the route planning module 550 or any other planning module for the route/motion/lane planning, much like what people do to manually start, e.g., cruise control in a car.
- the planning activities may also be activated by a certain configuration or an event.
- the vehicle may be configured to activate route planning whenever the vehicle accepts an input indicating the next destination. This may be regardless what the current location of the vehicle is.
- the planning modules may be always triggered on whenever the vehicle is on and depending on the situation, they may become engaged in different planning activities as needed. In different situations, they may also interact with each other in a manner called for by the situation.
- the lane planning module 570 may determine to change lane in certain circumstance. Such a planned lane control is output by the lane planning module 570 and may be fed to the motion planning module 560 so that a specific path trajectory (planned motion) appropriate for carrying out the planned lane changing may be further planned by the motion planning module 560.
- Output of a planning module may be fed into another within the planning module 440 for either further planning or for providing an input for the future planning of another.
- the output of the route planning module 550 (planned route 520) may be fed to the motion planning module 560 so that the route information may influence how the vehicle motion is planned.
- the output (planned lane control 540) of the lane planning module 570 may be fed to the motion planning module 560 so that the lane control behavior planned may be realized via planned motion control.
- the output of the motion planning module 560 (the planned motion 530) may also be fed to the lane planning module 570 to influence the planning of the lane control behavior.
- the motion planning module 560 may determine that the motion of the vehicle needs to be gentle due to the observation that the passenger of the vehicle prefers smooth motion. Such a determination is part of the motion planning and may be to be sent to the lane planning module 570 so that the lane control behavior of the vehicle may be carried out in a way that ensures smooth motion, e.g., change lane as little as possible.
- the route planning module 550, the motion planning module 560, and the lane planning module 570 also access the self-aware capability model 490 and use it to determine the planning strategy in a manner that takes into account of what the vehicle is actually capable of in the current scenario.
- Fig. 6A illustrates exemplary ways that the self- aware capability model 490 is realized, according to an embodiment of the present teaching.
- the self-aware capability model 490 may be constructed as a probabilistic model, a parameterized model, or a descriptive model. Such a model may be trained based on, e.g., learning.
- the model may include a variety of parameters to be used to characterize factors that may influence or have an impact on the actual ability of the vehicle.
- the model may be implemented as a probabilistic model with parameters being estimated probabilistically.
- the model may also be implemented as a parameterized model with explicit model attributes applicable to different real world conditions.
- the model 490 may also be provided as a descriptive model with enumerated conditions with values instantiated based on real time scenarios.
- the self-aware capability model 490 in any situation may include various parameters, each of which is associated with some factors that may impact the actual ability of the vehicle so that the vehicle planning (route, motion, or lane) has to consider.
- self-aware capability model and self-aware capability parameters will be used interchangeably.
- Fig. 6B illustrates an exemplary construct of the self-aware capability model or parameters 510, according to an embodiment of the present teaching.
- self-aware capability parameters 510 may include intrinsic capability parameters and extrinsic capability parameters.
- Intrinsic vehicle capability parameters may refer to parameters associated with the vehicle itself which may impact what the vehicle is capable of in operation and such parameters may be determined based on either how the vehicle is manufactured or how the vehicle is at the time.
- Extrinsic capability parameters may refer to the parameters or characteristics of the surrounding that are extrinsic to the vehicle but nevertheless may impact the way the vehicle can be operated.
- Fig. 6C illustrates exemplary types of intrinsic vehicle capability parameters, according to an embodiment of the present teaching.
- intrinsic vehicle capability parameters may include, but not limited to, characteristics of the vehicle in terms of, e.g., its engine, its safety measures, and its tires, etc.
- the intrinsic capability parameters may specify the maximum speed the vehicle is capable of, the control that can be exercised on the engine, including cruise control or any restrictions on manual control of the engine.
- the intrinsic capability parameters may include information on what sensors the vehicle is equipped with, specific parameters related to breaks, or information about the seats of the vehicle. For example, some vehicles may have seats that are backed by metal support (stronger) and some with only plastic support.
- the intrinsic capability parameters may also specify the type of the tires of the vehicle (which may have a bearing on what operation can be done) and whether the vehicle currently has snow tires installed or equipped with anti-skip measures. Such intrinsic vehicle capability parameters may be used to assess what types of routes and motions may be possible and which types of vehicle behaviors may be achievable. Thus, making such intrinsic capability parameters available to the planning modules allows the planning modules to plan appropriately without exceeding what the vehicle is actually capable of.
- Fig. 6D illustrates exemplary types of extrinsic capability parameters, according to an embodiment of the present teaching.
- extrinsic capability parameters specify information that are external to the vehicle but nevertheless may impact the ability of planning and such extrinsic capability parameters are used to determine appropriate planning given the conditions external to the vehicle. The ultimate output from the planning modules may be determined within the confine of both intrinsic and extrinsic capability parameters.
- extrinsic capability parameters may include parameters describing the conditions or situations the vehicle is in or may encounter, they likely will impact what should be planned. For example, the surrounding situations related to the road(s), either close to the vehicle or even relatively remote from the vehicle.
- the road condition related parameters may indicate how crowded the road is (so that the driving speed cannot be planned to be fast), whether there are speed limits on the road (regulated minimum and maximum speeds as well as the actual permissible speed due to traffic), whether there is any accident along the road, or the surface condition of the road (so that the motion cannot be too swift), or whether the surface of the road presents certain conditions that will impede the ability of the vehicle in its planning.
- the extrinsic parameters may include information about such nearby vehicles/objects, e.g., the nearby vehicle is a big truck or a bicycle, which may also impact how the planning decision is made.
- events occur along the road the vehicle is on may also impact the planning. For instance, whether the vehicle is currently on a road that is in a school zone or whether there is a construction going on along the road the vehicle is currently on may also be important information to the planning modules for obvious reasons.
- the extrinsic capability parameters may be acquired and updated continuously in time to support the planning modules to adapt their decisions based on external situations in real time. In some situations, the extrinsic capability parameters may also be predicted. For example, if the vehicle is driving on a road to the west in the afternoon, it may be predicted that there will be sun glare. Although such predicted extrinsic capability parameter may not be the real time information, it nevertheless will assist the planning module (e.g., the route planning module) to make an appropriate decision.
- the planning module e.g., the route planning module
- the route planning module 550 may accordingly decide to presently take the road heading to the north first and later take a road to head to the west after sun is down to avoid sun glare (safer).
- Such predicted extrinsic capability parameters may be determined based on other information such as the current location of the vehicle and the intended destination of the vehicle.
- Fig. 7 depicts an exemplary high level system diagram for a mechanism 700 for generating self-aware capability parameters, according to an embodiment of the present teaching.
- the mechanism 700 comprises a locale context determining unit 730 and a self-aware capability parameter generator 740.
- the locale context determining unit 730 is to gather information locale to where the vehicle is and/or will be (i.e., both where the vehicle is presently and where the vehicle will be on its way to the destination) based on, e.g., information about the current location of the vehicle and/or the destination the vehicle is heading to.
- the self-aware capability parameter generator 740 is to generate both intrinsic and extrinsic capability parameters, e.g., on a continuous basis, based on information related to the vehicle and the locale context information determined based on, e.g., the current and future location of the vehicle.
- the locale context determining unit 730 may retrieve information stored in a map configuration 750 and a road context configuration 760 based on the current location 720 and the destination information 710.
- the locale context information related to the roads may include the surrounding or contextual information of the road the vehicle is currently on and/or the roads that the vehicle will be on subsequently.
- the map configuration 750 may provide information about the roads from the current location to the intended destination, while the road context configuration 760 may provide some known or static information about the characteristics associated with roads, such as altitude, steepness, curviness of each road, etc. Such gathered static information about the roads may then be used by the self- aware capability parameter generator 740.
- the road conditions may change over time. For example, roads may become icy or slippery due to changes in weather conditions.
- Such dynamically changing context information about the roads may be acquired separately by, e.g., the self-aware capability parameter generator 740 on a continuous basis and used in generating extrinsic capability parameters that are reflective of the real time situations.
- both the current location and the source-destination information may also be sent to the self-aware capability parameter generator 740 in order for it to gather real time information about road conditions to determine the extrinsic capability parameters.
- information related to the vehicle may be accessed from a vehicle information storage 750.
- the vehicle information storage 750 may store vehicle parameters configured when the vehicle was manufactured such as whether the vehicle is equipped with cruise control or certain types of sensors.
- the storage 750 may also subsequently update information related to the parameters intrinsic to the vehicle. Such subsequent update may be generated due to, e.g., vehicle maintenance or repair or even update observed in real time.
- the self-aware capability parameter generator 740 includes also the mechanism to collect continuously any dynamic update of the vehicle related parameters consistent with the actual intrinsic capability of the vehicle.
- Fig. 8 depicts an exemplary high level system diagram of the self-aware capability parameter generator 740, according to an embodiment of the present teaching.
- the self-aware capability parameter generator 740 comprises a locale context information processor 810, a situation parameter determiner 820, a self-aware capability parameter updater 830, and various updaters that continuously and dynamically gather information of different aspects related to decision making of the vehicle.
- Such dynamic information updaters include, e.g., vehicle capability parameter updater 860-a, weather sensitive parameter updater 860-b, traffic sensitive parameter updater 860-c, orientation sensitive parameter updater 860-d, road sensitive parameter updater 860-e, ... , and time sensitive parameter updater 860-f.
- the locale context information processor 810 processes the received information and, e.g., extracts information related to the current route the vehicle is on and sends such information to the self-aware capability parameter updater 830.
- information related to the current route may include steepness or curviness of the route or other types of static information such as the altitude and orientation of the route.
- the situation parameter determiner 820 receives the current location 720 and, e.g., separates location and time information and sends the information to the self-aware capability parameter identifier 830 so that it may use that information to identify capability parameters specific to the location and the precise time.
- the self-aware capability parameter updater 830 may access intrinsic capability models 840 and/or extrinsic capability models 850 to retrieve capability related parameter values specific to the current location and time.
- the intrinsic capability models 840 may be configured to specify types of parameters relevant to the intrinsic capabilities of the vehicle and the current values thereof.
- the extrinsic capability models 850 may be configured to specify types of parameters that have an impact on the ability of the vehicle to operate and their current values.
- the intrinsic and extrinsic capability models may regularly trigger the updaters (860- a, ... , 860-f) to gather real time information and update the values of the corresponding parameters based on such gathered real time information.
- the intrinsic capability models 840 may be configured to have a mechanism to activate the vehicle capability parameter updater 860-a to gather updated information related to the intrinsic capabilities of the vehicle.
- Such a mechanism may specify different modes of triggering. For instance, it may be on a regular schedule, e.g., daily or hourly.
- the vehicle capability parameter updater 860-a may accept real time vehicle information from the sensor(s) and update the values/states of the relevant capability parameter in the intrinsic capability models to reflect that real time status of the vehicle. For instance, if during the operation of the vehicle, the headlight or a break may become non-functional. Such information sensed in real time may be gathered by the vehicle capability parameter updater 860-a and used to update the information stored in the intrinsic capability parameter storage 840. Such updated information relates to the vehicle may then be used by the self-aware capability parameters generator 740 to generate intrinsic capability parameters.
- the extrinsic capability models 850 may be configured to specify the update mechanism(s) for updating different types of extrinsic capability parameters.
- the update mechanism may specify regularly scheduled update or update to be triggered by some events.
- Different types of extrinsic capability parameters may be configured to be updated based on different triggering mechanisms. For example, for weather related extrinsic capability parameters or extrinsic capability parameters that may be keyed on weather, e.g., visibility in the vicinity of the vehicle, the update may be made regularly, e.g., every few minutes.
- traffic sensitive parameters e.g., the actual permissible speed which is usually direct result of the traffic condition, may also be updated regularly.
- Different types of parameters, although all regularly updated may have different update schedule, which may range from every few seconds to every few minutes or every few hours.
- some extrinsic capability related parameters may be made upon the occurrence of some events.
- an orientation sensitive parameter e.g., whether sun glare exists
- the update may be triggered when the vehicle is heading in certain directions. If the direction of the vehicle is heading changes from north to north-west at some afternoon time, this may trigger the orientation sensitive parameter updater 860-d to gather information related to sun glare and update the situation with regard to sun glare.
- the update may indicate that there is no sub glare, e.g., when it is a cloudy day. In some situations, the update may indicate that there is sun glare.
- such orientation sensitive information is then used to update the value of the corresponding extrinsic capability parameter stored in the extrinsic capability parameter storage 850.
- update of time sensitive parameters such as visibility of the vehicle due to time of the day, may be triggered based on detected location, time zone of the location, and the specific time of the day at the moment.
- the update of some of the capability parameters may also be triggered by event related to the detected updates of other capability parameter values.
- the update of road sensitive parameters such as slippery road condition may be triggered when the update for the weather condition indicates that it started to rain or snow.
- the weather sensitive parameter updater 860-b may receive both dynamic weather update and the updates of other weather sensitive capability parameters, e.g., precipitation, visibility, fog, or any other parameters that relate to weather and have the potential to impact the operation of the vehicle.
- Weather related information may be from multiple data sources that feed real time data.
- the traffic sensitive parameter updater 860-c may receive both dynamic traffic reports and other information relating to the traffic that may influence the operation of the vehicle. Examples include the extent of the traffic jam (which may be used to determine whether the route of the vehicle needs to be re-planned) or the time of the accident that had caused the traffic (to estimate how long the delay will continue in order to determine whether to re-do the route planning). Traffic or traffic related information may be received from one or more sources for real time data feed.
- the orientation sensitive parameter updater 860-d may be configured to gather information along the road in the direction of the vehicle.
- Such orientation sensitive information may include sun glare in certain directions (e.g., east or west) or any potential situations in the direction of the road the vehicle is on (e.g., landslide situation ahead of the road).
- the road sensitive parameter updater 860-e may, once triggered, gather information about various roads or road conditions with respect to the location of the vehicle, from one or more real time information feed sources. Such information may be related to the roads (e.g., open, close, detoured, school zone, etc.) or conditions thereof (e.g., slippery, icy, flooded, construction, etc.).
- the time sensitive parameter updater 860-f may be configured to collect from data source(s) real time data that depend on time.
- the visibility of the road may depend on the time of day at the zone the vehicle is in.
- the collected real time data may then be used to update the intrinsic capability models 840 and/or the extrinsic capability models 850. Such updated data may be time stamped.
- the self-aware capability parameter updater 830 may then access both the intrinsic and extrinsic capability models 840 and 850 to determine relevant capability parameters and the updated values thereof.
- the retrieved intrinsic/extrinsic capability parameters may then be output so that they can be used by various planning modules as shown in Fig. 5. Specifically such generated self-aware capability parameters 510 are used by the route planning module 550 for route planning, as will be discussed with reference to Figs. 10-13.
- the self-aware capability parameters are also used by the motion planning module 560 for personalized motion planning, which will be disclosed in detail with reference to Figs. 14-19.
- the self-aware capability parameters are also used by the lane planning module 570 for lane control, which will be detailed with reference to Figs. 20-22.
- Fig. 9 is a flowchart of an exemplary process of the self-aware capability parameter generator 740, according to an embodiment of the present teaching.
- the locale context information is received at 910 and location and time information are extracted at 920 and used at 930 by different updaters to obtain information feeds from different sources related to various aspects of intrinsic and extrinsic capabilities.
- Such acquired information is then used by different updaters at 940 to update the intrinsic capability parameters 840 and extrinsic capability parameters 850.
- the self-aware capability parameter updater 830 Based on the current location, time, and the received locale contextual information, the self-aware capability parameter updater 830 then identifies various intrinsic and extrinsic capability parameters 510 relevant to the vehicle at the present time to update, at 940, the intrinsic / extrinsic capability parameters and generates, at 950, the updated capability parameters. Such updated intrinsic/extrinsic capability parameters 510 are then output at 960. [0086] Such dynamically gathered self-aware capability parameters are to be used in various vehicle behavior planning operations, including route planning, motion planning, and lane related vehicle behavior planning. For example, in human driving, choosing a route to a destination is often done with consideration of factors captured by the self-aware capability parameters.
- a human driver may choose a route to a desired destination based on, e.g., what the vehicle is equipped with or capable of (intrinsic capability parameters). If the vehicle is in such a condition that it cannot handle steep road well, then such road needs to be avoided. In addition, a human driver may also consider other factors such as weather of the day, conditions of the roads considered, events known or scheduled at certain time of day (extrinsic capability parameters). For instance, it one road points to the west and the sun will be setting at that time, perhaps there will be too much glare so that it is better to take another alternative road. For both safety and reliability, autonomous vehicles ought to also consider such intrinsic and extrinsic capabilities with respect to route choices during route planning.
- the self-aware capability parameters 510 are considered by the planning module 450 in achieving different planning tasks, including the route planning module 550, the motion planning module 560, and the lane planning module 570.
- the route planning module 550 includes the route planning module 550, the motion planning module 560, and the lane planning module 570.
- Fig. 10 depicts an exemplary high level system diagram of the route planning module 550, according to an embodiment of the present teaching.
- the purpose of the route planning module 550 is to plan a route based on desired destination in a self-aware manner in terms of both intrinsic and extrinsic capabilities.
- conventional route planning technologies consider mainly criteria such as shortest distance, shortest time, most use of highways/local ways, etc. without taking into account the dynamic intrinsic capability parameters and the real time extrinsic capability parameters.
- the route planning module 550 comprises a route selection preference determiner 1030 and a global route planner 1020.
- the route selection preference determiner 1030 is to determine the preferences to be considered in selecting a route.
- the global route planner 1020 is to select an appropriate route based on a variety of information, including the self-aware capability parameters 150. In some embodiments, the route planning activities may be triggered based on the route planning trigger signal as shown.
- the global route planner 1020 may gather various types of dynamic information relevant to the present route planning operation. For example, the global route planner 1020 may rely on information related to the source/current location and the destination desired. The planning is performed with respect to the source/current location and the destination. The destination information may be determined in different ways. For example, it may optionally be received from a driver/passenger via an interface unit 1010.
- the global route planner 1020 may also take real time data 480 as input and plans a route accordingly.
- real-time data include information related to real time vehicle related information (position), information about passenger observed in the vehicle, ... , and road characteristics. Such real-time data provide surrounding information needed for the route planning.
- the global route planner 1020 also receives the self-aware capability parameters 510 that inform the planner what is possible given the dynamic intrinsic and extrinsic situations at the time of the planning. For instance, intrinsic capability parameters may indicate that the vehicle is currently not able to run fast due to some mechanical problems so that the global route planner 1020 may take that into account to, e.g., plan a route that involves mainly local roads and may possibly pass some car repair shops.
- extrinsic capability parameters may indicate that in the north direction of the current location of the vehicle, the sun glare is quite severe so that the global route planner may base that information to avoid a nearby route that is in the north direction before the sun is set.
- the real time data 480 and the self-aware capability parameters 510 provide information to the global route planner 1020 to enable it to plan a route that is appropriate given, e.g., the present time, the present location of the vehicle, the present weather, the present passenger’s situation, and present road conditions.
- the global route planner 1020 may also consider preference(s) to be applied in route planning. Such preferences may be specified by a driver/passenger via the user interface unit 1010 (which may be forwarded to the global route planner 1020) or may be obtained via other means (see disclosure below with reference to Fig. 12). In considering the preferences to be applied, information stored in route selection preference configuration 1050 may also be accessed and considered. Such route selection preference configuration may specify some general preference in route selection in different scenarios, e.g., avoid steep/curvy roads in raining/snow scenarios, avoid small streets at night, avoid roads with very few gas stations, etc.
- the global route planner 1020 may forward relevant information received from real-time data 480 and self-aware capability parameters 510 to the route selection preference determiner 1030, which is in turn used by the route selection determiner 1030 to retrieve certain route selection preference configuration from 1050. For example, if it is currently snowing (from real-time data 480) and the vehicle has no snow tire (from the intrinsic capability parameter 510), such dynamic information may be forwarded from the global route planner 1020 to the route selection preference determiner 1030 so that selection preference configuration related to such dynamic scenarios may be retrieved from the route selection preference configuration 1050 (e.g., avoid steep/curvy road) and sends back to the global route planner 1020 so that it can be relied on in selecting an appropriate route.
- the route selection preference determiner 1030 may forward relevant information received from real-time data 480 and self-aware capability parameters 510 to the route selection preference determiner 1030, which is in turn used by the route selection determiner 1030 to retrieve certain route selection preference configuration from 1050. For example, if it is currently snow
- the global route planner 1020 may also need to know additional information about the roads, such as what routes available from the current location of the vehicle to the intended destination.
- the map/road configuration 1060 may also store characteristic information about each of the available roads/routes.
- characteristic information of the roads/routes may include, but not limited to, geometric characteristics such as nature of the roads/routes (highway or not), dimension of the roads/routes, steepness/curviness, condition of the roads/routes, etc.
- the global route planner 1020 may first determine the roads/routes available between the current location of the vehicle to the desired destination. To select an appropriate route to the destination, for such available roads/routes, their characteristic information may also be accessed by the global route planner 1020 so that selections may be made based on such characteristic information.
- the global route planner 1020 may then select an appropriate route to the destination by matching the route selection preferences, determined by the route selection preference determiner 1030, with the
- the global route planner 1020 selects a planned route based on dynamic information from different sources, including real-time data 480 and self-aware capability parameters 510.
- the current location of the vehicle and the destination may also change in time, just like the real-time data 480 and the self-aware capability parameters 510.
- the real-time data associated with the current location may also change, e.g., from good weather associated with the previous spot to the raining condition associated with the current location. This may in turn lead to a change in terms of route selection preferences and ultimately, the route selected.
- the global route planner 1020 may interact with the route selection preference determiner 1030 in a bi-directional manner and dynamic manner. Whenever there is a change that may warrant a re-determination of route selection preferences, the global route planner 1020 may then activate the route selection preference determiner 1030 to modify or re-generate the preferences to be used by the global route planner 1020 to determine an appropriate route given the situation.
- Fig. 11 is a flowchart of an exemplary process for the route planning module 550, according to an embodiment of the present teaching.
- Information about the destination of the vehicle and optionally about the preferences is received at 1110.
- Real-time data 480 as well as self-aware capability parameters 510 are received by the global route planner 1020 at 1120 and various information related to the current scenario or situation of the vehicle may then be identified, at 1130, from the received real-time data and the self-aware capability parameters.
- preferences specific to the current scenario are determined, at 1140.
- the global route planner 1020 accesses, at 1150, information about available roads/routes with respect to the current location and the desired destination as well as the characteristic information of such available roads/routes.
- the global route planner 1020 selects a route appropriate for the current situation.
- Fig. 12 depicts an exemplary high level system diagram of the global route planner 1020, according to an embodiment of the present teaching. In this illustrated
- the global route planner 1020 comprises a self-aware capability parameter analyzer 1205, an intrinsic capability based filter generator 1210, and a route selection engine 1230.
- the global route planner 1020 also comprises a destination updater 1225 for dynamically determine and update the current destination.
- the global route planner 1020 also optionally include a mechanism for personalizing preferences of a driver/passenger so that the route in selecting a route.
- the route selection preference determiner 1030 is to determine the preferences related to selecting a route based on the specific situation the vehicle is currently in, which differs from obtaining personalized preferences directed to a specific driver/passenger.
- the optional mechanism to determine personalized preferences comprises a passenger driving data analyzer 1245, a preference personalization module 1250, and a passenger preference determiner 1240.
- the passenger driving data analyzer 1245 receives recorded human driving data 430 as input and analyzes or learns from such data to understand route preferences relevant to specific drivers/passengers. For example, from the recorded human driving data 430, it may be learned that a specific driver prefers to drive on local roads instead of highway or historically chose to use highways at nights even though it involved much longer distance. It may also learn preferences of all drivers associated with the vehicle. For instance, multiple people (husband, wife, and a child of a household) may be associated with the vehicle, i.e., anyone of these people may operate the vehicle.
- the passenger driving data analyzer 1245 may learn from the recorded human driving data 430 various types of information associated with the driving behavior of such drivers, which may enable the preference personalization module 1250, upon receiving such driving behavior information, to establish personal preferences of each of such individuals.
- the preference personalization module 1250 may then generate personalized preferences in terms of route choices. Such route related preferences may reflect not only route choices but also represent preferences of route choices in different situations such as specific time frames of a day, seasons, locations, etc. Such established preferences for each individual driver may then be stored in storage 1265.
- the passenger preference determiner 1240 receives the real-time data 480 and based on various types of information in the real-time data 480 (e.g., month/day/time, passenger information, locale weather, etc.), the passenger preference determiner 1240 may access, from the route selection preference storage 1265, relevant preferences that can be applied in current route planning.
- the passenger preference determiner 1240 may identify personalized route preferences in 1265 related to the current specific driver which are associated with the specific time frame and season of the year (e.g., a driver may prefer driving on highway in winter season). The personalized route selection preferences so identified may then be sent to the route selection engine 1230 so that personalized preferences of the driver/passenger at the time of the route planning can be considered in determining what route is to be selected.
- the route selection engine 1230 may also take the preferences estimated by the route selection preference determiner 1030 as input and use that in its route selection operation. In some embodiments, the route selection engine 1230 may rely on the preferences from 1030 without considering the personalized preferences of a driver, i.e., it may rely on merely the preferences identified by the route selection preference determiner 1030 in its route selection.
- the route selection engine 1230 may also receiving self-aware capability parameters 510.
- the self-aware capability parameter analyzer 1205 separates extrinsic capability parameters and intrinsic capability parameters and sends the extrinsic capability parameters to the route selection engine 1230 so that extrinsic conditions associated with the current situation the vehicle is in can be considered in selecting a route.
- the extrinsic capability parameters may indicate that there is on-going construction on Route 7, the route selection engine 1230 may consider that and avoid Route 7.
- the route selection engine 1230 may elect to choose Route 7, given all things considered.
- intrinsic capability parameters may also be considered in selecting an appropriate route.
- the intrinsic capability parameters are fed to the intrinsic capability based filter generator 1210, which may create different filters 1215 based on the intrinsic capability parameters so that such filters may be utilized by the route selection engine to filter out routes that are not appropriate given the intrinsic capability parameters. For example, if the intrinsic capability parameters indicate that the vehicle has no snow tire, any routes that are steep and/or curvy may not be appropriate on a snowy day.
- the route selection engine 1230 selects a route based on both the current location of the vehicle, tracked by a current location updater 1235, and a destination, tracked by a destination updater 1225. Depending on the situation, changed current location and destination may trigger the route selection engine 1230 to activate the route selection preference determiner 1030 to re-evaluate the preferences in route selection given the changed circumstance.
- Fig. 13 is a flowchart of an exemplary process for the global route planner 1020, according to an embodiment of the present teaching.
- self-aware capability parameters are received.
- Intrinsic capability parameters are used to generate, at 1320, intrinsic capability based filter(s) so that certain routes can be filtered out as inappropriate given the intrinsic conditions of the vehicle.
- Extrinsic capability parameters are extracted, at 1330, from the received self-aware capability parameters.
- real-time data 480 are continuously received at 1340 and recorded human driving data are received at 1350. Such data are then used to determine, at 1360, personalized route selection preferences relevant to the current driver, current situation, and current time.
- self-aware capability parameters and/or the personalized preferences of the driver may then be used to select a route appropriate given all factors considered.
- the selected route is output.
- the route planning according to the present teaching allows various types of information, such as real-time data and self-aware capability parameters, to be taken into account in route planning so that the planned routes are adaptive with respect to the vehicle condition at the time (via intrinsic capability parameters), the dynamic environment the vehicle is in at the time (via real-time data as well as extrinsic capability parameters), the passenger characteristics determined based on, e.g., dynamically updated real-time data (see Fig. 4B), as well as passenger personalized preferences.
- information may also be utilized in other types of planning operations so that the planned vehicle activities are adapt to the real-time situation, personalized based on individual preferences, and allows the vehicle to behave more human driver like. Below, more details are provided, with respect to Figs. 14-19 on personalized adaptive motion planning.
- human drivers also pay attention to the feedback or reaction of passengers who ride with them in the vehicle and respond to the vehicle motion. For example, some human drivers may prefer start and stop the vehicle smoothly. Some human drivers who usually start and stop the vehicle fairly abruptly may adjust their driving when they observe that passengers sitting in their vehicle respond in a certain way. Such human behavior may play an important role in autonomous vehicles. It is commonly recognized that driving behavior changes from person to person and how such behavior is to be adjusted in the presence of others in the same vehicle may also differ from person to person.
- autonomous vehicles may adopt planning models that are trained to capture the characteristics of human driving behavior of the general population. Such generalized models do not customize the planning approach based on individual driver/passenger preferences or intents.
- the present teaching aims to provide personalized motion planning based on knowledge of the driver/passenger as well as the dynamic observations of driver/passenger response to vehicle motions.
- Fig. 14A depicts an exemplary high level system diagram of the motion planning module 560, according to an embodiment of the present teaching.
- the motion planning module 560 aims at personalized, human-like, and adaptive motion planning, i.e., motions of the vehicle are planned in accordance with, e.g., general and personal likings, which may include what is known to be the preferences of a passenger and what is the reaction or feedback of a of a passenger of the current motion of the vehicle.
- the motion planning module 560 may comprise a generic motion planner 1450 and a passenger motion adapter 1460.
- the motion planning module 560 may plan vehicle motion based on various considerations, including real-time situations the vehicle is in (e.g., on a curvy road, raining day, dim lighting, etc.), vehicle conditions (via intrinsic capability parameters), and personal preferences of the passenger in the vehicle (known preferences or dynamically determined based on driver feedback observed). Given those considerations, vehicle motion may be planned based on motion planning models, which may be invoked in a manner suitable for different scenarios. Motion planning models may include different models appropriate for the given situation in hand.
- Fig. 14B illustrates exemplary types of motion planning models, according to an embodiment of the present teaching.
- motion planning models may include generic motion planning models (1450 in Fig. 14A), sub-category models (1480 in Fig. 14A), or personalized models (1430 in Fig. 14A).
- the generic motion planner 1450 may be preference based models or impact based models (see Fig. 14B).
- a preference based model may be provided specifying preferred vehicle motion in different scenarios based on general knowledge about vehicle operation. For instance, when roads are slippery or icy, it is preferred to plan motions that are slower without sharp turns.
- An impact based model may specify which kind of motion may cause what type of impact and such specification may be used to guide motion planning to achieve or avoid certain impact.
- a sub-category model for motion planning may be, as compared with the generic models, directed to a sub-category of vehicles or a sub-category of drivers/passengers.
- a sub-category model may be directed to sports cars and another sub-category model may be provided for vans.
- a sub-category model may be directed to teenager drivers and another sub-category model may be directed to senior citizens.
- Each sub-category model is tuned or specialized so that the motion planning for a matching sub-category can be performed more accurately.
- motion planning models may also include personalized models which may include individual models, each of which may specify preferences of each individual in terms of vehicle motions.
- a passenger’s individual preference model may specify that the passenger prefers smooth vehicle motion and another passenger’s individual preference model may specify some different preferences.
- Such generic, sub-category and individual models for motion planning may be derived based on recorded human driving data so that the motion planned based on such models are more human like.
- the motion planning module 560 receives various types of information and utilizes different motion planning models.
- the received information includes planned route 520 from the route planning module, surround information 420, real-time data 480, and self-aware capability parameters 510.
- the generic motion planner 1450 determines the situation the vehicle is in (e.g., raining, dark, etc.) and accordingly invokes appropriate generic motion planning models in 1440 to retrieve general motion planning information.
- the generic motion planner 1450 may also determine relevant sub-category of the vehicle and/or passenger so that associated sub-category motion planning models may be retrieved from 1480 and utilized for motion planning.
- the generic motion planning models 1440 may specify general motion planning strategies, e.g., if it is a snowy day or the vehicle is on a curvy road, it is preferred to make the vehicle motion slower and steady.
- Each sub-category model may be provided to specify the generic motion planning strategies for the sub-category (e.g., a type of cars such as sports car or a sub-group of passengers such as senior citizens).
- the motion planned by the generic motion planner 1450 may be further adjusted or adapted according to personalized preferences. In the illustrated embodiment, this is achieved by the passenger motion adapter 1460. There may be different ways to adapt the motion planned to meet personalized preferences.
- personalized preferences may be accessed from individual passenger models 1430. If the identity of the passenger is known, the associated individual passenger model for the passenger may be retrieved from 1430 and the specified preferences in vehicle motion may be used to determine how to achieve personalized motion planning. For instance, an individual model for a particular passenger may indicate that the passenger prefers a smooth ride without taking risks.
- real-time data 480 includes information related to passenger characteristics, which can be passenger condition, ... , and/or passenger reaction cues.
- Passenger condition may refer to mental, physical, and functional state of the passenger.
- Information to be used in personalized motion planning may also include other types of data collected related to the situation.
- a passenger observation analyzer 1420 may collect various types of information and extract relevant indications and then send such indications to the passenger motion adapter 1460 so that such dynamic and personalized information can be taken into account in motion planning. Details about the passenger observation analyzer 1420 are provided with reference to Figs. 15-16.
- Fig. 14C illustrates exemplary types of observations collected for consideration in motion planning, according to an embodiment of the present teaching.
- Observations may include explicit expression from, e.g., the passenger such as voice or text input which may explicitly indicate what the passenger desires. For instance, the passenger may shout “faster!” or“I am going to be late!” or“I really get there on time.” Such explicit expression detected may be relied on in motion planning.
- the observations may also include detected scenarios that may indicate something in terms of motion planning. Scenario information may include event involved (which may indicate the urgency of the passenger is faced with), the current state of the passenger (such as age, known health conditions, etc.), time of day which may imply certain task the passenger has at that time (e.g., pick up children from school) which requires certain safety level. Observations may also include observed physical reactions of the passenger which may be considered as relevant for motion planning.
- sensors inside the vehicle may capture any data that may indicate the emotion of the passenger, the body language of the passenger, or the tone in passenger’s voice, all of which may reflect the desire of the passenger in response to the current vehicle motion.
- the passenger may appear to be uneasy or even show anxiety, which may indicate that the vehicle motion is too rough for the passenger.
- a sharp tone in passenger’s voice may indicate the same.
- Certain physical behavior may also suggest certain reaction from the passenger to the vehicle motion. For example, if the passenger is napping, yawning, looking sleepy, or reading, it may indicate that the passenger is comfortable with the vehicle motion. On the other hand, if it is observed that the passenger keeps checking watch, it may indicate that the passenger feels the vehicle is moving too slow.
- motion planning in addition to personalized motion planning (e.g., not only with respect to sub- categories but also with respect to individuals), motion planning may also be adaptive to the current situation characterized by, e.g., self-aware capability parameters and real-time situations such as weather, road conditions, etc.
- the passenger motion adapter 1460 receives the extrinsic capability parameters from 1410 and plans motion accordingly. For example, if extrinsic capability parameters indicate that there is sun glare or foggy, motion may be planned accordingly (e.g., slow down).
- Fig. 15 depicts an exemplary high level system diagram of the passenger observation analyzer 1420, according to an embodiment of the present teaching.
- the passenger observation analyzer 1420 is provided for obtaining the dynamic preferences of the passenger in terms of vehicle motion so that the motion planning can be adapted to the personal likings.
- the dynamic preferences of the passenger are derived based on analysis of the passenger’s reaction cues to the current vehicle motion, observed via different sensors.
- the exemplary passenger observation analyzer 1420 comprises a sensor activator 1500, a plurality of in-situ sensors 1510, a passenger detector 1520, a passenger feature detector 1540, a visual-based reaction cue estimator 1580, an acoustic-based reaction cue estimator 1590, a passenger expression detector 1560, a passenger scenario detector 1570, and a user reaction generator 1595.
- the passenger observation analyzer 1420 is provided to determine the reaction or feedback of the passenger to the current vehicle motion to determine whether the vehicle motion needs to be adjusted. For example, if passenger reaction indicates that the passenger is not happy about the current vehicle motion, an adjustment may be made in motion planning accordingly.
- the passenger reaction is to be estimated based on different cues, including visual, acoustic, text, or contextual scenarios.
- the sensor activator 1500 activates the in-situ sensors 1510 to detect the passenger reaction.
- the in-situ sensors 1510 comprise a plurality of sensors including visual sensors, acoustic sensors, infrared sensors, ... , or communication sensors and the like that enable the detection of any expression of the passenger.
- the visual sensors included in the in-situ sensors 1510 may comprise a plurality of spatially distributed (within the vehicle) camera devices that are capable of capturing, processing and fusing images of a scene from a variety of viewpoints into some form more useful individual images/videos.
- the visual sensors may capture a gesture or facial expression of the passenger, which may be used to estimate the reaction of the passenger.
- the in-situ sensors may be selectively activated. For instance, at night, in order to observe accurately the passenger’s reaction, visual sensor may not work well and in this situation, infrared sensors may be activated instead.
- the passenger detector 1520 receives sensor data and detects the passenger based on, passenger detection models 1530.
- the detection may be based on either visual or acoustic information.
- the passenger detection models 1530 may include both visual and acoustic models associated with the passenger and can be either individually invoked to detect the passenger based on a single modal data or both invoked to detect the passenger based on both visual and acoustic features.
- the passenger detection models 1530 may include a face recognition model which can be used to detect the passenger based on video or pictorial data from one or more visual sensors.
- the passenger detection models 1530 may also include a speaker based passenger detection model by which the passenger may be recognized based on his/her voice.
- the sensor data may be continuously fed to the passenger feature detector 1540 to detect various passenger behavior features, which may include both visual and acoustic. For instance, certain body language may be detected that may reveal that the passenger is doing certain things, such as sleeping (napping), reading, yawning, or frequently checking his/her watch. Such detected passenger features may also include acoustic features. For instance, the passenger feature detector 1540 may detect that the passenger is saying“slow down.” Visual and acoustic cues may be simultaneously detected that reveal consistent reaction cues. For example, the passenger may constantly check the watch and say“faster!”
- the passenger features may be detected based on visual and acoustic feature detection models 1550.
- Such models may guide the passenger feature detector 1540 in terms of what feature to detect and provide, for each feature to be detected, a corresponding model that can be used to detect the feature.
- Those models may be personalized in the sense that what is to be detected may depend on the passenger. For instance, if the passenger is known to be mute, there is no reason to detect acoustic features associated with the passenger.
- Those feature detection models may be adaptive so that once they are trained and deployed on the vehicle, they may be configured to receive scheduled or dynamic update so that the models are adaptive to the changing situations.
- the detected passenger visual features are then sent to the visual-based reaction cue estimator 1580, which may then estimate the passenger’s reaction cues based on such visual cues. For example, if it is detected that the passenger is checking on the watch, the visual based reaction estimator 1580 may the reaction cue may be that the passenger is not happy with the speed of the vehicle and becomes impatient. Such an estimated cue may also be derived based on, e.g., a personalized visual feature model in 1550, which may be used to determine whether such a behavior (checking watch) is indicative of certain reaction cue associated with this particular passenger (may or may not be depending on the person).
- a personalized visual feature model in 1550 which may be used to determine whether such a behavior (checking watch) is indicative of certain reaction cue associated with this particular passenger (may or may not be depending on the person).
- the detected passenger acoustic features are sent to the acoustic based reaction cue estimator 1590, which may then estimate the passenger’s reaction cue based on such acoustic features. For example, if it is detected that the passenger is snoring, the acoustic based reaction cue estimator 1590 may estimate that the passenger is comfortable with or at least not unhappy with the current vehicle motion. Such an estimated cue may also be derived based on, e.g., a personalized acoustic feature model in 1550, which may be used to determine whether such a behavior is indicative of certain reaction cue of this particular passenger.
- the visual based and acoustic based reaction cue estimators 1580 and 1590 may be engaged in estimating the emotional state of the passenger. For example, from the body language observed from the passenger (e.g., restless or seems to be vomiting), it may indicate that the passenger feels uneasy which may be a cue of his/her reaction to the vehicle motion.
- the tone of the voice used when the passenger says“faster” or“slow down” may also be used to estimate a level of anxiety of the passenger which is a clue as to how unhappy the passenger is.
- Such estimated emotional state may be used in evaluating the severity of the reaction that the passenger exhibits in response to the current vehicle motion and may be used to guide whether and/or how to adjust the motion planning.
- the source of observation may be an input directly entered by the passenger via some communication interface (e.g., a touch screen display) within the vehicle.
- the passenger may input via the display interface in the vehicle that he/she wants the vehicle motion to be smoother. This may be detected by the passenger expression detector 1560 via different communication sensors, which could be textual or acoustic.
- the scenarios that the passenger is currently in may also influence how the motion should be planned.
- the passenger scenario detector 1570 is configured to detect any scenario parameters that may be relevant to motion planning.
- the various passenger reaction cues detected by the visual/acoustic based reaction cue estimators 1580 and 1590, the passenger expression detector 1560, and the passenger scenario detector 1570, are then send to the user reaction generator 1595, where the detected different parameters are selected and integrated to generate estimated user reaction and sent to the passenger motion adapter 1460 so that the motion planned by the generic motion planner 1450 may be adapted in accordance with the observed dynamic user reaction to the current vehicle motion.
- Fig. 16 is a flowchart of an exemplary process for the passenger observation analyzer 1420, according to an embodiment of the present teaching. To gather observations associated with the passenger, appropriate sensors are activated at 1610.
- Information from activated sensors is processed at 1620.
- the passenger is detected, at 1630, based on passenger detection models 1530. Once the identity of the passenger is ascertained, different types of features associated with the passenger may be obtained.
- any explicit expression from the passenger is detected.
- the scenario parameters associated with the passenger are detected at 1660. Such gathered explicit expression from and scenario parameters related to the passengers are then sent to the user reaction generator 1595.
- the estimated user reaction output by the passenger observation analyzer 1420 is sent to the passenger motion adapter 1460 so that the real-time passenger reaction to the current vehicle motion may be considered in determining how to adapt the planned motion based on the dynamic feedback from the passenger.
- Fig. 17 is a flowchart of an exemplary process for the motion planning module 560, according to an embodiment of the present teaching.
- real-time data 480 and the self-aware capability parameters are received, respectively.
- the self-aware capability parameters are processed, at 1730, and split into intrinsic and extrinsic capability parameters.
- the generic motion planner 1450 Based on real time data and the intrinsic capability parameters, the generic motion planner 1450 generates, at 1740, planned motion for the vehicle. Such planned motion is generated based on generic motion planning models 1440 as well as any applicable sub-category motion planning models 1480.
- the generically planned motion may then be adapted based on personalized information, which may include both known personal preferences and dynamically observed passenger reaction to the current vehicle motion.
- personalized information may include both known personal preferences and dynamically observed passenger reaction to the current vehicle motion.
- know passenger preferences are identified, at 1750, based on the individual passenger models 1430.
- dynamic passenger reaction/feedback is estimated, at 1760, based on information gathered from different sources/sensors.
- the personal preferences either already known or estimated dynamically, are then used to personalize the motion planned, e.g., by adapting, at 1770, the motion planned based on generic information.
- Such personalized planned motion is then output, at 1780, as the planned motion 530.
- Fig. 14A various models are used in motion planning, some being generic, some being semi-generic (sub-category models are semi-generic), and some being personalized.
- the motion planning scheme as disclosed herein also aims at behaving in a manner that is more human-like. Being adaptive to the dynamic reaction of the passenger may be part of it.
- the models used by the motion planning module 560 may also be generated to capture human-like behavior so that when they are applied in motion planning, the planned motion 530 will be more human-like.
- Fig. 18 depicts an exemplary high level system diagram of a motion planning model training mechanism 1800 for generating those models, according to an embodiment of the present teaching.
- the motion planning model training mechanism (MPMTM) 1800 comprises data pre-process portion and a model training portion.
- the data pre-processing portion comprises a sub-category training data classifier 1820, an individual training data extractor 1830, and an observation segmenter 1850.
- the model training portion comprises a model training engine 1810 and an independent impact model training engine 1840, and an independent impact model training engine 1840.
- Recorded human driving data 430 are utilized to train models so that the models can capture characteristics related to motion planning that are more human-like.
- the received recorded human driving data are sent to the model training engine 1810 and the trained models are saved as the generic motion planning models 1440.
- the recorded human driving data 430 are classified by the sub-category training data segmenter 1820 into training data sets for the sub-categories and then fed to the model training engine 1810 for training. For each sub category model, appropriate sub-category training data set is applied to derive the corresponding sub-category model and such trained sub-category models are then saved in 1480.
- recorded human driving data may be processed to generate different training sets by the individual training data extractor 1830, each for an individual, and used by the model training engine 1810 to derive individual passenger models that characterize the preferences of the corresponding individuals.
- the individual passenger models 1430 may also include models that characterize impact of vehicle motions on individual passengers observed from the reaction or feedback of passengers. The observed
- reaction/feedback may be positive or negative and can be used to influence how the motion should be planned in the future for passengers.
- Fig. 19 illustrates different types of reactions observed and their roles in model training, according to an embodiment of the present teaching.
- passenger reaction/feedback that can be used to train impact based models may include negative or position impact.
- Negative reaction (negative reinforcement) of a passenger to certain planned motion may be captured in a model so that similar motion may be avoided in the future as to this particular passenger.
- positive reaction to a planned motion or positive reinforcement observed may also be captured in the model for future motion planning.
- Some reaction may be neutral which may also be captured by the individual passenger models.
- the real-time data 480 which capture the passenger characteristics in terms of their behavioral, visual, acoustic cues as well as their conditions (including mental, physical and functional states during vehicle movement), may be segmented based on individuals and such segmented data may then be used to derive models that characterize how certain motions impact passengers.
- the mechanism 1800 includes an observation classifier 1850 that segments the real-time data 480 according to individual passengers and fed such segmented training data sets to the independent impact model training engine 1840 to derive individual impact models. Such derived individual impact models are then stored as part of the individual passenger models 1430.
- the planning module 440 also includes lane planning module 570, which may be directed to lane following and lane changing as illustrated in Fig. 20A.
- Lane following may refer to the behavior to stay in the lane while the vehicle is moving.
- Lane changing may refer to the behavior to move from one lane the vehicle is currently in to an adjacent lane while the vehicle is moving.
- Lane planning may refer to the planning of vehicle behavior in terms of either lane following or lane changing.
- FIG. 20B illustrates exemplary types of behavior related to lane following, according to an embodiment of the present teaching. As illustrated, there are multiple lanes (2010, 2020, and 2030) and a vehicle in each lane may follow its lane.
- the lane following behavior of individual vehicles may differ. For instance, as shown, when a vehicle merely tries to stay in a lane without turning, the vehicle may behave to stay in the middle of the lane. This is shown in Fig. 20B with respect to the vehicles in lane 2010 and 2020. This may be referred to as normal behavior 2040.
- a vehicle in lane 2030 needs to turn to the right, e.g., as shown 2050 in Fig. 20B, the vehicle in lane 2030 may behave differently. For example, instead of remain in the middle of lane 2030, vehicle in that lane may pull to the right side of the lane before the turn so that the turn is, e.g., safer and easier.
- the lane planning module 570 is configured to capture, e.g., via modeling, lane following behavior in different situations so that the autonomous driving vehicle may be controlled in a natural and human-like way.
- lane changing may involve behavior of the vehicle when it moves from one lane to an adjacent lane while the vehicle is moving. Different passengers may exhibit different lane changing behaviors. From safety considerations, there may be desirable lane changing behaviors for different situations. Lane planning in terms of lane changing is to plan the vehicle movement with respect to the lanes in a manner that is safe, natural, human-like, and personalized.
- FIG. 20C illustrates exemplary types of behavior related to lane changing, according to an embodiment of the present teaching. Illustrated are different lane changing behavior, i.e., changing from a current lane 2020 to the lane left to it (lane 2010) and changing from the current lane 2020 to the lane right to it (lane 2030). With respect to the lane changing from lane 2020 to lane 2010, different lane changing behaviors may be characterized in terms of (1) how fast to make the change and (2) in what manner the vehicle is to move to the next lane. For instance, as shown in Fig.
- FIG. 20B there are illustrated three plans to move to the lane 2010, which are left lane changing behavior 1 2060, left lane changing behavior 2 2070, and left lane changing behavior 3 2080, representing respectively different speeds to move to lane 2010.
- behavior 2060 the vehicle is to move to lane 2010 fastest.
- behavior 2080 the vehicle is to move to lane 2010 slowest.
- the speed to move to lane 2010 by behavior 2070 is in the middle.
- lane changing behavior e.g., right lane changing behavior 1 2065, right lane changing behavior 2 2075, and right lane changing behavior 2085, as shown in Fig. 20B.
- the lane changing behavior may also differ in terms of how the vehicle moves into the next lane.
- Fig. 20B when the vehicle is to move from lane 2020 to lane 2010 by employing left lane changing behavior 1 2060, there are different behaviors for the vehicle to adopt to ease into lane 2010, e.g., by following a straight line 2061, by following curve 2062 (cut in first and then straight out the vehicle), or by following curve 2063 (ease towards the edge of lane 2020 first and watch and then cut in when ready). So, with regard to lane changing, decisions as to vehicle behavior may be made at different levels.
- the lane planning module 570 is configured to learn different human behaviors in different circumstances and use such learned knowledge/models to achieve lane planning in autonomous driving.
- the present teaching utilizes lane detection models and lane planning models for lane planning and control. Both models are trained based on large amount of training data, some labeled and some are as collected. For lane detection, lane detection models are obtained using training data with labeled lanes to derive supervised models for lane detection. Such supervised models are to be trained using a large set of training data covering a wide range of environmental conditions to ensure the representativeness and robustness of the trained models.
- lane planning For lane planning, to achieve human-like lane planning behavior, large volume of human driving data are collected and used to train lane control models that, when used for lane planning, are to exhibit human-like behavior in maneuvering the vehicles. Although the lane detection models and the lane planning models are trained separately, in operation, the two sets of models are used in a cascade manner for inference in order to produce robust behavior in diverse types of environment or conditions with human-like operational behavior.
- the present teaching can be configured to further personalize to create personalized human-like lane planning models. With such personalized human-like lane planning models, an autonomous driving vehicle may be operated during lane planning/control in an adaptive manner, depending on who is the passenger in the vehicle.
- Fig. 21 depicts an exemplary high level system diagram of the lane planning module 570, according to an embodiment of the present teaching.
- the lane planning module 570 comprises two model training engines 2110 and 2140 for training lane detection models 2120 and lane planning models 2150, respectively.
- Such trained models are then used, in lane planning, in a cascaded manner by a driving lane detector 2130 and a driving lane planning unit 2160.
- the lane detection models 2120 are supervised models and are trained using training data with labeled lanes.
- Such supervised training data are processed and used by the driving lane detection model training engine 2110 to obtain the driving lane detection models 2120.
- the lane detection models 2120 may correspond to a generic model, capturing the characteristics of lane detection in different situations.
- the lane detection model 2120 may include different models, each of which may be for providing a model to detect lanes in a specific distinct situation.
- some model(s) may be for detecting lanes in normal road conditions, some may be for detecting lanes when the road is wet, some may be for detecting lanes when the road has glare or reflection, some may even be for estimating lanes when the roads are covered with, e.g., snow or other types of visual obstructing objects.
- the lane detection models may also provide separate models for different types of vehicle.
- some vehicles have higher gravity so that cameras capturing the ground image in front of the vehicle may be installed at higher positions relative to the ground.
- the lane detection models for such vehicles may be different from the lane detection models for vehicles with cameras installed at a level closer to the ground level.
- Each type of model may be trained using appropriate labeled training data that are related to the corresponding scenario.
- the driving lane planning model training engine 2140 takes recorded human driving data 430 as input and learns human-like behavior in terms of lane planning. As discussed above, such human driving data may be collected from a wide range of drivers/situations/conditions in order for the driving lane planning model training engine 2140 to learn and capture the characteristics of a wide range of human driving behavior in lane planning/control. In some embodiments, the driving lane planning model training engine 2140 may optionally take some supervised training data with labeled lanes as input, e.g., as seeds or some small set of data to drive the learning towards convergence more quickly.
- the driving lane planning model training engine 2140 may learn and/or train models for both lane following and lane changing.
- a generic model in 2150 for generic human behavior may be derived.
- the lane planning model training engine 2140 may also learn and/or train multiple models for lane planning, each of which may be for different known situations, e.g., lane following or lane changing for specific subgroups of the general population, or for particular different driving environment scenarios (wet road, dark light, crowded road). Such models for subgroups of the general population may also be stored in 2150.
- the human-like lane control models 2150 may also be personalized and stored in 2150.
- lane human driving data that meet the condition associated with each of different model may be extracted and used to train the models.
- lane planning including lane following and lane changing
- models for lane related behavior exhibited when driving on crowded roads may be learned based on human driving data related to lane driving behavior on crowded roads.
- the models for lane planning may also be personalized.
- the driving lane planning model training engine 2140 may derive a model for each individual passenger (e.g., with respect to each of lane following and lane changing) based on the passenger’s past driving data.
- information from a personal profile associated with the passenger may also be used during learning in order to obtain a model that is more accurately reflect the preferences of the passenger.
- Such obtained different types of lane planning/control models may then be stored in the driving lane control model storage 2150.
- different models for different situations may be organized and indexed for easy identification and quick access in real time during the operation of the vehicle.
- the driving lane detection model training engine 2110 and the driving lane planning model training engine 2140 may reside remotely from the vehicle and the learning may be performed in a centralized manner, i.e., they may be operating based on training data from different sources and the learning and update may be activated regularly.
- the trained models may be sent to distributed vehicles.
- personalized models for lane planning may be updated locally in each vehicle based on data acquired locally.
- each training engine may comprise a plurality of sub training engines, each for a specific (set of) models for some specific purposes and each may be configured and implemented differently in order to deriving the most effective models.
- Each training engine (2110 and 2140) may also include, in addition to learning, pre-processing mechanisms (not shown) for process the training data prior to being used by learning mechanism to derive trained models.
- the driving lane planning model training engine 2140 may be configured to derive a generic model for the general population, a personalized model for the driver/passenger of the vehicle, a model for lane planning in day light condition, a model for lane planning in night light condition, a model for lane planning in wet road condition, and a model for lane planning for snowy day condition.
- the pre-processing mechanism may then first group the received recorded human driving data 430 into different groups, each of which for one model planned so that the training engine may then use the appropriate training data group to learn the appropriate model.
- the models may be continuously updated when the new training data arrive. The update of the models may be performed by re learning based on all data received (batch mode) or by incremental mode.
- the models include the lane detection models 2120 and the driving lane control models 2150, are generated, they are used to plan lane related behavior for an autonomous driving vehicle in a human-like manner and in some instances personalized. As discussed previously, in operation, the obtained driving lane detection models 2120 and the driving lane control models 2150 are applied in a cascade manner. In the illustrated
- sensor(s) installed in the vehicle when the vehicle is on the road, sensor(s) installed in the vehicle take
- the driving lane detector 2130 may also receive the self-aware capability parameters 510. Via the self-aware capability parameters, the driving lane detector 2130 may determine various types of information, e.g., road condition, the vehicle’s capabilities, etc., in order to determine how it may proceed in a way that is appropriate. For example, if it is night time of the day, which may be indicated in the extrinsic capability parameters, the driving lane detector 2130 may proceed to invoke a lane detection model that is trained for detecting lanes in dark light situation to achieve reliable performance.
- the driving lane detector 2130 may proceed to invoke a lane detection model that is trained for detecting lanes in dark light situation to achieve reliable performance.
- the driving lane detector 2130 estimates segments of the lanes from the sensor data and optionally the estimated position of the vehicle. Such estimated lane segments and the vehicle position are then sent to the driving lane planning unit 2160, where appropriate driving lane planning model(s) may then be applied in a cascade manner for planning the lane control behavior of the vehicle.
- lane planning includes both lane following and lane changing.
- lane planning is directed to either controlling the vehicle behavior in lane following or the vehicle behavior in lane changing.
- the operation context may provide some indication as to whether lane following or lane changing planning is needed. For instance, if the vehicle needs to exit, it may need first to get into an exit lane from a current lane that does not lead to the exit. In this case, lane changing is implied so that the task involved in lane planning is for lane changing.
- the passenger in the vehicle may also provide an explicit lane control decision to indicate lane changing, e.g., by turning on the turn signal.
- an indication of lane changing may also be from the vehicle itself, e.g., the engine may experience some problem so that the autonomous driving system may send a lane control decision signal to the driving lane planning unit 2160, instructing to prepare for lane changing so that the vehicle can move to the emergency lane.
- the vehicle may assume a default mode of lane following in absence of any indication of entering into a lane changing mode.
- the driving lane planning unit 2160 receives, from different sources, various types of information (e.g., detected lanes, estimated vehicle position, lane planning decision, and self-aware capability parameters 510) and proceeds to lane planning accordingly. For example, if the lane control decision signal indicates that the current task is for lane following, models for lane following are to be retrieved and used for planning. If the current task is for lane changing, then models for lane changing are to be used.
- various types of information e.g., detected lanes, estimated vehicle position, lane planning decision, and self-aware capability parameters 510
- the driving lane planning unit 2160 may invoke the generic lane planning model from 2150 for the planning. It may also invoke different lane planning models that are appropriate for the situation in hand in order to enhance the performance. As discussed earlier, the self-aware capability parameters 510 provide both intrinsic and extrinsic capability parameters, which may indicate the weather condition, road condition, etc. which can be used by the driving lane planning unit 2160 to invoke appropriate lane planning models for the planning.
- personalized human-like models for the passenger in the event of a right turn from the current lane may be retrieved from 2150 and used to plan the vehicle behavior as to how to ease into a position in the current right lane and then make a right turn.
- the driving lane planning unit 2160 may appropriately access lane planning models trained for planning lane changing behavior on very wet roads. In some embodiments, such tasks may also be carried out using generic lane changing models. Based on selected models for the tasks in hand, the driving lane planning unit 2160 generates the planned lane control, which may then be sent to the vehicle control module 450 (Fig. 4A) so that the planned lane control behavior can be implemented.
- the driving lane planning unit 2160 may also perform personalized lane planning.
- the passenger currently present in the vehicle may be known, e.g., either via driver/passenger information sent to the driving lane planning unit 2160 or via detection of the passenger (now shown) from the sensor data.
- the driving lane planning unit 2160 may appropriately invoke lane control models suitable for the passenger.
- Such invoked customized models may be a model for a subgroup that the passenger belongs to or may be a model that is personalized for the passenger. Such customized models may then be used to control how the lane planning is performed in a personalized manner.
- Fig. 22 is a flowchart of an exemplary process for the lane planning module 570, according to an embodiment of the present teaching.
- training data with labeled lanes are first received at 2210 and such received supervised data are then used to obtain, at 2230 via training, driving lane detection models.
- driving lane planning models recorded human driving data, and optionally together with personalized profile information, are received at 2220 and used to derive, at 2240 via training, lane planning models. Once the models are obtained, they may be dynamically updated based on newly arrived training data (now shown).
- the sensors on the vehicle acquire sensor data including imagery of the road ahead of the vehicle with lanes present.
- Such sensor data are received at 2250 and are used to detect, at 2260, lanes in front of the vehicle based on the lane detection models.
- the relative position of the vehicle may also be optionally estimated.
- Such detected lanes and optionally estimated vehicle position may then be sent to the driving lane planning unit 2160.
- various types of information received at 2270 which include lane control decision, detected lanes, and self-aware capability parameters.
- Such information is used to determine the lane planning models to be used so that the lane planning can be achieved, at 2280, based on appropriated selected lane planning models.
- the learned lane planning models capture the characteristics of human behavior in lane planning so that when such models are used in autonomous driving, the vehicle can be controlled in a human-like manner.
- the learned lane planning models capture the characteristics of human behavior in lane planning so that when such models are used in autonomous driving, the vehicle can be controlled in a human-like manner.
- the lane planning behavior of the vehicle can be controlled in a manner that is familiar and comfortable for the passenger/driver in the vehicle.
- the output of the planning module 440 includes the planned route 520 from the route planning module 550, the planned motion 530 from the motion planning module 560, and the planned lane control 540 from the lane planning module 570 (see Fig. 5).
- Such output may be sent to different parts of the autonomous driving vehicle in order to carry out the planned vehicle behavior.
- the planned route 520 may be sent to the part of the vehicle that is responsible for guide the vehicle in terms of route control, e.g., such as the built in GPS.
- the planned motion 530 and the planned lane control 540 may be sent to the vehicle control module 450 (in Fig. 4) so that the planned vehicle behavior as to motion and lane control may be carried out on the vehicle via the vehicle control module 450.
- the vehicle control module 450 aims at delivering the planned action. According to the present teaching, the vehicle control module 450 also aims at learning how to control the vehicle according to the knowledge in terms of how the vehicle behaves or responds to different control signals in different situations so that the vehicle can be controlled to achieve the desired effect, including the planned vehicle behavior.
- Traditional approaches apply machine learning based control and derive vehicle dynamics models from classical mechanics, which often fail to model a variety of situations that occurred in real world. As a consequence, it often leads to poor performance and in some situations, may cause dangerous consequences.
- the present teaching discloses an approach that enables both achieving accurate simulation and safety of the vehicle performance. Instead of directly learn the vehicle dynamics model from the historic data, classical mechanics model is used as backbone model and learn how to adjust the predicted result from the historic data. In addition, limitation to the adjustment to be made is specified as a way to prevent a prediction result that significantly deviates from the normal situations.
- Fig. 23 A depicts a system diagram of traditional approach for generating vehicle control signal.
- a vehicle kinematic model 2310 is provided and used by a vehicle kinematic model (VKM) vehicle control signal generator 2320 based on the target motion and information about the current vehicle state. For example, if the current vehicle state is 30 miles per hour and the target motion is reaching 40 miles per hour in the next 5 seconds, the VKM vehicle control signal generator 2320 uses such information to determine, based on the vehicle kinematic model 2310, what kind of vehicle control is to be applied so that the velocity acceleration can enable the vehicle to achieve the target motion.
- VKM vehicle kinematic model
- the approach as depicted in Fig. 23 A is based on the traditional vehicle kinematic models 2310, which is merely a mechanical dynamic model.
- Fig. 23B depicts a high level system diagram of the vehicle control module 450 in Fig. 4A, according to an embodiment of the present teaching.
- the vehicle control module 450 as disclosed herein aims at providing the ability of generating vehicle control signals that can enable human-like driving behavior for autonomous vehicles.
- the vehicle control module 450 comprises a human-like vehicle control unit 2340 and a human-like vehicle control (HLVC) model 2330.
- HLVC human-like vehicle control
- the human-like vehicle control unit 2340 receives recorded human driving data 430 to uses that for learning human like vehicle control and generate the HLVC model 2330 that characterizes vehicle control behavior that is humanlike.
- the human-like vehicle control unit 2340 when the human-like vehicle control unit 2340 receives information related to a target motion and the current vehicle state, it generates a human-like vehicle control signal based on the HLVC model 2330 with respect to the real time situation associated with the vehicle (characterized by the real time data 480).
- the HLVC model 2330 may be dynamically updated or re-trained so that it captures the characteristics of human vehicle control behavior in a variety of situations.
- the dynamic update of the HLVC model 2330 may be triggered via a model update signal as shown in Fig. 23B.
- the model update signal may be triggered manually or automatically when certain conditions are met, e.g., set up with regular update with a pre-determined internal or when additional data available for update amounts to a certain level.
- the HLVC model 2330 may also be personalized. This is illustrated in Fig. 23 C.
- the HLVC model 2330 in this case may comprise a plurality of HLVC sub-models (e.g., 2330-1, 2330-2, ... , 2330-N), each of which may correspond to a sub population with, e.g., similar characteristics, and trained based on a part of the recorded human driving data 430 that are related to the sub-population.
- a HLVC sub-model may be directed to sub-population for people who may prefer to drive cautiously so that the model may be derived based on training data in the recorded human driving data 430 from the driving record of a corresponding sub-population that exhibit cautious driving record.
- a HLVC sub-model may also be personalized (e.g., a sub-model is for an individual) if appropriate training data are applied to derive a personalized sub-model.
- Fig. 24 depicts an exemplary internal high level architecture of the human-like vehicle control unit 2340, which comprises a human-like vehicle control model generator 2410 and a human like vehicle control signal generator 2420.
- the human like vehicle control model generator 2410 takes recorded human driving data 430 as input and uses that information for learning and training the HLVC model 2330.
- Exemplary types of data extracted from the recorded human driving data 430 for training may include, e.g., the vehicle control data applied to the vehicle and the vehicle states, which may include both the vehicles states prior to and after the vehicle control data are applied.
- Data to be used for deriving the HLVC model 2330 may also include environment data that characterize the surrounding condition under which the vehicle control data yielded the corresponding vehicle state.
- the environment data may include various types of information, e.g., road condition, whether condition, vehicle type and condition.
- the environment data may also include information about the passenger in the vehicle as well as characteristics of the passenger, e.g., gender, age, health situation, preferences, etc. All these different types of information from the human driving data may present some variables that may impact the passenger’s vehicle control behavior. For instance, when the road is wet or slippery, human drivers may exhibit different vehicle control behavior in terms of break the vehicle (e.g., apply pressure on the brake more slowly) than that when the road is not slippery.
- the HLVC model 2330 When the HLVC model 2330 is generated, it can be used by the human like vehicle control signal generator 2420 to generate a vehicle control signal, when it receives a desired target motion, to yield human-like vehicle control behavior in achieving the desired target motion.
- the vehicle control signal generator 2420 obtains real time data 480 comprising information about the surrounding of the vehicle at the time of the desired target motion and use such information in invoking the HLVC model 2330 to generate a human-like vehicle control signal.
- a target motion may be to accelerate, in 5 seconds, the vehicle to 40 miles per hour starting from the current velocity of 30 miles per hour.
- the real time data for that moment may indicate that the road the vehicle is on has a deep slope and the road is slippery because it is currently raining.
- Such real time data is relevant and may be provided as environment data to the HLVC model 2330.
- the human-like vehicle control signal generator 2420 may invoke the HLVC model 2330 with such parameters in order to obtain an inferred human-like vehicle control signal that enables the autonomous vehicle to achieve the desired target motion in a manner similar to human driving.
- Fig. 25 is a flowchart of an exemplary process of the human-like vehicle control unit 2340, according to an embodiment of the present teaching.
- recorded human driving data are received at 2510 to obtain training data, which are used in a training process, at 2520, to derive, at 2530, the HLVC model 2330.
- Such generated HLVC model 2330 is then used in operation when a requested target motion is received, at 2550.
- the human-like vehicle control signal generator 2420 obtains, at 2560, the real time data 480 in order to extract information related to the vehicle at the time, including environment data, current vehicle state, etc.
- information related to the passenger e.g., identification of the passenger or characteristics of the passenger
- the vehicle may also be obtained (not shown) so that such information may be used to personalize the process of generating an appropriate human-like vehicle control signal.
- the information obtained by the human-like vehicle control signal generator 2420 may then be applied to the HLVC model 2330 to generate, at 2570, the human- like vehicle control signal in accordance with the HLVC model 2330.
- the human-like vehicle control unit 2340 may check, at 2540, whether an update triggering signal is present. If an update signal is received, determined at 2540, the human-like vehicle control model generator 2410 proceeds to step 2510 to gather training data and re-train or adaptively adjust the HLVC model 2330 based on the dynamically collected human driving data.
- Fig. 26 depicts an exemplary high level system diagram of the human-like vehicle control model generator 2410, according to an embodiment of the present teaching.
- the human-like vehicle control model generator 2410 comprises a training data processing unit 2610, a VKM vehicle control prediction engine 2630, and a vehicle control model learning engine 2640.
- the training data processing unit 2610 takes the recorded human driving data 430 as input, processes the input to generate training data 2620 to be used to train the HLVC model 2330.
- the recorded human driving data 430 may include environment data 2620-1, vehicle state data 2620-2, ... , and current vehicle control data 2620-3.
- the environment data 2620-1 may include information such as the road condition such as the slope of the road, angle of the turn, slippage of the surface condition, wetness of the road, etc.
- the environment data may also include information related to limitations to the vehicle such as speed limit, time of the day, season, location, etc.
- Such environment data may be used as contextual information in training so that the HLVC model 2330 may learn human like vehicle control behavior in different context conditions such that the trained model characterizes human-like vehicle control behaviors exhibited in different situations. If personalization is needed, the obtained training data may also be classified into different sub-groups of training data (not shown), each of which may be used to train an HLVC sub-model specific for passengers that belong to that group.
- the vehicle state data 2620-2 may include information characterizing the state of the vehicle, including, e.g., position of the vehicle, velocity of the vehicle, roll/pitch/yaw of the vehicle, and steering angle of the vehicle, etc.
- the vehicle control data 2620-3 may provide information characterizing the control applied to the vehicle, such as brake applied with a certain force, steering by turning the steering wheel by a certain angle, or throttle.
- the present teaching rather than training the HLVC model 2330 to generate the vehicle control signal directly, the present teaching combines or fuses the traditional kinematic model based prediction approach with learning model, created by learning from human driving data, as to how to adjust a vehicle control signal predicted using the traditional kinematic model so that the adjustment yields human-like vehicle control behavior.
- learning model created by learning from human driving data
- Such an integrated approach enables not only more accurate vehicle control but also human-like feel as to control of the vehicle.
- the vehicle state data 2620-2 and vehicle control data 2620-3 are provided to the VKM vehicle control prediction engine 2630 to predict the motion achieved because of the control exercised.
- the VKM vehicle control prediction engine 2630 performs the prediction based on based on the vehicle kinematic model 2310, e.g., via traditional mechanical dynamics approach to generate VKM based prediction signal, as shown in Fig. 26.
- the VKM based prediction is then sent to the vehicle control model learning engine 2640 to be combined with other information from the training data 2620 to learn.
- the vehicle control model learning engine 2640 trains the HLVC model 2330 in, e.g., in an iterative process until the HLVC model 2330 converges. Once the model 2330 converges, it is used to derive human-like vehicle control signals given desired target motions.
- the vehicle control model learning engine 2640 may be triggered by the model update signal. When it is activated, the vehicle control model learning engine 2640 invokes the training data processing unit 2610 and the VKM vehicle control prediction engine 2630 to initiate the training process. In some embodiment, any subsequent training based on additional human driving data may be performed in a derivative manner or in a batch mode, i.e., re-train the HLVC model 2330.
- Fig. 27 is a flowchart of an exemplary process for the human-like vehicle control model generator 2410, according to an embodiment of the present teaching.
- Recorded human driving data 430 are first received at 2710.
- the received human driving data are processed, at 2720, to obtain training data.
- Some of the training data are used, at 2730, to generate VKM based prediction based on the traditional vehicle kinematic model 2310.
- various aspects of the training data are identified, at 2740, and used, at 2750, to train the HLVC model 2330. After convergence, the HLVC model 2330 is created at 2760.
- Fig. 28 depicts an exemplary high level system diagram of the human-like vehicle control signal generator 2420, according to an embodiment of the present teaching.
- the human-like vehicle control signal generator 2420 aims to generate, for a specified target motion, a human-like vehicle control signal based on the HLVC model 2330 so that when the human-like vehicle control signal is used to control the vehicle, the vehicle exhibits human-like vehicle control behavior.
- the human-like vehicle control signal generator 2420 comprises a VKM vehicle control signal inference engine 2810, a context data determiner 2820, and an HLVC model based fusion unit 2830.
- the VKM vehicle control signal inference engine 2810 obtains the current state of the vehicle and generates a VKM based vehicle control signal based on the vehicle kinematic model 2310.
- the use of the traditional approach to generate an inferred vehicle control signal based merely on the vehicle kinematic model 2310 aims at providing initially an inferred vehicle control signal based on purely on mechanical dynamics.
- the inferred VKM based vehicle control signal is to be further used as an input to the HLVC model based fusion unit 2830, where the VKM based vehicle control signal is used as the initial inferred result to be fused with the HLVC based approach so that the VKM based vehicle control signal may be adjusted in accordance with the learned HLVC model 2330.
- the HLVC model based fusion unit 2830 may activate, upon receiving the target motion, the context data determiner 2820 to obtain any information related to the surrounding of the vehicle.
- the context data determiner 2820 receives the real time data 480 and extracts relevant information such as environment data or passenger data, etc. and sends to the HLVC model based fusion unit 2830.
- the HLVC model based fusion unit 2830 accesses the HLVC model 2330 based on such input data to obtain a fused human-like vehicle control signal.
- the HLVC model 2330 may be created by learning the discrepancies between VKM model based predictions and the observed information from the recorded human driving data 430. As such, what the HLVC model 2330 captures and learns may correspond to adjustments to be made to the VKM based vehicle control signals to achieve human-like behavior. As discussed previously, as learning process may create overfitting situation, especially when the training data include outliers, to minimize the risks in vehicle control due to adjustment to the VKM based vehicle control signal, the human-like vehicle control signal generator 2420 may also optionally include preventative measures by limiting the adjustments to VKM vehicle control signals based on some fusion constraints 2840, as shown in Fig. 28. In this manner, the human-like vehicle control signal generated as a modified VKM based vehicle control signal can maximize the likelihood as to human behavior yet minimize the risks in vehicle control.
- information about the passenger in the vehicle may also be extracted from real time data 480 and can be used to access personalized HLVC sub model related to the passenger, which may be a HLVC sub-model for a group that the passenger belongs or a completely personalized HLVC sub-model for the passenger).
- personalized HLVC sub-model may allow the human-like vehicle control signal generator 2420 to generate personalized human-like vehicle control signal so that the vehicle control carried out based on it can be not only human-like but also to the personal liking of the passenger.
- Fig. 29 is a flowchart of an exemplary process of the human-like vehicle control signal generator 2420, according to an embodiment of the present teaching.
- Target motion information and the vehicle state data are first received at 2910.
- the vehicle kinematic model 2310 is accessed, at 2920, and used to infer, at 2930, the VKM vehicle control signal.
- Such inferred control signal based on mechanical dynamic model is sent to the HLVC model based fusion unit 2830.
- the context data determiner 2820 receives, at 2940, real time data 480 and extracts, at 2950, relevant information related to the vehicle.
- the HLVC model based fusion unit 2830 uses the context information as well as the VKM vehicle control signal to infer the human-like vehicle control signal based on the HLVC model 2330. Such inferred human like vehicle control signal is then output, at 2970, so that human-like vehicle control to achieve the target motion may be carried out in a human-like manner.
- Fig. 30 depicts the architecture of a mobile device which can be used to realize a specialized system implementing the present teaching.
- This mobile device 3000 includes, but is not limited to, a smart phone, a tablet, a music player, a handled gaming console, a global positioning system (GPS) receiver, and a wearable computing device (e.g., eyeglasses, wrist watch, etc.), or in any other form factor.
- GPS global positioning system
- wearable computing device e.g., eyeglasses, wrist watch, etc.
- the mobile device 3000 in this example includes one or more central processing units (CPUs) 3040, one or more graphic processing units (GPUs) 3030, a memory 3060, a communication platform 3010, such as a wireless communication module, storage 3090, one or more input/output (I/O) devices 3050, a display or a projection 3020-a for visual based presentation, and one or more multi-modal interface channels 3020-b.
- CPUs central processing units
- GPUs graphic processing units
- memory 3060 such as a wireless communication module
- I/O input/output
- I/O input/output
- the multi-modal channels may include acoustic channel or other media channels for signaling or communication. Any other suitable component, including but not limited to a system bus or a controller (not shown), may also be included in the mobile device 3000. As shown in Fig. 30, a mobile operating system 3070, e.g., iOS, Android, Windows Phone, etc., and one or more applications 3080 may be loaded into the memory 3060 from the storage 3090 in order to be executed by the CPU 3040. [00188] To implement various modules, units, and their functionalities described in the present disclosure, computer hardware platforms may be used as the hardware platform(s) for one or more of the elements described herein.
- a mobile operating system 3070 e.g., iOS, Android, Windows Phone, etc.
- applications 3080 may be loaded into the memory 3060 from the storage 3090 in order to be executed by the CPU 3040.
- computer hardware platforms may be used as the hardware platform(s) for one or more of the elements described herein.
- a computer with user interface elements may be used to implement a personal computer (PC) or other type of work station or terminal device, although a computer may also act as a server if appropriately programmed. It is believed that those skilled in the art are familiar with the structure, programming and general operation of such computer equipment and as a result the drawings should be self-explanatory.
- Fig. 31 depicts the architecture of a computing device which can be used to realize a specialized system implementing the present teaching.
- a specialized system incorporating the present teaching has a functional block diagram illustration of a hardware platform which includes user interface elements.
- the computer may be a general purpose computer or a special purpose computer. Both can be used to implement a specialized system for the present teaching.
- This computer 3100 may be used to implement any component or aspect of the present teachings, as described herein. Although only one such computer is shown, for convenience, the computer functions relating to the present teachings as described herein may be implemented in a distributed fashion on a number of similar platforms, to distribute the processing load.
- the computer 3100 for example, includes COM ports 3150 connected to and from a network connected thereto to facilitate data communications.
- the computer 3100 also includes a central processing unit (CPU) 3120, in the form of one or more processors, for executing program instructions.
- the exemplary computer platform includes an internal communication bus 3110, program storage and data storage of different forms, e.g., disk 3170, read only memory (ROM) 3130, or random access memory (RAM) 3140, for various data files to be processed and/or communicated by the computer, as well as possibly program instructions to be executed by the CPU.
- the computer 2600 also includes an I/O component 3160, supporting input/output flows between the computer and other components therein such as interface elements 3180 in different media forms.
- An exemplary type of interface element may correspond to different types of sensors 3180-a deployed on the autonomous driving vehicle.
- Another type of interface element may correspond to a display or a projection 3 l80-b for visual based communication.
- There may be additional components for other multi-modal interface channels such as acoustic device 3180-c for audio based communications and/or component 2680-d for signaling based on communication, e.g., signal that causes vibration on a vehicle component such as a car seat.
- the computer 3100 may also receive programming and data via network communications.
- aspects of the methods of the present teachings may be embodied in programming.
- Program aspects of the technology may be thought of as “products” or“articles of manufacture” typically in the form of executable code and/or associated data that is carried on or embodied in a type of machine readable medium.
- Tangible non-transitory“storage” type media include any or all of the memory or other storage for the computers, processors or the like, or associated modules thereof, such as various semiconductor memories, tape drives, disk drives and the like, which may provide storage at any time for the software programming.
- All or portions of the software may at times be communicated through a network such as the Internet or various other telecommunication networks.
- communications may enable loading of the software from one computer or processor into another, for example, from a management server or host computer of a search engine operator or other enhanced ad server into the hardware platform(s) of a computing environment or other system implementing a computing environment or similar functionalities in connection with the present teachings.
- another type of media that may bear the software elements includes optical, electrical and electromagnetic waves, such as used across physical interfaces between local devices, through wired and optical landline networks and over various air-links.
- the physical elements that carry such waves, such as wired or wireless links, optical links or the like, also may be considered as media bearing the software.
- terms such as computer or machine“readable medium” refer to any medium that participates in providing instructions to a processor for execution.
- a machine-readable medium may take many forms, including but not limited to, a tangible storage medium, a carrier wave medium or physical transmission medium.
- Non-volatile storage media include, for example, optical or magnetic disks, such as any of the storage devices in any computer(s) or the like, which may be used to implement the system or any of its components as shown in the drawings.
- Volatile storage media include dynamic memory, such as a main memory of such a computer platform.
- Tangible transmission media include coaxial cables; copper wire and fiber optics, including the wires that form a bus within a computer system.
- Carrier-wave transmission media may take the form of electric or electromagnetic signals, or acoustic or light waves such as those generated during radio frequency (RF) and infrared (IR) data communications.
- RF radio frequency
- IR infrared
- Common forms of computer-readable media therefore include for example: a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD or DVD-ROM, any other optical medium, punch cards paper tape, any other physical storage medium with patterns of holes, a RAM, a PROM and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave transporting data or instructions, cables or links transporting such a carrier wave, or any other medium from which a computer may read programming code and/or data.
- Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a physical processor for execution.
Landscapes
- Engineering & Computer Science (AREA)
- Automation & Control Theory (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Transportation (AREA)
- Mechanical Engineering (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Theoretical Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Power Engineering (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Social Psychology (AREA)
- Traffic Control Systems (AREA)
Abstract
La présente invention concerne un procédé, un système, un support et une mise en œuvre d'une planification de déplacement automatique dans un véhicule à conduite autonome. Des paramètres de capacité d'auto-positionnement sont obtenus, qui sont instanciés par rapport à un emplacement actuel du véhicule à conduite autonome et doivent être utilisés pour estimer la capacité opérationnelle du véhicule à conduite autonome par rapport à l'emplacement actuel. Une préférence d'un passager présent dans le véhicule à conduite autonome est estimée par rapport au déplacement du véhicule. Le déplacement du véhicule à conduite autonome est planifié automatiquement sur la base des paramètres de capacité d'auto-positionnement et de la préférence du passager.
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201780097496.0A CN111433101A (zh) | 2017-12-18 | 2017-12-18 | 用于自动驾驶车辆中的个性化运动规划的方法和*** |
PCT/IB2017/058085 WO2019122952A1 (fr) | 2017-12-18 | 2017-12-18 | Procédé et système pour planification de mouvement personnalisée dans des véhicules à conduite autonome |
EP17935411.3A EP3727980A4 (fr) | 2017-12-18 | 2017-12-18 | Procédé et système pour planification de mouvement personnalisée dans des véhicules à conduite autonome |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/IB2017/058085 WO2019122952A1 (fr) | 2017-12-18 | 2017-12-18 | Procédé et système pour planification de mouvement personnalisée dans des véhicules à conduite autonome |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2019122952A1 true WO2019122952A1 (fr) | 2019-06-27 |
Family
ID=66993988
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/IB2017/058085 WO2019122952A1 (fr) | 2017-12-18 | 2017-12-18 | Procédé et système pour planification de mouvement personnalisée dans des véhicules à conduite autonome |
Country Status (3)
Country | Link |
---|---|
EP (1) | EP3727980A4 (fr) |
CN (1) | CN111433101A (fr) |
WO (1) | WO2019122952A1 (fr) |
Cited By (24)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112785863A (zh) * | 2020-12-26 | 2021-05-11 | 浙江天行健智能科技有限公司 | 基于K-Means和熵加权的并道决策分类预警方法 |
JP2021195933A (ja) * | 2020-06-18 | 2021-12-27 | トヨタ自動車株式会社 | 機械学習用データ収集装置 |
US11403069B2 (en) | 2017-07-24 | 2022-08-02 | Tesla, Inc. | Accelerated mathematical engine |
US11409692B2 (en) | 2017-07-24 | 2022-08-09 | Tesla, Inc. | Vector computational unit |
US11487288B2 (en) | 2017-03-23 | 2022-11-01 | Tesla, Inc. | Data synthesis for autonomous control systems |
US11537811B2 (en) | 2018-12-04 | 2022-12-27 | Tesla, Inc. | Enhanced object detection for autonomous vehicles based on field view |
US11562231B2 (en) | 2018-09-03 | 2023-01-24 | Tesla, Inc. | Neural networks for embedded devices |
US11561791B2 (en) | 2018-02-01 | 2023-01-24 | Tesla, Inc. | Vector computational unit receiving data elements in parallel from a last row of a computational array |
US11567514B2 (en) | 2019-02-11 | 2023-01-31 | Tesla, Inc. | Autonomous and user controlled vehicle summon to a target |
US11610117B2 (en) | 2018-12-27 | 2023-03-21 | Tesla, Inc. | System and method for adapting a neural network model on a hardware platform |
WO2023057137A1 (fr) * | 2021-10-04 | 2023-04-13 | Bayerische Motoren Werke Aktiengesellschaft | Procédé de manœuvre d'un véhicule sur une route à plusieurs voies tout en tenant compte d'une pente dans une région d'une sortie de la route, système d'assistance au conducteur et véhicule |
US11636333B2 (en) | 2018-07-26 | 2023-04-25 | Tesla, Inc. | Optimizing neural network structures for embedded systems |
US11665108B2 (en) | 2018-10-25 | 2023-05-30 | Tesla, Inc. | QoS manager for system on a chip communications |
US11681649B2 (en) | 2017-07-24 | 2023-06-20 | Tesla, Inc. | Computational array microprocessor system using non-consecutive data formatting |
US11734562B2 (en) | 2018-06-20 | 2023-08-22 | Tesla, Inc. | Data pipeline and deep learning system for autonomous driving |
US11748620B2 (en) | 2019-02-01 | 2023-09-05 | Tesla, Inc. | Generating ground truth for machine learning from time series elements |
US11790664B2 (en) | 2019-02-19 | 2023-10-17 | Tesla, Inc. | Estimating object properties using visual image data |
US11816585B2 (en) | 2018-12-03 | 2023-11-14 | Tesla, Inc. | Machine learning models operating at different frequencies for autonomous vehicles |
US11841434B2 (en) | 2018-07-20 | 2023-12-12 | Tesla, Inc. | Annotation cross-labeling for autonomous control systems |
US11893393B2 (en) | 2017-07-24 | 2024-02-06 | Tesla, Inc. | Computational array microprocessor system with hardware arbiter managing memory requests |
US11893774B2 (en) | 2018-10-11 | 2024-02-06 | Tesla, Inc. | Systems and methods for training machine models with augmented data |
US12014553B2 (en) | 2019-02-01 | 2024-06-18 | Tesla, Inc. | Predicting three-dimensional features for autonomous driving |
EP4162337A4 (fr) * | 2020-06-05 | 2024-07-03 | Gatik Ai Inc | Procédé et système de prise de décision sensible au contexte d'un agent autonome |
US12037011B2 (en) | 2021-12-16 | 2024-07-16 | Gatik Ai Inc. | Method and system for expanding the operational design domain of an autonomous agent |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111923916B (zh) * | 2020-07-24 | 2022-03-29 | 江苏大学 | 一种基于视觉感知行为和anfis的仿人转向建模方法及转向控制*** |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120271500A1 (en) * | 2011-04-20 | 2012-10-25 | GM Global Technology Operations LLC | System and method for enabling a driver to input a vehicle control instruction into an autonomous vehicle controller |
US8634980B1 (en) * | 2010-10-05 | 2014-01-21 | Google Inc. | Driving pattern recognition and safety control |
US20150166069A1 (en) * | 2013-12-18 | 2015-06-18 | Ford Global Technologies, Llc | Autonomous driving style learning |
US20170217428A1 (en) * | 2014-04-02 | 2017-08-03 | Magna Electronics Inc. | Method for controlling a vehicle in accordance with parameters preferred by an identified driver |
US20170267256A1 (en) | 2016-03-15 | 2017-09-21 | Cruise Automation, Inc. | System and method for autonomous vehicle driving behavior modification |
Family Cites Families (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7797108B2 (en) * | 2006-10-19 | 2010-09-14 | Gm Global Technology Operations, Inc. | Collision avoidance system and method of aiding rearward vehicular motion |
CN101011967A (zh) * | 2006-12-22 | 2007-08-08 | 东风汽车有限公司 | 一种驻车警示***及使用该***的机动车 |
CN101596903B (zh) * | 2009-07-07 | 2012-02-15 | 清华大学 | 多功能汽车横向驾驶辅助方法及其辅助*** |
CN104331953B (zh) * | 2014-10-29 | 2016-09-07 | 云南大学 | 一种基于物联网技术的汽车行为数据识别与管理方法 |
-
2017
- 2017-12-18 WO PCT/IB2017/058085 patent/WO2019122952A1/fr unknown
- 2017-12-18 CN CN201780097496.0A patent/CN111433101A/zh active Pending
- 2017-12-18 EP EP17935411.3A patent/EP3727980A4/fr not_active Withdrawn
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8634980B1 (en) * | 2010-10-05 | 2014-01-21 | Google Inc. | Driving pattern recognition and safety control |
US20120271500A1 (en) * | 2011-04-20 | 2012-10-25 | GM Global Technology Operations LLC | System and method for enabling a driver to input a vehicle control instruction into an autonomous vehicle controller |
US20150166069A1 (en) * | 2013-12-18 | 2015-06-18 | Ford Global Technologies, Llc | Autonomous driving style learning |
US20170217428A1 (en) * | 2014-04-02 | 2017-08-03 | Magna Electronics Inc. | Method for controlling a vehicle in accordance with parameters preferred by an identified driver |
US20170267256A1 (en) | 2016-03-15 | 2017-09-21 | Cruise Automation, Inc. | System and method for autonomous vehicle driving behavior modification |
Non-Patent Citations (1)
Title |
---|
See also references of EP3727980A4 |
Cited By (31)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11487288B2 (en) | 2017-03-23 | 2022-11-01 | Tesla, Inc. | Data synthesis for autonomous control systems |
US12020476B2 (en) | 2017-03-23 | 2024-06-25 | Tesla, Inc. | Data synthesis for autonomous control systems |
US11403069B2 (en) | 2017-07-24 | 2022-08-02 | Tesla, Inc. | Accelerated mathematical engine |
US11681649B2 (en) | 2017-07-24 | 2023-06-20 | Tesla, Inc. | Computational array microprocessor system using non-consecutive data formatting |
US11409692B2 (en) | 2017-07-24 | 2022-08-09 | Tesla, Inc. | Vector computational unit |
US11893393B2 (en) | 2017-07-24 | 2024-02-06 | Tesla, Inc. | Computational array microprocessor system with hardware arbiter managing memory requests |
US11561791B2 (en) | 2018-02-01 | 2023-01-24 | Tesla, Inc. | Vector computational unit receiving data elements in parallel from a last row of a computational array |
US11797304B2 (en) | 2018-02-01 | 2023-10-24 | Tesla, Inc. | Instruction set architecture for a vector computational unit |
US11734562B2 (en) | 2018-06-20 | 2023-08-22 | Tesla, Inc. | Data pipeline and deep learning system for autonomous driving |
US11841434B2 (en) | 2018-07-20 | 2023-12-12 | Tesla, Inc. | Annotation cross-labeling for autonomous control systems |
US11636333B2 (en) | 2018-07-26 | 2023-04-25 | Tesla, Inc. | Optimizing neural network structures for embedded systems |
US11983630B2 (en) | 2018-09-03 | 2024-05-14 | Tesla, Inc. | Neural networks for embedded devices |
US11562231B2 (en) | 2018-09-03 | 2023-01-24 | Tesla, Inc. | Neural networks for embedded devices |
US11893774B2 (en) | 2018-10-11 | 2024-02-06 | Tesla, Inc. | Systems and methods for training machine models with augmented data |
US11665108B2 (en) | 2018-10-25 | 2023-05-30 | Tesla, Inc. | QoS manager for system on a chip communications |
US11816585B2 (en) | 2018-12-03 | 2023-11-14 | Tesla, Inc. | Machine learning models operating at different frequencies for autonomous vehicles |
US11908171B2 (en) | 2018-12-04 | 2024-02-20 | Tesla, Inc. | Enhanced object detection for autonomous vehicles based on field view |
US11537811B2 (en) | 2018-12-04 | 2022-12-27 | Tesla, Inc. | Enhanced object detection for autonomous vehicles based on field view |
US11610117B2 (en) | 2018-12-27 | 2023-03-21 | Tesla, Inc. | System and method for adapting a neural network model on a hardware platform |
US12014553B2 (en) | 2019-02-01 | 2024-06-18 | Tesla, Inc. | Predicting three-dimensional features for autonomous driving |
US11748620B2 (en) | 2019-02-01 | 2023-09-05 | Tesla, Inc. | Generating ground truth for machine learning from time series elements |
US11567514B2 (en) | 2019-02-11 | 2023-01-31 | Tesla, Inc. | Autonomous and user controlled vehicle summon to a target |
US11790664B2 (en) | 2019-02-19 | 2023-10-17 | Tesla, Inc. | Estimating object properties using visual image data |
EP4162337A4 (fr) * | 2020-06-05 | 2024-07-03 | Gatik Ai Inc | Procédé et système de prise de décision sensible au contexte d'un agent autonome |
JP7151743B2 (ja) | 2020-06-18 | 2022-10-12 | トヨタ自動車株式会社 | 車両用機械学習システム |
CN114084152B (zh) * | 2020-06-18 | 2024-02-27 | 丰田自动车株式会社 | 机器学习用数据收集装置 |
CN114084152A (zh) * | 2020-06-18 | 2022-02-25 | 丰田自动车株式会社 | 机器学习用数据收集装置 |
JP2021195933A (ja) * | 2020-06-18 | 2021-12-27 | トヨタ自動車株式会社 | 機械学習用データ収集装置 |
CN112785863A (zh) * | 2020-12-26 | 2021-05-11 | 浙江天行健智能科技有限公司 | 基于K-Means和熵加权的并道决策分类预警方法 |
WO2023057137A1 (fr) * | 2021-10-04 | 2023-04-13 | Bayerische Motoren Werke Aktiengesellschaft | Procédé de manœuvre d'un véhicule sur une route à plusieurs voies tout en tenant compte d'une pente dans une région d'une sortie de la route, système d'assistance au conducteur et véhicule |
US12037011B2 (en) | 2021-12-16 | 2024-07-16 | Gatik Ai Inc. | Method and system for expanding the operational design domain of an autonomous agent |
Also Published As
Publication number | Publication date |
---|---|
CN111433101A (zh) | 2020-07-17 |
EP3727980A1 (fr) | 2020-10-28 |
EP3727980A4 (fr) | 2021-08-04 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11643086B2 (en) | Method and system for human-like vehicle control prediction in autonomous driving vehicles | |
US20220185295A1 (en) | Method and system for personalized driving lane planning in autonomous driving vehicles | |
US11650586B2 (en) | Method and system for adaptive motion planning based on passenger reaction to vehicle motion in autonomous driving vehicles | |
US20190187705A1 (en) | Method and system for personalized self capability aware route planning in autonomous driving vehicles | |
EP3727980A1 (fr) | Procédé et système pour planification de mouvement personnalisée dans des véhicules à conduite autonome | |
CN110573978A (zh) | 用于自驾交通工具的动态传感器选择 | |
US11780466B1 (en) | Vehicle fleet remote ride comfort tuning management system | |
US11548515B2 (en) | Systems and methods for managing driver habits | |
EP3729001A1 (fr) | Procédé et système de planification de voie de circulation de type humain dans des véhicules à conduite autonome | |
EP3727979A1 (fr) | Procédé et système de planification d'itinéraire à capacité d'auto-positionnement dans des véhicules à conduite autonome |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 17935411 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
ENP | Entry into the national phase |
Ref document number: 2017935411 Country of ref document: EP Effective date: 20200720 |