WO2018085804A1 - System and method for driver distraction determination - Google Patents
System and method for driver distraction determination Download PDFInfo
- Publication number
- WO2018085804A1 WO2018085804A1 PCT/US2017/060307 US2017060307W WO2018085804A1 WO 2018085804 A1 WO2018085804 A1 WO 2018085804A1 US 2017060307 W US2017060307 W US 2017060307W WO 2018085804 A1 WO2018085804 A1 WO 2018085804A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- driver
- distraction
- vehicle
- distracted state
- output
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60Q—ARRANGEMENT OF SIGNALLING OR LIGHTING DEVICES, THE MOUNTING OR SUPPORTING THEREOF OR CIRCUITS THEREFOR, FOR VEHICLES IN GENERAL
- B60Q9/00—Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00, e.g. haptic signalling
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
- B60W40/08—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to drivers or passengers
- B60W40/09—Driving style or behaviour
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
- B60W50/08—Interaction between the driver and the control system
- B60W50/14—Means for informing the driver, warning the driver or prompting a driver intervention
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/94—Hardware or software architectures specially adapted for image or video understanding
- G06V10/95—Hardware or software architectures specially adapted for image or video understanding structured as a network, e.g. client-server architectures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
- G06V20/597—Recognising the driver's state or behaviour, e.g. attention or drowsiness
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
Definitions
- This invention relates generally to the vehicle interaction field, and more specifically to a new and useful system and method for driver distraction determination in the vehicle interaction field.
- FIGURE 1 is a flowchart representation of the method for driver distraction determination.
- FIGURE 2 is a schematic representation of an example system for driver distraction determination.
- FIGURE 3 is a schematic representation of an example of determination module updating.
- FIGURE 4 is a schematic representation of an example route associated with a plurality of determined driver distraction scores for each of a plurality of locations traversed by the vehicle along the route, as determined for a driving session.
- FIGURE 5 is a schematic depiction of an example information and processing flow path associated with an example implementation of the method.
- FIGURE 6 is a flowchart representation of a specific example implementation of the method.
- FIGURE 7 is a flowchart representation of a specific example implementation of the method. DESCRIPTION OF THE PREFERRED EMBODIMENTS
- the method 100 for driver distraction determination includes: sampling sensor measurements during a driving session Sno; determining distraction factor values based on the sensor measurements S120; and determining driver distraction based on the factor values S130.
- the method 100 functions to determine whether a driver is distracted during a driving session.
- the method 100 can determine the frequency, severity, location, surrounding context (e.g., the concurrent traffic parameters, the concurrent vehicle interior parameters, etc.), and/or any other suitable parameter of the driver distraction event (e.g., driver distraction parameter).
- the method can optionally include transmitting the sensor measurements to a remote computing system S140; generating a notification based on determining driver distraction S150; and, using the driver distraction determination in an application S160.
- All or part of the method 100 is preferably performed in real- or near-real time (e.g., during the driving session, as new sensor measurements are being sampled, etc.), but can alternatively be performed asynchronously (e.g., after the driving session has ended, when a suitable network link for off-vehicle data transmission is available, etc.) or at any other suitable time.
- the method 100 is preferably performed for a plurality of drivers and a plurality of driving sessions, but can alternatively be performed for a single driver, a single driving session, or any suitable population of drivers or driving sessions.
- the method 100 is preferably repeated multiple times throughout a driving session, but can alternatively be performed once per driving session, performed in response to occurrence of a determination event (e.g., when a distraction factor value exceeds a threshold; when a near-collision event occurs; when the user initiates distraction determination; etc.), or be performed at any other suitable time or frequency.
- a determination event e.g., when a distraction factor value exceeds a threshold; when a near-collision event occurs; when the user initiates distraction determination; etc.
- the method 100 can be entirely or partially performed by a system on-board the vehicle (e.g., an onboard system).
- the system can be an auxiliary system retrofitted onto the vehicle, the vehicle itself, a user device, or be any other suitable system.
- the system can include sensors (e.g., optical sensors, audio sensors, stereocamera, stereomicrophone, inertial sensors, accelerometer, gyroscope, magnetometer, etc.), a processing system (e.g., CPU, GPU), outputs (e.g., speaker, vibration mechanism, etc.), a location system (e.g., GPS, cellular trilateration system, etc.), communication systems (e.g., long range connections, such as cellular or WiFi, short range connections, such as BLE or NFC, wired connections such as a vehicle data bus, etc.), a power system (e.g., a vehicle power plug, a battery, etc.), or any other suitable component.
- sensors e.g., optical sensors, audio sensors, stereocamera, stereomicrophone, inertial sensors, accelerometer, gyroscope, magnetometer, etc.
- a processing system e.g., CPU, GPU
- outputs e.g., speaker, vibration mechanism, etc.
- a location system
- the system includes an interior-facing camera (e.g., directed toward the vehicle interior, toward the driver's head volume), an exterior-facing camera (e.g., directed toward the horizon, toward the road, etc.), a set of audio sensors, a set of inertial sensors, a location system, and a mount configured to mount the system to the vehicle dash, windshield, rear view mirror, or any other suitable part of the vehicle.
- the method loo is performed by a system substantially as described in U.S. Application serial number 15/705,043, filed 14-SEP-2017, which is hereby incorporated in its entirety herein by this reference. However, the method can be performed by any other suitable system.
- the relative locations and/or relative field of view relationship (e.g., x, y, z relationship; angular relationship; etc.) between the interior-facing camera and the exterior-facing camera are known and statically coupled, such that information extracted from the images sampled by the interior-facing camera (interior camera) can be spatially mapped to information extracted from the images sampled by the exterior-facing camera (exterior image).
- the driver's gaze direction determined from an interior image can be spatially mapped to physical regions in the vehicle ambient environment that appear in the exterior image.
- the spatial relationship between the sensors can be otherwise used.
- all or parts of the method 100 can be performed by a remote computing system (e.g., a remote server), by a secondary system, or by any other suitable system.
- a remote computing system e.g., a remote server
- sampling the sensor measurements associated with the distraction factors, processing the sensor measurements into distraction factor values, and identifying the distraction event based on the distraction factor values can all be performed by the on-board system, while distraction event verification and application to other processes (e.g., post-processing into a distraction heat map, filtering driving routes for optimal routes, etc.) can be performed by the remote computing system.
- the on-board system can transmit sensor measurements associated with the distraction event (e.g., sensor measurements recorded before, during, and/ or after the distraction event), derivative information associated with the distraction event (e.g., derivative information generated from one or more sensor or other input types), distraction event parameter values (e.g., time, driver, location, severity, etc.), or any other suitable information to the remote computing system: immediately upon distraction determination (e.g., using the communication system), upon connection to a high-bandwidth connection (e.g., upon connection to a home or office WiFi system), or at any other suitable time.
- the sensor measurements can be transmitted in real- or near-real time to the remote computing system, wherein the remote computing system performs the remainder of the method loo.
- the method 100 can be otherwise performed by any other suitable set of systems.
- the detected driver distraction can be used in one or more applications.
- the detected distraction event can be used to filter driving routes (traversal routes) out of a plurality of driving routes (e.g., traversal paths) being considered for optimal route planning, wherein the optimal route can be subsequently recommended to drivers, used to direct autonomous vehicles, or otherwise used.
- a first route or route section associated with a distraction event or a distraction score above a threshold level
- a second route or route section with no associated distraction event or a distraction event below a threshold level
- the detected distraction event or score thereof can be used to dynamically adjust notification parameters.
- the notification can be a notification for an imminent collision event, a notification to alert the driver that they are distracted, or be any other suitable notification.
- the imminent collision event can be an anticipated vehicle collision with an object moving into or already within the vehicle's anticipated traversal path, or be any other suitable collision event.
- the imminent collision event can be determined from images recorded by the external cameras (e.g., using object detection, tracking, etc.), from vehicle sensors (e.g., proximity sensors, RF sensors, etc.), or otherwise detected.
- the notification parameter values can be proportionally increased as a function of the driver distraction score (e.g., calculated for the detection time of the imminent collision event).
- the notification type can be selected based on the driver distraction score (e.g., calculated for the imminent collision event detection time). For example, the driving wheel can be vibrated when the distraction score is below a threshold score, while the vehicle's audio system can be controlled to emit an audio notification when the distraction score rises above the threshold score.
- the notification parameters can be otherwise dynamically adjusted based on the distraction score.
- the detected distraction event or score thereof can be fed into a near-collision determination module that determines whether a near-collision event has occurred.
- the distraction score can be used to determine whether the driver was the cause of a collision, or to determine the percentage of fault assigned to the driver. For example, if the driver was not distracted or had a distraction score below a threshold level during or within a time period preceding the collision, the driver can be assigned a lower fault percentage than if the driver was distracted or had a distraction score above a threshold level during or within the time period preceding the collision.
- the time period can be a constant duration or be variable depending on the collision type.
- the detected distraction event or score thereof can be stored in association with an identifier for the driver (e.g., in a driver profile).
- This driver profile can be used to determine driver-specific notifications, alerts, routes, or any other suitable information.
- the method 100 can include identifying contexts associated with a higher frequency of distraction for a given driver (e.g., based on historic distraction patterns) and automatically determine a route that minimizes the probability of distraction (e.g., minimizes the number of encountered distraction-associated contexts) for the driver (e.g., identified using the method disclosed in US Application Number 15/642,094 filed 05-JUL-2017, incorporated herein in its entirety; or any other suitable method).
- the method 100 can include preemptively notifying the driver when nearing locations historically associated with a distraction event for the driver.
- the method 100 can include providing coaching to a driver based on a historical distraction event data associated with the driver (e.g., providing feedback to the driver based on past behavior to prevent future distracted behavior, at such times as similar behavior patterns are determined via the method or at other suitable times).
- the driver profile can be otherwise suitably used.
- the detected distraction event or score thereof can be otherwise suitably used.
- the score (e.g., distraction score) is preferably determined by a scoring module, but can be otherwise suitably determined.
- the method 100 and/ or Block(s) thereof are preferably implemented by one or more modules (e.g., distraction module, object classification module, object detection module, scoring module, face detection module, any other suitable computing modules, etc.) of a computing system.
- Each module of the plurality can utilize one or more of: supervised learning (e.g., using logistic regression, using back propagation neural networks, using random forests, decision trees, etc.), unsupervised learning (e.g., using an Apriori algorithm, using K-means clustering), semi-supervised learning, reinforcement learning (e.g., using end-to-end learning, using a Q-learning algorithm, using temporal difference learning), and any other suitable learning style.
- supervised learning e.g., using logistic regression, using back propagation neural networks, using random forests, decision trees, etc.
- unsupervised learning e.g., using an Apriori algorithm, using K-means clustering
- semi-supervised learning e.g., using end-to-end learning, using a Q-learning algorithm,
- Each module of the plurality can implement any one or more of: a regression algorithm (e.g., ordinary least squares, logistic regression, stepwise regression, multivariate adaptive regression splines, locally estimated scatterplot smoothing, etc.), an instance-based method (e.g., k- nearest neighbor, learning vector quantization, self-organizing map, etc.), a regularization method (e.g., ridge regression, least absolute shrinkage and selection operator, elastic net, etc.), a decision tree learning method (e.g., classification and regression tree, iterative dichotomiser 3, C4.5, chi-squared automatic interaction detection, decision stump, random forest, multivariate adaptive regression splines, gradient boosting machines, etc.), a Bayesian method (e.g., naive Bayes, averaged one- dependence estimators, Bayesian belief network, etc.), a kernel method (e.g., a support vector machine, a radial basis function, a linear discriminate analysis, etc.), a clustering
- Each module can be generated, validated, verified, reinforced, calibrated, or otherwise updated based on newly received, up-to-date measurements; past measurements recorded during the operating session; historic measurements recorded during past operating sessions; synthetic data; or be updated based on any other suitable data.
- the distraction module and/ or facial recognition module can be trained using a set of synthetic images, which can include a set of real images of driver faces that are overexposed, edited (e.g., to include shadows or lines that simulate shadows from the vehicle frame), warped (e.g., to simulate different camera positions or differences), and/or otherwise modified to simulate different imaging conditions.
- any suitable set of data can be used, such as the data generated using the method disclosed in US Application Number 15/641,982 filed 05-JUL-2017, which is incorporated herein in its entirety by this reference, or any other suitable set of data.
- Each module can be run or updated: once; at a predetermined frequency; every time the method is performed; every time an unanticipated measurement value is received; or at any other suitable frequency.
- the set of modules can be run or updated concurrently with one or more other modules, serially, at varying frequencies, or at any other suitable time.
- Variants of the method 100 can confer various benefits and/or advantages.
- variants of the method can enable potential collisions to be avoided, by alerting (e.g., notifying) a driver that he or she is distracted or is likely to become distracted based on driver behavior.
- the method can include generating an audio alert when the gaze of the driver has drifted (e.g., beyond a threshold angular departure from the direction of travel of the vehicle for greater than a threshold amount of time), which can refocus the driver's attention on the road.
- variants of the method can enable a remote entity (e.g., a fleet manager) to simultaneously monitor the performance (e.g., the distraction levels, states, scores, etc.) of a plurality of drivers.
- a remote entity e.g., a fleet manager
- the performance e.g., the distraction levels, states, scores, etc.
- an onboard system installed in each of a fleet of vehicles can independently detect whether a driver is in a distracted state or an attentive state, and can transmit the state of the driver to a remote computing system associated with the remote entity.
- variants of the method can enable distributed computation for enhanced performance of the computing system and modules thereof.
- the method can include detecting the distraction state of the driver using a computational module of an onboard system, and can transmit the results of the first distraction state detection (and/ or underlying sensor data) to a remote computing system to confirm or refute the initial detection.
- the onboard system can avoid false negative detections (e.g., the onboard system can implement a lower detection threshold for distraction and/or a less complex computational model) and respond rapidly to perceived driver distraction, while the remote computing system can avoid false positive detections that could result in negative repercussions for the driver (e.g., higher insurance premiums, loss of his or her job, etc.).
- variants of the method can enable context-based actions (e.g., notifications).
- the sensor data can be fed into multiple analysis modules, wherein the output of one analysis module can function as a trigger event or as secondary validation for a different analysis module.
- the results of the multiple analysis modules can be fed into a downstream module (e.g., a scoring module, a notification module), wherein the downstream module can dynamically select action parameters (e.g., notification intensity, type, start time, end time, etc.) based on the modules' output values.
- action parameters e.g., notification intensity, type, start time, end time, etc.
- an urgent notification can be quickly presented when a driver is both distracted and tailgating (e.g., outputs from the distraction module and tailgating module, respectively), and a warning notification can be presented only after sustained detected distraction when the driver is distracted and not tailgating.
- notifications can be presented to the driver when a combination of a distraction event and any other relevant traffic situation (e.g., approaching a red or yellow traffic signal, approaching a stop sign, proximity to a pedestrian or vehicle, etc.) is detected.
- the analysis modules can be otherwise used.
- variants of the method 100 can additionally or alternatively confer any suitable benefits and/or advantages.
- the method 100 includes Block Sno, which includes: sampling sensor measurements. Sampling the sensor measurements functions to sample signals indicative of driver distraction for a given time point or time period.
- the sensor measurements are preferably sampled (e.g., recorded) during a driving session, but can alternatively be performed immediately before or after a driving session, substantially before or after a driving session, across multiple driving sessions, or at any other suitable time.
- the sensor measurements can be recorded at a predetermined frequency (e.g., 2-5HZ, 10Hz, 100Hz, etc.), be recorded in response to occurrence of a recordation event (e.g., when the proximity sensors detect proximal objects, when the vehicle velocity exceeds a threshold velocity, etc.), or be recorded at any other suitable time, in response to and/or based on any suitable trigger or event.
- a recordation event e.g., when the proximity sensors detect proximal objects, when the vehicle velocity exceeds a threshold velocity, etc.
- the sensor measurements are preferably recorded by the on-board system, but can alternatively be recorded by any other suitable set of sensors.
- the sensor measurements include an image or video
- Block S110 can include sampling sensor measurements at an onboard system of the vehicle, wherein the sensor measurements include an image of the driver (e.g., Snoa, S110').
- the sensor measurements include an image of the driver (e.g., Snoa, S110').
- the camera's field of view preferably includes all or a portion of the driver's head, and can optionally include all or a portion of the passenger-side volume (e.g., the passenger's body or head), all or a portion of the rear seats, all or a portion of the rear window, the driver's side window(s), the passenger side window(s), or include any other suitable portion of the vehicle interior.
- the image(s) can be used to optically identify and/or track the driver's head position, head movement, eye position (e.g., gaze direction), eye movement, parameters thereof (e.g., duration, angle, etc.), or determine any other suitable distraction factor value.
- the image(s) can additionally be used to determine the driving context, such as whether there is a passenger in the vehicle or where the passenger is located, the environment around the vehicle visible through the vehicle windows, and/or any other suitable context derivable from the image(s).
- the image(s) can additionally be used to verify or validate other distraction factor values. For example, driver interaction with a user device (e.g., phone) can be verified when the user device is present in the concurrently recorded image.
- the image can be a visual spectrum image, hyperspectral image, IR image, or be sampled at any other suitable wavelength.
- the image can be recorded using ambient light, light emitted by a light emitter (e.g., from an LED flash, etc.), or using any other suitable light source.
- the image can be a stereoimage (e.g., recorded by a stereocamera), a single image, or be any other suitable image.
- the sensor measurements include cabin audio (e.g., vehicle interior noise), which can be sampled by the audio sensor arranged proximal the driver's volume (e.g., integrated with the onboard system, part of a user device associated with the driver, etc.) or otherwise sampled.
- the cabin audio can be used to infer the driver head position, the driving context (e.g., whether the driver is calm, whether there is a child in the vehicle, etc.), or be otherwise used.
- the cabin audio preferably includes driver audio, and can optionally include passenger-side audio, rear-seat audio, vehicle audio (e.g., generated by the vehicle), road noise, radio audio, notification audio, or any other suitable audio stream.
- the cabin audio can be a single stream (e.g., recorded by a single audio sensor), multiple streams (e.g., recorded by multiple collocated or distal audio sensors, wherein the cabin audio is recorded in stereo, etc.), or include any suitable number of audio streams.
- the sensor measurements include user device operation parameters, which can be determined by the user device or otherwise determined (example shown in FIGURE 2).
- User device operation parameters can include the user device's: processing load (e.g., as determined from an activity monitor, the CPU load, the GPU load, etc.), processing load changes, power consumption, touch input operation, sensor measurements, or any other suitable set of operation parameters.
- User device sensor measurements can include user device kinematics, pressure, temperature, ambient light obstruction, or any other suitable parameter measurement indicative of driver interaction with the user device.
- the user device operation parameters can be determined by one or more clients (e.g., applications, SDKs on multiple applications, etc.) running on the user device, or be otherwise determined.
- processing load increases can be indicative of user interaction with the user device.
- touch input operation can be indicative of user entering data into the user device.
- user device vibration can be indicative of the user device being knocked by the user.
- user device translation can be indicative of the user device being lifted or picked up by the user.
- an increase in the sampled user device ambient pressure or temperature can be indicative of the user device being held.
- any other suitable user device parameter can be monitored for user interaction with the user device.
- the sensor measurements include vehicle sensor data, which can be determined by the vehicle or otherwise determined (example shown in FIGURE 2).
- vehicle sensor data can be received from the vehicle through the vehicle data bus, wirelessly communicated from the vehicle, or otherwise received from the vehicle.
- vehicle sensor data can include: control panel signals (e.g., whether the radio is being changed, whether the user interface is being used, etc.), steering wheel signals (e.g., number of touches, high pressure regions, high temperature regions, etc. on the steering wheel), seat sensor signals (e.g., driver's weight distribution or change over time, etc.), or any other suitable set of measurements.
- control panel signals e.g., whether the radio is being changed, whether the user interface is being used, etc.
- steering wheel signals e.g., number of touches, high pressure regions, high temperature regions, etc. on the steering wheel
- seat sensor signals e.g., driver's weight distribution or change over time, etc.
- any other suitable set of sensor measurements can be sampled.
- the sensor measurements include an image or video
- the camera's field of view preferably includes all or a portion of the roadway oriented toward the front the vehicle (e.g., the foreground toward the front of the vehicle, the lane in which the vehicle is moving, the adjacent lane or lanes, etc.), and can optionally include the region(s) to the sides of the vehicle (e.g., the driver's side, the passenger's side), to the rear of the vehicle, above the vehicle (e.g., through a sunroof and/or moon-roof), or include any other suitable portion of the environment outside the vehicle.
- the image(s) can be used to optically identify and/ or track the vehicle's position within the environment (e.g., localize the vehicle), position within a lane, position relative to other objects (e.g., vehicles, pedestrians, etc.), or determine any other suitable parameter value.
- the image(s) can additionally be used to determine the driving context, such as whether there is a second vehicle in front of or beside the vehicle in which the sensor is emplaced.
- the image(s) can additionally be used to determine, verify, and/or validate other distraction factor values. For example, driver attentiveness can be measured based on an amount and/ or frequency of lateral drifting within a lane of the roadway.
- Drifting within a lane can be based on vehicle path extraction (e.g., extraction of the vehicle trajectory from the image or images) and comparison to lane markers extracted from images of the road.
- the image can be a visual spectrum image, hyperspectral image, IR image, or be sampled at any other suitable wavelength.
- the image can be recorded using ambient light, light emitted by a light emitter (e.g., from an LED flash, etc.), or using any other suitable light source.
- the image can be a stereoimage (e.g., recorded by a stereocamera), a single image, or be any other suitable image.
- the sensor measurements include a kinematic signal.
- the kinematic signal is preferably indicative of a kinematic variable of the vehicle (e.g., acceleration, velocity, position, jerk, etc.), but can additionally or alternatively be indicative of any suitable motion variable associated with the vehicle and/or vehicle occupants (e.g., the driver, a passenger, etc.).
- the kinematic signal is preferably collected by a kinematic measurement mechanism (e.g., inertial measurement unit) of the onboard system, but can additionally or alternatively be collected by any other suitable component.
- the sensor measurements include an accelerometer output of an inertial measurement unit of the onboard system.
- the sensor measurements can otherwise include any suitable kinematic signal, vehicle information (e.g., steering wheel position, pedal position), indicator, or other information.
- the method 100 includes Block S120, which includes: determining distraction factor values based on the sensor measurements. Determining the distraction factor values functions to characterize the driver's attention, distraction level, or other suitable indicator of driver attention to the set of tasks associated with driving a vehicle. Distraction factors for which values can be determined can include: driver gaze direction, driver eye motion, driver head direction, driver head motion, driver body position, vehicle cabin noise, user device interactions (e.g., phone interactions, vehicle control panel interactions, or any other suitable factor indicative of imminent, instantaneous, or prior driver distraction. One or more distraction factor values can be determined concurrently or asynchronously and used for distraction event detection.
- the distraction factor values are preferably determined based on the sensor measurements, but can additionally or alternatively be retrieved, received (e.g., from a preprocessing system or database), or otherwise determined.
- the distraction factor values can be determined from a single sensor measurement type, determined from multiple sensor measurement types (e.g., using sensor synthesis), or otherwise determined. Additionally or alternatively, the determined factor values can be verified using a second sensor measurement type.
- the distraction factor values are preferably implicitly determined within a model of driver distraction, but can additionally or alternatively be explicitly determined and/ or otherwise computed.
- Block S120 is implemented at least in part as a learning module trained in an "end-to-end" manner, wherein inputs are provided to the module and the output is labeled as either positive (e.g., distracted) or negative (e.g., not distracted), but the features (e.g., distraction factor values) of the model implemented by the learning module are not pre-specified.
- Block S120 can be otherwise suitable implemented as a non-learning module, and/or as a learning module trained in any other suitable manner.
- Each determined distraction factor value can be associated with the recordation time of the underlying sensor measurement, the vehicle location associated with the recordation time (e.g., determined using the system location system, vehicle location system, etc.), the driver associated with the driving session and/or recordation time (e.g., as determined from driver biometrics, the driver's user device, etc.), or with any other suitable concurrent parameter.
- the distraction factor values can be associated with any other suitable time, location, driver, or information.
- the distraction factor value can be determined (e.g., by a distraction detection module, of which an example is shown in FIGURE 2) using equations, regression, classification, neural networks (e.g., convolutional neural networks, deep neural networks), heuristics, selection (e.g., from a library, graph, or chart), instance- based methods (e.g., nearest neighbor), regularization methods (e.g., ridge regression), decision trees, Baysean methods, kernel methods, probability, deterministics, a combination of the above, or any other suitable method.
- neural networks e.g., convolutional neural networks, deep neural networks
- heuristics selection (e.g., from a library, graph, or chart), instance- based methods (e.g., nearest neighbor), regularization methods (e.g., ridge regression), decision trees, Baysean methods, kernel methods, probability, deterministics, a combination of the above, or any other suitable method.
- the distraction factor value can be a binary state (e.g., distracted and not distracted), a score (e.g., a distraction score, etc.), a severity (e.g., a likelihood of a distraction state or event to lead to material loss, a risk factor associated with a distracted state, etc.) or be any other suitable measure of a driver's distraction level.
- a binary state e.g., distracted and not distracted
- a score e.g., a distraction score, etc.
- a severity e.g., a likelihood of a distraction state or event to lead to material loss, a risk factor associated with a distracted state, etc.
- Block S120 can include receiving the sensor measurement (e.g., an image) at a face detection module of the onboard system.
- This example can include generating, at the face detection module, an output, wherein the output is indicative of the presence of a face in the image (e.g., Si2oa, S120').
- the output can be used as a check (e.g., one of a plurality of checks) on an output of the driver distraction module to determine if the driver distraction module has erroneously labeled a distraction event and/ or distraction state.
- the first output indicative of the presence of a face in the image can be otherwise suitably used (e.g., to identify a driver).
- Block S120 can include receiving the sensor measurement (e.g., an image, a sequence of images, etc.) at a driver distraction module of the onboard system.
- This example can include generating an output, at the driver distraction module, indicative of a distracted state (e.g., Si2ob, S120").
- the output of the driver distraction module is preferably generated in parallel to the output of the face detection module (e.g., in real time, in near real time, simultaneously, contemporaneously, within a microsecond interval, within a second interval, etc.), but can additionally or alternatively be generated sequentially (e.g., before or after the output of the face detection module) and/or having any suitable temporal relationship to any suitable generated output of other suitable modules.
- the driver distraction module can operate independently of a face detection module, which can be omitted in some variations.
- determining the distraction factor value can include determining the parameters of the driver's gaze, such as gaze direction, gaze duration, gaze angle (e.g., relative to a reference point), rate of change of gaze direction, or any other suitable gaze parameter.
- This can include identifying the driver's corneal reflectance, pupil, retinal patterns, or other eye parameter in the sampled images or video frames (e.g., the image segment associated with the driver's volume), and determining the driver's gaze direction using gaze estimation, head pose determination, or any other suitable technique.
- the gaze direction can optionally be classified as one of a set of predetermined directions (e.g., forward, right side, left side, rearward, etc.) or otherwise characterized.
- determining the distraction factor values can include determining the driver's head motion. This can include: using object recognition to recognize the driver's head within the image, tracking the head (e.g., recognized object) motion through a series of images (e.g., using object tracking), determining parameters of the head motion (e.g., turn direction, motion duration, etc.). This variant can optionally include determining the current or anticipated head pose based on the head motion parameters.
- determining the distraction factor values can include determining the driver's head pose. This can include: recognizing the driver's head within the image, retrieving a reference image of the driver's head while the driver was looking forward, and determining the driver's head pose based on differences between the driver's head in the new image and the driver's head in the reference image. In one embodiment, this can include: recognizing the driver's head within the image using object recognition, determining the head's pose using head pose determination methods (e.g., iterative closest curve matching), and comparing the determined head pose to the head pose in the reference image.
- head pose determination methods e.g., iterative closest curve matching
- this can include: recognizing the driver's head within the current image using object recognition, identifying one or more reference points on the driver's head (e.g., eyes, nose, moles, freckles, etc.), identifying the corresponding reference points on the reference image, and determining the driver's current head pose based on the difference between the reference points' positions in the current image and the reference image.
- object recognition e.g., object recognition, identifying one or more reference points on the driver's head (e.g., eyes, nose, moles, freckles, etc.), identifying the corresponding reference points on the reference image, and determining the driver's current head pose based on the difference between the reference points' positions in the current image and the reference image.
- the reference image can be a prerecorded image with the driver in the vehicle (e.g., where the driver is instructed to gaze forward during image recordation), a prerecorded image of the driver (e.g., driver's license standard image) superimposed within the driver head volume, an image composited from multiple images recorded over one or more driving sessions (e.g., the average head pose), or be any other suitable reference image.
- This variation can optionally include recognizing objects within the vehicle (e.g., children, passengers, etc.), determining the object's location within the vehicle, and evaluating the driver's interaction with the object.
- a driver's head turn can be classified as a blind spot check with a low distraction score when no passengers are in the vehicle, and but be classified as a child interaction with a high distraction score when a child appears within the camera frame.
- the head pose can be otherwise determined.
- determining the distraction factor values can include determining the driver's head pose based on the cabin acoustics.
- the head pose can be determined from a single acoustic frame, recorded by a set of stereo microphones.
- the method 100 can include: extracting the acoustic signal associated with the driver's voice from the first and second signal from the first and second microphone, respectively (e.g., using pattern matching, noise reduction, etc.); comparing the magnitude or amplitude of the driver's voice between the first and second signals; and determining the driver's head pose as turned toward the microphone that measured the louder voice.
- the head motion can be determined from a set of acoustic frames recorded by a set of stereo microphones.
- a decrease in acoustic intensity sampled by a first microphone paired with an increase in acoustic intensity sampled by a second microphone can be interpreted as a head turn toward the second microphone (e.g., using pattern matching, etc.).
- the cabin acoustics can be used to determine driving context.
- the cabin acoustics can be used to determine whether a passenger is present in the vehicle (e.g., based on detection of a second voice), used to classify the passenger (e.g., as a child, adult, potential distraction, etc.), used to determine the passenger's location, or otherwise used.
- determining the distraction factor values can include determining the degree of user interaction (e.g., frequency, duration, intensity, etc.) with a user device.
- the method 100 can include receiving a user input at the user device (e.g., touch input, haptic input, etc.) and determining that the user has interacted with the user device.
- the method 100 can include determining parameters of the user input (e.g., frequency of inputs; input types, such as targeted inputs or noisy inputs; frequency of application switching; types of applications interacted with; etc.) and determining the degree of user interaction based on the user input parameters.
- targeted user inputs e.g., precise key entries
- high frequencies of user inputs and messaging applications can be associated with a high degree of user interaction
- noisy user inputs low input frequencies
- map applications can be associated with a low degree of user interaction
- the method 100 can include identifying entire or portions of driver body parts (e.g., hands, torso) in the sampled signals (e.g., image(s)), characterizing the body part (e.g., size, position, pose, context, action, etc.), and determining the distraction factor values based on the characterization.
- a high distraction values can be associated with sideways torso orientation (e.g., instead of forward-facing), less than a threshold proportion of hands detected on the steering wheel, or with any other suitable body part characterization.
- Determining the degree of user interaction with the device can optionally include determining the identity of the user interacting with the device.
- the user identity can be determined using the user's biometrics (e.g., fingerprint), as measured by the user device.
- the user identity can be determined using the user device's location within the vehicle during the interaction session.
- the user device's location within the vehicle can be determined from one or more short-range connections established between the user device and one or more secondary endpoint(s) with known locations (e.g., beacons attached to the A-pillars, beacon attached proximal the driver volume).
- the user device location can be determined based on the strength of the connection; for example, the interaction session can be classified as a passenger interaction session (instead of a driver interaction session) if the user device is weakly connected to a driver-side beacon, but strongly connected to a passenger-side beacon.
- vehicle motion can be otherwise filtered from the user device motion.
- the distraction factor values can be determined based on the amount of driver's gaze overlap with an external object or based on the driver's anticipated gaze overlap with the anticipated position of the external object relative to the vehicle.
- the method 100 can include identifying external objects (e.g., from external images), assigning a collision risk to each identified object (e.g., based on the vehicle's kinematics, planned route, etc.), determining the driver's gaze direction (e.g., from images sampled by the internal-facing camera), optionally determining the driver's field of view, and determining a distraction score based on the amount of driver's gaze or field of view overlap with identified objects having collision risks above a threshold risk value.
- the external signals can be otherwise used.
- the distraction factor values can be determined based on the rate at which the object of a driver's gaze changes.
- the distraction score can be increased based on the rate falling below a threshold level, which can be indicative that the driver is not adequately scanning the roadway while operating the vehicle and is instead gazing at a fixed point (e.g., and maybe lost in thought).
- the distraction factor values can be otherwise determined.
- the method 100 can optionally include determining the context of the distraction state, which functions to enable the severity of the distraction state to be characterized.
- determining the context includes determining scaling factor values, which can function to fine-tune the severity of the determined distraction (e.g., determined distraction score). Additionally or alternatively, the scaling factor values can be used as inputs into the distraction determination module or be otherwise used.
- the scaling factor values can be determined based on the vehicle operation parameters (example shown in FIGURE 2).
- Vehicle operation parameters can include: vehicle kinematics (e.g., velocity, acceleration), vehicle location, vehicle notifications, vehicle driving instrument position (e.g., brake position, accelerator position, steering wheel position, wheel position, etc.), or any other suitable operation parameter.
- vehicle operation parameters can be determined by the vehicle, the system, the user device, or any other suitable system.
- the scaling factor values can vary with a collision probability, as determined based on the vehicle operation parameters. However, the scaling factor values can be otherwise determined.
- the distraction score can be scaled as a function of the vehicle kinematics magnitude (e.g., be higher when the vehicle is moving faster, be lower when the vehicle is stopped).
- a different score threshold e.g., used to determine whether the distraction event is recorded, to determine whether a notification should be presented, etc.
- the vehicle kinematics magnitude e.g., the velocity of the vehicle.
- a different scaling value e.g., weight
- the location e.g., geographic location
- the scaling value assigned to the location can be specific to the driver (e.g., determined based on the driver profile), specific to the driver demographic, universal, or otherwise shared.
- the scaling value assigned to the location can be static, dynamic (e.g., vary with traffic proximal the location, traffic upstream from the location, etc.), or otherwise determined.
- the scaling factor values can be determined based on signals indicative of the driving environment (example shown in FIGURE 2). These signals can include external signals (e.g., of the vehicle ambient environment), internal signals (e.g., of the vehicle interior environment), or any other suitable set of signals.
- External signals can include traffic parameters (e.g., traffic density, traffic flow rate, etc.), road type (e.g., highway or urban), traffic signals (e.g., road signs, stop signs, stop lights, etc.), objects or obstructions (e.g., in the anticipated traversal path), weather conditions (e.g., based on a weather forecast or determined in near-real time for the ambient environment), adjacent drivers' profiles, adjacent drivers' distraction levels, or any other suitable external parameter associated with driving risk.
- traffic parameters e.g., traffic density, traffic flow rate, etc.
- road type e.g., highway or urban
- traffic signals e.g., road signs, stop signs, stop lights, etc.
- objects or obstructions e.g., in the anticipated traversal path
- weather conditions e.g., based on a weather forecast or determined in near-real time for the ambient environment
- adjacent drivers' profiles e.g., adjacent drivers' distraction levels, or any other suitable external parameter associated with driving risk.
- Traffic parameters can be crowdsourced (e.g., manually generated by a plurality of users inputting traffic information; automatically generated from a plurality of systems each with an external camera, etc.), automatically determined (e.g., using the images sampled by the external- facing camera), determined from the vehicle's location and historic traffic patterns for the location and the recurrent time period, or otherwise determined.
- the road type can be determined based on a municipal map and the vehicle's location, determined based on the vehicle's velocity (e.g., classified as a highway when the velocity is over 6omph), determined based on features or objects extracted from the external images, or otherwise determined.
- the distraction score can be scaled higher (e.g., the scaling factors can be larger) when the driver is distracted while in heavy traffic.
- the distraction score can be scaled lower when a stop sign or red light appears in the external image (e.g., image sampled by the external-facing camera), and scaled higher when no traffic sign or a green light appears in the external image.
- the distraction score can be scaled higher (e.g., increased in weight) when a stop sign or red light appears in an image of the external environment and the vehicle velocity exceeds a threshold value (e.g., o mph, smph) at a predetermined location relative to the traffic signal (e.g., o ft. away, 5 ft. away, etc.), indicative that the vehicle is entering a scenario requiring enhanced attention.
- the distraction score can additionally or alternatively be scaled higher or lower based on any suitable determination.
- Internal signals can include: passenger presence, passenger type, passenger activity (e.g., all of which can be determined from the sensor measurements or otherwise determined), vehicle kinematics (e.g., rapid starting or stopping), recurrent time (e.g., time of day, month, year), driver calendar, passenger calendar, or any other suitable internal parameter associated with driving risk.
- the internal signals can be received from vehicle sensors, system sensors, user device sensors, secondary user accounts associated with the user account (e.g., a calendar associated with the user account through OAuth), or otherwise determined.
- the scaling factor values can be determined based on vehicle parameters (example shown in FIGURE 2).
- Vehicle parameters can include the vehicle autonomy type (e.g., autonomous, semi-autonomous, manual, etc.), the vehicle class (e.g., sedan, truck, etc.), vehicle ride height, vehicle braking power, vehicle horsepower, vehicle torque, or any other suitable vehicle parameter.
- the vehicle parameters can be received from the user (e.g., entered by the user), automatically determined from a database, automatically determined from the vehicle electronic control unit, automatically determined from the sensor measurements (e.g., from the proportion of the road seen in the external image, by matching vehicle interiors to the vehicle interior from the internal image), or otherwise determined.
- the distraction score can be heavily discounted when the vehicle is a fully-autonomous vehicle, and scaled up when the vehicle is a level o or level 1 SAE automated vehicle class.
- the distraction threshold used for notification presentation, distraction score storage, or for other applications can be selected based on the vehicle parameter values. For example, a first threshold can be selected for a level-o or level-i vehicle, a second threshold higher than the first can be selected for a semi-autonomous vehicle, and a third threshold higher than the second threshold can be selected for an autonomous vehicle. The second threshold can be closer to the first threshold than to the third threshold, but the thresholds can be otherwise distributed. However, the factor values and/ or scaling values can be otherwise determined.
- the method 100 includes Block S130, which includes: determining that the driver of the vehicle is characterized by the distracted state (e.g., determining driver distraction) based on the factor values. Determining driver distraction based on distraction factor values functions to determine a measure of driver distraction for a given time period.
- the driver distraction can be determined (e.g., by a driver distraction module, example shown in FIGURE 2) using equations, regression, classification, neural networks (e.g., convolutional neural networks, deep neural networks), heuristics, selection (e.g., from a library, graph, or chart), instance-based methods (e.g., nearest neighbor), correlation methods, regularization methods (e.g., ridge regression), decision trees, Baysean methods, kernel methods, probability, deterministics, a combination of the above, or any other suitable method.
- neural networks e.g., convolutional neural networks, deep neural networks
- heuristics selection (e.g., from a library, graph, or chart), instance-based methods (e.g., nearest neighbor), correlation methods, regularization methods (e.g., ridge regression), decision trees, Baysean methods, kernel methods, probability, deterministics, a combination of the above, or any other suitable method.
- neural networks e.g., convolutional neural networks
- the driver distraction can be a binary classification or binary score (e.g., distracted or not), a score (e.g., continuous or discontinuous; consistent or inconsistent with a uniform distraction scale; etc.), a classification (e.g., high, medium, low), or be any other suitable measure of distraction.
- the driver distraction can be determined at a predetermined frequency (e.g., every 5 seconds, continuously, etc.), in response to occurrence of a determination event (e.g., in response to obstacle identification in the vehicle path, in response to the driver's gaze being directed outside of a predetermined angular range, etc.), at a frequency selected based on available computing resources, or determined at any other suitable time.
- the driver distraction can be associated with a time or time period, a driver, a location, a driving context, route (e.g., as shown by example in FIGURE 4), or any other suitable parameter value.
- the parameter values associated with driver distraction are preferably extracted from the parameter values associated with the driver distraction factor values used to determine the driver distraction, but can be any other suitable parameter value.
- the time associated with the determined driver distraction is preferably the sampling time for the underlying data, not the driver distraction determination time.
- the time associated with the determined driver distraction can be the driver distraction determination time or be any other suitable time.
- the driver distraction is preferably determined based on distraction factor values, but can be determined based on any other suitable information.
- the distraction factor values used to determine the driver distraction are preferably sampled within a predetermined time period (e.g., within 2s of a reference time, wherein the reference time can be an image sampling time, a location sampling time, a randomly selected time, or be any other suitable time), but can alternatively be otherwise related.
- the driver is preferably deemed distracted (e.g., characterized by a distracted state) after a threshold number of image frames produce distraction factor values indicative of distraction, but can alternatively be deemed distracted after the values from a single frame are indicative of distraction, when the values satisfy a predetermined condition (e.g., surpass a threshold value), or otherwise determined.
- the threshold number of image frames can be predetermined, dynamically determined (e.g., based on other modules' output values, such as the external obstacle proximity, facial detection, vehicle kinematics, etc.), or otherwise determined. For example, the threshold number of frames can decrease with decreased leading vehicle separation and/or increased vehicle velocity.
- the time series of image frames can be contiguous or noncontiguous (e.g., the frames associated with a distracted state can be separated by frames associated with undistracted states), wherein the number or frequency of undistracted frames can be predetermined, dynamically adjusted based on context (e.g., other modules' output values), or otherwise determined.
- the driver distraction module can determine a first driver distraction score when the driver is gazing forward (e.g., as determined from the internal image), and determine a second driver distraction score higher than the first score when the driver is gazing downward or to the side.
- the driver distraction module can further determine a third driver distraction score higher than the second score when the driver is gazing toward the side and a noisy child is in the backseat, or when the driver is gazing away from or has a field of view non-overlapping with an obstruction detected in the vehicle path (e.g., determined from vehicle proximity sensors, the external image, etc.).
- the driver distraction module can determine any other suitable driver distraction score given any other suitable set of inputs.
- the driver distraction module can determine that the driver of the vehicle is characterized by the distracted state based on checking the result against a face detection module output (e.g., contemporaneously generated, asynchronously generated, etc.).
- Block S130 can include determining that the driver of the vehicle is characterized by a distracted state based on a combination of the output of the driver distraction module and the output of the face detection module (e.g., Si3oa, S130').
- the driver distraction module determines whether or not the input data (e.g., image data) is indicative of driver distraction or not, and the face detection module determines whether the input data includes a face or not. For this example, in cases wherein a face is not detected but the driver distraction module determines that the driver is distracted, the output of the driver distraction module is presumed to be a false positive.
- Block S130 of the method 100 can include determining a driver score, which can be distinct from a distraction score (e.g., S130C).
- the driver score is preferably a cumulative score based on a history of distraction scores, determined and recorded during prior driving sessions and a current driving session.
- An individual driver preferably has a single driver score that is associated with the driver, but can additionally or alternatively be associated with multiple driver scores.
- the driver score can be utilized in various Blocks of the method 100 in a similar manner to other scores (e.g., distraction scores) as described above, but can be otherwise suitably utilized.
- Block S130 can, in variations, be partially or entirely implemented at a remote computing system.
- Block S130 can include checking the result of a first distraction determination at a second driver distraction module implemented at the remote computing system, by providing the sensor measurements to the remote computing system.
- Block S130 includes generating, at a driver distraction module of the remote computing system, an output indicative that the driver is characterized by the distracted state, based on the sensor measurements (e.g., Si3ob).
- the onboard driver distraction module can be configured to be computationally faster but less accurate than a remote driver distraction module (e.g., the onboard system can be implemented as a neural network having fewer neurons than the remote system), and the remote driver distraction module can be used to correct false positives generated by the onboard driver distraction module.
- a remote driver distraction module e.g., the onboard system can be implemented as a neural network having fewer neurons than the remote system
- Block S130 can include determining a context of the distracted state (e.g., S130").
- the context can be determined based on sensor measurements as described above, or otherwise suitably determined.
- Block S130 can further include determining a score (e.g., as described above), based on the context of the distracted state (e.g., S130'").
- determining the context of the distracted state can include determining that the traffic level is sparse, and the sparsity of the traffic can be used to weight the score (e.g., reduce the severity of the distraction level due to the low risk of vehicle or personal loss).
- the context can additionally or alternatively be otherwise suitably determined.
- Determining the driver distraction can optionally include selecting a distraction model to use for driver distraction determination. This can be useful when a plurality of distraction models are available for driver distraction determination.
- the distraction model can be selected based on the distraction factor values.
- the distraction model can be selected based on the distraction factor value for a limited subset of distraction factors, such as the driver's gaze direction or the current vehicle location.
- a first distraction model e.g., a correlation model
- a second distraction model e.g., a convolutional neural network or other more complex model
- a more complex model can be selected in the latter instance to handle the increased complexities associated with the underlying cause(s) of the turned head.
- any other suitable model can be selected.
- a first distraction model e.g., a correlation model
- a second distraction model e.g., a convolutional neural network or other more complex model
- the distraction model can be selected based on the numerosity of distraction factors concurrently exceeding their respective thresholds.
- a first distraction model e.g., an equation
- a second model selected when a subset of the distraction factors exceed their thresholds (e.g., the driver's gaze is directed forward and there are no distractions within the vehicle, but there are obstructions in the vehicle path)
- a third model selected when a different number of distraction factors exceed their thresholds (e.g., the driver's gaze is directed backward, there are distractions within the vehicle, and there are obstructions in the vehicle path).
- the distraction model can be otherwise selected.
- the method 100 can optionally include updating the distraction models.
- the distraction models can be updated automatically, manually, or otherwise updated.
- the distraction models can be updated periodically (e.g., at a predetermined frequency), in response to occurrence of an update event (e.g., in response to the determined distraction score differing beyond a threshold difference from an actual distraction score), or at any other suitable time.
- Updating the distraction model can include: retraining the model using labeled training data or otherwise updating the distraction model.
- the training data can be automatically generated, manually generated, or otherwise determined.
- the method 100 can include determining a driver distraction score using a distraction module based on the sensor measurements sampled during a first time period, identifying a distraction-associated event (e.g., a collision, a near-collision, vehicle swerving, sudden braking, etc.) based on subsequent sensor measurements sampled during a second time period after the first time period (e.g., within a predetermined duration of the first time period), determining an actual distraction score based on the distraction-associated event, tagging the sensor measurements sampled during a first time period with the actual distraction score, and calibrating or retraining the distraction module until the determined driver distraction score substantially matches the actual distraction score.
- the distraction module or model can be otherwise updated.
- the method 100 can optionally include transmitting the sensor measurements to a remote computing system S140.
- Block S140 functions to provide the sensor measurements to the remote computing system, where sensor measurements can be aggregated and/or acted upon by additional or alternative modules to those available at the onboard system.
- Block S140 is preferably performed asynchronously with collection of the sensor measurements, and in variations can be performed in response to the connection of the onboard system to a network data link having a predetermined characteristic (e.g., capacity, uplink speed, bandwidth, connection quality, etc.), in response to determination of a distraction score above a threshold distraction score value, or in response to any other suitable event.
- a predetermined characteristic e.g., capacity, uplink speed, bandwidth, connection quality, etc.
- Block S140 can include transmitting the sensor measurements to a remote computing system in response to determining that the driver of the vehicle is characterized by the distracted state (e.g., Si4oa).
- Block S140 can additionally or alternatively be performed in real- or near real-time, substantially real time, simultaneously, contemporaneously, or with any other suitable temporal characteristics relative to sensor measurement.
- the method 100 can optionally include generating a notification based on determining driver distraction S150.
- Block S150 functions to notify an entity (e.g., the driver, a remote fleet manager, a passenger) that the driver has been determined to be distracted.
- the notification can have any suitable format, including a textual format, an audio format, a video format (e.g., a video clip recording during over a time period at which the distraction state is determined), or have any other suitable format.
- Block S150 is preferably performed substantially immediately upon determination of driver distraction and/or a score or other characterization thereof, but can additionally or alternatively be performed asynchronously, in response to a trigger event (e.g., connection to a network link having a suitable characteristic), and/or with any other suitable temporal relationship.
- a trigger event e.g., connection to a network link having a suitable characteristic
- Block S150 can include notifying the driver, at an output of the onboard system of the vehicle and in substantially real-time, based on the distracted state determined in Block S130, the context of the distracted state, and the score of the distracted state (e.g., S150').
- Block S150 can include notifying a fleet manager, at an interface of the remote computing system (e.g., a local PC terminal, a user device associated with the fleet manager and communicatively coupled to the remote computing system, etc., that the driver score of a driver has fallen below a threshold as a result of consecutive determinations of a distracted state within a predetermined period of time (e.g., a day, five minutes, etc.) in response to updating the driver score (e.g., in real time, asynchronously, etc.).
- Block S150 can additionally or alternatively include generating a notification in any suitable manner, and transmitting and/or rendering the notification to any suitable entity in any suitable manner.
- the method can include sampling sensor measurements, wherein the sensor measurements include image data and signal data (e.g., time series signals).
- the method can include receiving the sampled sensor measurements at a plurality of modules, including an object detection and classification module, a face detection module, a distraction module, and a driver behavior module.
- the face detection module determines whether a face is present in an input image (e.g., via a machine learning algorithm, via any suitable image processing technique, Hough transform, etc.) in parallel to determination of driver distraction by the distraction module, based on the sensor measurements received as inputs to the distraction module.
- the output of the face detection module can be used to eliminate false positive outputs of the distraction module (e.g., in cases wherein a face is not detected, an indication that the driver is distracted that is output by the distraction module can be negated).
- the outputs of the object detection & classification module, the distraction module (e.g., checked against the output of the face detection module), and the driver behavior module e.g., that determines driver and/or operator behavior substantially as described in U.S. Application Serial No.
- 15/705,043, filed 14-SEP-2017, which is incorporated herein in its entirety by this reference) can be received at a rule-based comparator, which functions in this variation to determine whether the driver is characterized by a distracted state, and can also function to determine the degree of distraction (e.g., severity, distraction score, etc.) based on the outputs of the modules.
- a rule-based comparator which functions in this variation to determine whether the driver is characterized by a distracted state, and can also function to determine the degree of distraction (e.g., severity, distraction score, etc.) based on the outputs of the modules.
- the transformation of inputs e.g., sensor measurements, outputs of upstream modules, etc.
- outputs is preferably performed continuously and in real- or near-real-time, but can additionally or alternatively be performed at any suitable frequency and/ or with any suitable temporal characteristics (e.g., asynchronously, synchronously, in response to a trigger event, at a frequency of 4 Hz, at any suitable frequency, etc.).
- the method 100 includes sampling sensor measurements at an onboard system of the vehicle, wherein the sensor measurements comprise an image of the driver Snoa; receiving the image at a face detection module of the onboard system, and generating a first output, wherein the first output is indicative of the presence of a face in the image Si2oa; receiving the image at a driver distraction module of the onboard system, and generating a second output contemporaneously with generating the first output, wherein the second output is indicative of a distracted state Si2ob; determining that the driver of the vehicle is characterized by the distracted state, based on a combination of the first output and the second output Si30a; transmitting the sensor measurements to a remote computing system in response to determining that the driver of the vehicle is characterized by the distracted state Si4oa; generating, at a second driver distraction module of the remote computing system, a third output indicative that the driver is characterized by the distracted state, based on the sensor measurements Si3ob; computing a
- the method 100 includes sampling sensor measurements at an onboard system of the vehicle, wherein the sensor measurements comprise an image of the driver S110'; receiving the image at a face detection module of the onboard system, and generating a first output, wherein the first output is indicative of the presence of a face in the image S120'; receiving the image at a driver distraction module of the onboard system, and generating a second output contemporaneously with generating the first output, wherein the second output is indicative of a distracted state S120"; determining that the driver of the vehicle is characterized by the distracted state, based on a combination of the first output and the second output S130'; determining a context of the distracted state S130"; determining a score of the distracted state based on the context of the distracted state S130'"; notifying the driver, at an output of the onboard system of the vehicle and in substantially realtime, based on the distracted state, the context of the distracted state, and the score of the distracted state S150
- Embodiments of the system and/ or method can include every combination and permutation of the various system components and the various method processes, wherein the method processes can be performed in any suitable order, sequentially or concurrently.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Mechanical Engineering (AREA)
- Automation & Control Theory (AREA)
- Ophthalmology & Optometry (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Transportation (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Traffic Control Systems (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
Abstract
A method for determining distraction of a driver of a vehicle, including sampling sensor measurements at an onboard system of the vehicle; generating an output indicative of a distracted state; determining that the driver of the vehicle is characterized by the distracted state, based on the output; generating, at a second distraction detection module of a remote computing system, a second output indicative that the driver is characterized by the distracted state, based on the sensor measurements; computing a distraction score, at a scoring module of the remote computing system, in response to generating the second output and based on the sensor measurements and the distracted state.
Description
SYSTEM AND METHOD FOR DRIVER DISTRACTION DETERMINATION
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims the benefit of U.S. Provisional Application serial number 62/418,655, filed 07-NOV-2016, which is incorporated herein in its entirety by this reference.
TECHNICAL FIELD
[0002] This invention relates generally to the vehicle interaction field, and more specifically to a new and useful system and method for driver distraction determination in the vehicle interaction field.
BRIEF DESCRIPTION OF THE FIGURES
[0003] FIGURE 1 is a flowchart representation of the method for driver distraction determination.
[0004] FIGURE 2 is a schematic representation of an example system for driver distraction determination.
[0005] FIGURE 3 is a schematic representation of an example of determination module updating.
[0006] FIGURE 4 is a schematic representation of an example route associated with a plurality of determined driver distraction scores for each of a plurality of locations traversed by the vehicle along the route, as determined for a driving session.
[0007] FIGURE 5 is a schematic depiction of an example information and processing flow path associated with an example implementation of the method.
[0008] FIGURE 6 is a flowchart representation of a specific example implementation of the method.
[0009] FIGURE 7 is a flowchart representation of a specific example implementation of the method.
DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] The following description of the preferred embodiments of the invention is not intended to limit the invention to these preferred embodiments, but rather to enable any person skilled in the art to make and use this invention,
l. Overview.
[0011 ] As shown in FIGURE l, the method 100 for driver distraction determination includes: sampling sensor measurements during a driving session Sno; determining distraction factor values based on the sensor measurements S120; and determining driver distraction based on the factor values S130. The method 100 functions to determine whether a driver is distracted during a driving session. In some variations, the method 100 can determine the frequency, severity, location, surrounding context (e.g., the concurrent traffic parameters, the concurrent vehicle interior parameters, etc.), and/or any other suitable parameter of the driver distraction event (e.g., driver distraction parameter). The method can optionally include transmitting the sensor measurements to a remote computing system S140; generating a notification based on determining driver distraction S150; and, using the driver distraction determination in an application S160.
[0012] All or part of the method 100 is preferably performed in real- or near-real time (e.g., during the driving session, as new sensor measurements are being sampled, etc.), but can alternatively be performed asynchronously (e.g., after the driving session has ended, when a suitable network link for off-vehicle data transmission is available, etc.) or at any other suitable time. The method 100 is preferably performed for a plurality of drivers and a plurality of driving sessions, but can alternatively be performed for a single driver, a single driving session, or any suitable population of drivers or driving sessions. The method 100 is preferably repeated multiple times throughout a driving session, but can alternatively be performed once per driving session, performed in response to occurrence of a determination event (e.g., when a distraction factor value exceeds a threshold; when a near-collision event occurs; when the user initiates distraction determination; etc.), or be performed at any other suitable time or frequency.
[0013] The method 100 can be entirely or partially performed by a system on-board the vehicle (e.g., an onboard system). The system can be an auxiliary system retrofitted onto the vehicle, the vehicle itself, a user device, or be any other suitable system. The system can include sensors (e.g., optical sensors, audio sensors, stereocamera, stereomicrophone, inertial sensors, accelerometer, gyroscope, magnetometer, etc.), a processing system (e.g., CPU, GPU), outputs (e.g., speaker, vibration mechanism, etc.), a location system (e.g., GPS, cellular trilateration system, etc.), communication systems (e.g., long range connections, such as cellular or WiFi, short range connections, such as BLE or NFC, wired connections such as a vehicle data bus, etc.), a power system (e.g., a vehicle power plug, a battery, etc.), or any other suitable component. In one variation, the system includes an interior-facing camera (e.g., directed toward the vehicle interior, toward the driver's head volume), an exterior-facing camera (e.g., directed toward the horizon, toward the road, etc.), a set of audio sensors, a set of inertial sensors, a location system, and a mount configured to mount the system to the vehicle dash, windshield, rear view mirror, or any other suitable part of the vehicle. In another variation, the method loo is performed by a system substantially as described in U.S. Application serial number 15/705,043, filed 14-SEP-2017, which is hereby incorporated in its entirety herein by this reference. However, the method can be performed by any other suitable system. In one example, the relative locations and/or relative field of view relationship (e.g., x, y, z relationship; angular relationship; etc.) between the interior-facing camera and the exterior-facing camera are known and statically coupled, such that information extracted from the images sampled by the interior-facing camera (interior camera) can be spatially mapped to information extracted from the images sampled by the exterior-facing camera (exterior image). In a specific example, the driver's gaze direction determined from an interior image can be spatially mapped to physical regions in the vehicle ambient environment that appear in the exterior image. However, the spatial relationship between the sensors can be otherwise used.
[0014] Additionally or alternatively, all or parts of the method 100 can be performed by a remote computing system (e.g., a remote server), by a secondary system, or by any other suitable system. In one variation of split computing, sampling the sensor
measurements associated with the distraction factors, processing the sensor measurements into distraction factor values, and identifying the distraction event based on the distraction factor values (e.g., the imminent, occurring, or previously occurred distraction event) can all be performed by the on-board system, while distraction event verification and application to other processes (e.g., post-processing into a distraction heat map, filtering driving routes for optimal routes, etc.) can be performed by the remote computing system. In this variation, the on-board system can transmit sensor measurements associated with the distraction event (e.g., sensor measurements recorded before, during, and/ or after the distraction event), derivative information associated with the distraction event (e.g., derivative information generated from one or more sensor or other input types), distraction event parameter values (e.g., time, driver, location, severity, etc.), or any other suitable information to the remote computing system: immediately upon distraction determination (e.g., using the communication system), upon connection to a high-bandwidth connection (e.g., upon connection to a home or office WiFi system), or at any other suitable time. In a second variation, the sensor measurements can be transmitted in real- or near-real time to the remote computing system, wherein the remote computing system performs the remainder of the method loo. However, the method 100 can be otherwise performed by any other suitable set of systems.
[0015] The detected driver distraction (distraction event, distraction state) can be used in one or more applications. In a first application, the detected distraction event can be used to filter driving routes (traversal routes) out of a plurality of driving routes (e.g., traversal paths) being considered for optimal route planning, wherein the optimal route can be subsequently recommended to drivers, used to direct autonomous vehicles, or otherwise used. For example, a first route or route section associated with a distraction event (or a distraction score above a threshold level) can be removed from consideration, while a second route or route section with no associated distraction event (or a distraction event below a threshold level) can be retained for optimal route determination.
[0016] In a second application, the detected distraction event or score thereof can be used to dynamically adjust notification parameters. The notification can be a
notification for an imminent collision event, a notification to alert the driver that they are distracted, or be any other suitable notification. The imminent collision event can be an anticipated vehicle collision with an object moving into or already within the vehicle's anticipated traversal path, or be any other suitable collision event. The imminent collision event can be determined from images recorded by the external cameras (e.g., using object detection, tracking, etc.), from vehicle sensors (e.g., proximity sensors, RF sensors, etc.), or otherwise detected. In one variation, the notification parameter values (e.g., volume, brightness, duration, etc.) can be proportionally increased as a function of the driver distraction score (e.g., calculated for the detection time of the imminent collision event). In a second variation, the notification type can be selected based on the driver distraction score (e.g., calculated for the imminent collision event detection time). For example, the driving wheel can be vibrated when the distraction score is below a threshold score, while the vehicle's audio system can be controlled to emit an audio notification when the distraction score rises above the threshold score. However, the notification parameters can be otherwise dynamically adjusted based on the distraction score.
[0017] In a third application, the detected distraction event or score thereof can be fed into a near-collision determination module that determines whether a near-collision event has occurred. In a fourth application, the distraction score can be used to determine whether the driver was the cause of a collision, or to determine the percentage of fault assigned to the driver. For example, if the driver was not distracted or had a distraction score below a threshold level during or within a time period preceding the collision, the driver can be assigned a lower fault percentage than if the driver was distracted or had a distraction score above a threshold level during or within the time period preceding the collision. The time period can be a constant duration or be variable depending on the collision type.
[0018] In a fifth application, the detected distraction event or score thereof can be stored in association with an identifier for the driver (e.g., in a driver profile). This driver profile can be used to determine driver-specific notifications, alerts, routes, or any other suitable information. In one variation, the method 100 can include identifying contexts associated with a higher frequency of distraction for a given driver (e.g., based on historic
distraction patterns) and automatically determine a route that minimizes the probability of distraction (e.g., minimizes the number of encountered distraction-associated contexts) for the driver (e.g., identified using the method disclosed in US Application Number 15/642,094 filed 05-JUL-2017, incorporated herein in its entirety; or any other suitable method). In a second variation, the method 100 can include preemptively notifying the driver when nearing locations historically associated with a distraction event for the driver. In a third variation, the method 100 can include providing coaching to a driver based on a historical distraction event data associated with the driver (e.g., providing feedback to the driver based on past behavior to prevent future distracted behavior, at such times as similar behavior patterns are determined via the method or at other suitable times). However, the driver profile can be otherwise suitably used. Furthermore, the detected distraction event or score thereof can be otherwise suitably used. In this and related applications, the score (e.g., distraction score) is preferably determined by a scoring module, but can be otherwise suitably determined.
[0019] The method 100 and/ or Block(s) thereof are preferably implemented by one or more modules (e.g., distraction module, object classification module, object detection module, scoring module, face detection module, any other suitable computing modules, etc.) of a computing system. Each module of the plurality can utilize one or more of: supervised learning (e.g., using logistic regression, using back propagation neural networks, using random forests, decision trees, etc.), unsupervised learning (e.g., using an Apriori algorithm, using K-means clustering), semi-supervised learning, reinforcement learning (e.g., using end-to-end learning, using a Q-learning algorithm, using temporal difference learning), and any other suitable learning style. Each module of the plurality can implement any one or more of: a regression algorithm (e.g., ordinary least squares, logistic regression, stepwise regression, multivariate adaptive regression splines, locally estimated scatterplot smoothing, etc.), an instance-based method (e.g., k- nearest neighbor, learning vector quantization, self-organizing map, etc.), a regularization method (e.g., ridge regression, least absolute shrinkage and selection operator, elastic net, etc.), a decision tree learning method (e.g., classification and regression tree, iterative dichotomiser 3, C4.5, chi-squared automatic interaction
detection, decision stump, random forest, multivariate adaptive regression splines, gradient boosting machines, etc.), a Bayesian method (e.g., naive Bayes, averaged one- dependence estimators, Bayesian belief network, etc.), a kernel method (e.g., a support vector machine, a radial basis function, a linear discriminate analysis, etc.), a clustering method (e.g., k-means clustering, expectation maximization, etc.), an associated rule learning algorithm (e.g., an Apriori algorithm, an Eclat algorithm, etc.), an artificial neural network model (e.g., a Perceptron method, a back-propagation method, a Hopfield network method, a self-organizing map method, a learning vector quantization method, etc.), a deep learning algorithm (e.g., a restricted Boltzmann machine, a deep belief network method, a convolutional network method, a stacked auto-encoder method, etc.), a dimensionality reduction method (e.g., principal component analysis, partial least squares regression, Sammon mapping, multidimensional scaling, projection pursuit, etc.), an ensemble method (e.g., boosting, bootstrapped aggregation, AdaBoost, stacked generalization, gradient boosting machine method, random forest method, etc.), and any suitable form of machine learning algorithm. Each module can additionally or alternatively be a: probabilistic module, heuristic module, deterministic module, or be any other suitable module leveraging any other suitable computation method, machine learning method, or combination thereof.
Each module can be generated, validated, verified, reinforced, calibrated, or otherwise updated based on newly received, up-to-date measurements; past measurements recorded during the operating session; historic measurements recorded during past operating sessions; synthetic data; or be updated based on any other suitable data. For example, the distraction module and/ or facial recognition module can be trained using a set of synthetic images, which can include a set of real images of driver faces that are overexposed, edited (e.g., to include shadows or lines that simulate shadows from the vehicle frame), warped (e.g., to simulate different camera positions or differences), and/or otherwise modified to simulate different imaging conditions. However, any suitable set of data can be used, such as the data generated using the method disclosed in US Application Number 15/641,982 filed 05-JUL-2017, which is incorporated herein in its entirety by this reference, or any other suitable set of data. Each module can be run or
updated: once; at a predetermined frequency; every time the method is performed; every time an unanticipated measurement value is received; or at any other suitable frequency. The set of modules can be run or updated concurrently with one or more other modules, serially, at varying frequencies, or at any other suitable time.
2. Benefits
[0020] Variants of the method 100 can confer various benefits and/ or advantages.
[0021] First, variants of the method can enable potential collisions to be avoided, by alerting (e.g., notifying) a driver that he or she is distracted or is likely to become distracted based on driver behavior. For example, the method can include generating an audio alert when the gaze of the driver has drifted (e.g., beyond a threshold angular departure from the direction of travel of the vehicle for greater than a threshold amount of time), which can refocus the driver's attention on the road.
[0022] Second, variants of the method can enable a remote entity (e.g., a fleet manager) to simultaneously monitor the performance (e.g., the distraction levels, states, scores, etc.) of a plurality of drivers. For example, an onboard system installed in each of a fleet of vehicles can independently detect whether a driver is in a distracted state or an attentive state, and can transmit the state of the driver to a remote computing system associated with the remote entity.
[0023] Third, variants of the method can enable distributed computation for enhanced performance of the computing system and modules thereof. For example, the method can include detecting the distraction state of the driver using a computational module of an onboard system, and can transmit the results of the first distraction state detection (and/ or underlying sensor data) to a remote computing system to confirm or refute the initial detection. Thus, in the aforementioned example, the onboard system can avoid false negative detections (e.g., the onboard system can implement a lower detection threshold for distraction and/or a less complex computational model) and respond rapidly to perceived driver distraction, while the remote computing system can avoid false positive detections that could result in negative repercussions for the driver (e.g., higher insurance premiums, loss of his or her job, etc.).
[0024] Fourth, variants of the method can enable context-based actions (e.g., notifications). For example, the sensor data can be fed into multiple analysis modules, wherein the output of one analysis module can function as a trigger event or as secondary validation for a different analysis module. Additionally or alternatively, the results of the multiple analysis modules can be fed into a downstream module (e.g., a scoring module, a notification module), wherein the downstream module can dynamically select action parameters (e.g., notification intensity, type, start time, end time, etc.) based on the modules' output values. In a specific example, an urgent notification can be quickly presented when a driver is both distracted and tailgating (e.g., outputs from the distraction module and tailgating module, respectively), and a warning notification can be presented only after sustained detected distraction when the driver is distracted and not tailgating. In other examples, notifications can be presented to the driver when a combination of a distraction event and any other relevant traffic situation (e.g., approaching a red or yellow traffic signal, approaching a stop sign, proximity to a pedestrian or vehicle, etc.) is detected. However, the analysis modules can be otherwise used.
[0025] However, variants of the method 100 can additionally or alternatively confer any suitable benefits and/or advantages.
3. Method.
[0026] The method 100 includes Block Sno, which includes: sampling sensor measurements. Sampling the sensor measurements functions to sample signals indicative of driver distraction for a given time point or time period. The sensor measurements are preferably sampled (e.g., recorded) during a driving session, but can alternatively be performed immediately before or after a driving session, substantially before or after a driving session, across multiple driving sessions, or at any other suitable time. The sensor measurements can be recorded at a predetermined frequency (e.g., 2-5HZ, 10Hz, 100Hz, etc.), be recorded in response to occurrence of a recordation event (e.g., when the proximity sensors detect proximal objects, when the vehicle velocity exceeds a threshold velocity, etc.), or be recorded at any other suitable time, in response to and/or based on
any suitable trigger or event. The sensor measurements are preferably recorded by the on-board system, but can alternatively be recorded by any other suitable set of sensors.
[0027] In a first variation, the sensor measurements include an image or video
(e.g., set of images) of the cabin interior, which can be sampled by the inward-facing camera directed toward the driver's side head volume or otherwise sampled (example shown in FIGURE 2). For example, Block S110 can include sampling sensor measurements at an onboard system of the vehicle, wherein the sensor measurements include an image of the driver (e.g., Snoa, S110'). The camera's field of view preferably includes all or a portion of the driver's head, and can optionally include all or a portion of the passenger-side volume (e.g., the passenger's body or head), all or a portion of the rear seats, all or a portion of the rear window, the driver's side window(s), the passenger side window(s), or include any other suitable portion of the vehicle interior. The image(s) can be used to optically identify and/or track the driver's head position, head movement, eye position (e.g., gaze direction), eye movement, parameters thereof (e.g., duration, angle, etc.), or determine any other suitable distraction factor value. The image(s) can additionally be used to determine the driving context, such as whether there is a passenger in the vehicle or where the passenger is located, the environment around the vehicle visible through the vehicle windows, and/or any other suitable context derivable from the image(s). The image(s) can additionally be used to verify or validate other distraction factor values. For example, driver interaction with a user device (e.g., phone) can be verified when the user device is present in the concurrently recorded image. The image can be a visual spectrum image, hyperspectral image, IR image, or be sampled at any other suitable wavelength. The image can be recorded using ambient light, light emitted by a light emitter (e.g., from an LED flash, etc.), or using any other suitable light source. The image can be a stereoimage (e.g., recorded by a stereocamera), a single image, or be any other suitable image.
[0028] In a second variation, the sensor measurements include cabin audio (e.g., vehicle interior noise), which can be sampled by the audio sensor arranged proximal the driver's volume (e.g., integrated with the onboard system, part of a user device associated with the driver, etc.) or otherwise sampled. The cabin audio can be used to infer the driver
head position, the driving context (e.g., whether the driver is calm, whether there is a child in the vehicle, etc.), or be otherwise used. The cabin audio preferably includes driver audio, and can optionally include passenger-side audio, rear-seat audio, vehicle audio (e.g., generated by the vehicle), road noise, radio audio, notification audio, or any other suitable audio stream. The cabin audio can be a single stream (e.g., recorded by a single audio sensor), multiple streams (e.g., recorded by multiple collocated or distal audio sensors, wherein the cabin audio is recorded in stereo, etc.), or include any suitable number of audio streams.
[0029] In a third variation, the sensor measurements include user device operation parameters, which can be determined by the user device or otherwise determined (example shown in FIGURE 2). User device operation parameters can include the user device's: processing load (e.g., as determined from an activity monitor, the CPU load, the GPU load, etc.), processing load changes, power consumption, touch input operation, sensor measurements, or any other suitable set of operation parameters. User device sensor measurements can include user device kinematics, pressure, temperature, ambient light obstruction, or any other suitable parameter measurement indicative of driver interaction with the user device. The user device operation parameters can be determined by one or more clients (e.g., applications, SDKs on multiple applications, etc.) running on the user device, or be otherwise determined. In one example of the variation, processing load increases can be indicative of user interaction with the user device. In a second example of the variation, touch input operation can be indicative of user entering data into the user device. In a third example of the variation, user device vibration can be indicative of the user device being knocked by the user. In a fourth example, user device translation can be indicative of the user device being lifted or picked up by the user. In a fifth example, an increase in the sampled user device ambient pressure or temperature can be indicative of the user device being held. However, any other suitable user device parameter can be monitored for user interaction with the user device.
[0030] In a fourth variation, the sensor measurements include vehicle sensor data, which can be determined by the vehicle or otherwise determined (example shown in FIGURE 2). The vehicle sensor data can be received from the vehicle through the vehicle
data bus, wirelessly communicated from the vehicle, or otherwise received from the vehicle. The vehicle sensor data can include: control panel signals (e.g., whether the radio is being changed, whether the user interface is being used, etc.), steering wheel signals (e.g., number of touches, high pressure regions, high temperature regions, etc. on the steering wheel), seat sensor signals (e.g., driver's weight distribution or change over time, etc.), or any other suitable set of measurements. However, any other suitable set of sensor measurements can be sampled.
[0031] In a fifth variation, the sensor measurements include an image or video
(e.g., set of images) of the vehicle exterior, which can be sampled by the outward-facing camera directed toward the front of the vehicle (e.g., through the front windshield) or otherwise sampled (e.g., examples as shown in FIGURE 2). The camera's field of view preferably includes all or a portion of the roadway oriented toward the front the vehicle (e.g., the foreground toward the front of the vehicle, the lane in which the vehicle is moving, the adjacent lane or lanes, etc.), and can optionally include the region(s) to the sides of the vehicle (e.g., the driver's side, the passenger's side), to the rear of the vehicle, above the vehicle (e.g., through a sunroof and/or moon-roof), or include any other suitable portion of the environment outside the vehicle. The image(s) can be used to optically identify and/ or track the vehicle's position within the environment (e.g., localize the vehicle), position within a lane, position relative to other objects (e.g., vehicles, pedestrians, etc.), or determine any other suitable parameter value. The image(s) can additionally be used to determine the driving context, such as whether there is a second vehicle in front of or beside the vehicle in which the sensor is emplaced. The image(s) can additionally be used to determine, verify, and/or validate other distraction factor values. For example, driver attentiveness can be measured based on an amount and/ or frequency of lateral drifting within a lane of the roadway. Drifting within a lane can be based on vehicle path extraction (e.g., extraction of the vehicle trajectory from the image or images) and comparison to lane markers extracted from images of the road. The image can be a visual spectrum image, hyperspectral image, IR image, or be sampled at any other suitable wavelength. The image can be recorded using ambient light, light emitted by a light emitter (e.g., from an LED flash, etc.), or using any other suitable light source. The
image can be a stereoimage (e.g., recorded by a stereocamera), a single image, or be any other suitable image.
[0032] In a sixth variation, the sensor measurements include a kinematic signal.
The kinematic signal is preferably indicative of a kinematic variable of the vehicle (e.g., acceleration, velocity, position, jerk, etc.), but can additionally or alternatively be indicative of any suitable motion variable associated with the vehicle and/or vehicle occupants (e.g., the driver, a passenger, etc.). The kinematic signal is preferably collected by a kinematic measurement mechanism (e.g., inertial measurement unit) of the onboard system, but can additionally or alternatively be collected by any other suitable component. In a first example, the sensor measurements include an accelerometer output of an inertial measurement unit of the onboard system. However, the sensor measurements can otherwise include any suitable kinematic signal, vehicle information (e.g., steering wheel position, pedal position), indicator, or other information.
[0033] The method 100 includes Block S120, which includes: determining distraction factor values based on the sensor measurements. Determining the distraction factor values functions to characterize the driver's attention, distraction level, or other suitable indicator of driver attention to the set of tasks associated with driving a vehicle. Distraction factors for which values can be determined can include: driver gaze direction, driver eye motion, driver head direction, driver head motion, driver body position, vehicle cabin noise, user device interactions (e.g., phone interactions, vehicle control panel interactions, or any other suitable factor indicative of imminent, instantaneous, or prior driver distraction. One or more distraction factor values can be determined concurrently or asynchronously and used for distraction event detection.
[0034] The distraction factor values are preferably determined based on the sensor measurements, but can additionally or alternatively be retrieved, received (e.g., from a preprocessing system or database), or otherwise determined. The distraction factor values can be determined from a single sensor measurement type, determined from multiple sensor measurement types (e.g., using sensor synthesis), or otherwise determined. Additionally or alternatively, the determined factor values can be verified using a second sensor measurement type.
[0035] The distraction factor values are preferably implicitly determined within a model of driver distraction, but can additionally or alternatively be explicitly determined and/ or otherwise computed. In a first example, Block S120 is implemented at least in part as a learning module trained in an "end-to-end" manner, wherein inputs are provided to the module and the output is labeled as either positive (e.g., distracted) or negative (e.g., not distracted), but the features (e.g., distraction factor values) of the model implemented by the learning module are not pre-specified. However, Block S120 can be otherwise suitable implemented as a non-learning module, and/or as a learning module trained in any other suitable manner.
[0036] Each determined distraction factor value can be associated with the recordation time of the underlying sensor measurement, the vehicle location associated with the recordation time (e.g., determined using the system location system, vehicle location system, etc.), the driver associated with the driving session and/or recordation time (e.g., as determined from driver biometrics, the driver's user device, etc.), or with any other suitable concurrent parameter. However, the distraction factor values can be associated with any other suitable time, location, driver, or information.
[0037] The distraction factor value can be determined (e.g., by a distraction detection module, of which an example is shown in FIGURE 2) using equations, regression, classification, neural networks (e.g., convolutional neural networks, deep neural networks), heuristics, selection (e.g., from a library, graph, or chart), instance- based methods (e.g., nearest neighbor), regularization methods (e.g., ridge regression), decision trees, Baysean methods, kernel methods, probability, deterministics, a combination of the above, or any other suitable method. The distraction factor value can be a binary state (e.g., distracted and not distracted), a score (e.g., a distraction score, etc.), a severity (e.g., a likelihood of a distraction state or event to lead to material loss, a risk factor associated with a distracted state, etc.) or be any other suitable measure of a driver's distraction level.
[0038] In a specific example, Block S120 can include receiving the sensor measurement (e.g., an image) at a face detection module of the onboard system. This example can include generating, at the face detection module, an output, wherein the
output is indicative of the presence of a face in the image (e.g., Si2oa, S120'). The output can be used as a check (e.g., one of a plurality of checks) on an output of the driver distraction module to determine if the driver distraction module has erroneously labeled a distraction event and/ or distraction state. However, the first output indicative of the presence of a face in the image can be otherwise suitably used (e.g., to identify a driver).
[0039] In another specific example, Block S120 can include receiving the sensor measurement (e.g., an image, a sequence of images, etc.) at a driver distraction module of the onboard system. This example can include generating an output, at the driver distraction module, indicative of a distracted state (e.g., Si2ob, S120"). The output of the driver distraction module is preferably generated in parallel to the output of the face detection module (e.g., in real time, in near real time, simultaneously, contemporaneously, within a microsecond interval, within a second interval, etc.), but can additionally or alternatively be generated sequentially (e.g., before or after the output of the face detection module) and/or having any suitable temporal relationship to any suitable generated output of other suitable modules. In further alternatives, the driver distraction module can operate independently of a face detection module, which can be omitted in some variations.
[0040] In a first variation, determining the distraction factor value can include determining the parameters of the driver's gaze, such as gaze direction, gaze duration, gaze angle (e.g., relative to a reference point), rate of change of gaze direction, or any other suitable gaze parameter. This can include identifying the driver's corneal reflectance, pupil, retinal patterns, or other eye parameter in the sampled images or video frames (e.g., the image segment associated with the driver's volume), and determining the driver's gaze direction using gaze estimation, head pose determination, or any other suitable technique. The gaze direction can optionally be classified as one of a set of predetermined directions (e.g., forward, right side, left side, rearward, etc.) or otherwise characterized. Additionally or alternatively, the number, frequency, duration, or other parameter of the gazes can be determined for each head pose, time duration, or from any other set of images.
[0041 ] In a second variation, determining the distraction factor values can include determining the driver's head motion. This can include: using object recognition to recognize the driver's head within the image, tracking the head (e.g., recognized object) motion through a series of images (e.g., using object tracking), determining parameters of the head motion (e.g., turn direction, motion duration, etc.). This variant can optionally include determining the current or anticipated head pose based on the head motion parameters.
[0042] In a third variation, determining the distraction factor values can include determining the driver's head pose. This can include: recognizing the driver's head within the image, retrieving a reference image of the driver's head while the driver was looking forward, and determining the driver's head pose based on differences between the driver's head in the new image and the driver's head in the reference image. In one embodiment, this can include: recognizing the driver's head within the image using object recognition, determining the head's pose using head pose determination methods (e.g., iterative closest curve matching), and comparing the determined head pose to the head pose in the reference image. In a second embodiment, this can include: recognizing the driver's head within the current image using object recognition, identifying one or more reference points on the driver's head (e.g., eyes, nose, moles, freckles, etc.), identifying the corresponding reference points on the reference image, and determining the driver's current head pose based on the difference between the reference points' positions in the current image and the reference image. The reference image (or reference head pose) can be a prerecorded image with the driver in the vehicle (e.g., where the driver is instructed to gaze forward during image recordation), a prerecorded image of the driver (e.g., driver's license standard image) superimposed within the driver head volume, an image composited from multiple images recorded over one or more driving sessions (e.g., the average head pose), or be any other suitable reference image. This variation can optionally include recognizing objects within the vehicle (e.g., children, passengers, etc.), determining the object's location within the vehicle, and evaluating the driver's interaction with the object. For example, a driver's head turn can be classified as a blind spot check with a low distraction score when no passengers are in the vehicle, and but be
classified as a child interaction with a high distraction score when a child appears within the camera frame. However, the head pose can be otherwise determined.
[0043] In a fourth variation, determining the distraction factor values can include determining the driver's head pose based on the cabin acoustics. In one embodiment, the head pose can be determined from a single acoustic frame, recorded by a set of stereo microphones. In this embodiment, the method 100 can include: extracting the acoustic signal associated with the driver's voice from the first and second signal from the first and second microphone, respectively (e.g., using pattern matching, noise reduction, etc.); comparing the magnitude or amplitude of the driver's voice between the first and second signals; and determining the driver's head pose as turned toward the microphone that measured the louder voice. In a second embodiment, the head motion can be determined from a set of acoustic frames recorded by a set of stereo microphones. In one example, a decrease in acoustic intensity sampled by a first microphone paired with an increase in acoustic intensity sampled by a second microphone can be interpreted as a head turn toward the second microphone (e.g., using pattern matching, etc.).
[0044] Additionally or alternatively, the cabin acoustics can be used to determine driving context. In one example, the cabin acoustics can be used to determine whether a passenger is present in the vehicle (e.g., based on detection of a second voice), used to classify the passenger (e.g., as a child, adult, potential distraction, etc.), used to determine the passenger's location, or otherwise used.
[0045] In a fifth variation, determining the distraction factor values can include determining the degree of user interaction (e.g., frequency, duration, intensity, etc.) with a user device. In one example, the method 100 can include receiving a user input at the user device (e.g., touch input, haptic input, etc.) and determining that the user has interacted with the user device. In a second example, the method 100 can include determining parameters of the user input (e.g., frequency of inputs; input types, such as targeted inputs or noisy inputs; frequency of application switching; types of applications interacted with; etc.) and determining the degree of user interaction based on the user input parameters. In a specific example, targeted user inputs (e.g., precise key entries), high frequencies of user inputs, and messaging applications can be associated with a high
degree of user interaction, while noisy user inputs, low input frequencies, and map applications can be associated with a low degree of user interaction. In a third example, the method 100 can include identifying entire or portions of driver body parts (e.g., hands, torso) in the sampled signals (e.g., image(s)), characterizing the body part (e.g., size, position, pose, context, action, etc.), and determining the distraction factor values based on the characterization. For example, a high distraction values can be associated with sideways torso orientation (e.g., instead of forward-facing), less than a threshold proportion of hands detected on the steering wheel, or with any other suitable body part characterization.
[0046] Determining the degree of user interaction with the device can optionally include determining the identity of the user interacting with the device. In one variation, the user identity can be determined using the user's biometrics (e.g., fingerprint), as measured by the user device. In a second variation, the user identity can be determined using the user device's location within the vehicle during the interaction session. In one example, the user device's location within the vehicle can be determined from one or more short-range connections established between the user device and one or more secondary endpoint(s) with known locations (e.g., beacons attached to the A-pillars, beacon attached proximal the driver volume). In a specific example, the user device location can be determined based on the strength of the connection; for example, the interaction session can be classified as a passenger interaction session (instead of a driver interaction session) if the user device is weakly connected to a driver-side beacon, but strongly connected to a passenger-side beacon. However, the user identity can be otherwise determined. Determining the degree of user interaction with the device can optionally include filtering vehicle-induced device motion from user-induced device motion. The motion can be filtered using: concurrently measured temperature (e.g., whether the motion is associated with an increase in ambient temperature), pattern matching (e.g., using patterns previously associated with vehicle motion, given the instantaneous measured vehicle kinematics), classification, or otherwise determined. However, vehicle motion can be otherwise filtered from the user device motion.
[0047] In another example, the distraction factor values can be determined based on the amount of driver's gaze overlap with an external object or based on the driver's anticipated gaze overlap with the anticipated position of the external object relative to the vehicle. In a specific example, the method 100 can include identifying external objects (e.g., from external images), assigning a collision risk to each identified object (e.g., based on the vehicle's kinematics, planned route, etc.), determining the driver's gaze direction (e.g., from images sampled by the internal-facing camera), optionally determining the driver's field of view, and determining a distraction score based on the amount of driver's gaze or field of view overlap with identified objects having collision risks above a threshold risk value. However, the external signals can be otherwise used.
[0048] In yet another example, the distraction factor values can be determined based on the rate at which the object of a driver's gaze changes. For example, the distraction score can be increased based on the rate falling below a threshold level, which can be indicative that the driver is not adequately scanning the roadway while operating the vehicle and is instead gazing at a fixed point (e.g., and maybe lost in thought).
[0049] However, the distraction factor values can be otherwise determined.
[0050] The method 100 can optionally include determining the context of the distraction state, which functions to enable the severity of the distraction state to be characterized. In variations, determining the context includes determining scaling factor values, which can function to fine-tune the severity of the determined distraction (e.g., determined distraction score). Additionally or alternatively, the scaling factor values can be used as inputs into the distraction determination module or be otherwise used.
[0051 ] In a first variation, the scaling factor values can be determined based on the vehicle operation parameters (example shown in FIGURE 2). Vehicle operation parameters can include: vehicle kinematics (e.g., velocity, acceleration), vehicle location, vehicle notifications, vehicle driving instrument position (e.g., brake position, accelerator position, steering wheel position, wheel position, etc.), or any other suitable operation parameter. The vehicle operation parameters can be determined by the vehicle, the system, the user device, or any other suitable system. In one embodiment, the scaling factor values can vary with a collision probability, as determined based on the vehicle
operation parameters. However, the scaling factor values can be otherwise determined. In one example, the distraction score can be scaled as a function of the vehicle kinematics magnitude (e.g., be higher when the vehicle is moving faster, be lower when the vehicle is stopped). In a second example, a different score threshold (e.g., used to determine whether the distraction event is recorded, to determine whether a notification should be presented, etc.) can be selected based on the vehicle kinematics magnitude (e.g., the velocity of the vehicle). In a third example, a different scaling value (e.g., weight) can be selected based on the location (e.g., geographic location) associated with the distraction event. The scaling value assigned to the location can be specific to the driver (e.g., determined based on the driver profile), specific to the driver demographic, universal, or otherwise shared. The scaling value assigned to the location can be static, dynamic (e.g., vary with traffic proximal the location, traffic upstream from the location, etc.), or otherwise determined.
[0052] In a second variation, the scaling factor values can be determined based on signals indicative of the driving environment (example shown in FIGURE 2). These signals can include external signals (e.g., of the vehicle ambient environment), internal signals (e.g., of the vehicle interior environment), or any other suitable set of signals.
[0053] External signals can include traffic parameters (e.g., traffic density, traffic flow rate, etc.), road type (e.g., highway or urban), traffic signals (e.g., road signs, stop signs, stop lights, etc.), objects or obstructions (e.g., in the anticipated traversal path), weather conditions (e.g., based on a weather forecast or determined in near-real time for the ambient environment), adjacent drivers' profiles, adjacent drivers' distraction levels, or any other suitable external parameter associated with driving risk. Traffic parameters can be crowdsourced (e.g., manually generated by a plurality of users inputting traffic information; automatically generated from a plurality of systems each with an external camera, etc.), automatically determined (e.g., using the images sampled by the external- facing camera), determined from the vehicle's location and historic traffic patterns for the location and the recurrent time period, or otherwise determined. The road type can be determined based on a municipal map and the vehicle's location, determined based on the vehicle's velocity (e.g., classified as a highway when the velocity is over 6omph),
determined based on features or objects extracted from the external images, or otherwise determined.
[0054] In one example, the distraction score can be scaled higher (e.g., the scaling factors can be larger) when the driver is distracted while in heavy traffic. In a second example, the distraction score can be scaled lower when a stop sign or red light appears in the external image (e.g., image sampled by the external-facing camera), and scaled higher when no traffic sign or a green light appears in the external image. In a third example, the distraction score can be scaled higher (e.g., increased in weight) when a stop sign or red light appears in an image of the external environment and the vehicle velocity exceeds a threshold value (e.g., o mph, smph) at a predetermined location relative to the traffic signal (e.g., o ft. away, 5 ft. away, etc.), indicative that the vehicle is entering a scenario requiring enhanced attention. However, the distraction score can additionally or alternatively be scaled higher or lower based on any suitable determination.
[0055] Internal signals can include: passenger presence, passenger type, passenger activity (e.g., all of which can be determined from the sensor measurements or otherwise determined), vehicle kinematics (e.g., rapid starting or stopping), recurrent time (e.g., time of day, month, year), driver calendar, passenger calendar, or any other suitable internal parameter associated with driving risk. The internal signals can be received from vehicle sensors, system sensors, user device sensors, secondary user accounts associated with the user account (e.g., a calendar associated with the user account through OAuth), or otherwise determined.
[0056] In a third variation, the scaling factor values can be determined based on vehicle parameters (example shown in FIGURE 2). Vehicle parameters can include the vehicle autonomy type (e.g., autonomous, semi-autonomous, manual, etc.), the vehicle class (e.g., sedan, truck, etc.), vehicle ride height, vehicle braking power, vehicle horsepower, vehicle torque, or any other suitable vehicle parameter. The vehicle parameters can be received from the user (e.g., entered by the user), automatically determined from a database, automatically determined from the vehicle electronic control unit, automatically determined from the sensor measurements (e.g., from the proportion of the road seen in the external image, by matching vehicle interiors to the
vehicle interior from the internal image), or otherwise determined. In one example, the distraction score can be heavily discounted when the vehicle is a fully-autonomous vehicle, and scaled up when the vehicle is a level o or level 1 SAE automated vehicle class. In a second example, the distraction threshold used for notification presentation, distraction score storage, or for other applications can be selected based on the vehicle parameter values. For example, a first threshold can be selected for a level-o or level-i vehicle, a second threshold higher than the first can be selected for a semi-autonomous vehicle, and a third threshold higher than the second threshold can be selected for an autonomous vehicle. The second threshold can be closer to the first threshold than to the third threshold, but the thresholds can be otherwise distributed. However, the factor values and/ or scaling values can be otherwise determined.
[0057] The method 100 includes Block S130, which includes: determining that the driver of the vehicle is characterized by the distracted state (e.g., determining driver distraction) based on the factor values. Determining driver distraction based on distraction factor values functions to determine a measure of driver distraction for a given time period. The driver distraction can be determined (e.g., by a driver distraction module, example shown in FIGURE 2) using equations, regression, classification, neural networks (e.g., convolutional neural networks, deep neural networks), heuristics, selection (e.g., from a library, graph, or chart), instance-based methods (e.g., nearest neighbor), correlation methods, regularization methods (e.g., ridge regression), decision trees, Baysean methods, kernel methods, probability, deterministics, a combination of the above, or any other suitable method. The driver distraction can be a binary classification or binary score (e.g., distracted or not), a score (e.g., continuous or discontinuous; consistent or inconsistent with a uniform distraction scale; etc.), a classification (e.g., high, medium, low), or be any other suitable measure of distraction. The driver distraction can be determined at a predetermined frequency (e.g., every 5 seconds, continuously, etc.), in response to occurrence of a determination event (e.g., in response to obstacle identification in the vehicle path, in response to the driver's gaze being directed outside of a predetermined angular range, etc.), at a frequency selected based on available computing resources, or determined at any other suitable time.
[0058] The driver distraction can be associated with a time or time period, a driver, a location, a driving context, route (e.g., as shown by example in FIGURE 4), or any other suitable parameter value. The parameter values associated with driver distraction are preferably extracted from the parameter values associated with the driver distraction factor values used to determine the driver distraction, but can be any other suitable parameter value. For example, the time associated with the determined driver distraction is preferably the sampling time for the underlying data, not the driver distraction determination time. However, the time associated with the determined driver distraction can be the driver distraction determination time or be any other suitable time.
[0059] The driver distraction is preferably determined based on distraction factor values, but can be determined based on any other suitable information. The distraction factor values used to determine the driver distraction are preferably sampled within a predetermined time period (e.g., within 2s of a reference time, wherein the reference time can be an image sampling time, a location sampling time, a randomly selected time, or be any other suitable time), but can alternatively be otherwise related. The driver is preferably deemed distracted (e.g., characterized by a distracted state) after a threshold number of image frames produce distraction factor values indicative of distraction, but can alternatively be deemed distracted after the values from a single frame are indicative of distraction, when the values satisfy a predetermined condition (e.g., surpass a threshold value), or otherwise determined. The threshold number of image frames can be predetermined, dynamically determined (e.g., based on other modules' output values, such as the external obstacle proximity, facial detection, vehicle kinematics, etc.), or otherwise determined. For example, the threshold number of frames can decrease with decreased leading vehicle separation and/or increased vehicle velocity. The time series of image frames can be contiguous or noncontiguous (e.g., the frames associated with a distracted state can be separated by frames associated with undistracted states), wherein the number or frequency of undistracted frames can be predetermined, dynamically adjusted based on context (e.g., other modules' output values), or otherwise determined.
[0060] In a first example, the driver distraction module can determine a first driver distraction score when the driver is gazing forward (e.g., as determined from the internal
image), and determine a second driver distraction score higher than the first score when the driver is gazing downward or to the side. The driver distraction module can further determine a third driver distraction score higher than the second score when the driver is gazing toward the side and a noisy child is in the backseat, or when the driver is gazing away from or has a field of view non-overlapping with an obstruction detected in the vehicle path (e.g., determined from vehicle proximity sensors, the external image, etc.). However, the driver distraction module can determine any other suitable driver distraction score given any other suitable set of inputs.
[0061] In a second example, the driver distraction module can determine that the driver of the vehicle is characterized by the distracted state based on checking the result against a face detection module output (e.g., contemporaneously generated, asynchronously generated, etc.). In this example, Block S130 can include determining that the driver of the vehicle is characterized by a distracted state based on a combination of the output of the driver distraction module and the output of the face detection module (e.g., Si3oa, S130'). In this example, the driver distraction module determines whether or not the input data (e.g., image data) is indicative of driver distraction or not, and the face detection module determines whether the input data includes a face or not. For this example, in cases wherein a face is not detected but the driver distraction module determines that the driver is distracted, the output of the driver distraction module is presumed to be a false positive.
[0062] Block S130 of the method 100 can include determining a driver score, which can be distinct from a distraction score (e.g., S130C). The driver score is preferably a cumulative score based on a history of distraction scores, determined and recorded during prior driving sessions and a current driving session. An individual driver preferably has a single driver score that is associated with the driver, but can additionally or alternatively be associated with multiple driver scores. The driver score can be utilized in various Blocks of the method 100 in a similar manner to other scores (e.g., distraction scores) as described above, but can be otherwise suitably utilized. The driver score is preferably determined, at least in part, by the scoring module (e.g., as in S13OC, in conjunction with a database including a time history of distraction scores associated with the driver, etc.).
[0063] Block S130 can, in variations, be partially or entirely implemented at a remote computing system. In a first example, Block S130 can include checking the result of a first distraction determination at a second driver distraction module implemented at the remote computing system, by providing the sensor measurements to the remote computing system. In this example, Block S130 includes generating, at a driver distraction module of the remote computing system, an output indicative that the driver is characterized by the distracted state, based on the sensor measurements (e.g., Si3ob). In this and other examples, the onboard driver distraction module can be configured to be computationally faster but less accurate than a remote driver distraction module (e.g., the onboard system can be implemented as a neural network having fewer neurons than the remote system), and the remote driver distraction module can be used to correct false positives generated by the onboard driver distraction module.
[0064] Block S130 can include determining a context of the distracted state (e.g., S130"). The context can be determined based on sensor measurements as described above, or otherwise suitably determined. Block S130 can further include determining a score (e.g., as described above), based on the context of the distracted state (e.g., S130'"). For example, determining the context of the distracted state can include determining that the traffic level is sparse, and the sparsity of the traffic can be used to weight the score (e.g., reduce the severity of the distraction level due to the low risk of vehicle or personal loss). However, the context can additionally or alternatively be otherwise suitably determined.
[0065] Determining the driver distraction can optionally include selecting a distraction model to use for driver distraction determination. This can be useful when a plurality of distraction models are available for driver distraction determination. In one variation, the distraction model can be selected based on the distraction factor values. In one example, the distraction model can be selected based on the distraction factor value for a limited subset of distraction factors, such as the driver's gaze direction or the current vehicle location. In a specific example, a first distraction model (e.g., a correlation model) can be selected when the driver's gaze is directed forward, and a second distraction model (e.g., a convolutional neural network or other more complex model) can be selected when
the driver's gaze is directed backward or to the side. A more complex model can be selected in the latter instance to handle the increased complexities associated with the underlying cause(s) of the turned head. However, any other suitable model can be selected. In a second specific example, a first distraction model (e.g., a correlation model) can be selected when there are children in the vehicle, and a second distraction model (e.g., a convolutional neural network or other more complex model) can be selected when there are no children in the vehicle.
[0066] In a second example, the distraction model can be selected based on the numerosity of distraction factors concurrently exceeding their respective thresholds. In a specific example, a first distraction model (e.g., an equation) can be selected when no distraction factors exceed their thresholds (e.g., the driver's gaze is directed forward, there are no distractions within the vehicle, there are no obstructions in the vehicle path, etc.), a second model selected when a subset of the distraction factors exceed their thresholds (e.g., the driver's gaze is directed forward and there are no distractions within the vehicle, but there are obstructions in the vehicle path), and a third model selected when a different number of distraction factors exceed their thresholds (e.g., the driver's gaze is directed backward, there are distractions within the vehicle, and there are obstructions in the vehicle path). However, the distraction model can be otherwise selected.
[0067] The method 100 can optionally include updating the distraction models. The distraction models can be updated automatically, manually, or otherwise updated. The distraction models can be updated periodically (e.g., at a predetermined frequency), in response to occurrence of an update event (e.g., in response to the determined distraction score differing beyond a threshold difference from an actual distraction score), or at any other suitable time. Updating the distraction model can include: retraining the model using labeled training data or otherwise updating the distraction model. The training data can be automatically generated, manually generated, or otherwise determined.
[0068] As shown in FIGURE 3, in a first variation, the method 100 can include determining a driver distraction score using a distraction module based on the sensor measurements sampled during a first time period, identifying a distraction-associated
event (e.g., a collision, a near-collision, vehicle swerving, sudden braking, etc.) based on subsequent sensor measurements sampled during a second time period after the first time period (e.g., within a predetermined duration of the first time period), determining an actual distraction score based on the distraction-associated event, tagging the sensor measurements sampled during a first time period with the actual distraction score, and calibrating or retraining the distraction module until the determined driver distraction score substantially matches the actual distraction score. However, the distraction module or model can be otherwise updated.
[0069] The method 100 can optionally include transmitting the sensor measurements to a remote computing system S140. Block S140 functions to provide the sensor measurements to the remote computing system, where sensor measurements can be aggregated and/or acted upon by additional or alternative modules to those available at the onboard system. Block S140 is preferably performed asynchronously with collection of the sensor measurements, and in variations can be performed in response to the connection of the onboard system to a network data link having a predetermined characteristic (e.g., capacity, uplink speed, bandwidth, connection quality, etc.), in response to determination of a distraction score above a threshold distraction score value, or in response to any other suitable event. In an example, Block S140 can include transmitting the sensor measurements to a remote computing system in response to determining that the driver of the vehicle is characterized by the distracted state (e.g., Si4oa). However, Block S140 can additionally or alternatively be performed in real- or near real-time, substantially real time, simultaneously, contemporaneously, or with any other suitable temporal characteristics relative to sensor measurement.
[0070] The method 100 can optionally include generating a notification based on determining driver distraction S150. Block S150 functions to notify an entity (e.g., the driver, a remote fleet manager, a passenger) that the driver has been determined to be distracted. The notification can have any suitable format, including a textual format, an audio format, a video format (e.g., a video clip recording during over a time period at which the distraction state is determined), or have any other suitable format. Block S150 is preferably performed substantially immediately upon determination of driver
distraction and/or a score or other characterization thereof, but can additionally or alternatively be performed asynchronously, in response to a trigger event (e.g., connection to a network link having a suitable characteristic), and/or with any other suitable temporal relationship.
[0071] In an example, Block S150 can include notifying the driver, at an output of the onboard system of the vehicle and in substantially real-time, based on the distracted state determined in Block S130, the context of the distracted state, and the score of the distracted state (e.g., S150'). In another example, Block S150 can include notifying a fleet manager, at an interface of the remote computing system (e.g., a local PC terminal, a user device associated with the fleet manager and communicatively coupled to the remote computing system, etc., that the driver score of a driver has fallen below a threshold as a result of consecutive determinations of a distracted state within a predetermined period of time (e.g., a day, five minutes, etc.) in response to updating the driver score (e.g., in real time, asynchronously, etc.). However, Block S150 can additionally or alternatively include generating a notification in any suitable manner, and transmitting and/or rendering the notification to any suitable entity in any suitable manner.
[0072] In one variation of the method, an example of which is shown in FIGURE 5, the method can include sampling sensor measurements, wherein the sensor measurements include image data and signal data (e.g., time series signals). The method can include receiving the sampled sensor measurements at a plurality of modules, including an object detection and classification module, a face detection module, a distraction module, and a driver behavior module. In this variation, the face detection module determines whether a face is present in an input image (e.g., via a machine learning algorithm, via any suitable image processing technique, Hough transform, etc.) in parallel to determination of driver distraction by the distraction module, based on the sensor measurements received as inputs to the distraction module. The output of the face detection module (e.g., a binary output indicating that a face was or was not detected in the image) can be used to eliminate false positive outputs of the distraction module (e.g., in cases wherein a face is not detected, an indication that the driver is distracted that is output by the distraction module can be negated). The outputs of the object detection &
classification module, the distraction module (e.g., checked against the output of the face detection module), and the driver behavior module (e.g., that determines driver and/or operator behavior substantially as described in U.S. Application Serial No. 15/705,043, filed 14-SEP-2017, which is incorporated herein in its entirety by this reference) can be received at a rule-based comparator, which functions in this variation to determine whether the driver is characterized by a distracted state, and can also function to determine the degree of distraction (e.g., severity, distraction score, etc.) based on the outputs of the modules. In this variation, the transformation of inputs (e.g., sensor measurements, outputs of upstream modules, etc.) to outputs is preferably performed continuously and in real- or near-real-time, but can additionally or alternatively be performed at any suitable frequency and/ or with any suitable temporal characteristics (e.g., asynchronously, synchronously, in response to a trigger event, at a frequency of 4 Hz, at any suitable frequency, etc.).
[0073] In a first specific example, as shown in FIGURE 6, the method 100 includes sampling sensor measurements at an onboard system of the vehicle, wherein the sensor measurements comprise an image of the driver Snoa; receiving the image at a face detection module of the onboard system, and generating a first output, wherein the first output is indicative of the presence of a face in the image Si2oa; receiving the image at a driver distraction module of the onboard system, and generating a second output contemporaneously with generating the first output, wherein the second output is indicative of a distracted state Si2ob; determining that the driver of the vehicle is characterized by the distracted state, based on a combination of the first output and the second output Si30a; transmitting the sensor measurements to a remote computing system in response to determining that the driver of the vehicle is characterized by the distracted state Si4oa; generating, at a second driver distraction module of the remote computing system, a third output indicative that the driver is characterized by the distracted state, based on the sensor measurements Si3ob; computing a driver score, at a scoring module of the remote computing system, in response to generating the third output and based on the sensor measurements and the distracted state S130C.
[0074] In a second specific example, as shown in FIGURE 7, the method 100 includes sampling sensor measurements at an onboard system of the vehicle, wherein the sensor measurements comprise an image of the driver S110'; receiving the image at a face detection module of the onboard system, and generating a first output, wherein the first output is indicative of the presence of a face in the image S120'; receiving the image at a driver distraction module of the onboard system, and generating a second output contemporaneously with generating the first output, wherein the second output is indicative of a distracted state S120"; determining that the driver of the vehicle is characterized by the distracted state, based on a combination of the first output and the second output S130'; determining a context of the distracted state S130"; determining a score of the distracted state based on the context of the distracted state S130'"; notifying the driver, at an output of the onboard system of the vehicle and in substantially realtime, based on the distracted state, the context of the distracted state, and the score of the distracted state S150'.
[0075] Embodiments of the system and/ or method can include every combination and permutation of the various system components and the various method processes, wherein the method processes can be performed in any suitable order, sequentially or concurrently.
[0076] As a person skilled in the art will recognize from the previous detailed description and from the figures and claims, modifications and changes can be made to the preferred embodiments of the invention without departing from the scope of this invention defined in the following claims.
Claims
1. A method for determining distraction of a driver of a vehicle, comprising:
• sampling sensor measurements at an onboard system of the vehicle, wherein the sensor measurements comprise an image of the driver;
• receiving the image at a face detection module of the onboard system, and
generating a first output, wherein the first output is indicative of the presence of a face in the image;
• receiving the image at a driver distraction module of the onboard system, and generating a second output contemporaneously with generating the first output, wherein the second output is indicative of a distracted state;
• determining that the driver of the vehicle is characterized by the distracted state, based on a combination of the first output and the second output;
• determining a context of the distracted state;
• determining a score of the distracted state based on the context of the distracted state; and
• notifying the driver, at an output of the onboard system of the vehicle and in
substantially real-time, based on the distracted state, the context of the distracted state, and the score of the distracted state.
2. The method of Claim l, wherein the distracted state comprises a gaze direction of the driver, and wherein generating the second output is based on identifying the gaze direction of the driver from the image.
3. The method of Claim 1, wherein the context of the distracted state comprises a duration of the distracted state, and wherein determining the severity comprises determining the score of the distracted state based on the duration exceeding a threshold duration.
4. The method of Claim 3, wherein the threshold duration is predetermined by a remote entity associated with the driver, and further comprising transmitting a notification to the remote entity based on the determined score of the distracted state.
5. The method of Claim 1, wherein the sensor measurements comprise an accelerometer output, and further comprising:
• determining driver behavior at a driver behavior analysis module of the onboard system based on the accelerometer output; and
• wherein determining that the driver of the vehicle is characterized by the
distracted state is based on the determined driver behavior.
6. The method of Claim 1, further comprising transmitting the sensor
measurements to a remote computing system in response to the score of the distracted state exceeding a threshold score.
7. The method of Claim 6, wherein transmitting the sensor measurements to the remote computing system is performed substantially instantaneously subsequent to determining that the score of the distracted state exceeds the threshold score.
8. The method of Claim 6, further comprising generating, at a second driver distraction module of the remote computing system, a third output indicative that the driver is characterized by the distracted state, based on the sensor measurements.
9. The method of Claim 8, further comprising transmitting a notification to a remote entity associated with the driver, in response to generating the third output, wherein the notification comprises the distracted state, the score of the distracted state, the context of the distracted state, and a time point of distracted state determination.
10. The method of Claim 6, further comprising computing a driver score, at a scoring module of the remote computing system, based on the sensor measurements, the determined score, and a time history of scores associated with the driver.
11. A method for determining distraction of a driver of a vehicle, comprising:
• sampling sensor measurements at an onboard system of the vehicle, during a driving session, wherein the sensor measurements comprise an image of the driver;
• receiving the image at a face detection module of the onboard system, and
generating a first output, wherein the first output is indicative of the presence of a face in the image;
• receiving the image at a driver distraction module of the onboard system, and generating a second output contemporaneously with generating the first output, wherein the second output is indicative of a distracted state;
• determining that the driver of the vehicle is characterized by the distracted state, based on a combination of the first output and the second output;
• transmitting the sensor measurements to a remote computing system in response to determining that the driver of the vehicle is characterized by the distracted state;
• generating, at a second driver distraction module of the remote computing
system, a third output indicative that the driver is characterized by the distracted state, based on the sensor measurements;
• computing a driver score, at a scoring module of the remote computing system, in response to generating the third output and based on the sensor measurements and the distracted state.
12. The method of Claim 11, wherein the sensor measurements comprise a sequence of images, wherein the distracted state comprises a rate of change of a gaze direction of the driver, and wherein the rate of change of the gaze direction is computed based on the image in combination with the sequence of images.
13. The method of Claim 11, wherein transmitting the sensor measurements to the remote computing system is performed substantially instantaneously subsequent to determining that the driver of the vehicle is characterized by the distracted state.
14. The method of Claim 11, further comprising determining a context of the distracted state, at the onboard system of the vehicle, based on the sensor
measurements.
15. The method of Claim 14, wherein the sensor measurements further comprise a second image of a region in front of the vehicle, and wherein determining the context of the distracted state comprises determining a distance between an object depicted in the second image and the vehicle.
16. The method of Claim 14, wherein the context comprises a duration of the distracted state, and further comprising determining a score of the distracted state based on the duration exceeding a threshold duration.
17. The method of Claim 14, further comprising determining a score of the distracted state based on the context of the distracted state.
18. The method of Claim 17, further comprising notifying the driver, at an output of the onboard system of the vehicle and in substantially real-time, based on the distracted state, the context of the distracted state, and the score of the distracted state.
19. The method of Claim 17, wherein transmitting the sensor measurements to the remote computing system is performed in response to the score of the distracted state exceeding a threshold score.
20. The method of Claim 11, further comprising transmitting a notification to a remote entity associated with the driver, in response to generating the third output, wherein the notification comprises the distracted state, a time point of distracted state determination, and the score.
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP17867780.3A EP3535646A4 (en) | 2016-11-07 | 2017-11-07 | System and method for driver distraction determination |
CN201780082650.7A CN110178104A (en) | 2016-11-07 | 2017-11-07 | System and method for determining driver distraction |
JP2019522480A JP7290567B2 (en) | 2016-11-07 | 2017-11-07 | Systems and methods for driver distraction determination |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201662418655P | 2016-11-07 | 2016-11-07 | |
US62/418,655 | 2016-11-07 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2018085804A1 true WO2018085804A1 (en) | 2018-05-11 |
Family
ID=62066042
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2017/060307 WO2018085804A1 (en) | 2016-11-07 | 2017-11-07 | System and method for driver distraction determination |
Country Status (5)
Country | Link |
---|---|
US (3) | US10246014B2 (en) |
EP (1) | EP3535646A4 (en) |
JP (1) | JP7290567B2 (en) |
CN (1) | CN110178104A (en) |
WO (1) | WO2018085804A1 (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
DE102020123744A1 (en) | 2020-09-11 | 2022-03-17 | Bayerische Motoren Werke Aktiengesellschaft | Determining a line of sight of a person |
EP3992789A1 (en) | 2020-11-02 | 2022-05-04 | BSH Hausgeräte GmbH | Coupling of a domestic appliance with a mobile device |
WO2022161810A1 (en) * | 2021-02-01 | 2022-08-04 | Volkswagen Aktiengesellschaft | Estimating a degree of driver distraction |
US11571969B2 (en) * | 2020-06-04 | 2023-02-07 | Toyota Motor Engineering & Manufacturing North America, Inc. | External communication suppression device for driving automation |
Families Citing this family (92)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11465640B2 (en) * | 2010-06-07 | 2022-10-11 | Affectiva, Inc. | Directed control transfer for autonomous vehicles |
US9524269B1 (en) | 2012-12-19 | 2016-12-20 | Allstate Insurance Company | Driving event data analysis |
US9535878B1 (en) * | 2012-12-19 | 2017-01-03 | Allstate Insurance Company | Driving event data analysis |
US9944296B1 (en) * | 2015-01-13 | 2018-04-17 | State Farm Mutual Automobile Insurance Company | Apparatuses, systems and methods for determining distractions associated with vehicle driving routes |
JP6193912B2 (en) * | 2015-04-24 | 2017-09-06 | 株式会社パイ・アール | Drive recorder |
FR3038770B1 (en) * | 2015-07-10 | 2021-03-19 | Innov Plus | OPERATOR VIGILANCE MONITORING SYSTEM |
US11194405B2 (en) * | 2015-10-08 | 2021-12-07 | Panasonic Intellectual Property Corporation Of America | Method for controlling information display apparatus, and information display apparatus |
US10717410B2 (en) * | 2016-02-04 | 2020-07-21 | Vasile CIUPERGER | Vehicle perimeter alert system |
EP3535646A4 (en) * | 2016-11-07 | 2020-08-12 | Nauto, Inc. | System and method for driver distraction determination |
US10328913B2 (en) * | 2016-11-18 | 2019-06-25 | International Business Machines Corporation | Facilitation of automatic adjustment of a braking system |
WO2018125796A1 (en) * | 2016-12-27 | 2018-07-05 | Denso International America, Inc. | System and method for microlocation sensor communication |
JP6982767B2 (en) * | 2017-03-31 | 2021-12-17 | パナソニックIpマネジメント株式会社 | Detection device, learning device, detection method, learning method, and program |
US10902577B2 (en) * | 2017-06-19 | 2021-01-26 | Apeel Technology, Inc. | System and method for hyperspectral image processing to identify object |
US10902581B2 (en) * | 2017-06-19 | 2021-01-26 | Apeel Technology, Inc. | System and method for hyperspectral image processing to identify foreign object |
US10592785B2 (en) * | 2017-07-12 | 2020-03-17 | Futurewei Technologies, Inc. | Integrated system for detection of driver condition |
WO2019028798A1 (en) * | 2017-08-10 | 2019-02-14 | 北京市商汤科技开发有限公司 | Method and device for monitoring driving condition, and electronic device |
US10907980B2 (en) * | 2017-08-11 | 2021-02-02 | Here Global B.V. | Method and apparatus for detecting false positive slippery road reports using mapping data |
CN107679448B (en) * | 2017-08-17 | 2018-09-25 | 平安科技(深圳)有限公司 | Eyeball action-analysing method, device and storage medium |
US10373500B1 (en) * | 2017-09-22 | 2019-08-06 | State Farm Mutual Automobile Insurance Company | Technology for using image data to assess vehicular risks and communicate notifications |
US20190120640A1 (en) | 2017-10-19 | 2019-04-25 | rideOS | Autonomous vehicle routing |
US10572745B2 (en) * | 2017-11-11 | 2020-02-25 | Bendix Commercial Vehicle Systems Llc | System and methods of monitoring driver behavior for vehicular fleet management in a fleet of vehicles using driver-facing imaging device |
JP6777060B2 (en) * | 2017-11-15 | 2020-10-28 | オムロン株式会社 | Inattentive judgment device, driving support system, inattentive judgment method and program for inattentive judgment |
JP2019091267A (en) * | 2017-11-15 | 2019-06-13 | オムロン株式会社 | Inattentive driving determination device, inattentive driving determination method, and program |
US20200283028A1 (en) * | 2017-11-17 | 2020-09-10 | Sony Semiconductor Solutions Corporation | Information processing apparatus and information processing method |
US10783346B2 (en) * | 2017-12-11 | 2020-09-22 | Invensense, Inc. | Enhancing quality of a fingerprint image |
DE102017223005A1 (en) * | 2017-12-18 | 2019-06-19 | Robert Bosch Gmbh | Method and device for providing injury information about an injury to an unprotected road user in a collision with a vehicle |
US11775688B2 (en) * | 2017-12-20 | 2023-10-03 | Felica Networks, Inc. | Information processing device and information processing method |
JP6844568B2 (en) * | 2018-03-27 | 2021-03-17 | 日本電気株式会社 | Extra findings determination device, extra findings determination system, extra findings determination method, program |
US11565693B2 (en) * | 2018-05-03 | 2023-01-31 | Honda Motor Co., Ltd. | Systems and methods for distracted driving detection |
US10970571B2 (en) | 2018-06-04 | 2021-04-06 | Shanghai Sensetime Intelligent Technology Co., Ltd. | Vehicle control method and system, vehicle-mounted intelligent system, electronic device, and medium |
CN109002757A (en) * | 2018-06-04 | 2018-12-14 | 上海商汤智能科技有限公司 | Drive management method and system, vehicle intelligent system, electronic equipment, medium |
US10915769B2 (en) | 2018-06-04 | 2021-02-09 | Shanghai Sensetime Intelligent Technology Co., Ltd | Driving management methods and systems, vehicle-mounted intelligent systems, electronic devices, and medium |
CN110633593A (en) * | 2018-06-21 | 2019-12-31 | 北京嘀嘀无限科技发展有限公司 | Malignant event prediction method and system |
CN113056390A (en) * | 2018-06-26 | 2021-06-29 | 伊泰·卡茨 | Situational driver monitoring system |
US10655978B2 (en) * | 2018-06-27 | 2020-05-19 | Harman International Industries, Incorporated | Controlling an autonomous vehicle based on passenger behavior |
US11180158B1 (en) * | 2018-07-31 | 2021-11-23 | United Services Automobile Association (Usaa) | Routing or driving systems and methods based on sleep pattern information |
GB2577270B (en) * | 2018-09-19 | 2021-04-07 | Jaguar Land Rover Ltd | Apparatus and method for monitoring vehicle operation |
US10746112B2 (en) * | 2018-10-18 | 2020-08-18 | Ford Global Technologies, Llc | Method and system for NVH control |
EP3871141A4 (en) * | 2018-10-23 | 2022-08-10 | 5DT, Inc | Operator behavior monitoring system |
US10807605B2 (en) * | 2018-12-19 | 2020-10-20 | Waymo Llc | Systems and methods for detecting and dynamically mitigating driver fatigue |
US11577734B2 (en) | 2018-12-20 | 2023-02-14 | Nauto, Inc. | System and method for analysis of driver behavior |
US10635917B1 (en) * | 2019-01-30 | 2020-04-28 | StradVision, Inc. | Method and device for detecting vehicle occupancy using passenger's keypoint detected through image analysis for humans' status recognition |
CN111661059B (en) * | 2019-03-08 | 2022-07-08 | 虹软科技股份有限公司 | Method and system for monitoring distracted driving and electronic equipment |
GB2582775B (en) * | 2019-04-02 | 2022-04-06 | Jaguar Land Rover Ltd | Attention level determination |
KR20200117641A (en) * | 2019-04-05 | 2020-10-14 | 현대자동차주식회사 | Apparatus and method for guiding path |
JP7031998B2 (en) * | 2019-04-19 | 2022-03-08 | 矢崎総業株式会社 | Audio control system and audio control method |
US12056922B2 (en) * | 2019-04-26 | 2024-08-06 | Samsara Inc. | Event notification system |
US10672249B1 (en) | 2019-05-06 | 2020-06-02 | Cambridge Mobile Telematics Inc. | Determining, scoring, and reporting mobile phone distraction of a driver |
US10759441B1 (en) * | 2019-05-06 | 2020-09-01 | Cambridge Mobile Telematics Inc. | Determining, scoring, and reporting mobile phone distraction of a driver |
US11281920B1 (en) * | 2019-05-23 | 2022-03-22 | State Farm Mutual Automobile Insurance Company | Apparatuses, systems and methods for generating a vehicle driver signature |
CN110232341B (en) * | 2019-05-30 | 2022-05-03 | 重庆邮电大学 | Semi-supervised learning image identification method based on convolution-stacking noise reduction coding network |
JP7228472B2 (en) * | 2019-06-07 | 2023-02-24 | 本田技研工業株式会社 | Recognition device, recognition method, and program |
WO2021003172A1 (en) * | 2019-06-30 | 2021-01-07 | Moment Ai, Inc. | Facial recognition and monitoring device, system, and method |
JP7493507B2 (en) | 2019-07-17 | 2024-05-31 | 株式会社クラレ | Water-soluble films and packaging |
US11496617B2 (en) | 2019-07-22 | 2022-11-08 | Jeremy Davis | Car surveillance system |
KR102218526B1 (en) * | 2019-07-26 | 2021-02-19 | 엘지전자 주식회사 | Method, system, and vehicle for preventing drowsy driving |
US10993647B2 (en) * | 2019-08-21 | 2021-05-04 | Micron Technology, Inc. | Drowsiness detection for vehicle control |
JP7328828B2 (en) * | 2019-08-27 | 2023-08-17 | フォルシアクラリオン・エレクトロニクス株式会社 | State estimation device, state estimation program, and state estimation method |
KR20190116187A (en) * | 2019-09-23 | 2019-10-14 | 엘지전자 주식회사 | Vehicle terminal and operation method thereof |
US11587461B2 (en) * | 2019-10-23 | 2023-02-21 | GM Global Technology Operations LLC | Context-sensitive adjustment of off-road glance time |
CN110816543B (en) * | 2019-10-28 | 2021-04-06 | 东南大学 | Driver distraction driving detection and early warning system and method under vehicle turning and lane changing scenes |
US10861144B1 (en) * | 2019-11-27 | 2020-12-08 | Panasonic Intellectual Property Management Co., Ltd. | Image-processing method |
JP7358958B2 (en) * | 2019-12-11 | 2023-10-11 | トヨタ自動車株式会社 | Driving awareness estimation device |
EP3845431B1 (en) * | 2020-01-06 | 2024-08-07 | Aptiv Technologies AG | Driver-monitoring system |
CN111317469B (en) * | 2020-01-10 | 2022-12-02 | 深圳市宏智力科技有限公司 | Brain wave monitoring equipment, system and monitoring method |
EP4097706A4 (en) * | 2020-01-29 | 2023-07-19 | Netradyne, Inc. | Combination alerts |
KR20210111558A (en) * | 2020-03-03 | 2021-09-13 | 현대자동차주식회사 | Driver assist apparatus and adaptive warning method thereof |
US11568655B2 (en) | 2020-03-26 | 2023-01-31 | Intel Corporation | Methods and devices for triggering vehicular actions based on passenger actions |
DE102020110417A1 (en) * | 2020-04-16 | 2021-10-21 | Valeo Schalter Und Sensoren Gmbh | Method and system for assisting a driver |
US20210334289A1 (en) * | 2020-04-22 | 2021-10-28 | Morgan State University | System and method for synchronization of asynchronous datasets |
US20220032924A1 (en) * | 2020-07-30 | 2022-02-03 | Morgan State University | System and method for driver distraction detection and classification |
JP7359099B2 (en) * | 2020-07-31 | 2023-10-11 | トヨタ自動車株式会社 | Mobile object interference detection device, mobile object interference detection system, and mobile object interference detection program |
US11748407B2 (en) | 2020-10-12 | 2023-09-05 | Robert Bosch Gmbh | Activity level based management and upload of ride monitoring data of rides of a mobility service provider |
US11941150B2 (en) | 2020-10-12 | 2024-03-26 | Robert Bosch Gmbh | In-vehicle system for monitoring rides of a mobility service provider |
US11518401B2 (en) * | 2020-10-14 | 2022-12-06 | Magna Electronics Inc. | Vehicular driving assist with driver monitoring |
KR20220055209A (en) * | 2020-10-26 | 2022-05-03 | 현대자동차주식회사 | Vehicle and method for controlling thereof |
US11341786B1 (en) | 2020-11-13 | 2022-05-24 | Samsara Inc. | Dynamic delivery of vehicle event data |
US11352013B1 (en) | 2020-11-13 | 2022-06-07 | Samsara Inc. | Refining event triggers using machine learning model feedback |
US11643102B1 (en) * | 2020-11-23 | 2023-05-09 | Samsara Inc. | Dash cam with artificial intelligence safety event detection |
US20220180109A1 (en) * | 2020-12-04 | 2022-06-09 | Nauto, Inc. | Devices and methods for monitoring drivers of vehicles |
US11545029B2 (en) * | 2021-02-16 | 2023-01-03 | Atieva, Inc. | Distraction-sensitive traffic drive-off alerts |
US20210312193A1 (en) * | 2021-06-15 | 2021-10-07 | Nauto, Inc. | Devices and methods for predicting intersection violations and/or collisions |
EP4355626A1 (en) * | 2021-06-15 | 2024-04-24 | Nauto, Inc. | Devices and methods for predicting collisions, predicting intersection violations, and/or determining region of interest for object detection in camera images |
US11945435B2 (en) * | 2021-06-15 | 2024-04-02 | Nauto, Inc. | Devices and methods for predicting collisions and/or intersection violations |
US20220410908A1 (en) * | 2021-06-29 | 2022-12-29 | Bendix Commercial Vehicle Systems, Llc | System and method for controlling vehicle functions based on evaluated driving team composition |
US12031374B2 (en) * | 2021-08-17 | 2024-07-09 | Ford Global Technologies, Llc | Vehicle having imaging device for driver and window monitoring |
US11386325B1 (en) | 2021-11-12 | 2022-07-12 | Samsara Inc. | Ensemble neural network state machine for detecting distractions |
US11352014B1 (en) | 2021-11-12 | 2022-06-07 | Samsara Inc. | Tuning layers of a modular neural network |
US12043115B2 (en) * | 2021-12-01 | 2024-07-23 | GM Global Technology Operations LLC | Control system comprising first, second, and third inputs and first, second, and third displays, and a method of operating the same |
CN114386763B (en) * | 2021-12-10 | 2023-05-23 | 斑马网络技术有限公司 | Vehicle interaction method, vehicle interaction device and storage medium |
CN114187581B (en) * | 2021-12-14 | 2024-04-09 | 安徽大学 | Driver distraction fine granularity detection method based on unsupervised learning |
CN115923809A (en) * | 2022-11-30 | 2023-04-07 | 蔚来软件科技(上海)有限公司 | Man-machine common driving control method, device and system and vehicle |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140139655A1 (en) * | 2009-09-20 | 2014-05-22 | Tibet MIMAR | Driver distraction and drowsiness warning and sleepiness reduction for accident avoidance |
US20140210978A1 (en) * | 2013-01-25 | 2014-07-31 | Toyota Motor Engineering & Manufacturing North America, Inc. | Method and apparatus for early detection of dynamic attentive states for providing an inattentive warning |
US20160267335A1 (en) * | 2015-03-13 | 2016-09-15 | Harman International Industries, Incorporated | Driver distraction detection system |
Family Cites Families (232)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
DE69032348T2 (en) | 1990-07-31 | 1998-09-24 | Hewlett Packard Co | Object based system |
US8604932B2 (en) | 1992-05-05 | 2013-12-10 | American Vehicular Sciences, LLC | Driver fatigue monitoring system and method |
US7788008B2 (en) | 1995-06-07 | 2010-08-31 | Automotive Technologies International, Inc. | Eye monitoring system and method for vehicular occupants |
US5638116A (en) | 1993-09-08 | 1997-06-10 | Sumitomo Electric Industries, Ltd. | Object recognition apparatus and method |
US7421321B2 (en) | 1995-06-07 | 2008-09-02 | Automotive Technologies International, Inc. | System for obtaining vehicular information |
US5642106A (en) | 1994-12-27 | 1997-06-24 | Siemens Corporate Research, Inc. | Visual incremental turn detector |
US5961571A (en) | 1994-12-27 | 1999-10-05 | Siemens Corporated Research, Inc | Method and apparatus for automatically tracking the location of vehicles |
US5798949A (en) | 1995-01-13 | 1998-08-25 | Kaub; Alan Richard | Traffic safety prediction model |
US6662141B2 (en) | 1995-01-13 | 2003-12-09 | Alan R. Kaub | Traffic safety prediction model |
US20070154063A1 (en) | 1995-06-07 | 2007-07-05 | Automotive Technologies International, Inc. | Image Processing Using Rear View Mirror-Mounted Imaging Device |
US6720920B2 (en) | 1997-10-22 | 2004-04-13 | Intelligent Technologies International Inc. | Method and arrangement for communicating between vehicles |
US7085637B2 (en) | 1997-10-22 | 2006-08-01 | Intelligent Technologies International, Inc. | Method and system for controlling a vehicle |
US5898390A (en) | 1995-09-14 | 1999-04-27 | Zexel Corporation | Method and apparatus for calibration of a distance sensor in a vehicle navigation system |
WO1997029437A1 (en) | 1996-02-09 | 1997-08-14 | Sarnoff Corporation | Method and apparatus for training a neural network to detect and classify objects with uncertain training data |
CN100484228C (en) | 1996-12-04 | 2009-04-29 | 松下电器产业株式会社 | Optical disk reproduction apparatus |
DE19752127A1 (en) | 1997-11-25 | 1999-07-29 | Stockhausen Chem Fab Gmbh | Process for the production of synthetic polymers with a very low residual monomer content, products produced thereafter and their use |
US6240367B1 (en) | 1998-11-27 | 2001-05-29 | Ching-Fang Lin | Full fusion positioning method for vehicle |
JP3243236B2 (en) | 1999-09-24 | 2002-01-07 | 松下電器産業株式会社 | Position data thinning device |
AUPQ896000A0 (en) * | 2000-07-24 | 2000-08-17 | Seeing Machines Pty Ltd | Facial image processing system |
US6502033B1 (en) | 2000-10-05 | 2002-12-31 | Navigation Technologies Corp. | Turn detection algorithm for vehicle positioning |
FR2816200A1 (en) | 2000-11-06 | 2002-05-10 | Praxim | DETERMINING THE POSITION OF A KNEE PROSTHESIS |
US6496117B2 (en) | 2001-03-30 | 2002-12-17 | Koninklijke Philips Electronics N.V. | System for monitoring a driver's attention to driving |
US6927694B1 (en) | 2001-08-20 | 2005-08-09 | Research Foundation Of The University Of Central Florida | Algorithm for monitoring head/eye motion for driver alertness with one camera |
WO2003032143A2 (en) | 2001-10-12 | 2003-04-17 | Hrl Laboratories, Llc | Vision-based pointer tracking method and apparatus |
DE10210130B4 (en) | 2002-03-08 | 2014-12-24 | Robert Bosch Gmbh | Method and device for driver warning |
AU2003280516A1 (en) | 2002-07-01 | 2004-01-19 | The Regents Of The University Of California | Digital processing of video images |
US20040051659A1 (en) | 2002-09-18 | 2004-03-18 | Garrison Darwin A. | Vehicular situational awareness system |
BR0315384A (en) | 2002-10-15 | 2005-09-06 | Volvo Technology Corp | Method and disposition to interpret head and eye activity of individuals |
WO2004038335A1 (en) | 2002-10-22 | 2004-05-06 | Hitachi, Ltd. | Map data delivering method for communication-type navigation system |
WO2004059900A2 (en) | 2002-12-17 | 2004-07-15 | Evolution Robotics, Inc. | Systems and methods for visual simultaneous localization and mapping |
JP4305057B2 (en) | 2003-05-22 | 2009-07-29 | 日産自動車株式会社 | Armpit detector |
DE10323915A1 (en) | 2003-05-23 | 2005-02-03 | Daimlerchrysler Ag | Camera-based position detection for a road vehicle |
EP1638801A1 (en) | 2003-06-06 | 2006-03-29 | Volvo Technology Corporation | Method and arrangement for controlling vehicular subsystems based on interpreted driver activity |
US7212651B2 (en) | 2003-06-17 | 2007-05-01 | Mitsubishi Electric Research Laboratories, Inc. | Detecting pedestrians using patterns of motion and appearance in videos |
DE10336638A1 (en) | 2003-07-25 | 2005-02-10 | Robert Bosch Gmbh | Apparatus for classifying at least one object in a vehicle environment |
WO2005069675A1 (en) | 2004-01-20 | 2005-07-28 | Omron Corporation | Device and method for telephone countermeasure in using telephone during driving |
US20050234679A1 (en) | 2004-02-13 | 2005-10-20 | Evolution Robotics, Inc. | Sequential selective integration of sensor data |
US7689321B2 (en) | 2004-02-13 | 2010-03-30 | Evolution Robotics, Inc. | Robust sensor fusion for mapping and localization in a simultaneous localization and mapping (SLAM) system |
CA2559726C (en) | 2004-03-24 | 2015-10-20 | A9.Com, Inc. | System and method for displaying images in an online directory |
JP4329622B2 (en) | 2004-06-02 | 2009-09-09 | 日産自動車株式会社 | VEHICLE DRIVE OPERATION ASSISTANCE DEVICE AND VEHICLE HAVING VEHICLE DRIVE OPERATION ASSISTANCE DEVICE |
DE102004031557B4 (en) | 2004-06-29 | 2016-12-22 | Conti Temic Microelectronic Gmbh | Method and crash sensor for a device for occupant-relevant activation of occupant protection devices in a motor vehicle in crash cases |
US7195394B2 (en) | 2004-07-19 | 2007-03-27 | Vijay Singh | Method for resonant wave mixing in closed containers |
US7228230B2 (en) | 2004-11-12 | 2007-06-05 | Mitsubishi Denki Kabushiki Kaisha | System for autonomous vehicle navigation with carrier phase DGPS and laser-scanner augmentation |
JP4811019B2 (en) | 2005-01-17 | 2011-11-09 | 株式会社豊田中央研究所 | Impact behavior control device |
CA2934721C (en) | 2005-03-18 | 2018-05-01 | Gatekeeper Systems, Inc. | Two-way communication system for tracking locations and statuses of wheeled vehicles |
EP1876411A4 (en) | 2005-04-25 | 2011-06-29 | Geo Technical Lab Co Ltd | Imaging position analyzing method |
EP2428413B1 (en) | 2005-07-11 | 2013-03-27 | Volvo Technology Corporation | Methods and arrangement for performing driver identity verification |
US20070050108A1 (en) | 2005-08-15 | 2007-03-01 | Larschan Bradley R | Driver activity and vehicle operation logging and reporting |
EP1754621B1 (en) | 2005-08-18 | 2009-10-14 | Honda Research Institute Europe GmbH | Driver assistance system |
US7933786B2 (en) | 2005-11-01 | 2011-04-26 | Accenture Global Services Limited | Collaborative intelligent task processor for insurance claims |
US7423540B2 (en) | 2005-12-23 | 2008-09-09 | Delphi Technologies, Inc. | Method of detecting vehicle-operator state |
US7646922B2 (en) | 2005-12-30 | 2010-01-12 | Honeywell International Inc. | Object classification in video images |
US9477639B2 (en) * | 2006-03-08 | 2016-10-25 | Speed Demon Inc. | Safe driving monitoring system |
TWI302879B (en) | 2006-05-12 | 2008-11-11 | Univ Nat Chiao Tung | Real-time nighttime vehicle detection and recognition system based on computer vision |
JP4724043B2 (en) | 2006-05-17 | 2011-07-13 | トヨタ自動車株式会社 | Object recognition device |
JP4680131B2 (en) | 2006-05-29 | 2011-05-11 | トヨタ自動車株式会社 | Own vehicle position measuring device |
US8487775B2 (en) * | 2006-06-11 | 2013-07-16 | Volvo Technology Corporation | Method and apparatus for determining and analyzing a location of visual interest |
US9558505B2 (en) | 2006-07-18 | 2017-01-31 | American Express Travel Related Services Company, Inc. | System and method for prepaid rewards |
US7853072B2 (en) | 2006-07-20 | 2010-12-14 | Sarnoff Corporation | System and method for detecting still objects in images |
US7912288B2 (en) | 2006-09-21 | 2011-03-22 | Microsoft Corporation | Object detection and recognition system |
US7579942B2 (en) * | 2006-10-09 | 2009-08-25 | Toyota Motor Engineering & Manufacturing North America, Inc. | Extra-vehicular threat predictor |
US8174568B2 (en) | 2006-12-01 | 2012-05-08 | Sri International | Unified framework for precise vision-aided navigation |
US20080243378A1 (en) | 2007-02-21 | 2008-10-02 | Tele Atlas North America, Inc. | System and method for vehicle navigation and piloting including absolute and relative coordinates |
US8606512B1 (en) | 2007-05-10 | 2013-12-10 | Allstate Insurance Company | Route risk mitigation |
US7787689B2 (en) | 2007-07-06 | 2010-08-31 | Topcon Corporation | Location measuring device and method |
JP2009117832A (en) | 2007-11-06 | 2009-05-28 | Asml Netherlands Bv | Method of preparing substrate for lithography, substrate, device manufacturing method, sealing coating applicator, and sealing coating measuring device |
US8577828B2 (en) | 2007-12-12 | 2013-11-05 | New York University | System, method and computer-accessible medium for normalizing databased through mixing |
US8022831B1 (en) | 2008-01-03 | 2011-09-20 | Pamela Wood-Eyre | Interactive fatigue management system and method |
TWI534719B (en) | 2008-03-03 | 2016-05-21 | 艾威吉隆專利第2控股公司 | Dynamic object classification |
US8374420B2 (en) | 2008-07-23 | 2013-02-12 | Daifuku Co., Ltd. | Learning device and learning method for article transport facility |
JP5267570B2 (en) | 2008-12-25 | 2013-08-21 | トヨタ自動車株式会社 | Driving assistance device |
US7868821B2 (en) | 2009-01-15 | 2011-01-11 | Alpine Electronics, Inc | Method and apparatus to estimate vehicle position and recognized landmark positions using GPS and camera |
DE102009005730A1 (en) | 2009-01-22 | 2010-07-29 | Hella Kgaa Hueck & Co. | Method for monitoring concentration of driver of motor vehicle, involves determining reference direction and time target based on position of indicator, information about road course and/or information of lane maintenance assistants |
US8854199B2 (en) | 2009-01-26 | 2014-10-07 | Lytx, Inc. | Driver risk assessment system and method employing automated driver log |
US20100209881A1 (en) | 2009-02-18 | 2010-08-19 | Gm Global Technology Operations, Inc. | Driving skill recognition based on behavioral diagnosis |
US20100209891A1 (en) | 2009-02-18 | 2010-08-19 | Gm Global Technology Operations, Inc. | Driving skill recognition based on stop-and-go driving behavior |
US8254670B2 (en) | 2009-02-25 | 2012-08-28 | Toyota Motor Engineering & Manufacturing North America, Inc. | Self-learning object detection and classification systems and methods |
JP5142047B2 (en) | 2009-02-26 | 2013-02-13 | アイシン・エィ・ダブリュ株式会社 | Navigation device and navigation program |
JP5041099B2 (en) | 2009-02-27 | 2012-10-03 | トヨタ自動車株式会社 | Vehicle relative position estimation device and vehicle relative position estimation method |
US8266132B2 (en) | 2009-03-03 | 2012-09-11 | Microsoft Corporation | Map aggregation |
JP4788798B2 (en) | 2009-04-23 | 2011-10-05 | トヨタ自動車株式会社 | Object detection device |
KR101004664B1 (en) | 2009-06-03 | 2011-01-04 | 주식회사 하이닉스반도체 | Semiconductor memory apparatus and method for operating the same |
US8369608B2 (en) | 2009-06-22 | 2013-02-05 | Toyota Motor Engineering & Manufacturing North America, Inc. | System and method for detecting drowsy facial expressions of vehicle drivers under changing illumination conditions |
WO2010151603A1 (en) | 2009-06-23 | 2010-12-29 | L&P Property Management Company | Drowsy driver detection system |
US9491420B2 (en) | 2009-09-20 | 2016-11-08 | Tibet MIMAR | Vehicle security with accident notification and embedded driver analytics |
US8502860B2 (en) | 2009-09-29 | 2013-08-06 | Toyota Motor Engineering & Manufacturing North America (Tema) | Electronic control system, electronic control unit and associated methodology of adapting 3D panoramic views of vehicle surroundings by predicting driver intent |
AU2009243442B2 (en) | 2009-11-30 | 2013-06-13 | Canon Kabushiki Kaisha | Detection of abnormal behaviour in video objects |
US8805707B2 (en) | 2009-12-31 | 2014-08-12 | Hartford Fire Insurance Company | Systems and methods for providing a safety score associated with a user location |
JP2011180873A (en) | 2010-03-02 | 2011-09-15 | Panasonic Corp | Driving support device and driving support method |
EP2395478A1 (en) | 2010-06-12 | 2011-12-14 | Toyota Motor Europe NV/SA | Monocular 3D pose estimation and tracking by detection |
US8676498B2 (en) | 2010-09-24 | 2014-03-18 | Honeywell International Inc. | Camera and inertial measurement unit integration with navigation data feedback for feature tracking |
JP5257433B2 (en) | 2010-09-30 | 2013-08-07 | ブラザー工業株式会社 | Image reading device |
KR101231510B1 (en) | 2010-10-11 | 2013-02-07 | 현대자동차주식회사 | System for alarming a danger coupled with driver-viewing direction, thereof method and vehicle for using the same |
DE102010049351A1 (en) | 2010-10-23 | 2012-04-26 | Daimler Ag | A method of operating a brake assist device and brake assist device for a vehicle |
US8447519B2 (en) | 2010-11-10 | 2013-05-21 | GM Global Technology Operations LLC | Method of augmenting GPS or GPS/sensor vehicle positioning using additional in-vehicle vision sensors |
US9146558B2 (en) | 2010-11-30 | 2015-09-29 | Irobot Corporation | Mobile robot and method of operating thereof |
KR101306286B1 (en) | 2010-12-17 | 2013-09-09 | 주식회사 팬택 | Apparatus and method for providing augmented reality based on X-ray view |
US8957847B1 (en) * | 2010-12-28 | 2015-02-17 | Amazon Technologies, Inc. | Low distraction interfaces |
US8862395B2 (en) | 2011-01-31 | 2014-10-14 | Raytheon Company | Coded marker navigation system and method |
US8620026B2 (en) | 2011-04-13 | 2013-12-31 | International Business Machines Corporation | Video-based detection of multiple object types under varying poses |
US20140049601A1 (en) | 2011-05-05 | 2014-02-20 | Panono Gmbh | Camera system for capturing images and methods thereof |
JP2013014225A (en) | 2011-07-04 | 2013-01-24 | Toyota Motor Corp | Vehicle collision alarm device |
US8195394B1 (en) | 2011-07-13 | 2012-06-05 | Google Inc. | Object detection and classification for autonomous vehicles |
US8761439B1 (en) | 2011-08-24 | 2014-06-24 | Sri International | Method and apparatus for generating three-dimensional pose using monocular visual sensor and inertial measurement unit |
US8606492B1 (en) | 2011-08-31 | 2013-12-10 | Drivecam, Inc. | Driver log generation |
US8744642B2 (en) | 2011-09-16 | 2014-06-03 | Lytx, Inc. | Driver identification based on face data |
US9235750B1 (en) | 2011-09-16 | 2016-01-12 | Lytx, Inc. | Using passive driver identification and other input for providing real-time alerts or actions |
US11074495B2 (en) | 2013-02-28 | 2021-07-27 | Z Advanced Computing, Inc. (Zac) | System and method for extremely efficient image and pattern recognition and artificial intelligence platform |
US20130093886A1 (en) | 2011-10-18 | 2013-04-18 | Ariel Inventions, Llc | Method and system for using a vehicle-based digital imagery system to identify another vehicle |
US9111147B2 (en) | 2011-11-14 | 2015-08-18 | Massachusetts Institute Of Technology | Assisted video surveillance of persons-of-interest |
US20130147661A1 (en) | 2011-12-07 | 2013-06-13 | International Business Machines Corporation | System and method for optical landmark identification for gps error correction |
US8811938B2 (en) | 2011-12-16 | 2014-08-19 | Microsoft Corporation | Providing a user interface experience based on inferred vehicle state |
WO2013101054A1 (en) | 2011-12-29 | 2013-07-04 | Intel Corporation | Systems, methods, and apparatus for identifying an occupant of a vehicle |
US9784843B2 (en) | 2012-01-17 | 2017-10-10 | Limn Tech LLC | Enhanced roadway mark locator, inspection apparatus, and marker |
JP5863481B2 (en) | 2012-01-30 | 2016-02-16 | 日立マクセル株式会社 | Vehicle collision risk prediction device |
US8457827B1 (en) | 2012-03-15 | 2013-06-04 | Google Inc. | Modifying behavior of autonomous vehicle based on predicted behavior of other vehicles |
JP5729345B2 (en) | 2012-04-10 | 2015-06-03 | 株式会社デンソー | Emotion monitoring system |
US8634822B2 (en) | 2012-06-24 | 2014-01-21 | Tango Networks, Inc. | Automatic identification of a vehicle driver based on driving behavior |
SE536586C2 (en) | 2012-07-02 | 2014-03-11 | Scania Cv Ab | Device and method for assessing accident risk when driving a vehicle |
US9854159B2 (en) * | 2012-07-20 | 2017-12-26 | Pixart Imaging Inc. | Image system with eye protection |
JP5944781B2 (en) | 2012-07-31 | 2016-07-05 | 株式会社デンソーアイティーラボラトリ | Mobile object recognition system, mobile object recognition program, and mobile object recognition method |
US8510196B1 (en) | 2012-08-16 | 2013-08-13 | Allstate Insurance Company | Feedback loop in mobile damage assessment and claims processing |
US9365162B2 (en) | 2012-08-20 | 2016-06-14 | Magna Electronics Inc. | Method of obtaining data relating to a driver assistance system of a vehicle |
DE102012016800A1 (en) | 2012-08-23 | 2014-02-27 | Audi Ag | Method and device for determining a vehicle position in a mapped environment |
DE102012216386A1 (en) | 2012-09-14 | 2014-03-20 | Robert Bosch Gmbh | Method for operating a driver assistance system of a vehicle |
GB2506365B (en) | 2012-09-26 | 2017-12-20 | Masternaut Risk Solutions Ltd | Vehicle incident detection |
US9488492B2 (en) | 2014-03-18 | 2016-11-08 | Sri International | Real-time system for multi-modal 3D geospatial mapping, object recognition, scene annotation and analytics |
JP2014081831A (en) * | 2012-10-17 | 2014-05-08 | Denso Corp | Vehicle driving assistance system using image information |
US9535878B1 (en) * | 2012-12-19 | 2017-01-03 | Allstate Insurance Company | Driving event data analysis |
US9081650B1 (en) | 2012-12-19 | 2015-07-14 | Allstate Insurance Company | Traffic based driving analysis |
EP2752348A1 (en) | 2013-01-04 | 2014-07-09 | Continental Automotive Systems, Inc. | Adaptive emergency brake and steer assist system based on driver focus |
CN104903946B (en) | 2013-01-09 | 2016-09-28 | 三菱电机株式会社 | Vehicle surrounding display device |
US9367065B2 (en) | 2013-01-25 | 2016-06-14 | Google Inc. | Modifying behavior of autonomous vehicles based on sensor blind spots and limitations |
US9439036B2 (en) | 2013-01-25 | 2016-09-06 | Visa International Service Association | Systems and methods to select locations of interest based on distance from route points or route paths |
US8952819B2 (en) * | 2013-01-31 | 2015-02-10 | Lytx, Inc. | Direct observation event triggering of drowsiness |
US9092986B2 (en) | 2013-02-04 | 2015-07-28 | Magna Electronics Inc. | Vehicular vision system |
US8799034B1 (en) | 2013-03-08 | 2014-08-05 | Allstate University Company | Automated accident detection, fault attribution, and claims processing |
US20140267703A1 (en) | 2013-03-15 | 2014-09-18 | Robert M. Taylor | Method and Apparatus of Mapping Landmark Position and Orientation |
GB2512317A (en) | 2013-03-26 | 2014-10-01 | Jaguar Land Rover Ltd | Vehicle control system and method |
US9349113B2 (en) | 2013-03-26 | 2016-05-24 | 3 Strike, Llc | Storage container with inventory control |
JP5878491B2 (en) | 2013-03-27 | 2016-03-08 | 株式会社日本自動車部品総合研究所 | Driving assistance device |
US9260095B2 (en) | 2013-06-19 | 2016-02-16 | Magna Electronics Inc. | Vehicle vision system with collision mitigation |
EP3017403A2 (en) | 2013-07-01 | 2016-05-11 | Agent Video Intelligence Ltd. | System and method for abnormality detection |
US20150025917A1 (en) | 2013-07-15 | 2015-01-22 | Advanced Insurance Products & Services, Inc. | System and method for determining an underwriting risk, risk score, or price of insurance using cognitive information |
JP6398347B2 (en) | 2013-08-15 | 2018-10-03 | 株式会社リコー | Image processing apparatus, recognition object detection method, recognition object detection program, and moving object control system |
US9201424B1 (en) | 2013-08-27 | 2015-12-01 | Google Inc. | Camera calibration using structure from motion techniques |
EP2851870B1 (en) | 2013-09-20 | 2019-01-23 | Application Solutions (Electronics and Vision) Limited | Method for estimating ego motion of an object |
US20150084757A1 (en) | 2013-09-23 | 2015-03-26 | Agero, Inc. | Methods and systems for determining auto accidents using mobile phones and initiating emergency response |
US9495602B2 (en) | 2013-10-23 | 2016-11-15 | Toyota Motor Engineering & Manufacturing North America, Inc. | Image and map-based detection of vehicles at intersections |
US9305214B1 (en) | 2013-10-29 | 2016-04-05 | The United States Of America, As Represented By The Secretary Of The Navy | Systems and methods for real-time horizon detection in images |
CN103661375A (en) * | 2013-11-25 | 2014-03-26 | 同济大学 | Lane departure alarming method and system with driving distraction state considered |
JP6325806B2 (en) | 2013-12-06 | 2018-05-16 | 日立オートモティブシステムズ株式会社 | Vehicle position estimation system |
US9327743B2 (en) | 2013-12-19 | 2016-05-03 | Thales Canada Inc | Guideway mounted vehicle localization system |
JP6248208B2 (en) * | 2014-02-04 | 2017-12-13 | フラウンホーファー−ゲゼルシャフト・ツール・フェルデルング・デル・アンゲヴァンテン・フォルシュング・アインゲトラーゲネル・フェライン | 3D image analysis apparatus for determining a line-of-sight direction |
US10586570B2 (en) | 2014-02-05 | 2020-03-10 | Snap Inc. | Real time video processing for changing proportions of an object in the video |
US9342888B2 (en) | 2014-02-08 | 2016-05-17 | Honda Motor Co., Ltd. | System and method for mapping, localization and pose correction of a vehicle based on images |
SG11201606716VA (en) | 2014-02-14 | 2016-09-29 | Nec Corp | Video analysis system |
WO2015160900A1 (en) | 2014-04-15 | 2015-10-22 | Maris, Ltd | Assessing asynchronous authenticated data sources for use in driver risk management |
US9158962B1 (en) | 2014-05-07 | 2015-10-13 | Lytx, Inc. | Passive driver identification |
EP2950294B1 (en) | 2014-05-30 | 2019-05-08 | Honda Research Institute Europe GmbH | Method and vehicle with an advanced driver assistance system for risk-based traffic scene analysis |
CN106463054B (en) | 2014-06-03 | 2019-04-02 | 宝马股份公司 | The adaptive alarm management of advanced driving assistance system (ADAS) |
CN112528737B (en) | 2014-06-03 | 2022-03-01 | 御眼视觉技术有限公司 | System and method for detecting objects |
US9242654B2 (en) | 2014-06-27 | 2016-01-26 | International Business Machines Corporation | Determining vehicle collision risk |
EP3161572B1 (en) | 2014-06-27 | 2019-01-23 | Crown Equipment Corporation | Lost vehicle recovery utilizing associated feature pairs |
US20160063761A1 (en) | 2014-08-27 | 2016-03-03 | Toyota Jidosha Kabushiki Kaisha | Communication of spatial information based on driver attention assessment |
JP6441616B2 (en) | 2014-08-29 | 2018-12-19 | 株式会社ゼンリン | Positioning device, driving support device, and control program |
US20160086021A1 (en) | 2014-09-24 | 2016-03-24 | 1A Smart Start, Inc. | Substance Testing Systems and Methods with Test Subject Identification Using Electronic Facial Recognition Techniques |
US9519289B2 (en) | 2014-11-26 | 2016-12-13 | Irobot Corporation | Systems and methods for performing simultaneous localization and mapping using machine vision systems |
US9886856B2 (en) | 2014-12-04 | 2018-02-06 | Here Global B.V. | Near miss system |
DE102014226185B4 (en) | 2014-12-17 | 2022-09-29 | Bayerische Motoren Werke Aktiengesellschaft | Method and line of sight recognition system for determining a line of sight of a person, and use of the line of sight recognition system in a motor vehicle |
US9573592B2 (en) | 2014-12-23 | 2017-02-21 | Toyota Motor Engineering & Manufacturing North America, Inc. | Risk mitigation for autonomous vehicles relative to oncoming objects |
EP3057061B1 (en) | 2015-02-16 | 2017-08-30 | Application Solutions (Electronics and Vision) Limited | Method and device for the estimation of car egomotion from surround view images |
US20160244022A1 (en) | 2015-02-24 | 2016-08-25 | Ford Global Technologies, Llc | Vehicle control action sequence for operator authentication |
US11113941B2 (en) | 2015-02-27 | 2021-09-07 | Carrier Corporation | Ambient light sensor in a hazard detector and a method of using the same |
GB201503413D0 (en) | 2015-02-27 | 2015-04-15 | Caring Community Sa | Improved navigation system |
JP6411917B2 (en) | 2015-02-27 | 2018-10-24 | 株式会社日立製作所 | Self-position estimation apparatus and moving body |
US10416764B2 (en) * | 2015-03-13 | 2019-09-17 | Apple Inc. | Method for operating an eye tracking device for multi-user eye tracking and eye tracking device |
US9505413B2 (en) * | 2015-03-20 | 2016-11-29 | Harman International Industries, Incorporated | Systems and methods for prioritized driver alerts |
KR20160114992A (en) | 2015-03-25 | 2016-10-06 | 한국전자통신연구원 | Bin-picking system and method for bin-picking |
JP2016197083A (en) | 2015-04-06 | 2016-11-24 | ソニー株式会社 | Control device, method, and program |
DE102015206200A1 (en) | 2015-04-08 | 2016-10-13 | Robert Bosch Gmbh | Method and device for attention recognition of a driver |
US20160300242A1 (en) | 2015-04-10 | 2016-10-13 | Uber Technologies, Inc. | Driver verification system for transport services |
US9767625B1 (en) | 2015-04-13 | 2017-09-19 | Allstate Insurance Company | Automatic crash detection |
US20160325680A1 (en) | 2015-05-04 | 2016-11-10 | Kamama, Inc. | System and method of vehicle sensor management |
KR101693991B1 (en) | 2015-05-18 | 2017-01-17 | 현대자동차주식회사 | System and method for cutting high voltage of battery for vehicle |
US10035509B2 (en) | 2015-08-06 | 2018-07-31 | Safer Technology Solutions LLC | Early warning intersection device |
US9679480B2 (en) | 2015-08-07 | 2017-06-13 | Ford Global Technologies, Llc | Vehicle driver responsibility factor assessment and broadcast |
US9845097B2 (en) | 2015-08-12 | 2017-12-19 | Ford Global Technologies, Llc | Driver attention evaluation |
KR20170020036A (en) | 2015-08-13 | 2017-02-22 | 현대자동차주식회사 | System applied to secret mode about biometric information of driver and Method for operating thereof |
US10586102B2 (en) | 2015-08-18 | 2020-03-10 | Qualcomm Incorporated | Systems and methods for object tracking |
US20170053555A1 (en) | 2015-08-21 | 2017-02-23 | Trimble Navigation Limited | System and method for evaluating driver behavior |
JP6406171B2 (en) | 2015-08-25 | 2018-10-17 | トヨタ自動車株式会社 | Blink detection device |
US9996756B2 (en) | 2015-08-31 | 2018-06-12 | Lytx, Inc. | Detecting risky driving with machine vision |
US11006162B2 (en) | 2015-08-31 | 2021-05-11 | Orcam Technologies Ltd. | Systems and methods for analyzing information collected by wearable systems |
US10150448B2 (en) | 2015-09-18 | 2018-12-11 | Ford Global Technologies. Llc | Autonomous vehicle unauthorized passenger or object detection |
US11307042B2 (en) | 2015-09-24 | 2022-04-19 | Allstate Insurance Company | Three-dimensional risk maps |
US9914460B2 (en) | 2015-09-25 | 2018-03-13 | Mcafee, Llc | Contextual scoring of automobile drivers |
EP3358550A4 (en) | 2015-09-30 | 2019-06-12 | Sony Corporation | Information processing device and information processing method |
US9718468B2 (en) | 2015-10-13 | 2017-08-01 | Verizon Patent And Licensing Inc. | Collision prediction system |
US9981662B2 (en) | 2015-10-15 | 2018-05-29 | Ford Global Technologies, Llc | Speed limiting comfort enhancement |
US10515417B2 (en) | 2015-10-16 | 2019-12-24 | Accenture Global Services Limited | Device based incident detection and notification |
EP3159853B1 (en) | 2015-10-23 | 2019-03-27 | Harman International Industries, Incorporated | Systems and methods for advanced driver assistance analytics |
US9734455B2 (en) | 2015-11-04 | 2017-08-15 | Zoox, Inc. | Automated extraction of semantic information to enhance incremental mapping modifications for robotic vehicles |
US10019637B2 (en) | 2015-11-13 | 2018-07-10 | Honda Motor Co., Ltd. | Method and system for moving object detection with single camera |
US9491374B1 (en) | 2015-12-11 | 2016-11-08 | Fuji Xerox Co., Ltd. | Systems and methods for videoconferencing input and display management based on activity |
WO2017106846A2 (en) | 2015-12-18 | 2017-06-22 | Iris Automation, Inc. | Real-time visual situational awareness system |
CN105654753A (en) * | 2016-01-08 | 2016-06-08 | 北京乐驾科技有限公司 | Intelligent vehicle-mounted safe driving assistance method and system |
US10460600B2 (en) | 2016-01-11 | 2019-10-29 | NetraDyne, Inc. | Driver behavior monitoring |
US10269075B2 (en) | 2016-02-02 | 2019-04-23 | Allstate Insurance Company | Subjective route risk mapping and mitigation |
US9892558B2 (en) | 2016-02-19 | 2018-02-13 | The Boeing Company | Methods for localization using geotagged photographs and three-dimensional visualization |
US10289113B2 (en) | 2016-02-25 | 2019-05-14 | Ford Global Technologies, Llc | Autonomous occupant attention-based control |
JP6724425B2 (en) | 2016-03-04 | 2020-07-15 | アイシン精機株式会社 | Parking assistance device |
US9701307B1 (en) | 2016-04-11 | 2017-07-11 | David E. Newman | Systems and methods for hazard mitigation |
US10235585B2 (en) | 2016-04-11 | 2019-03-19 | The Nielsen Company (US) | Methods and apparatus to determine the dimensions of a region of interest of a target object from an image using target object landmarks |
US10247565B2 (en) | 2016-04-11 | 2019-04-02 | State Farm Mutual Automobile Insurance Company | Traffic risk avoidance for a route selection system |
US10323952B2 (en) | 2016-04-26 | 2019-06-18 | Baidu Usa Llc | System and method for presenting media contents in autonomous vehicles |
US10126141B2 (en) | 2016-05-02 | 2018-11-13 | Google Llc | Systems and methods for using real-time imagery in navigation |
JP6702543B2 (en) | 2016-05-31 | 2020-06-03 | 株式会社東芝 | Information processing apparatus, method and program |
JP6384521B2 (en) | 2016-06-10 | 2018-09-05 | トヨタ自動車株式会社 | Vehicle driving support device |
US10007854B2 (en) | 2016-07-07 | 2018-06-26 | Ants Technology (Hk) Limited | Computer vision based driver assistance devices, systems, methods and associated computer executable code |
IL247101B (en) | 2016-08-03 | 2018-10-31 | Pointgrab Ltd | Method and system for detecting an occupant in an image |
EP3497405B1 (en) | 2016-08-09 | 2022-06-15 | Nauto, Inc. | System and method for precision localization and mapping |
US20180046869A1 (en) | 2016-08-10 | 2018-02-15 | Surround.IO Corporation | Method and Apparatus for Providing Information Via Collected and Stored Metadata Using Inferred Attentional Model |
AU2017315462B2 (en) | 2016-08-26 | 2021-02-04 | Allstate Insurance Company | Automatic hail damage detection and repair |
EP3513265A4 (en) * | 2016-09-14 | 2020-04-22 | Nauto Global Limited | Systems and methods for near-crash determination |
US9739627B1 (en) | 2016-10-18 | 2017-08-22 | Allstate Insurance Company | Road frustration index risk mapping and mitigation |
EP3535646A4 (en) * | 2016-11-07 | 2020-08-12 | Nauto, Inc. | System and method for driver distraction determination |
US20180176173A1 (en) | 2016-12-15 | 2018-06-21 | Google Inc. | Detecting extraneous social media messages |
US10259452B2 (en) | 2017-01-04 | 2019-04-16 | International Business Machines Corporation | Self-driving vehicle collision management system |
JP6867184B2 (en) | 2017-02-13 | 2021-04-28 | トヨタ自動車株式会社 | Driving support device |
US10078790B2 (en) | 2017-02-16 | 2018-09-18 | Honda Motor Co., Ltd. | Systems for generating parking maps and methods thereof |
US11347054B2 (en) | 2017-02-16 | 2022-05-31 | Magic Leap, Inc. | Systems and methods for augmented reality |
-
2017
- 2017-11-07 EP EP17867780.3A patent/EP3535646A4/en active Pending
- 2017-11-07 CN CN201780082650.7A patent/CN110178104A/en active Pending
- 2017-11-07 WO PCT/US2017/060307 patent/WO2018085804A1/en unknown
- 2017-11-07 JP JP2019522480A patent/JP7290567B2/en active Active
- 2017-11-07 US US15/805,348 patent/US10246014B2/en active Active
-
2019
- 2019-01-03 US US16/239,326 patent/US10703268B2/en active Active
-
2020
- 2020-07-03 US US16/946,749 patent/US11485284B2/en active Active
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140139655A1 (en) * | 2009-09-20 | 2014-05-22 | Tibet MIMAR | Driver distraction and drowsiness warning and sleepiness reduction for accident avoidance |
US20140210978A1 (en) * | 2013-01-25 | 2014-07-31 | Toyota Motor Engineering & Manufacturing North America, Inc. | Method and apparatus for early detection of dynamic attentive states for providing an inattentive warning |
US20160267335A1 (en) * | 2015-03-13 | 2016-09-15 | Harman International Industries, Incorporated | Driver distraction detection system |
Non-Patent Citations (1)
Title |
---|
See also references of EP3535646A4 * |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11571969B2 (en) * | 2020-06-04 | 2023-02-07 | Toyota Motor Engineering & Manufacturing North America, Inc. | External communication suppression device for driving automation |
DE102020123744A1 (en) | 2020-09-11 | 2022-03-17 | Bayerische Motoren Werke Aktiengesellschaft | Determining a line of sight of a person |
EP3992789A1 (en) | 2020-11-02 | 2022-05-04 | BSH Hausgeräte GmbH | Coupling of a domestic appliance with a mobile device |
DE102020213755A1 (en) | 2020-11-02 | 2022-05-05 | BSH Hausgeräte GmbH | Pairing a home appliance with a mobile device |
DE102020213755B4 (en) | 2020-11-02 | 2023-05-04 | BSH Hausgeräte GmbH | Pairing a home appliance with a mobile device |
WO2022161810A1 (en) * | 2021-02-01 | 2022-08-04 | Volkswagen Aktiengesellschaft | Estimating a degree of driver distraction |
Also Published As
Publication number | Publication date |
---|---|
US20180126901A1 (en) | 2018-05-10 |
EP3535646A4 (en) | 2020-08-12 |
CN110178104A (en) | 2019-08-27 |
JP2020501227A (en) | 2020-01-16 |
US10703268B2 (en) | 2020-07-07 |
JP7290567B2 (en) | 2023-06-13 |
EP3535646A1 (en) | 2019-09-11 |
US11485284B2 (en) | 2022-11-01 |
US20200331387A1 (en) | 2020-10-22 |
US10246014B2 (en) | 2019-04-02 |
US20190152390A1 (en) | 2019-05-23 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11485284B2 (en) | System and method for driver distraction determination | |
US11375338B2 (en) | Method for smartphone-based accident detection | |
US11281944B2 (en) | System and method for contextualized vehicle operation determination | |
US10769456B2 (en) | Systems and methods for near-crash determination | |
KR101276770B1 (en) | Advanced driver assistance system for safety driving using driver adaptive irregular behavior detection | |
US10336252B2 (en) | Long term driving danger prediction system | |
EP3698340A1 (en) | Method and system for vehicular-related communications | |
JP2012048310A (en) | Driving support system, on-vehicle device and information distribution device | |
JP2020521978A (en) | Systems and methods for determining safe routes | |
JP2024525153A (en) | DEVICE AND METHOD FOR PREDICTING COLLASONS, PREDICTING INTERSECTION VIOLATIONS AND/OR DETERMINING AREAS OF INTEREST FOR OBJECT DETECTION IN CAMERA IMAGES - Patent application | |
JP5742180B2 (en) | Gaze point estimation device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 17867780 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2019522480 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
ENP | Entry into the national phase |
Ref document number: 2017867780 Country of ref document: EP Effective date: 20190607 |