WO2020078465A1 - 驾驶状态分析方法和装置、驾驶员监控***、车辆 - Google Patents

驾驶状态分析方法和装置、驾驶员监控***、车辆 Download PDF

Info

Publication number
WO2020078465A1
WO2020078465A1 PCT/CN2019/111932 CN2019111932W WO2020078465A1 WO 2020078465 A1 WO2020078465 A1 WO 2020078465A1 CN 2019111932 W CN2019111932 W CN 2019111932W WO 2020078465 A1 WO2020078465 A1 WO 2020078465A1
Authority
WO
WIPO (PCT)
Prior art keywords
driver
state
distraction
information
action
Prior art date
Application number
PCT/CN2019/111932
Other languages
English (en)
French (fr)
Inventor
秦仁波
杨大乾
Original Assignee
上海商汤智能科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 上海商汤智能科技有限公司 filed Critical 上海商汤智能科技有限公司
Priority to KR1020207029610A priority Critical patent/KR102469234B1/ko
Priority to JP2020551931A priority patent/JP2021516829A/ja
Priority to SG11202009420RA priority patent/SG11202009420RA/en
Publication of WO2020078465A1 publication Critical patent/WO2020078465A1/zh
Priority to US17/031,030 priority patent/US11386679B2/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
    • G06V20/597—Recognising the driver's state or behaviour, e.g. attention or drowsiness
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60Q—ARRANGEMENT OF SIGNALLING OR LIGHTING DEVICES, THE MOUNTING OR SUPPORTING THEREOF OR CIRCUITS THEREFOR, FOR VEHICLES IN GENERAL
    • B60Q9/00—Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00, e.g. haptic signalling
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
    • B60W40/08—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to drivers or passengers
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/08—Interaction between the driver and the control system
    • B60W50/14—Means for informing the driver, warning the driver or prompting a driver intervention
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
    • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
    • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/107—Static hand or arm
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • G06V40/171—Local features and components; Facial parts ; Occluding parts, e.g. glasses; Geometrical relationships
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G06V40/19—Sensors therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
    • G—PHYSICS
    • G08—SIGNALLING
    • G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
    • G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
    • G08B21/02—Alarms for ensuring the safety of persons
    • G08B21/06—Alarms for ensuring the safety of persons indicating a condition of sleep, e.g. anti-dozing alarms
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/08—Interaction between the driver and the control system
    • B60W50/14—Means for informing the driver, warning the driver or prompting a driver intervention
    • B60W2050/143—Alarm means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30268—Vehicle interior

Definitions

  • the present disclosure relates to computer vision technology, and in particular, to a driving state analysis method and device, a driver monitoring system, and a vehicle.
  • the embodiments of the present disclosure provide a technical solution for driving state analysis.
  • a driving state analysis method includes:
  • a driving state analysis device includes:
  • the driving state detection module is used to detect the driver's fatigue state and the distraction state on the driver image, and obtain the fatigue state detection result and the distraction state detection result;
  • An alarm module configured to respond to one of the fatigue state detection result and the distraction state detection result satisfying a predetermined alarm condition, outputting alarm information of a corresponding detection result satisfying the predetermined alarm condition; and / or in response to Both the fatigue state detection result and the distraction state detection result both satisfy a predetermined alarm condition, and output alarm information of the fatigue state detection result satisfying the predetermined alarm condition.
  • a driver monitoring system includes:
  • Driving state analysis device used to perform driver's fatigue state detection and distraction state detection on the driver's image; in response to one of the fatigue state detection result and the distraction state detection result satisfying a predetermined alarm condition, outputting a predetermined alarm Alarm information of the corresponding detection result of the condition; and / or, in response to both the fatigue state detection result and the distraction state detection result satisfying the predetermined alarm condition, outputting the alarm information of the fatigue state detection result satisfying the predetermined alarm condition.
  • an electronic device includes:
  • Memory used to store computer programs
  • the processor is configured to execute a computer program stored in the memory, and when the computer program is executed, implement the method described in any one of the above embodiments of the present disclosure.
  • a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, the computer program can be implemented as described in any of the above embodiments of the present disclosure Methods.
  • a vehicle including a central control system, and further comprising: a driving state analysis device according to any of the above embodiments of the present disclosure, or any of the above embodiments of the present disclosure Driver monitoring system.
  • the driver monitoring system Based on the driving state analysis method and device, the driver monitoring system, the vehicle, the electronic equipment, and the media provided by the above embodiments of the present disclosure, it is possible to realize the common detection of the driver's fatigue state and the driver's distraction state on the driver image.
  • the alarm information of the corresponding detection result meeting the predetermined alarm condition is output; and / or both the fatigue state detection result and the distraction state detection result
  • the alarm information of the fatigue state detection results that meet the predetermined alarm conditions is output to remind the driver to improve driving safety and reduce the incidence of road traffic accidents; and, the fatigue state detection results and points
  • both of the heart state detection results meet the predetermined alarm condition, only the alarm information of the fatigue state detection result satisfying the predetermined alarm condition is output, which can avoid the distraction and disgust of the driver caused by too many or too frequent alarms.
  • Alarm strategy improves driving safety And user experience.
  • FIG. 1 is a flowchart of an embodiment of a driving state analysis method of the present disclosure.
  • FIG. 2 is a flowchart of another embodiment of a driving state analysis method of the present disclosure.
  • FIG. 3 is a flowchart of an embodiment of performing predetermined distraction detection on a driver image in an embodiment of the present disclosure.
  • FIG. 4 is a schematic structural diagram of an embodiment of a driving state analysis device of the present disclosure.
  • FIG. 5 is a schematic structural diagram of another embodiment of a driving state analysis device of the present disclosure.
  • FIG. 6 is a schematic structural diagram of an embodiment of a driver monitoring system of the present disclosure.
  • FIG. 7 is a schematic structural diagram of an application embodiment of an electronic device of the present disclosure.
  • FIG. 8 is a schematic structural diagram of an embodiment of the disclosed vehicle.
  • a plurality may refer to two or more, and “at least one” may refer to one, two, or more than two.
  • the term "and / or” in the disclosure is just an association relationship that describes the related objects, indicating that there can be three relationships, for example, A and / or B, which can mean: A exists alone, A and B exist at the same time, There are three cases of B alone.
  • the character “/” in the present disclosure generally indicates that the related objects before and after are in an “or” relationship.
  • Embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general-purpose or special-purpose computing system environments or configurations.
  • Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with terminal devices, computer systems, servers, and other electronic devices include, but are not limited to: in-vehicle devices, personal computer systems, server computer systems, thin clients , Thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above, and many more.
  • Electronic devices such as terminal devices, computer systems, and servers may be described in the general context of computer system executable instructions (such as program modules) executed by the computer system.
  • program modules may include routines, programs, target programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types.
  • the computer system / server can be implemented in a distributed cloud computing environment. In a distributed cloud computing environment, tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules may be located on local or remote computing system storage media including storage devices.
  • the neural network in each embodiment of the present disclosure may be a multi-layer neural network (ie, deep neural network), wherein the neural network may be a multi-layer convolutional neural network, such as LeNet, AlexNet, GoogLeNet, VGG , ResNet and other arbitrary neural network models.
  • Each neural network may use a neural network of the same type and structure, or a neural network of a different type and / or structure. The embodiments of the present disclosure do not limit this.
  • FIG. 1 is a flowchart of an embodiment of a driving state analysis method of the present disclosure. As shown in FIG. 1, the driving state analysis method of this embodiment includes:
  • the operation 102 may be executed by the processor invoking the corresponding instruction stored in the memory, or by the driving state detection module executed by the processor.
  • the fatigue state detection result and the distraction state detection result satisfying the predetermined alarm condition In response to one of the fatigue state detection result and the distraction state detection result satisfying the predetermined alarm condition, output alarm information corresponding to the detection result satisfying the predetermined alarm condition, for example, by sound (such as voice or ringing, etc.) / Light (such as lighting or flashing lights) / vibration to alarm; and / or, in response to both the fatigue state detection result and the distraction state detection result satisfying the predetermined alarm condition, output the fatigue state detection meeting the predetermined alarm condition
  • the resulting alarm information for example, sound (such as voice or ringing, etc.) / light (such as lighting or flashing lights) / vibration and other ways to alarm.
  • outputting alarm information of a corresponding detection result satisfying the predetermined alarm condition may include: When the fatigue state detection result is the fatigue state, a prompt / alarm message corresponding to the fatigue state detection result is output; and / or, when the distraction state detection result is the distraction state, a prompt corresponding to the distraction state detection result is output / Alarm information.
  • the operation 104 may be executed by the processor invoking the corresponding instruction stored in the memory, or may be executed by the alarm module executed by the processor.
  • One of the results of the fatigue state detection and the distraction state detection is When the predetermined alarm condition is satisfied, output the alarm information of the corresponding detection result that meets the predetermined alarm condition; and / or, when both the fatigue state detection result and the distraction state detection result satisfy the predetermined alarm condition, output the fatigue that meets the predetermined alarm condition.
  • the alarm information of the state detection result in order to remind the driver to improve driving safety and reduce the incidence of road traffic accidents; and, when both the fatigue state detection result and the distraction state detection result meet the predetermined alarm condition, only Outputting the alarm information of the fatigue state detection result that satisfies the predetermined alarm condition can prevent the driver from being distracted and disgusted by too many or too frequent alarms.
  • the present disclosure improves the safety of assisted driving and the user experience by optimizing the alarm strategy.
  • it may further include:
  • the alarm information corresponding to the fatigue state detection result Within a preset time after outputting the alarm information corresponding to the fatigue state detection result, suppress the alarm information corresponding to other detection results (such as the distraction state detection result) that meet the predetermined alarm condition; and / or, output the Within a preset time after the alarm information corresponding to the detection result of the distraction state, the alarm information corresponding to other detection results (such as the fatigue state detection result) satisfying the predetermined alarm condition is suppressed.
  • This embodiment can further avoid distracting and disgusting of the driver caused by excessive or too frequent alarms. By further optimizing the alarm strategy, the safety and user experience of assisted driving are further improved.
  • FIG. 2 is a flowchart of another embodiment of a driving state analysis method of the present disclosure. As shown in FIG. 2, the driving state analysis method of this embodiment includes:
  • operation 204 If the deviation angle of the driver's head position exceeds the preset range, operation 204 is performed. Otherwise, if the deviation angle of the driver's head position does not exceed the preset range, operation 206 is performed.
  • the operation 202 may be executed by the processor invoking the corresponding instruction stored in the memory, or may be executed by the first determination module executed by the processor.
  • performing the driver's distraction state detection on the driver image may include: performing a head posture detection and / or eye state detection on the driver image to obtain head posture information and / or eye Status information; based on head posture information and / or eye status information, determine the driver's distraction status detection result, for example, based on head posture information and / or eye status information, determine the distraction used to characterize the driver
  • the parameter value of the state index determines the driver's distraction state detection result based on the parameter value of the index used to characterize the driver's distraction state.
  • performing the driver's fatigue state detection and the distraction state detection on the driver image may include: performing head posture detection, eye state detection, and / or mouth state detection on the driver image to obtain Head posture information, eye state information and / or mouth state information; based on head posture information, eye state information and / or mouth state information, determine the driver's fatigue state detection results and distraction state detection results, For example, based on the head posture information, eye state information, and / or mouth state information, the parameter values of the index used to characterize the fatigue state of the driver and the parameter values of the index used to characterize the driver's distraction state are determined, The driver's fatigue state detection result is determined based on the parameter value of the index characterizing the driver's fatigue state, and the driver's distraction state detection result is determined based on the parameter value of the index characterizing the driver's distraction state .
  • the parameter value of the index used to characterize the fatigue state of the driver is determined according to the head posture information, eye state information, and mouth state information.
  • the operations 204-206 may be executed by the processor invoking the corresponding instruction stored in the memory, or by the driving state detection module executed by the processor.
  • the operation 208 may be executed by the processor by calling a corresponding instruction stored in the memory, or by an alarm module executed by the processor.
  • the driver when the deviation angle of the head position exceeds the preset range, the driver may be in a distracted state, so only the detection of the driver's distracted state, not the detection of fatigue, can be achieved
  • the effect of driving state monitoring can save the computing resources required for fatigue state detection and improve the efficiency of driving state analysis.
  • the driver When the deviation angle of the head position does not exceed the preset range, the driver may be in a distracted state or a fatigue state. Therefore, the driver is simultaneously tested for the distraction state and the fatigue state to realize the common monitoring of the driving state. To ensure the safety of driving.
  • performing head posture detection, eye state detection, and / or mouth state detection on the driver image to obtain head posture information, eye state information, and / or mouth state information may include:
  • the head pose information when the head pose information is obtained according to the detected face key points, the head pose information may be obtained based on the face key points via the first neural network.
  • the head pose information when the head pose information is obtained according to the detected face key points, for example, the head pose information may be obtained based on the face key points via the first neural network.
  • the head posture can be represented by the head posture angle / Eulerian angle in the normalized spherical coordinate system (that is, the camera coordinate system where the camera is located) under normal driving conditions, where the posture angle / Eulerian Angles include: pitch angle ⁇ (pitch), yaw angle ⁇ (yaw), roll angle ⁇ (roll), and head attitude information includes: (pitch, yaw, roll).
  • the pitch angle is used to indicate the angle of the face down or head up in the vertical direction
  • the yaw angle is used to indicate the angle of the side of the face in the horizontal direction (that is, turning the head)
  • the roll angle is used to indicate the vertical direction The angle of the face tilting its head (ie leaning against the shoulder).
  • the detection result of the distraction state may be determined to be a distraction state (ie, inattentiveness).
  • the pitch angle changes from 0 degrees to a certain angle within a preset short period of time and then returns to 0 degrees (corresponding to the nodding movement of the head from the normal position to lower the head suddenly, and then return to the normal position to doze) can Determine the fatigue state test result as fatigue state (ie fatigue driving level).
  • the head posture information can be determined based on the angle between the camera and the camera when the camera is directly in front of the driver's position and facing the driver's position as the reference head posture.
  • the specific implementation can be known based on the records of the embodiments of the present disclosure, which will not be repeated here.
  • the first neural network can be pre-trained based on deep learning technology.
  • using the first neural network to obtain head posture information based on key points of the face can improve the accuracy of the obtained head posture information, thereby improving the accuracy of the driver state detection result.
  • face key point detection can also be performed through a pre-trained neural network, thereby improving the accuracy of face key point detection results, further improving the accuracy of head posture information, and thus improving the driver state detection results accuracy.
  • obtaining the eye state information according to the detected key points of the face may include: determining the eye area image in the driver image according to the face key points; based on the second neural network, the eye area The image is used to detect the upper eyelid line and the lower eyelid line; according to the interval between the upper eyelid line and the lower eyelid line, the driver's eye closure state information is determined.
  • the closed eye state includes an open eye state, a semi-closed eye state, or an closed eye state
  • the above eye state information includes the closed eye state information.
  • the eye key points in the face key points may be used to locate the eyes in the driver image to obtain an eye area image , And use the eye area image to obtain the upper eyelid line and the lower eyelid line, and calculate the interval between the upper eyelid line and the lower eyelid line to obtain the eye closure state information.
  • face key points of the driver image can also be detected, and the key points of the eyes in the detected face key points can be directly used to calculate, so as to obtain the closed eye state according to the calculation result information.
  • the closed-eye state information can be used to detect the driver ’s closed eyes, such as whether the driver is half-closed (“half” indicates a state of incompletely closed eyes, such as squinting in a drowsy state, etc.) , Eye closure times, eye closure range, etc.
  • the eye-opening state information may be information after normalizing the height of the eye-opening.
  • the second neural network can be pre-trained based on deep learning technology.
  • using the second neural network to detect the upper and lower eyelid lines can accurately detect the positions of the upper and lower eyelid lines, thereby improving the accuracy of the eye closure state information to improve the driver The accuracy of the state's test results.
  • obtaining eye state information according to the detected face key points may include: determining the eye area image in the driver image according to the face key points; based on the third neural network pair The eye area image is subjected to the classification process of opening and closing eyes to obtain a classification result of opening or closing eyes, corresponding to indicating that the eyes are in an open or closed state, and the above eye state information includes the classification of the open or closed eyes The eye-open state or eye-closed state corresponding to the result.
  • the third neural network can perform feature extraction and classification of closed eyes for the input eye area image, and output the open eyes probability (value range can be 0 to 1) or the closed eyes probability (value range can be 0 ⁇ 1)
  • This classification result based on the open-eye probability or closed-eye probability, it can be determined that the eyes are in the open-eye state or closed-eye state, thereby obtaining the driver's eye state.
  • the third neural network can be based on deep learning technology, directly trained with the open-eye sample image and the closed-eye sample image, and the trained third neural network can directly obtain the classification results of open-eye or closed-eye for the input image, and There is no need to calculate the degree of eye closure.
  • obtaining the driver's eye state in the eye area image based on the third neural network can improve the accuracy and detection efficiency of the eye state information, thereby improving the accuracy and detection efficiency of the driver state detection result .
  • acquiring the mouth state information according to the detected face key points may include: determining the mouth area image in the driver image according to the face key points; based on the fourth neural network, the mouth area The image detects the upper and lower lip lines; the driver's mouth opening and closing state information is determined according to the interval between the upper and lower lip lines.
  • the mouth opening and closing state may include an open state of the mouth (ie, open mouth state), a closed state (ie, closed mouth state), a semi-closed state (ie, half open mouth state), and the like.
  • the mouth state information includes the mouth opening and closing state information.
  • the mouth key points in the face key points may be used to locate the mouth in the driver image by cutting, etc.
  • an image of the mouth area can be obtained, and the upper lip line and the lower lip line can be obtained by using the image of the mouth area.
  • the mouth opening and closing state information can be obtained.
  • the mouth key points in the face key points can be directly used for calculation, so as to obtain mouth opening and closing state information according to the calculation result.
  • the above mouth opening and closing state information may be used to detect a driver's yawning, for example, to detect whether the driver yawns and the number of yawnings.
  • the mouth opening and closing state information may be information after the mouth opening height is normalized.
  • the fourth neural network can be pre-trained based on deep learning technology.
  • the fourth neural network is used to detect the upper lip line and the lower lip line, which can accurately detect the position of the upper lip line and the lower lip line, thereby improving the accuracy of the mouth opening and closing state information to improve the driver's state. The accuracy of the test results.
  • acquiring the state information of the mouth according to the detected key points of the face may include: determining the mouth area image in the driver image according to the key points of the face; based on the fifth neural network The partial region image is subjected to the classification process of opening and closing the mouth to obtain a classification result of opening or closing the mouth, which corresponds to indicating that the mouth is in the open or closed state; wherein, the above-mentioned mouth state information includes the open or closed state.
  • the fifth neural network can perform feature extraction and classification processing on the input mouth area image, and output the probability of opening the mouth (that is, the mouth is open) (value range can be 0 to 1) or closing the mouth (that is, the mouth Closed state) probability (value range can be 0 to 1), based on the open mouth probability or closed mouth probability, it can be determined that the mouth is in the open state or closed state, so as to obtain the driver's mouth state information.
  • the fifth neural network can be based on deep learning technology, directly pre-trained with open mouth sample images and closed mouth sample images, and the trained fifth neural network can directly obtain the classification results of open mouth or closed mouth for the input image without the need The detection of the upper lip line and the lower lip line and the calculation of the interval between them are performed.
  • obtaining the mouth state information of the driver in the mouth area image based on the fifth neural network can improve the accuracy and detection efficiency of the mouth state information, thereby improving the accuracy and detection of the driver state detection results effectiveness.
  • the indicators used to characterize the fatigue state of the driver may include, for example, but not limited to: the degree of doze, the degree of eye closure, the degree of blinking, the degree of yawning, etc .; and / or
  • the indicators of the member's distraction status may include, for example, but not limited to: the degree of head position deviation, the degree of face orientation deviation, the degree of sight line deviation, the degree of daze, and so on.
  • determining the parameter value of the index used to characterize the driver's distraction state based on head posture information and / or eye state information may include:
  • the parameter value of the degree of head position deviation may include, but is not limited to, any one or more of the following: head position deviation state, head position deviation direction, head position deviation angle in the head position deviation direction, Duration of head position deviation, frequency of head position deviation; and / or,
  • the face orientation information may include, for example, the direction and angle of rotation of the face, and the direction of rotation here may be to the left , Turn to the right, turn down and / or turn up, etc.
  • the parameter value of the degree of deviation of the face orientation may include, but is not limited to, any one or more of the following: the number of times of turning the head, the duration of turning the head, the frequency of turning the head, etc .; and / or,
  • the parameter value of the degree of sight line deviation may include, but is not limited to, any one or more of the following: the angle of sight line deviation, the length of sight line deviation, the frequency of sight line deviation, etc .; and / or,
  • the parameter value of the degree of daze is obtained.
  • the parameter value of the degree of daze may include, but is not limited to, any one or more of the following: the degree of eye opening, the duration of eye opening, the ratio of the cumulative time of eye opening to the statistical time window, and so on.
  • determining the parameter value of the index used to characterize the fatigue state of the driver based on the head posture information, eye state information, and / or mouth state information may include:
  • the parameter value of the snooze degree may include, but is not limited to, any one or more of the following: the state of nodding, the width of the nodding, the number of nodding, the frequency of the nodding, the frequency of the nodding, the duration of the nodding, etc .; and / or,
  • the parameter value of the degree of closed eyes is obtained.
  • the parameter value of the degree of closed eyes may include, but is not limited to, any one or more of the following: the number of closed eyes, the frequency of closed eyes, the duration of closed eyes, the amplitude of closed eyes, the number of semi-closed eyes, the frequency of semi-closed eyes, the closed eyes The ratio of the accumulated eye duration to the statistical time window, etc .; and / or,
  • the parameter value of the degree of blinking is acquired.
  • the process of the eyes from the open state, to the closed state, and then to the open state can be considered to complete a blinking action, and the time required for a blinking action can be, for example, 0.2s ⁇ Around 1s.
  • the parameter value of the degree of blinking may include, but is not limited to, any one or more of the following: the number of blinks, the frequency of blinking, the duration of blinking, the ratio of the cumulative duration of blinking to the statistical time window, etc .; and / or,
  • the parameter value of the yawn degree may include, but is not limited to, any one or more of the following: yawning status, yawning frequency, yawning duration, yawning frequency, and so on.
  • the head posture information can be obtained based on the deep learning technology, and the driver's head position, face orientation, and line of sight in the driver image can be determined based on the head posture information, which improves the head position information, person
  • the accuracy of face orientation information and line-of-sight information makes the parameter value of the indicator used to characterize the driver's state determined based on the head posture information more accurate, thereby helping to improve the accuracy of the detection result of the driver's state.
  • the head position information can be used to determine whether the driver's head position is normal, such as determining whether the driver is looking down, tilting his head, tilting his head, or turning his head.
  • the head position information can be optionally determined by the pitch angle, yaw angle and roll angle of the head.
  • Face orientation information can be used to determine whether the driver's face orientation is normal, such as determining whether the driver is looking sideways or turning back.
  • the face orientation information may optionally be an angle between the front of the driver's face and the front of the vehicle driven by the driver.
  • the above-mentioned line-of-sight information can be used to determine whether the driver's line-of-sight direction is normal, such as determining whether the driver looks forward, etc.
  • the line-of-sight direction information can be used to determine whether the driver's line of sight has deviated.
  • the line-of-sight direction information may optionally be an angle between the driver's line of sight and the front of the vehicle driven by the driver.
  • N1 frames for example, 9 frames or 10 frames, etc.
  • N1 frames for example, 9 frames or 10 frames, etc.
  • the driver appears After a long-time large-angle head turn, you can record a long-time large-angle head turn, or you can record the duration of the current head turn; after judging that the face orientation information is not greater than the first orientation, greater than the second orientation, and in The phenomenon that is not greater than the first orientation and greater than the second orientation continues for N1 frames (N1 is an integer greater than 0, for example, lasts 9 frames or 10 frames, etc.), then it is determined that the driver has experienced a long-term small-angle turn Head phenomenon, you can record the deviation of the head at a small angle, or you can record the duration of the head.
  • the angle between the line-of-sight direction information and the front of the vehicle is determined to be greater than the first angle, and the phenomenon that is greater than the first angle continues for N2 frames (for example, 8 frames or 9 frames, etc.), it is determined that the driver has experienced a serious line of sight deviation, and can record the serious line of sight deviation or the duration of the current line of sight deviation; when judging the angle between the line of sight information and the front of the vehicle
  • the phenomenon is not greater than the first included angle and greater than the second included angle, and the phenomenon of not greater than the first included angle and greater than the second included angle lasts N2 frames (N2 is an integer greater than 0, for example, lasts 9 frames or 10 frames, etc.), it is determined that the driver has a line of sight deviation phenomenon, which can record the line of sight deviation once, or the duration of the current line of sight deviation.
  • the values of the first orientation, the second orientation, the first included angle, the second included angle, N1, and N2 may be set according to actual conditions, and the present disclosure does not limit the value.
  • the eye state information can be obtained based on the deep learning technology, and the parameter value of the degree of closed eyes, the parameter value of the degree of daze and the parameter value of the degree of blinking can be determined according to the eye state information, and the parameter of the degree of closed eyes is improved
  • the accuracy of the parameter value, the parameter value of the degree of daze and the parameter value of the degree of blinking make the parameter value of the indicator used to characterize the driver's state determined based on the eye state information more accurate, thereby helping to improve the detection result of the driver's state Accuracy.
  • the mouth state information can be obtained based on the deep learning technology, and the parameter value used to characterize the yawning degree can be determined according to the mouth state information, which improves the accuracy of the parameter value of the yawning degree so that the mouth-based
  • the parameter value of the indicator used to characterize the driver state determined by the state information is more accurate, thereby helping to improve the accuracy of the detection result of the driver state.
  • the sixth neural network can be obtained by pre-training with sample images based on deep learning technology. After training, the sixth neural network can directly output line-of-sight direction information to the input image to improve the accuracy of line-of-sight direction information. Therefore, the accuracy of the detection result of the driver state is improved.
  • the sixth neural network can be trained in various ways, which is not limited in the present disclosure.
  • the first line of sight direction may be determined according to the camera that captures the sample image and the pupils in the sample image, the sample image includes at least the eye image; the line of sight direction of the sample image is detected via the sixth neural network, Obtain the first detected sight line direction; train the sixth neural network according to the first sight line direction and the first detected sight line direction.
  • the first coordinate of the pupil reference point in the sample image in the first camera coordinate system is determined, and the second coordinate of the cornea reference point in the sample image in the first camera coordinate system is determined .
  • the sample image includes at least the eye image; determine the second line-of-sight direction of the sample image according to the first and second coordinates; perform line-of-sight detection on the sample image via the sixth neural network to obtain the second detected line-of-sight direction; Train the sixth neural network in the second line of sight direction and the second detected line of sight direction.
  • determining the driver's line of sight direction in the driver image according to the head posture information to obtain the line of sight direction information may include: determining the pupil edge according to the eye image located at the eye key point in the face key point Position, and calculate the pupil center position according to the pupil edge position; obtain the head posture information corresponding to the head posture information according to the pupil center position and the eye center position; the eye corner information under the head posture; determine the driver's line of sight according to the head posture information and the eye corner information To get line of sight information.
  • determining the position of the pupil edge according to the eye image located at the eye key point in the face key point may include: detecting the pupil edge position of the eye image in the image segmented according to the face key point based on the seventh neural network And obtain the pupil edge position according to the information output by the seventh neural network.
  • the eye image can be cut and enlarged from the driver image, and the cut and enlarged eye image can be provided to the seventh neural network for pupil positioning for pupil key point detection and output detection
  • the key point of the pupil is obtained according to the key point of the pupil output by the seventh neural network, and the center position of the pupil can be obtained by calculating the position of the edge of the pupil (for example, calculating the position of the center of the circle).
  • the center position of the eye can be obtained based on the above upper and lower eyelid lines, for example, the coordinate information of all key points of the upper and lower eyelid lines are added, and the upper and lower eyelid lines are subtracted For the number of all key points of the eyelid line, the coordinate information obtained after division is taken as the center position of the eye.
  • the eye center position can also be obtained in other ways, for example, the eye key points in the detected face key points are calculated to obtain the eye center position; the disclosure does not limit the implementation manner of obtaining the eye center position.
  • a more accurate pupil center position can be obtained; by obtaining the eye center position based on the positioning of the eyelid line, a more accurate eye center can be obtained Position, so that when using the center position of the pupil and the center position of the eye to determine the line-of-sight direction, more accurate line-of-sight direction information can be obtained.
  • the realization of the direction of the line of sight is accurate and easy to achieve .
  • the present disclosure may adopt an existing neural network to implement the detection of the position of the pupil edge and the detection of the center position of the eye.
  • the seventh neural network can be pre-trained based on deep learning technology.
  • using the seventh neural network to detect the position of the pupil edge can accurately detect the position of the pupil edge, thereby improving the accuracy of line-of-sight direction information.
  • the parameter of the degree of deviation of the line-of-sight direction when the parameter value of the degree of deviation of the line-of-sight direction is obtained based on the line-of-sight direction information over a period of time, the parameter of the degree of deviation of the line-of-sight direction may be obtained according to the deviation angle of the line-of-sight direction information relative to the reference line-of-sight direction within a period of time value.
  • the reference sight line direction may be preset; or, the average sight line direction determined based on the first N frames of the driver image in the video where the driver image is located may be used as the reference sight line direction.
  • N is an integer greater than 1.
  • obtaining the parameter value of the degree of daze based on eye state information over a period of time may include: according to the above eye state information, when the driver's eyes are in the open state and continue to reach the preset daze time When it is determined that the driver is in a daze state; according to the eye state information for a period of time, the parameter value of the daze degree is obtained.
  • the period of time includes the above-mentioned preset daze time.
  • the head suddenly lowers its head downward from the normal head position, and then returns to the normal head position (that is, the pitch angle in the head attitude information is changed from 0 degrees during normal driving to a preset
  • the process of changing to a certain angle in a short time and then returning to 0 degrees) can be regarded as a nod.
  • obtaining the parameter value of the doze level according to the head position information within a period of time may include: according to the head position information, the deviation of the driver's head position from the preset reference head position When the degree reaches the preset deviation range within the first preset time and returns to the preset reference head position within the second preset time, it is determined that the driver is in a nap state; according to the head position information within a period of time, obtain The parameter value of the doze level; where a period of time includes a first preset time and a second preset time.
  • the process of the mouth from the closed state, to the open state, and then to the closed state can be considered to complete one yawning action, and the time required for one yawning action is usually greater than 400 ms.
  • obtaining the parameter value of the yawn degree according to the mouth state information over a period of time may include: according to the mouth state information, the driver ’s mouth changes from the closed state to the open state, and then resumes When the time to the closed state is within the preset time range, it is determined that the driver has completed a yawning action; according to the mouth state information within a period of time, the parameter value of the yawning degree is obtained.
  • the period of time includes the time when the driver's mouth changes from the closed state to the open state, and then returns to the closed state.
  • determining the detection result of the fatigue state according to the parameter value of the index used to characterize the fatigue state of the driver may include: in any one or more of the parameter values of the index used to characterize the fatigue state of the driver satisfy When the fatigue condition is predetermined, the fatigue state test result is determined to be the fatigue state; and / or, when all the parameter values used to characterize the driver's fatigue state do not satisfy the predetermined fatigue condition, the fatigue state test result is determined to be the non-fatigue state.
  • the predetermined fatigue condition may include multiple fatigue level conditions.
  • the determination of the fatigue state detection result as the fatigue state includes: according to the indicators used to characterize the driver's fatigue state The fatigue level condition satisfied by the parameter value of is to determine the fatigue state level; the determined fatigue state level is used as the fatigue state detection result.
  • the fatigue state detection result is expressed as a degree of fatigue driving.
  • the degree of fatigue driving may include, for example, a normal driving level (that is, a non-fatigue state) and a fatigue driving level (that is, a fatigue state level); where the fatigue driving level may be
  • a fatigue state level can also be divided into a number of different fatigue state levels.
  • the above fatigue driving level can be divided into: prompt fatigue level (also called mild fatigue level) and warning fatigue level (also called Is a severe fatigue level).
  • the degree of fatigue driving can also be divided into more levels, for example, a mild fatigue level, a moderate fatigue level, and a severe fatigue level.
  • the present disclosure does not limit the different fatigue state levels included in the degree of fatigue driving.
  • each fatigue state level included in the fatigue driving degree corresponds to a fatigue level condition
  • the non-fatigue state where the parameter value of the index used to characterize the fatigue state of the driver does not satisfy all the fatigue level conditions is determined as the degree of fatigue driving.
  • the preset condition corresponding to the normal driving level (that is, the non-fatigue state) (that is, not meeting the predetermined fatigue condition) may include:
  • the fatigue level condition corresponding to the fatigue level may include:
  • the fatigue level conditions corresponding to the warning fatigue level may include:
  • Condition 20d there is an eye-closing phenomenon or the number of closed eyes within a period of time reaches a preset number of times or the period of closed eyes within a period of time reaches a preset time;
  • condition 20d and condition 20e If any of the above condition 20d and condition 20e is satisfied, the driver is currently in the warning fatigue level.
  • determining the detection result of the distraction state based on the parameter value of the indicator used to characterize the driver's distraction state may include: in any one or more of the indicators used to characterize the driver's distraction state When the parameter value satisfies the predetermined distraction condition, the detection result of the distraction state is determined to be the distraction state; and / or, when all the parameter values of the indicators characterizing the driver's distraction state do not satisfy the predetermined distraction condition, the determination The result of the heart state detection is a non-distracted state.
  • the aforementioned predetermined distraction condition may include multiple distraction level conditions.
  • the detection result of the distraction state is determined to be the distraction state, including: The distraction level condition satisfied by the parameter value of the index of the distraction state determines the distraction state level; the determined distraction state level is used as the detection result of the distraction state.
  • the detection result of the distracted state may be expressed as a degree of distracted driving.
  • the degree of distracted driving may include, for example: driver concentration (the driver's attention is not distracted, non-distracted state), driver attention Distracted (distracted state). For example, if the deviation angle of the line of sight direction, the deviation angle of the face orientation, and the deviation angle of the head position are all less than the first preset angle, and the duration of opening the eyes is less than the first preset duration, it is the driver ’s concentration (driver ’s attention) Undispersed, non-distracted state).
  • the driver's distraction level may include, for example, a slight distraction of the driver's attention, a moderate distraction of the driver's attention, and a severe distraction of the driver's attention.
  • the driver's level of distraction can be determined by the distraction level condition satisfied by the parameter value of the index used to characterize the driver's distraction state.
  • the deviation angle of the face towards the deviating angle and the head position is not less than the preset angle, and the duration is not greater than the first preset duration, and is less than the second preset duration, or the eyes continue
  • the duration is not greater than the first preset duration and less than the second preset duration, which is a slight distraction of the driver's attention; if the deviation of the line of sight direction and the angle of face orientation is not less than the preset angle, and the duration is not greater than The second preset duration is less than the third preset duration, or the duration of opening the eyes is not greater than the second preset duration and less than the third preset duration, which means that the driver's attention is moderately distracted; Any deviation of the face orientation angle is not less than the preset angle and the duration is not less than the third preset duration, or the duration of opening the eyes is not less than the third preset duration, which is a serious distraction for the driver.
  • FIG. 3 is a flowchart of still another embodiment of the driving state detection method of the present disclosure. Relative to the above shown in FIG. 1 or FIG. 2, the driving state detection method of this embodiment further includes operations related to the detection of a predetermined distraction action on the driver image. As shown in FIG. 3, the predetermined distraction on the driver image Examples of motion detection include:
  • the predetermined distraction motion in the embodiment of the present disclosure may be any distraction motion that may distract the driver's attention, such as a smoking motion, a water drinking motion, a diet motion, a phone call motion, an entertainment motion, and a makeup motion.
  • eating actions such as eating fruits, snacks and other foods
  • entertainment actions such as sending messages, playing games, karaoke, etc.
  • electronic devices such as mobile phone terminals, PDAs, game consoles, etc.
  • the operation 302 may be executed by the processor invoking the corresponding instruction stored in the memory, or may be executed by the second detection module executed by the processor.
  • operation 304 is performed. Otherwise, if the predetermined distraction action does not occur, the subsequent process of this embodiment is not executed.
  • the parameter value of the degree of distraction may include, but is not limited to, any one or more of the following: the number of predetermined distraction actions, the duration of the predetermined distraction action, the frequency of the predetermined distraction action, and so on. For example, the number, duration, and frequency of smoking actions; the number, duration, and frequency of water drinking actions; the number, duration, and frequency of phone calls; and so on.
  • the operation 304 may be executed by the processor invoking the corresponding instruction stored in the memory, or may be executed by the first acquiring module executed by the processor.
  • the operation 306 may be executed by the processor invoking the corresponding instruction stored in the memory, or may be executed by the fourth determination module executed by the processor.
  • performing a predetermined distraction motion detection on the driver image to determine whether the predetermined distraction motion occurs may include:
  • Multiple candidate frames that may include predetermined distraction actions based on feature extraction
  • the action target frame is determined based on a plurality of candidate frames, where the action target frame includes a local area of a human face and an action interactant, or may further selectively include a hand area.
  • the local area of the human face may include, for example, but not limited to any one or more of the following: mouth area, ear area, eye area, etc .; and / or, the action interactive object may include, for example, but not limited to the following Any one or more: containers, cigarettes, mobile phones, food, tools, beverage bottles, glasses, masks, etc .;
  • the classification detection of the predetermined distraction action is performed to determine whether the predetermined distraction action occurs.
  • performing a predetermined distraction motion detection on the driver image to determine whether the predetermined distraction motion occurs may include: performing target object detection corresponding to the predetermined distraction motion on the driver image to obtain the target The detection frame of the object; according to the detection frame of the target object, determine whether a predetermined distraction occurs.
  • This embodiment provides an implementation solution for detecting a predetermined distraction action for a driver, by detecting a target object corresponding to the predetermined distraction action, and determining whether a predetermined distraction action occurs according to the detection frame of the detected target object, thereby judging the driver Whether it is distracted or not helps to obtain accurate detection results of the driver's predetermined distraction action, thereby helping to improve the accuracy of the detection result of the driving state.
  • the above-mentioned predetermined distraction motion detection of the driver image to determine whether the predetermined distraction motion occurs may include: performing face detection on the driver image via the eighth neural network to obtain face detection Frame, and extract the feature information of the face detection frame; determine whether smoking occurs through the eighth neural network according to the feature information of the face detection frame.
  • the above Detecting the predetermined distraction action of the driver image to determine whether the predetermined distraction action occurs may include: presetting the eating action / drinking action / calling action / entertainment action / make-up action on the driver image via the ninth neural network Target object detection to obtain a detection frame of a preset target object, where the preset target object includes: hands, mouth, eyes, action interaction objects; action interaction objects may include but are not limited to any one or more of the following types, for example : Container, food, electronic equipment, cosmetics, etc .; determine whether a predetermined distraction occurs according to the detection frame of the preset target object, and the determination result of whether the predetermined distraction occurs may include one of the following: No dietary movement / drinking occurs Water action / phone call action
  • the predetermined distracting action is a dieting action / a drinking action / a calling action / entertainment action (ie, a dieting action and / or a drinking action and / or a calling action and / or entertainment action) / make-up action
  • determining whether a predetermined distraction occurs according to the detection frame of the preset target object which may include: detecting the detection frame of the hand, the detection frame of the mouth, the detection frame of the eye, and the detection frame of the action interaction object, And according to whether the detection frame of the hand and the detection frame of the action interaction object overlap, the type of the action interaction object, and whether the distance between the detection frame of the action interaction object and the detection frame of the mouth or the eye meets the preset condition To determine whether a predetermined distraction occurs.
  • the type of the action interaction object is a container or food, and the detection frame of the action interaction object overlaps with the detection frame of the mouth, it is determined that the eating action, Drinking water action or make-up action; and / or, if the detection frame of the hand overlaps the detection frame of the action interaction object, the type of the action interaction object is an electronic device, and between the detection frame of the action interaction object and the detection frame of the mouth.
  • the minimum distance is less than the first preset distance, or the minimum distance between the detection frame of the action interaction object and the detection frame of the eye is less than the second preset distance, and it is determined that an entertainment action or a phone call action occurs.
  • the detection frame of the hand, the detection frame of the mouth, and the detection frame of any motion interaction are not simultaneously detected, and the detection frame of the hand, the detection frame of the eye, and the detection of any motion interaction are not simultaneously detected Frame, to determine whether a predetermined distraction action has occurred.
  • the result of the determination is that no eating, drinking, calling, entertainment, and makeup actions have been detected; and / or if the detection frame of the hand does not overlap with the detection frame of the action interaction ,
  • the result of the determination of whether a distracting action occurs is that no eating action, drinking action, phone call action, entertainment action, and makeup action are detected; and / or, if the type of action interaction is container or food, and the action interaction
  • the detection frame of the mouth and the detection frame of the mouth, and / or the type of action interaction object is an electronic device, and the minimum distance between the detection frame of the action interaction object and the detection frame of the mouth is not less than the first Set the distance, or the minimum distance between the detection frame of the action interaction object and the detection frame of the eye to be not less than the second preset distance, and determine whether the determination result of the predetermined distraction action is undetected Action to diet, drink plenty of water action, a call action, action and entertainment make-up action.
  • the eighth neural network and the ninth neural network can perform predetermined distraction detection on the driver image, and output the probability of occurrence of various predetermined distraction movements, which can be selected to be greater than the preset probability threshold and the highest probability value As the detection result of the predetermined distraction motion, if the probability of each predetermined distraction motion is lower than the preset probability threshold, it can be considered that the predetermined distraction motion is not detected.
  • detecting a predetermined distraction motion on the driver image it may further include: if it is determined that the predetermined distraction motion occurs, prompting the detected distraction motion, for example, when a smoking motion is detected, prompting detection Smoking; when drinking water is detected, it prompts the detection of drinking water; when a call is detected, it prompts the detection of a call to remind the driver to focus on driving.
  • the predetermined distraction motion may further include:
  • the determined distraction action level is used as the detection result of the driver's predetermined distraction action.
  • the detection result of the driver's predetermined distraction action may include: no predetermined distraction action occurs (that is, no predetermined distraction action is detected, which may also be referred to as a concentration driving level), and a predetermined distraction action occurs (that is, detection To the scheduled distraction).
  • the detection result of the driver's predetermined distraction action may also be a distraction action level, for example, a prompt distraction level (also referred to as a mild distraction level) and a warning distraction level (also referred to as severe distraction) level).
  • the distraction action level can also be divided into more levels, such as: undistracted level, mild distraction level, moderate distraction level, and severe distraction level.
  • the distraction action levels of the embodiments of the present disclosure may also be divided according to other situations, and are not limited to the above-mentioned level division situations.
  • the level of the distraction action may be determined by the condition of the level of the distraction action that is satisfied by the parameter value of the index used to characterize the degree of distraction. For example, the detection result of the driver's scheduled distraction motion is that the scheduled distraction motion does not occur; if it is determined that the scheduled distraction motion occurs, the duration of detecting the scheduled distraction motion is less than the first preset duration and the frequency is less than the first preset Frequency, the level of distraction action is a mild distraction level; if the duration of a predetermined distraction action is detected to be greater than the first preset duration, and / or the frequency is greater than the first preset frequency, the distraction action level is a severe distraction level .
  • it may further include: in response to the detection result of the predetermined distraction action satisfying the predetermined alarm condition, outputting alarm information corresponding to the detection result of the predetermined distraction action satisfying the predetermined alarm condition.
  • outputting alarm information of a corresponding detection result satisfying the predetermined alarm condition may include: responding As one of the fatigue state detection result, the distraction state detection result, and the predetermined distraction action detection result meets a predetermined alarm condition, an alarm message that outputs a corresponding detection result that meets the predetermined alarm condition is output.
  • outputting alarm information of the fatigue state detection result satisfying the predetermined alarm condition may include: The fatigue state detection result, the distraction state detection result, and the predetermined distraction motion detection result all satisfy the predetermined alarm condition, and output alarm information of the fatigue state detection result satisfying the predetermined alarm condition; and / or, in response to the fatigue state detection result, Two of the detection results of the heart state and the detection results of the predetermined distraction action satisfy the predetermined alarm condition, and the alarm information is output according to the preset alarm strategy.
  • the alarm information in response to the fatigue state detection result, the distraction state detection result, and the predetermined distraction action detection result, two of which satisfy the predetermined alarm condition, the alarm information is output according to the preset alarm strategy, include:
  • the alarm information corresponding to other detection results that meet the predetermined alarm condition may be suppressed within a preset time after the alarm information corresponding to the detection result of the predetermined distraction action that satisfies the predetermined alarm condition is output, Therefore, the purpose of prompting the driver to drive safely can be achieved, and repeated output of multiple prompt / warning information can be avoided to interfere with the normal driving of the driver, thereby improving driving safety.
  • outputting the alarm information of the fatigue state detection result satisfying the predetermined alarm condition may include: outputting a corresponding prompt or warning message according to the fatigue state level; and / or .
  • outputting the prompt / alarm information corresponding to the distraction status detection result satisfying the predetermined alarm condition may include: outputting the corresponding prompt or alarm information according to the distraction status level; and /
  • outputting the alarm information corresponding to the detection result of the predetermined distraction action satisfying the predetermined alarm condition may include: outputting a corresponding prompt or alarm message according to the distraction action level .
  • outputting prompt / warning information corresponding to the detection result of the fatigue state may include:
  • the fatigue state detection result is determined to be a fatigue state, and fatigue prompt information is output. If the driver has been dozing and nodding, the driver's fatigue will continue to deepen (that is, the fatigue state level will increase). The fatigue level will be light fatigue level, moderate fatigue level to severe fatigue level, and it will be output every X seconds. A fatigue alarm message until the fatigue state detection result is a non-fatigue state, that is, the driver returns to the normal driving state, where X is a value greater than 0;
  • the fatigue state detection result is determined to be a fatigue state, and fatigue prompt information is output. If the duration of the driver ’s closed eyes continues to increase or the blink rate has been too high, the driver ’s fatigue will continue to deepen (that is, the fatigue state level will increase), and the fatigue level will be mild fatigue level, moderate fatigue level, and then severe Fatigue level, fatigue alarm information is output every X seconds until the fatigue state detection result is non-fatigue state, that is, the driver returns to normal driving state;
  • the driver is found to be yawning, the fatigue state detection result is determined to be the fatigue state, and the fatigue prompt information is output. If the driver has been yawning, the driver's fatigue level will continue to deepen (that is, the fatigue state level will increase). The fatigue level will be light fatigue level, moderate fatigue level to severe fatigue level, and it will be output every X seconds. A fatigue warning message until the fatigue state detection result is non-fatigue state, that is, the driver returns to normal driving state;
  • outputting prompt / alarm information corresponding to the distraction state detection result may include:
  • the detection result of the distraction state is determined to be the distraction state, and a distraction prompt message is output, where Y is greater than 0 Value.
  • the deviation time of the driver's head position exceeds the preset range, the deviation time will continue to increase, the driver's distraction will continue to deepen (that is, the level of distraction state will increase), and the degree of distraction will be slightly distracting for the driver in turn.
  • the driver ’s attention is moderately distracted, the driver ’s attention is severely distracted, etc., and the distraction alarm information is output every X seconds until the detection result of the distraction state is non-distracted, that is, the driver returns to the normal driving state;
  • the detection result of the distraction state is determined to be Distraction status, output distraction prompt information.
  • the duration of the deviation of the line-of-sight direction exceeds the preset line-of-sight safety range, the driver ’s distraction will continue to deepen (that is, the level of distraction status will increase), and the driver ’s distraction will in turn be slightly distracting. Attention is moderately distracted, driver's attention is severely distracted, etc., and a distraction alarm message is output every X seconds until the detection result of the distraction state is non-distracted, that is, the driver returns to the normal driving state;
  • the detection result of the distraction state is determined to be the distraction state, and the distraction prompt information is output.
  • the degree of distraction of the driver will continue to deepen (that is, the level of distraction state will increase).
  • the degree of distraction will be slightly distracted by the driver, moderately distracted by the driver, severely distracted by the driver, etc., every X
  • the distraction alarm information is output once per second until the detection result of the distraction state is a non-distraction state, that is, the driver returns to the normal driving state.
  • it may further include: when any one or more of the detection results of the fatigue state detection result, the distraction state detection result, and the detection result of the predetermined distraction action satisfy the predetermined driving mode switching condition, Switch the driving mode to the automatic driving mode.
  • the driving mode may be switched to the automatic driving mode to achieve safe driving and avoid occurrence of Road traffic accidents.
  • the driving mode is switched to the automatic driving mode, and the safe driving of the vehicle is achieved through the automatic driving mode to avoid road traffic accidents.
  • the detection result of the driver state may also be output, for example, the detection result of the driver state is output locally and / or the detection of the driver state is output remotely result.
  • the local output of the driver state detection result means that the driver state detection result is output through the driving state detection device or the driver monitoring system, or the driver state detection result is output to the central control system in the vehicle, so that the vehicle is based on the driving
  • the detection results of the state of the staff perform intelligent driving control on the vehicle.
  • the driver state detection result can be sent to the cloud server or management node, so that the cloud server or management node can collect, analyze, and / or manage the driver state detection result, or The vehicle is remotely controlled based on the detection result of the driver state.
  • the detection result of the driving state can also be stored in the user information of the driver in the database, and the detection result of the driving state of the driver can be recorded in order to facilitate the subsequent driving state of the driver Check the test results, or analyze and count the driver's driving habits.
  • it may further include: image acquisition through an infrared camera, for example, image acquisition through an infrared camera deployed at at least one location in the vehicle to obtain a driver image.
  • the driver image in the embodiment of the present disclosure is generally an image frame in a video captured by an infrared camera (including a near-infrared camera, etc.) for the cab.
  • an infrared camera including a near-infrared camera, etc.
  • the wavelength of the infrared camera may include 940nm or 850nm.
  • the infrared camera can be set in any position where the driver can be photographed in the cab of the vehicle.
  • the infrared camera can be deployed in any one or more of the following positions: above or near the instrument panel, above or near the center console, A-pillar or nearby location, rear-view mirror or nearby location.
  • the infrared camera can be set above the instrument panel (such as the position directly above), facing the front position; it can be set above the center console (such as the middle position), facing the front position; or It can be installed on the A-pillar (for example, it can be attached to the glass near the A-pillar), facing the driver's face; it can also be installed on the rear-view mirror (for example, it can be attached to the glass above the rear-view mirror) and facing the driver's face.
  • the optional position can be determined according to the camera's angle of view and the driver's position.
  • the infrared camera when the infrared camera is set above the instrument panel, the infrared camera can be oriented towards the driver To ensure that the camera's angle of view is not blocked by the steering wheel; when it is set above the center console, if the camera's angle of view is large enough, it can be aimed at the rear to ensure that the driver is in the camera's field of view, if the angle of view is not large enough, it can face the driver To ensure that the driver appears in the perspective of the infrared camera.
  • the quality of the driver image captured by the infrared camera is often better than that of the driver image captured by the ordinary camera, especially at night In a dark environment such as a cloudy day or a tunnel, the driver image captured by the infrared camera is usually significantly better than the driver image captured by the ordinary camera, which is conducive to improving the detection and distraction of the driver's distraction state
  • the accuracy of motion detection is in turn beneficial to improve the accuracy of driving state monitoring.
  • an easy-to-deploy and easy-to-use infrared camera is used to obtain driver images in real time, and the camera is installed in a variety of positions.
  • the neural network of learning technology realizes the detection of driver fatigue state and distraction state. It has good robustness and a wide range of applications. It can achieve a good driving state detection effect in scenarios such as day, night, strong light, and low light.
  • the original image acquired by the camera is often not directly usable due to various conditions and random interference.
  • the driver image captured by the infrared camera may be used Grayscale pre-processing, convert the red, green, and blue (RGB) 3-channel images into grayscale images, and then perform driver identity authentication, distraction status detection, and distraction motion detection operations to improve identity authentication and distraction Accuracy of state detection and distraction detection.
  • RGB red, green, and blue
  • the image can be collected by an infrared camera to obtain the driver image in the following scenarios:
  • image acquisition is performed through an infrared camera to obtain an image of the driver; and / or,
  • an infrared camera After detecting the ignition of the vehicle, an infrared camera is used for image acquisition to obtain an image of the driver; and / or,
  • the infrared camera is used for image acquisition to obtain the driver image; and / or,
  • an infrared camera is used for image acquisition to obtain a driver image.
  • the infrared camera can be started to collect the driver image for driving state detection.
  • an infrared (including near-infrared) camera may be used for image acquisition to obtain a driver image, and then sent to a single-chip microcomputer, FPGA, ARM, CPU, GPU, or microprocessor that can load a neural network. It is realized by electronic devices such as smart mobile phones, notebook computers, tablet computers (PAD), desktop computers, or servers.
  • the electronic devices can run computer programs (also called program codes).
  • the computer programs can be stored in flash memory, cache, or hard disk. Or in a computer-readable storage medium such as an optical disc.
  • any of the driving state analysis methods provided by the embodiments of the present disclosure may be executed by any appropriate device with data processing capabilities, including but not limited to: terminal devices and servers.
  • any driving state analysis method provided by an embodiment of the present disclosure may be executed by a processor.
  • the processor executes any driving state analysis method mentioned in the embodiment of the present disclosure by calling a corresponding instruction stored in a memory. I will not repeat them below.
  • the driving state analysis device of this embodiment may be used to implement the above driving state analysis method embodiments of the present disclosure.
  • the driving state analysis device of this embodiment includes: a driving state detection module, which is used to perform driver fatigue state detection and distraction state detection on the driver image to obtain fatigue state detection results and distraction state detection Results; an alarm module for responding to one of the fatigue state detection result and the distraction state detection result satisfying a predetermined alarm condition, outputting alarm information of a corresponding detection result satisfying the predetermined alarm condition; and / or, responding to the fatigue state Both the detection result and the distraction state detection result satisfy the predetermined alarm condition, and output the alarm information of the fatigue state detection result satisfying the predetermined alarm condition.
  • the alarm module responds to one of the fatigue state detection result and the distraction state detection result to satisfy the predetermined alarm condition, and outputs the alarm information of the corresponding detection result satisfying the predetermined alarm condition when the alarm state
  • a prompt / alarm message corresponding to the fatigue state detection result is output; and / or, when the distraction state detection result is a distraction state, a prompt / alarm information corresponding to the distraction state detection result is output.
  • One of the results of the fatigue state detection and the distraction state detection is When the predetermined alarm condition is satisfied, output the alarm information of the corresponding detection result that meets the predetermined alarm condition; and / or, when both the fatigue state detection result and the distraction state detection result satisfy the predetermined alarm condition, output the fatigue that meets the predetermined alarm condition.
  • the alarm information of the state detection result in order to remind the driver to improve driving safety and reduce the incidence of road traffic accidents; and, when both the fatigue state detection result and the distraction state detection result meet the predetermined alarm condition, only Outputting the alarm information of the fatigue state detection result that satisfies the predetermined alarm condition can prevent the driver from being distracted and disgusted by too many or too frequent alarms.
  • the present disclosure improves the safety of assisted driving and the user experience by optimizing the alarm strategy.
  • the alarm module may be further used to: suppress the alarm information corresponding to other detection results that satisfy the predetermined alarm condition within a preset time after the alarm information corresponding to the fatigue state detection result is output; and / or, Within the preset time after outputting the alarm information corresponding to the detection result of the distraction state, the alarm information corresponding to the other detection results satisfying the predetermined alarm condition is suppressed.
  • This embodiment can avoid distracting and disgusting of the driver caused by excessive or too frequent alarms, and further improve the safety and user experience of assisted driving
  • the driving state analysis device of this embodiment may further include: a first determination module for determining the deviation angle of the driver's head position in the driver image Whether it exceeds the preset range.
  • the driving state detection module is used to detect the driver's distraction state on the driver's image when the deviation angle of the driver's head position exceeds a preset range to obtain the distraction state detection result ; And / or, when the deviation angle of the driver's head position does not exceed the preset range, perform the driver's fatigue state detection and distraction state detection operation on the driver image to obtain the fatigue state detection result and the distraction state Test results.
  • the driving state detection module may include: a first detection module, configured to perform head posture detection, eye state detection, and / or mouth state detection on the driver image to obtain Head posture information, eye state information and / or mouth state information; second determination module, used to determine the driver's distraction state detection result based on head posture information and / or eye state information; third determination The module is used to determine the driver's fatigue state detection result based on head posture information, eye state information, and / or mouth state information.
  • the second determining module may include: a first determining unit, configured to determine the parameter of the indicator used to characterize the driver's distraction state based on the head posture information and / or eye state information
  • the second determination unit is used to determine the detection result of the driver's distraction state according to the parameter value of the index used to characterize the driver's distraction state.
  • the third determining module may include: a third determining unit, configured to determine the characterization of the fatigue state of the driver based on head posture information, eye state information, and / or mouth state information The parameter value of the index; the fourth determining unit is used to determine the driver's fatigue state detection result based on the parameter value of the index used to characterize the driver's fatigue state.
  • the first detection module may include: a key point detection unit for detecting face key points of the driver image; a first acquisition unit for acquiring the head according to the detected face key points Posture information, eye state information, and / or mouth state information.
  • the first obtaining unit when the first obtaining unit obtains the head pose information according to the detected face key points, it is used to obtain the head pose information based on the face key points via the first neural network.
  • the first acquiring unit when the first acquiring unit acquires eye state information according to the detected key points of the face, it is used to determine the eye area image in the driver image according to the key points of the face; based on the second neural network pair The eye area image is used to detect the upper eyelid line and the lower eyelid line; the driver's eye closure state information is determined according to the interval between the upper eyelid line and the lower eyelid line; wherein, the eye state information includes: eye closed state information .
  • the first obtaining unit when the first obtaining unit obtains eye state information according to the detected face key points, it is used to determine the eye area image in the driver image according to the face key points; based on the third neural network pair The eye region image is subjected to the classification process of opening and closing the eyes to obtain the classification result of opening or closing the eyes; wherein, the eye state information includes: an open-eye state or an closed-eye state.
  • the first acquiring unit when the first acquiring unit acquires mouth state information according to the detected face key points, it is used to determine the mouth area image in the driver image according to the face key points; based on the fourth neural network pair The mouth area image is used to detect the upper lip line and the lower lip line; the mouth opening and closing state information of the driver is determined according to the interval between the upper lip line and the lower lip line; wherein, the mouth state information includes the mouth opening and closing state information.
  • the first acquisition unit when the first acquisition unit acquires mouth state information according to the detected face key points, it is used to determine the mouth area image in the driver image according to the face key points; based on the fifth neural network pair The mouth area image is subjected to the classification process of opening and closing the mouth to obtain the classification result of opening or closing the mouth; wherein, the mouth state information includes the opening or closing state.
  • the third determining unit is configured to determine the parameter value of the index used to characterize the fatigue state of the driver based on the head posture information, eye state information, and mouth state information.
  • the first determining unit is configured to: determine the head position of the driver in the driver image based on the head posture information to obtain head position information; and obtain the head according to the head position information within a period of time Parameter value of the degree of partial position deviation; and / or, determine the face orientation of the driver in the driver image based on the head posture information to obtain face orientation information; obtain the degree of face orientation deviation based on the face orientation information within a period of time Parameter value of; and / or, determine the driver ’s line of sight direction in the driver image based on the head posture information, obtain line of sight direction information, and obtain the parameter value of the degree of line of sight deviation based on the line of sight direction information over a period of time; or, based on the person Face keypoints determine the eye area image in the driver image, obtain the driver's sight line direction information in the eye area image based on the sixth neural network, and obtain the parameter value of the degree of sight line direction deviation based on the sight line direction information over a period of time; and
  • the third determining unit is configured to: determine the driver's head position in the driver image based on the head posture information to obtain head position information; and obtain napping according to the head position information within a period of time The parameter value of the degree; and / or the parameter value of the degree of eye closure based on the eye state information over a period of time; and / or the parameter value of the degree of blinking based on the eye state information over a period of time; and / or or,
  • the indicators used to characterize the fatigue state of the driver include any one or more of the following: the degree of doze, the degree of eye closure, the degree of blinking, the degree of yawning; and / or
  • the status indicators include any one or more of the following: the degree of head position deviation, the degree of face orientation deviation, the degree of sight direction deviation, and the degree of daze.
  • the first determining unit or the third determining unit determines the head position of the driver in the driver image according to the head posture information, and is used to obtain the pitch in the head posture information when the head position information is obtained
  • the angle is used as the head position; and / or, the first determining unit determines the face orientation of the driver in the driver image based on the head pose information, and when obtaining face orientation information, it is used to obtain the pitch angle and The yaw angle serves as the face orientation.
  • the first determining unit determines the driver's line of sight direction in the driver image based on the head posture information, and when obtaining the line of sight direction information, it is used to locate the eye image based on the eye key point in the face key point Determine the position of the pupil edge and calculate the pupil center position according to the pupil edge position; obtain the head posture information according to the pupil center position and the eye center position corresponding to the eye angle information under the head posture; determine the driver based on the head posture information and the eye angle information Direction of the line of sight to obtain line of sight information.
  • the first determining unit determines the edge position of the pupil according to the eye image located at the eye key point in the face key point, it is used to segment the image according to the face key point based on the seventh neural network
  • the image of the eye area is used to detect the position of the pupil edge, and the position of the pupil edge is obtained according to the information output by the seventh neural network.
  • the first determining unit when the first determining unit obtains the parameter value of the degree of deviation of the line-of-sight direction based on the line-of-sight direction information for a period of time, it is used to obtain the line of sight based on the deviation angle of the line-of-sight direction information with respect to the reference line-of-sight direction for a period of time
  • the parameter value of the degree of direction deviation may be set in advance, or may be an average sight line direction determined based on the first N frames of the driver image in the video where the driver image is located; where N is an integer greater than 1.
  • the first determining unit when the first determining unit obtains the parameter value of the degree of daze based on the eye state information for a period of time, it is used to, according to the eye state information, when the driver's eyes are in the open state and continue to reach the pre- When setting the daze time, it is determined that the driver is in a daze state; according to the eye state information within a period of time, the parameter value of the daze degree is obtained; wherein, the period of time includes the preset daze time.
  • the third determining unit when the third determining unit obtains the parameter value of the doze level according to the head position information within a period of time, it is used to determine the driver's head position relative to the preset reference head according to the head position information When the deviation degree of the part position reaches the preset deviation range within the first preset time and returns to the preset reference head position within the second preset time, it is determined that the driver is in a nap state; according to the head within a period of time Position information to obtain the parameter value of the doze level; wherein, a period of time includes a first preset time and a second preset time.
  • the third determining unit when the third determining unit obtains the parameter value of the yawn degree according to the mouth state information within a period of time, it is used to change the mouth of the driver from the closed state to the open mouth according to the mouth state information
  • the state and the time to return to the closed state are within the preset time range, determine that the driver has completed a yawning action; according to the mouth state information within a period of time, obtain the parameter value of the yawning degree; The time when the driver's mouth changes from the closed state to the open state, and then returns to the closed state.
  • the parameter value of the head position deviation degree includes any one or more of the following: head position deviation state, head position deviation direction, head position deviation angle in the head position deviation direction, Head position deviation duration, head position deviation frequency; and / or, the parameter value of the degree of face orientation deviation includes any one or more of the following: number of rotations, duration of rotation, and frequency of rotation; and / or ,
  • the parameter value of the degree of sight direction deviation includes any one or more of the following: the angle of sight direction deviation, the length of sight direction deviation, and the frequency of sight direction deviation; and / or the parameter value of the degree of daze includes any one or more of the following: Eye opening amplitude, duration of eye opening duration, and ratio of eye opening cumulative duration to the statistical time window; and / or the parameter values of the degree of doze include any one or more of the following: the status of nodding, the amplitude of nodding, the number of nodding, the number of nodding, Nodding frequency and duration of nodding; and / or the parameter value of the degree of closed
  • the second determining unit is configured to determine the detection of the driver's distraction state when any one or more parameter values of indicators for characterizing the driver's distraction state satisfy a predetermined distraction condition The result is a distracted state; and / or, when all the parameter values of the index used to characterize the driver's distracted state do not satisfy the predetermined distracted condition, it is determined that the driver's distracted state detection result is a non-distracted state.
  • the predetermined distraction conditions include multiple distraction level conditions.
  • the predetermined distraction condition when any one or more of the parameter values of the index used to characterize the driver's distraction state meets the predetermined distraction condition, when the detection result of the driver's distraction state is determined to be the distraction state, It is used to determine the distraction state level according to the distraction level condition satisfied by the parameter value of the index used to characterize the driver's distraction state; the determined distraction state level is used as the detection result of the driver's distraction state.
  • the fourth determining unit is used to determine that the fatigue state detection result of the driver is a fatigue state when any one or more parameter values of the indicators used to characterize the fatigue state of the driver meet predetermined fatigue conditions ; And / or, when all the parameter values of the indicators used to characterize the fatigue state of the driver do not satisfy the predetermined fatigue condition, the fatigue state detection result of the driver is determined to be a non-fatigue state.
  • the predetermined fatigue condition includes multiple fatigue level conditions.
  • the fatigue state level is determined; the determined fatigue state level is used as the fatigue state detection result of the driver.
  • the driving state detection device of the present disclosure may further include: a second detection module, configured to perform a predetermined distraction motion detection on the driver image and determine whether a predetermined distraction motion occurs
  • the first obtaining module is used to obtain the parameter value of the index used to characterize the degree of distraction of the driver according to the determination result of whether the predetermined distracting action occurs within a period of time when the predetermined distracting action occurs; the fourth determining module , Used to determine the detection result of the driver's predetermined distraction action according to the parameter value of the index used to characterize the driver's distraction degree.
  • the parameter value of the degree of distraction may include any one or more of the following: the number of predetermined distractions, the duration of the predetermined distractions, the frequency of the predetermined distractions, and so on.
  • the predetermined distraction action may include any one or more of the following: smoking action, drinking action, eating and drinking action, phone call action, entertainment action, makeup action, and so on.
  • the second detection module is used to extract features of the driver image; extract multiple candidate frames that may include predetermined distracting actions based on the feature; determine an action target frame based on the multiple candidate frames, wherein the action target The frame includes the local area of the human face and the action interaction object, or may also include the hand area; based on the action target frame, classification detection of the predetermined distraction action is performed to determine whether the predetermined distraction action occurs.
  • the local area of the human face may include any one or more of the following: the mouth area, the ear area, and the eye area; and / or, the action interaction object may include any one or more of the following: container, smoke, Mobile phones, food, tools, beverage bottles, glasses, masks.
  • the second detection module is configured to perform face detection on the driver image via the eighth neural network to obtain a face detection frame and extract feature information of the face detection frame;
  • the feature information of the face detection frame determines whether smoking occurs.
  • the second detection module is configured to detect the preset target object corresponding to the eating action / drinking action / calling action / entertainment action / make-up action on the driver image via the ninth neural network Set the detection frame of the target object.
  • the preset target objects include: hands, mouth, eyes, action interaction objects.
  • the action interaction objects include any one or more of the following categories: container, food, electronic equipment, cosmetics; according to the preset
  • the detection frame of the target object determines whether a predetermined distraction action occurs; where the determination result of whether a predetermined distraction action occurs includes one of the following: no eating action / drinking action / calling action / entertainment action / make-up action; appearing Eating action, drinking action, calling action, entertainment action, makeup action.
  • the second detection module determines whether a predetermined distraction motion occurs according to the detection frame of the preset target object, which is used to detect the detection frame of the hand, the detection frame of the mouth, and the eye Frame and motion interaction object detection frame, and according to whether the hand detection frame and the motion interaction object detection frame overlap, the type of the motion interaction object and the motion interaction object detection frame and the mouth detection frame or the eye detection frame Whether the distance between them satisfies the preset condition and determines whether a predetermined distraction occurs.
  • the second detection module is based on whether the detection frame of the hand and the detection frame of the action interaction object overlap, and the detection frame of the action interaction object and the detection frame of the mouth or the detection frame of the eye Whether the positional relationship satisfies the preset condition and determines whether a predetermined distraction action occurs, it is used to: if the detection frame of the hand overlaps the detection frame of the action interaction object, the type of action interaction object is container or food, and the detection of the action interaction object The frame overlaps with the detection frame of the mouth to determine the occurrence of eating or drinking movements; and / or if the detection frame of the hand overlaps the detection frame of the action interaction object, the type of action interaction object is an electronic device and the action interaction
  • the minimum distance between the detection frame of the object and the detection frame of the mouth is less than the first preset distance, or the minimum distance between the detection frame of the action interaction object and the detection frame of the eye is less than the second preset distance, it is determined that entertainment occurs Action or call action.
  • the second detection module is also used to: if the detection frame of the hand, the detection frame of the mouth, and the detection frame of any motion interaction are not simultaneously detected, and the detection of the hand is not simultaneously detected A frame, an eye detection frame, and any motion interaction object detection frame, to determine whether the predetermined distraction motion is determined to be the result of not detecting eating movements, drinking water movements, phone call movements, entertainment movements, and makeup movements; and / or If the detection frame of the hand does not overlap with the detection frame of the action interaction object, it is determined whether the determination result of the predetermined distraction motion is that no eating motion, drinking water motion, phone call motion, entertainment motion, makeup motion is detected; and / or, If the type of action interaction object is a container or food, and there is no overlap between the detection frame of the action interaction object and the detection frame of the mouth, and / or, the type of the action interaction object is an electronic device, and the detection frame of the action interaction object is The minimum distance between the detection frame of the mouth is not less than the first preset distance, or the minimum
  • the driving state detection device of the present disclosure may further include: a fifth determination module for distracting action levels satisfied according to the parameter value of the index characterizing the degree of distraction Conditions determine the level of distraction.
  • the fourth determination module is used to use the determined distraction action level as the detection result of the driver's predetermined distraction action.
  • the driving state detection device of the present disclosure may further include: an alarm module for alarming according to the detection result of the driver state; and / or a driving control module for Based on the detection results of the driver's state, intelligent driving control is performed.
  • the detection result of the driver state includes any one or more of the following: the detection result of the fatigue state, the detection result of the distraction state, and the detection result of the predetermined distraction action.
  • the alarm module may be further configured to output alarm information corresponding to the detection result of the predetermined distraction action satisfying the predetermined alarm condition in response to the detection result of the predetermined distraction action satisfying the predetermined alarm condition.
  • the alarm module responds to fatigue when one of the fatigue state detection result and the distraction state detection result meets a predetermined alarm condition and outputs alarm information of a corresponding detection result satisfying the predetermined alarm condition
  • One of the state detection result, the distraction state detection result, and the detection result of the predetermined distraction action satisfies the predetermined alarm condition, and outputs alarm information of the corresponding detection result that meets the predetermined alarm condition.
  • the alarm module is responsive to fatigue in response to fatigue state detection results and distraction state detection results both satisfying predetermined alarm conditions and outputting alarm information of fatigue state detection results satisfying predetermined alarm conditions
  • the state detection result, the distraction state detection result, and the detection result of the predetermined distraction action all satisfy the predetermined alarm condition, and output alarm information of the fatigue state detection result satisfying the predetermined alarm condition; and / or, in response to the fatigue state detection result, distraction Two of the state detection result and the detection result of the predetermined distraction action satisfy the predetermined alarm condition, and output the alarm information according to the preset alarm strategy.
  • the alarm module responds to the fatigue state detection result, the distraction state detection result, and the predetermined distraction action detection result.
  • Two of the three meet the predetermined alarm condition and output alarm information according to the preset alarm strategy, which can be used to: respond to fatigue
  • the state detection result and the detection result of the predetermined distraction action satisfy the predetermined alarm condition, and output alarm information of the fatigue state detection result satisfying the predetermined alarm condition; and / or, in response to the detection result of the distraction state and the detection result of the predetermined distraction action
  • Two of them satisfy the predetermined alarm condition and output the alarm information of the detection result of the distraction state satisfying the predetermined alarm condition, or output the alarm information corresponding to the detection result of the predetermined distraction action that meets the predetermined alarm condition.
  • the alarm module may also be used to suppress alarms corresponding to other detection results that satisfy the predetermined alarm condition within a preset time after outputting alarm information corresponding to the detection result of the predetermined distraction action that satisfies the predetermined alarm condition information.
  • the alarm module outputs a corresponding prompt or warning message according to the fatigue state level when the fatigue state detection result is the fatigue state level; and / or, when the distraction state detection result is the distraction state level, according to The corresponding prompt or alarm information is output in the distraction state level; and / or, when the detection result of the predetermined distraction motion is the distraction motion level, the corresponding prompt or alarm information is output according to the distraction motion level.
  • the driving state detection device of the present disclosure may further include: a driving control module for responding to a fatigue state detection result, a distraction state detection result, and a predetermined distraction action When any one or more of the detection results meet the predetermined driving mode switching condition, the driving mode is switched to the automatic driving mode.
  • the driving control module is configured to switch the driving mode to the automatic driving mode when the fatigue state level and / or the distraction state level and / or the distraction action level meet predetermined driving mode switching conditions.
  • an infrared camera which is used for image acquisition to obtain a driver image.
  • the infrared camera is deployed in at least one location in the vehicle, for example, any one or more of the following locations: above or near the instrument panel, above or near the center console, A-pillar or near location, rearview mirror or near location.
  • the infrared camera is used to: collect images when the vehicle is in the driving state to obtain the driver image; and / or, collect images when the driving speed of the vehicle exceeds the preset speed to obtain the driver image ; And / or, after the ignition of the vehicle is detected, image acquisition is performed to obtain the driver image; and / or, when the start command of the vehicle is detected, the image acquisition is performed to obtain the driver image; and / or, the vehicle is detected Or when the control commands of the components or systems in the vehicle are collected, the driver image is obtained.
  • the driver monitoring system of this embodiment can be used to implement the embodiments of the above-mentioned driving state detection methods of the present disclosure.
  • the driver monitoring system of this embodiment includes: a display device for displaying a driver image; a driving state analysis device for detecting the driver's fatigue state and distraction state of the driver image; In response to one of the fatigue state detection result and the distraction state detection result satisfying a predetermined alarm condition, output alarm information corresponding to the detection result meeting the predetermined alarm condition; and / or, in response to the fatigue state detection result and the distraction state detection As a result, both meet the predetermined alarm condition, and output the alarm information of the fatigue state detection result satisfying the predetermined alarm condition.
  • the driver state detection device may be implemented by the driver state detection device of any of the above embodiments of the present disclosure.
  • One of the results of the fatigue state detection and the distraction state detection is When the predetermined alarm condition is satisfied, output the alarm information of the corresponding detection result that meets the predetermined alarm condition; and / or, when both the fatigue state detection result and the distraction state detection result satisfy the predetermined alarm condition, output the fatigue that meets the predetermined alarm condition.
  • the alarm information of the state detection result in order to remind the driver to improve driving safety and reduce the incidence of road traffic accidents; and, when both the fatigue state detection result and the distraction state detection result meet the predetermined alarm condition, only Outputting the alarm information of the fatigue state detection result that satisfies the predetermined alarm condition can prevent the driver from being distracted and disgusted by too many or too frequent alarms.
  • the present disclosure improves the safety of assisted driving and user experience by optimizing the alarm strategy.
  • another electronic device provided by an embodiment of the present disclosure includes:
  • Memory used to store computer programs
  • the processor is used to execute the computer program stored in the memory, and when the computer program is executed, the driving state analysis method of any of the above embodiments of the present disclosure is implemented.
  • FIG. 7 is a schematic structural diagram of an application embodiment of an electronic device of the present disclosure.
  • the electronic device includes one or more processors, a communication section, etc.
  • processors such as one or more central processing units (CPUs), and / or one or more image processors (GPU), etc.
  • the processor can perform various appropriate actions and processes according to the executable instructions stored in the read-only memory (ROM) or the executable instructions loaded from the storage section into the random access memory (RAM).
  • the communication unit may include, but is not limited to, a network card.
  • the network card may include, but is not limited to, an IB (Infiniband) network card.
  • the processor may communicate with a read-only memory and / or a random access memory to execute executable instructions. Communicate with other target devices via the communication unit to complete operations corresponding to any of the methods provided in the embodiments of the present disclosure, for example, the driver's fatigue state detection and distraction state detection of the driver image, the fatigue state detection result, the distraction state detection Results; in response to one of the fatigue state detection result and the distraction state detection result satisfying the predetermined alarm condition, outputting alarm information corresponding to the detection result of the predetermined alarm condition; and / or, in response to the fatigue state detection result and distraction Both of the state detection results satisfy the predetermined alarm condition, and output the alarm information of the fatigue state detection result satisfying the predetermined alarm condition.
  • IB Intelligent Binary
  • various programs and data necessary for the operation of the device can be stored in the RAM.
  • the CPU, ROM, and RAM are connected to each other through a bus.
  • ROM is an optional module.
  • the RAM stores executable instructions, or writes executable instructions to the ROM at runtime, and the executable instructions cause the processor to perform operations corresponding to any of the above methods of the present disclosure.
  • the input / output (I / O) interface is also connected to the bus.
  • the communication unit may be integrated, or may be configured to have multiple sub-modules (for example, multiple IB network cards), and be on the bus link.
  • the following components are connected to the I / O interface: input parts including keyboard, mouse, etc .; output parts such as cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers; storage parts including hard disks;
  • the communication part of network interface cards such as LAN cards and modems.
  • the communication section performs communication processing via a network such as the Internet.
  • the drive is also connected to the I / O interface as needed.
  • Removable media such as magnetic disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on the drive as needed, so that the computer program read out therefrom is installed into the storage portion as needed.
  • FIG. 7 is only an optional implementation method.
  • the number and types of the components in FIG. 7 can be selected, deleted, added, or replaced according to actual needs; Separate settings or integrated settings can also be used for the setting of different functional components.
  • GPU and CPU can be set separately or the GPU can be integrated on the CPU.
  • the communication department can be set separately or integrated on the CPU or GPU. and many more.
  • the process described above with reference to the flowchart may be implemented as a computer software program.
  • embodiments of the present disclosure include a computer program product including a computer program tangibly contained on a machine-readable medium, the computer program containing program code for performing the method shown in the flowchart, the program code may include a corresponding The instruction corresponding to the method step provided by any embodiment of the present disclosure is executed.
  • the computer program may be downloaded and installed from the network through the communication section, and / or installed from a removable medium.
  • an embodiment of the present disclosure also provides a computer program, including computer instructions, when the computer instructions run in the processor of the device, to implement the driving state analysis method of any of the embodiments of the present disclosure.
  • an embodiment of the present disclosure also provides a computer-readable storage medium on which a computer program is stored.
  • the computer program is executed by a processor, the driving state analysis method of any of the foregoing embodiments of the present disclosure is implemented.
  • FIG. 8 is a schematic structural diagram of an embodiment of the disclosed vehicle. As shown in FIG. 8, the vehicle of this embodiment includes a central control system, and further includes: the driving state analysis device or the driver monitoring system of any of the above embodiments of the present disclosure.
  • the auxiliary driving is improved Driving safety and user experience
  • the central control system is used to switch the driving mode to the automatic driving mode when the detection result of the driver state meets the predetermined driving mode switching condition, and perform automatic driving control on the vehicle in the automatic driving mode.
  • the central control system may also be used to switch the driving mode to the manual driving mode when receiving a driving instruction to switch to manual driving.
  • the vehicle of the above embodiment may further include: an entertainment system for outputting reminder / alarm information corresponding to the predetermined condition of the reminder / alarm according to the control instruction of the central control system; and / or Control instructions to adjust the warning effect of the prompt / alarm information or the playing effect of entertainment items.
  • the entertainment system may include speakers, buzzers, lighting equipment, etc., for example.
  • the vehicle of the above embodiment may further include: at least one infrared camera for image acquisition.
  • the infrared camera in the vehicle may be deployed in at least one location within the vehicle, for example, in any one or more of the following locations: above or near the dashboard, above or near the center console, A Post or nearby location, rearview mirror or nearby location, etc.
  • the method, apparatus and device of the present disclosure may be implemented in many ways.
  • the method, apparatus, and device of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware.
  • the above sequence of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the above-described sequence unless otherwise specifically stated.
  • the present disclosure may also be implemented as programs recorded in a recording medium, and these programs include machine-readable instructions for implementing the method according to the present disclosure.
  • the present disclosure also covers the recording medium storing the program for executing the method according to the present disclosure.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mechanical Engineering (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Automation & Control Theory (AREA)
  • Molecular Biology (AREA)
  • Biomedical Technology (AREA)
  • General Engineering & Computer Science (AREA)
  • Ophthalmology & Optometry (AREA)
  • Transportation (AREA)
  • Mathematical Physics (AREA)
  • Social Psychology (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Psychiatry (AREA)
  • Biodiversity & Conservation Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Emergency Management (AREA)
  • Business, Economics & Management (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Traffic Control Systems (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
  • Image Analysis (AREA)
  • Eye Examination Apparatus (AREA)

Abstract

本公开实施例公开了一种驾驶状态分析方法和装置、驾驶员监控***、车辆，其中，驾驶状态分析方法包括：对驾驶员图像进行驾驶员的疲劳状态检测和分心状态检测，得到疲劳状态检测结果和分心状态检测结果；响应于疲劳状态检测结果和分心状态检测结果二者其中之一满足预定报警条件，输出满足预定报警条件的相应检测结果的报警信息；和/或，响应于疲劳状态检测结果和分心状态检测结果二者均满足预定报警条件，输出满足预定报警条件的疲劳状态检测结果的报警信息。本公开实施例可以基于驾驶员图像实现对驾驶员疲劳状态的和分心状态的共同检测，还可以避免过多或者过于频繁的报警引起驾驶员的分心和反感，本公开通过优化报警策略，提高了辅助驾驶的安全性和用户体验。

Description

驾驶状态分析方法和装置、驾驶员监控***、车辆
本公开要求在2018年10月19日提交中国专利局、申请号为CN 201811224316.3、发明名称为“驾驶状态分析方法和装置、驾驶员监控***、车辆”的中国专利申请的优先权，其全部内容通过引用结合在本公开中。
技术领域
本公开涉及计算机视觉技术，尤其是涉及一种驾驶状态分析方法和装置、驾驶员监控***、车辆。
背景技术
随着车辆的不断普及，交通事故也随之增多，驾驶员的驾驶状态对安全行车的影响非常严重。在驾驶员的驾驶状态较差时，例如由于疲劳过度、睡眠不足、注意力分散等问题导致驾驶状态较差时，可能导致判断能力下降、造成反应迟缓，甚至出现精神恍惚或瞬间记忆消失，导致驾驶动作迟误或过早、操作停顿或修正时间不当等不安全因素，极易发生道路交通事故。当驾驶员在驾驶过程中由于顾及手机等其他事物将其分心状态分散到手机等其他事物时，会增加驾驶安全隐患。
发明内容
本公开实施例提供一种驾驶状态分析的技术方案。
根据本公开实施例的一个方面，提供的一种驾驶状态分析方法，包括：
对驾驶员图像进行驾驶员的疲劳状态检测和分心状态检测，得到疲劳状态检测结果和分心状态检测结果；
响应于所述疲劳状态检测结果和所述分心状态检测结果二者其中之一满足预定报警条件，输出满足预定报警条件的相应检测结果的报警信息；和/或，响应于所述疲劳状态检测结果和所述分心状态检测结果二者均满足预定报警条件，输出满足预定报警条件的所述疲劳状态检测结果的报警信息。
根据本公开实施例的另一个方面，提供的一种驾驶状态分析装置，包括：
驾驶状态检测模块，用于对驾驶员图像进行驾驶员的疲劳状态检测和分心状态检测，得到疲劳状态检测结果和分心状态检测结果；
报警模块，用于响应于所述疲劳状态检测结果和所述分心状态检测结果二者其中之一满足预定报警条件，输出满足预定报警条件的相应检测结果的报警信息；和/或，响应于所述疲劳状态检测结果和所述分心状态检测结果二者均满足预定报警条件，输出满足预定报警条件的所述疲劳状态检测结果的报警信息。
根据本公开实施例的又一个方面，提供的一种驾驶员监控***，包括：
显示装置，用于显示驾驶员图像；
驾驶状态分析装置，用于对驾驶员图像进行驾驶员的疲劳状态检测和分心状态检测；响应于疲劳状态检测结果和分心状态检测结果二者其中之一满足预定报警条件，输出满足预定报警条件的相应检测结果的报警信息；和/或，响应于疲劳状态检测结果和分心状态检测结果二者均满足预定报警条件，输出满足预定报警条件的疲劳状态检测结果的报警信息。
根据本公开实施例的再一个方面，提供的一种电子设备，包括：
存储器，用于存储计算机程序；
处理器，用于执行所述存储器中存储的计算机程序，且所述计算机程序被执行时，实现本公开上述任一实施例所述的方法。
根据本公开实施例的再一个方面，提供的一种计算机可读存储介质，其上存储有计算机程序，其特征在于，该计算机程序被处理器执行时，实现本公开上述任一实施例所述的方法。
根据本公开实施例的再一个方面，提供的一种车辆，包括中控***，还包括：本公开上述任一实施例所述的驾驶状态分析装置，或者本公开上述任一实施例所述的驾驶员监控***。
基于本公开上述实施例提供的驾驶状态分析方法和装置、驾驶员监控***、车辆、电子设备、介质，可以对驾驶员图像实现驾驶员疲劳状态检测和驾驶员分心状态的共同检测，在疲劳状态检测结果和分心状态检测结果二者其中之一满足预定报警条件时，输出满足预定报警条件的相应检测结果的报警信息；和/或，在疲劳状态检测结果和分心状态检测结果二者均满足预定报警条件时，输出满足预定报警条件的疲劳状态检测结果的报警信息，以便于提醒驾驶员注意，以提高驾驶安全性，降低道路交通事故发生率；并且，在疲劳状态检测结果和分心状态检测结果二者均满足预定报警条件时，只输出满足预定报警条件的疲劳状态检测结果的报警信息，可以避免过多或者过于频繁的报警引起驾驶员的分心和反感，本公开通过优化报警策略，提高了驾驶的安全性和用户体验。
下面通过附图和实施例，对本公开的技术方案做进一步的详细描述。
附图说明
构成说明书的一部分的附图描述了本公开的实施例，并且连同描述一起用于解释本公开的原理。
参照附图，根据下面的详细描述，可以更加清楚地理解本公开，其中：
图1为本公开驾驶状态分析方法一个实施例的流程图。
图2为本公开驾驶状态分析方法另一个实施例的流程图。
图3为本公开实施例中对驾驶员图像进行预定分心动作检测一个实施例的流程图。
图4为本公开驾驶状态分析装置一个实施例的结构示意图。
图5为本公开驾驶状态分析装置另一个实施例的结构示意图。
图6为本公开驾驶员监控***一个实施例的结构示意图。
图7为本公开电子设备一个应用实施例的结构示意图。
图8为本公开车辆一个实施例的结构示意图。
具体实施方式
现在将参照附图来详细描述本公开的各种示例性实施例。应注意到：除非另外可选说明，否则在这些实施例中阐述的部件和步骤的相对布置、数字表达式和数值不限制本公开的范围。
应理解，本公开实施例中的“第一”、“第二”等术语仅仅是为了区分，而不应理解成对本公开实施例的限定。
还应理解，在本公开实施例中，“多个”可以指两个或两个以上，“至少一个”可以指一个、两个或两个以上。
还应理解，对于本公开实施例中提及的任一部件、数据或结构，在没有明确限定或者在前后文给出相反启示的情况下，一般可以理解为一个或多个。
还应理解，本公开对各个实施例的描述着重强调各个实施例之间的不同之处，其相同或相似之处可以相互参考，为了简洁，不再一一赘述。
同时，应当明白，为了便于描述，附图中所示出的各个部分的尺寸并不是按照实际的比例关系绘制的。
以下对至少一个示例性实施例的描述实际上仅仅是说明性的，决不作为对本公开及其应用或使用的任何限制。
对于相关领域普通技术人员已知的技术、方法和设备可能不作详细讨论，但在适当情况下，所述技术、方法和设备应当被视为说明书的一部分。
应注意到：相似的标号和字母在下面的附图中表示类似项，因此，一旦某一项在一个附图中被定义，则在随后的附图中不需要对其进行进一步讨论。
另外，公开中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本公开中字符“/”，一般表示前后关联对象是一种“或”的关系。
本公开实施例可以应用于终端设备、计算机***、服务器等电子设备，其可与众多其它通用或专用计算***环境或配置一起操作。适于与终端设备、计算机***、服务器等电子设备一起使用的众所周知的终端设备、计算***、环境和/或配置的例子包括但不限于：车载设备、个人计算机***、服务器计算机***、瘦客户机、厚客户机、手持或膝上设备、基于微处理器的***、机顶盒、可编程消费电子产品、网络个人电脑、小型计算机***﹑大型计算机***和包括上述任何***的分布式云计算技术环境，等等。
终端设备、计算机***、服务器等电子设备可以在由计算机***执行的计算机***可执行指令(诸如程序模块)的一般语境下描述。通常，程序模块可以包括例程、程序、目标程序、组件、逻辑、数据结构等等，它们执行特定的任务或者实现特定的抽象数据类型。计算机***/服务器可以在分布式云计算环境中实施，分布式云计算环境中，任务是由通过通信网络链接的远程处理设备执行的。在分布式云计算环境中，程序模块可以位于包括存储设备的本地或远程计算***存储介质上。
本公开各实施例中的神经网络，分别可以是一个多层神经网络(即：深度神经网络)，其中的神经网络可以是多层的卷积神经网络，例如可以是LeNet、AlexNet、GoogLeNet、VGG、ResNet等任意神经网络模型。各神经网络可以采用相同类型和结构的神经网络，也可以采用不同类型和/或结构的神经网络。本公开实施例不对此进行限制。
图1为本公开驾驶状态分析方法一个实施例的流程图。如图1所示，该实施例的驾驶状态分析方法包括：
102，对驾驶员图像进行驾驶员的疲劳状态检测和分心状态检测，得到疲劳状态检测结果和分心状态检测结果。
在一个可选示例中，该操作102可以由处理器调用存储器存储的相应指令执行，也可以由被处理器运行的驾驶状态检测模块执行。
104，响应于疲劳状态检测结果和分心状态检测结果二者其中之一满足预定报警条件，输出满足预定报警条件的相应检测结果的报警信息，例如，通过声(例如语音或者响铃等)/光(例如亮灯或者灯光闪烁等)/震动等方式进行报警；和/或，响应于疲劳状态检测结果和分心状态检测结果二者均满足预定报警条件，输出满足预定报警条件的疲劳状态检测结果的报警信息，例如，通过声(例如语音或者响铃等)/光(例如亮灯或者灯光闪烁等)/震动等方式进行报警。
在其中一些实施方式中，上述操作104中，响应于疲劳状态检测结果和分心状态检测结果二者其中之一满足预定报警条件，输出满足预定报警条件的相应检测结果的报警信息，可以包括：在疲劳状态检测结果为疲劳状态时，输出与疲劳状态的检测结果相应的提示/告警信息；和/或，在分心状态检测结果为分心状态时，输出与分心状态检测结果相应的提示/告警信息。
在一个可选示例中，该操作104可以由处理器调用存储器存储的相应指令执行，也可以由被处理器运行的报警模块执行。
基于本公开上述实施例提供的驾驶状态分析方法，可以对驾驶员图像实现驾驶员疲劳状态检测和驾驶员分心状态的共同检测，在疲劳状态检测结果和分心状态检测结果二者其中之一满足预定报警条件时，输出满足预定报警条件的相应检测结果的报警信息；和/或，在疲劳状态检测结果和分心状态检测结果二者均满足预定报警条件时，输出满足预定报警条件的疲劳状态检测结果的报警信息，以便于提醒驾驶员注意，以提高驾驶安全性，降低道路交通事故发生率；并且，在疲劳状态检测结果和分心状态检测结果二者均满足预定报警条件时，只输出满足预定报警条件的疲劳状态检测结果的报警信息，可以避免过多或者过于频繁的报警引起驾驶员的分心和反感，本公开通过优化报警策略，提高了辅助驾驶的安全性和用户体验。
在本公开驾驶状态分析方法的另一个实施例中，还可以包括：
在输出所述疲劳状态检测结果相应的报警信息后的预设时间内，抑制掉满足预定报警条件的其他检测结果(例如分心状态检测结果)相应的报警信息；和/或，在输出所述分心状态检测结果相应的报警信息后的预设时间内，抑制掉满足预定报警条件的其他检测结果(例如疲劳状态检测结果)相应的报警信息。
本实施例可以进一步避免过多或者过于频繁的报警引起驾驶员的分心和反感，通过进一步优化报警策略，进一步提高了辅助驾驶的安全性和用户体验。
图2为本公开驾驶状态分析方法另一个实施例的流程图。如图2所示，该实施例的驾驶状态分析方法包括：
202，确定驾驶员图像中驾驶员的头部位置的偏离角度是否超出预设范围。
若驾驶员的头部位置的偏离角度超出预设范围，执行操作204。否则，若驾驶员的头部位置的偏离角度未超出预设范围，执行操作206。
在一个可选示例中，该操作202可以由处理器调用存储器存储的相应指令执行，也可以由被处理器运行的第一确定模块执行。
204，对驾驶员图像进行驾驶员分心状态的检测，得到驾驶员分心状态检测结果。
在其中一些实施方式中，对驾驶员图像进行驾驶员的分心状态检测，可以包括：对驾驶员图像进行头部姿态检测和/或眼部状态检测，得到头部姿态信息和/或眼部状态信息；根据头部姿态信息和/或眼部状态信息，确定驾驶员的分心状态检测结果，例如，根据头部姿态信息和/或眼部状态信息，确定用于表征驾驶员的分心状态的指标的参数值，根据用于表征驾驶员的分心状态的指标的参数值，确定驾驶员的分心状态检测结果。
之后，执行操作208。
206，对驾驶员图像进行驾驶员的疲劳状态检测和分心状态检测，得到疲劳状态检测结果和分心状态检测结果。
在其中一些实施方式中，对驾驶员图像进行驾驶员的疲劳状态检测和分心状态检测，可以包括：对驾驶员图像进行头部姿态检测、眼部状态检测和/或嘴部状态检测，得到头部姿态信息、眼部状态信息和/或嘴部状态信息；根据头部姿态信息、眼部状态信息和/或嘴部状态信息，确定驾驶员的疲劳状态检测结果和分心状态检测结果，例如，根据头部姿态信息、眼部状态信息和/或嘴部状态信息，确定用于表征驾驶员的疲劳状态的指标的参数值和用于表征驾驶员的分心状态的指标的参数值，根据用于表征驾驶员的疲劳状态的指标的参数值，确定驾驶员的疲劳状态检测结果，以及根据用于表征驾驶员的分心状态的指标的参数值，确定驾驶员的分心状态检测结果。
在其中一些可选示例中，据头部姿态信息、眼部状态信息和嘴部状态信息，确定用于表征驾驶员的疲劳状态的指标的参数值。
在一个可选示例中，该操作204-206可以由处理器调用存储器存储的相应指令执行，也可以由被处理器运行的驾驶状态检测模块执行。
208，响应于疲劳状态检测结果和分心状态检测结果二者其中之一满足预定报警条件，输出满足预定报警条件的相应检测结果的报警信息；和/或，响应于疲劳状态检测结果和分心状态检测结果二者均满足预定报警条件，输出满足预定报警条件的疲劳状态检测结果的报警信息。
在一个可选示例中，该操作208可以由处理器调用存储器存储的相应指令执行，也可以由被处理器运行的报警模块执行。
在上述实施例中，在头部位置的偏离角度超出预设范围时，驾驶员有可能会处于分心状态，因此仅对驾驶员进行分心状态检测、而不进行疲劳状态检测，便可实现驾驶状态监测的效果，可以节省由于疲劳状态检测所需的计算资源，提高驾驶状态分析的效率。在头部位置的偏离角度未超出预设范围时，驾驶员有可能会处于分心状态或者疲劳状态，因此对驾驶员同时进行分心状态检测和疲劳状态检测，可实现驾驶状态的共同监测，以保证驾驶的安全性。
在其中一些实施方式中，对驾驶员图像进行头部姿态检测、眼部状态检测和/或嘴部状态检测，得到头部姿态信息、眼部状态信息和/或嘴部状态信息，可以包括：
对驾驶员图像进行人脸关键点检测；
根据检测到的人脸关键点获取头部姿态信息、眼部状态信息和/或嘴部状态信息。
在一些可选示例中，根据检测到的人脸关键点获取头部姿态信息时，可以经第一神经网络基于人脸关键点得到头部姿态信息。
其中，在一些可选示例中，根据检测到的人脸关键点获取头部姿态信息时，例如可以经第一神经网络基于人脸关键点得到头部姿态信息。
在其中一些实施方式中，头部姿态可以通过正常驾驶状态下归一化球坐标系(即摄像头所在的相机坐标系)中头部的姿态角/欧拉角表示，其中的姿态角/欧拉角包括：俯仰角θ(pitch)、偏航角ψ(yaw)、滚转角Φ(roll)，头部姿态信息即包括：(pitch，yaw，roll)。其中，俯仰角用于表示竖直方向上人脸低头或仰头的角度，偏航角用于表示水平方向上人脸侧脸(即转头)的角度，滚转角用于表示竖直方向上人脸歪头(即靠向肩膀)的角度。
在人脸大小一定的情况下，以一种可能的应用场景为例，拍摄驾驶员图像的摄像头位于驾驶员位正前方、正对驾驶员位置时，偏航角度、俯仰角度、滚转角越小，人脸越正，驾驶状态员的驾驶状态更好，可以设置偏航角、俯仰角、滚转角均为0时，为基准头部姿态，此时驾驶员处于最佳的驾驶状态。当俯仰角和/或偏航角大于一个预设角度阈值、且持续时间大于一个预设时间阈值时可以确定分心状态检测结果为分心状态(即注意力不集中)。当俯仰角由0度在一个预设较短时间内变化到一定角度然后再恢复到0度时(对应于头部由正常位置突然向下低头、然后又恢复到正常位置的打盹点头动作)可以确定疲劳状态检测结果为疲劳状态(即疲劳驾驶级别)。对于摄像头位于其他位置的应用场景，可以基于该位置时摄像头与摄像头位于驾驶员位正前方、正对驾驶员位置时的夹角为基准头部姿态，来确定头部姿态信息，本领域技术人员基于本公开实施例的记载可以知悉其具体实现，此处不再赘述。
其中，第一神经网络可以基于深度学习技术预先训练完成。本实施例中，利用第一神经网络基于人脸关键点获取头部姿态信息，可以提高获取的头部姿态信息的精确度，从而提升驾驶员状态的检测结果的准确性。
另外，也可以通过一个预先训练好的神经网络进行人脸关键点检测，从而提升人脸关键点检测结果的精确性，进一步提高头部姿态信息的精确度，从而提升驾驶员状态的检测结果的准确性。
在一些可选示例中，根据检测到的人脸关键点获取眼部状态信息，例如可以包括：根据人脸关键点确定驾驶员图像中的眼部区域图像；基于第二神经网络对眼部区域图像进行上眼睑线和下眼睑线的检测；根据上眼睑线和下眼睑线之间的间隔确定驾驶员的眼睛睁合状态信息。其中，眼睛睁合状态包括睁眼状态、半闭眼状态或闭眼状态，上述眼部状态信 息包括该眼睛睁合状态信息。在一个可选示例中，可以先利用人脸关键点中的眼睛关键点(例如，眼睛关键点在驾驶员图像中的坐标信息)对驾驶员图像中的眼睛进行定位，以获得眼部区域图像，并利用该眼部区域图像获得上眼睑线和下眼睑线，通过计算上眼睑线和下眼睑线之间的间隔，获得眼睛睁合状态信息。或者，在另一个可选示例中，也可以对驾驶员图像进行人脸关键点检测，直接利用所检测出的人脸关键点中的眼睛关键点进行计算，从而根据计算结果获得眼睛睁合状态信息。该眼睛睁合状态信息可以用于进行驾驶员的闭眼检测，如检测驾驶员是否半闭眼(“半”表示非完全闭眼的状态，如瞌睡状态下的眯眼等)、是否闭眼、闭眼次数、闭眼幅度等。可选的，眼睛睁合状态信息可以为对眼睛睁开的高度进行归一化处理后的信息。
其中，第二神经网络可以基于深度学习技术预先训练完成。本实施例中，利用第二神经网络进行上眼睑线和下眼睑线的检测，可以实现上眼睑线和下眼睑线位置的精确检测，从而提高眼睛睁合状态信息的准确性、以提升驾驶员状态的检测结果的准确性。
另外，在另一些可选示例中，根据检测到的人脸关键点获取眼部状态信息，例如可以包括：根据人脸关键点确定驾驶员图像中的眼部区域图像；基于第三神经网络对该眼部区域图像进行睁闭眼的分类处理，得到睁眼或闭眼的分类结果，对应于表示眼睛处于睁眼状态或者闭眼状态，上述眼部状态信息包括该睁眼或闭眼的分类结果对应的睁眼状态或者闭眼状态。例如，第三神经网络可以针对输入的眼部区域图像进行特征提取和睁闭眼的分类处理，输出睁眼概率(取值范围可以为0～1)或者闭眼概率(取值范围可以为0～1)这一分类结果，基于该睁眼概率或者闭眼概率可以确定眼睛处于睁眼状态或闭眼状态，从而得到驾驶员的眼部状态。
其中，第三神经网络可以基于深度学习技术，直接用睁眼样本图像和闭眼样本图像训练得到，训练完成的第三神经网络可以针对输入的图像直接得到睁眼或闭眼的分类结果，而无需进行眼睛睁合程度的计算。本实施例中，基于第三神经网络得到眼部区域图像中驾驶员的眼部状态，可以提高眼部状态信息的准确性和检测效率，从而提升驾驶员状态的检测结果的准确性和检测效率。
在一些可选示例中，根据检测到的人脸关键点获取嘴部状态信息，例如可以包括：根据人脸关键点确定驾驶员图像中的嘴部区域图像；基于第四神经网络对嘴部区域图像进行上唇线和下唇线的检测；根据上唇线和下唇线之间的间隔确定驾驶员的嘴巴开合状态信息。其中，嘴巴开合状态可以包括嘴巴的张开状态(即张嘴状态)、闭合状态(即闭嘴状态)、半闭合状态(即半张嘴状态)等。上述嘴部状态信息包括该嘴巴开合状态信息。例如，在一个可选示例中，可以先利用人脸关键点中的嘴巴关键点(例如，嘴巴关键点在驾驶员图像中的坐标信息)对驾驶员图像中的嘴巴进行定位，通过剪切等方式可以获得嘴部区域图像，并利用该嘴部区域图像获得上唇线和下唇线，通过计算上唇线和下唇线之间的间隔，获得嘴巴开合状态信息。在另一个可选示例中，可以直接利用人脸关键点中的嘴部关键点进行计算，从而根据计算结果获得嘴巴开合状态信息。
上述嘴巴开合状态信息可以用于进行驾驶员的打哈欠检测，例如检测驾驶员是否打哈欠、打哈欠次数等。可选的，嘴巴开合状态信息可以为对嘴巴张开的高度进行归一化处理后的信息。
其中，第四神经网络可以基于深度学习技术预先训练完成。本实施例中，利用第四神经网络进行上唇线和下唇线的检测，可以实现上唇线和下唇线位置的精确检测，从而提高嘴巴开合状态信息的准确性、以提升驾驶员状态的检测结果的准确性。
在另一些可选示例中，根据检测到的人脸关键点获取嘴部状态信息，例如可以包括：根据人脸关键点确定驾驶员图像中的嘴部区域图像；基于第五神经网络对该嘴部区域图像进行张闭嘴的分类处理，得到张嘴或闭嘴的分类结果，对应于表示嘴巴处于张嘴状态或者闭嘴状态；其中，上述嘴部状态信息包括张嘴状态或闭嘴状态。例如，第五神经网络可以针对输入的嘴部区域图像进行特征提取和张闭嘴的分类处理，输出张嘴(即嘴巴开状态)概率(取值范围可以为0～1)或者闭嘴(即嘴巴闭状态)概率(取值范围可以为0～1)，基于该张嘴概率或者闭嘴概率可以确定嘴巴处于张嘴状态或者闭嘴状态，从而得到驾驶员的嘴部状态信息。
其中，第五神经网络可以基于深度学习技术，直接用张嘴样本图像和闭嘴样本图像预先训练得到，训练完成的第五神经网络可以针对输入的图像直接得到张嘴或闭嘴的分类结果，而无需进行上唇线和下唇线的检测和二者之间间隔的计算。本实施例中，基于第五神经网络得到嘴部区域图像中驾驶员的嘴部状态信息，可以提高嘴部状态信息的准确性和检测效率，从而提升驾驶员状态的检测结果的准确性和检测效率。
在其中一些实施方式中，其中，用于表征驾驶员疲劳状态的指标例如可以包括但不限于：打盹程度、闭眼程度、眨眼程度、打哈欠程度，等等；和/或，用于表征驾驶员分心状态的指标例如可以包括但不限于：头部位置偏离程度、人脸朝向偏离程度、视线方向偏离程度、发呆程度，等等。
在其中一些实施方式中，上述各实施例中，根据头部姿态信息和/或眼部状态信息，确定用于表征驾驶员的分心状态的指标的参数值，可以包括：
根据头部姿态信息确定驾驶员图像中驾驶员的头部位置，得到头部位置信息，例如，可以获取头部姿态信息中的俯仰角作为头部位置；根据一段时间内的头部位置信息，获取头部位置偏离程度的参数值。其中，头部位置偏离程度的参数值例如可以包括但不限于以下任意一项或多项：头部位置偏离状态，头部位置偏离方向，头部位置在头部位置偏离方向上的偏离角度，头部位置偏离持续时长，头部位置偏离频率；和/或，
根据头部姿态信息确定驾驶员图像中驾驶员的人脸朝向，得到人脸朝向信息，该人脸朝向信息例如可以包括人脸转动的方向以及角度，这里的转动的方向可以为向左转动、向右转动、向下转动和/或者向上转动等，例如，可以获取头部姿态信息中的俯仰角和偏航角作为人脸朝向；根据一段时间内的人脸朝向信息获取人脸朝向偏离程度的参数值。其中，人脸朝向偏离程度的参数值例如可以包括但不限于以下任意一项或多项：转头次数、转头持续时长、转头频率，等等；和/或，
根据头部姿态信息确定驾驶员图像中驾驶员的视线方向，得到视线方向信息，根据一段时间内的视线方向信息获取视线方向偏离程度的参数值；或者，根据人脸关键点确定驾驶员图像中的眼部区域图像，基于第六神经网络得到眼部区域图像中驾驶员的视线方向信息，根据一段时间内的视线方向信息获取视线方向偏离程度的参数值。其中，视线方向偏离程度的参数值例如可以包括但不限于以下任意一项或多项：视线方向偏离角度、视线方向偏离时长、视线方向偏离频 率，等等；和/或，
根据一段时间内的眼部状态信息，获取发呆程度的参数值。其中，发呆程度的参数值例如可以包括但不限于以下任意一项或多项：睁眼幅度、睁眼持续时长、睁眼累计时长占统计时间窗的比值，等等。
本实施例通过检测驾驶员图像的任意一项或多项用于表征驾驶员分心状态的指标的参数值，并据此确定分心状态检测结果，以判断驾驶员是否集中注意力驾驶，通过对驾驶员分心状态的指标进行量化，将驾驶专注程度量化为头部位置偏离程度、人脸朝向偏离程度、视线方向偏离程度、发呆程度的指标中的至少一个，有利于及时客观的衡量驾驶员的专注驾驶状态。
在其中一些实施方式中，上述各实施例中，根据头部姿态信息、眼部状态信息和/或嘴部状态信息，确定用于表征驾驶员的疲劳状态的指标的参数值，可以包括：
根据头部姿态信息确定驾驶员图像中驾驶员的头部位置，得到头部位置信息；根据一段时间内的头部位置信息，获取打盹程度的参数值。其中，打盹程度的参数值例如可以包括但不限于以下任意一项或多项：打盹点头状态、打盹点头幅度、打盹点头次数、打盹点头频率、打盹点头持续时长，等等；和/或，
根据一段时间内的眼部状态信息，获取闭眼程度的参数值。其中，闭眼程度的参数值例如可以包括但不限于以下任意一项或多项：闭眼次数、闭眼频率、闭眼持续时长、闭眼幅度、半闭眼次数、半闭眼频率、闭眼累计时长占统计时间窗的比值，等等；和/或，
根据一段时间内的眼部状态信息，获取眨眼程度的参数值。本公开实施例中，根据眼部状态信息，眼睛由睁眼状态、到闭眼状态、再到睁眼状态的过程，可以认为完成一次眨眼动作，一次眨眼动作所需时长例如可以为0.2s～1s左右。其中，眨眼程度的参数值例如可以包括但不限于以下任意一项或多项：眨眼次数、眨眼频率、眨眼持续时长、眨眼累计时长占统计时间窗的比值，等等；和/或，
根据一段时间内的嘴部状态信息，获取打哈欠程度的参数值。其中，打哈欠程度的参数值例如可以包括但不限于以下任意一项或多项：打哈欠状态、打哈欠次数、打哈欠持续时长、打哈欠频率，等等。
在上述实施例中，可以基于深度学习技术获取头部姿态信息，根据该头部姿态信息确定驾驶员图像中驾驶员的头部位置、人脸朝向和视线方向，提高了头部位置信息、人脸朝向信息和视线方向信息的准确性，使得基于头部姿态信息确定的用于表征驾驶员状态的指标的参数值更精确，从而有助于提升驾驶员状态的检测结果的准确性。
其中，头部位置信息可以用于确定驾驶员的头部位置是否正常，如确定驾驶员是否低头、是否仰头、是否歪头或者是否转头等。头部位置信息可选的可以通过头部的俯仰角、偏航角和滚转角确定。人脸朝向信息可以用于确定驾驶员的人脸方向是否正常，如确定驾驶员是否侧脸或者是否回头等。人脸朝向信息可选的可以为驾驶员人脸正前方与驾驶员所驾驶的车辆正前方之间的夹角。上述视线方向信息可以用于确定驾驶员的视线方向是否正常，如确定驾驶员是否目视前方等，视线方向信息可以用于判断驾驶员的视线是否发生了偏离现象等。视线方向信息可选的可以为驾驶员的视线与驾驶员所驾驶的车辆正前方之间的夹角。
在一个可选示例中，在判断出人脸朝向信息大于第一朝向，且大于第一朝向的这一现象持续了N1帧(例如，持续了9帧或者10帧等)，则确定驾驶员出现了一次长时间大角度转头现象，可以记录一次长时间大角度转头，也可以记录本次转头持续时长；在判断出人脸朝向信息不大于第一朝向、大于第二朝向，且在不大于第一朝向、大于第二朝向的这一现象持续了N1帧(N1为大于0的整数，例如，持续了9帧或者10帧等)，则确定驾驶员出现了一次长时间小角度转头现象，可以记录一次小角度转头偏离，也可以记录本次转头持续时长。
在一个可选示例中，在判断出视线方向信息和车辆正前方之间的夹角大于第一夹角，且大于第一夹角的这一现象持续了N2帧(例如，持续了8帧或9帧等)，则确定驾驶员出现了一次视线严重偏离现象，可以记录一次视线严重偏离，也可以记录本次视线严重偏离持续时长；在判断出视线方向信息和车辆正前方之间的夹角不大于第一夹角、大于第二夹角，且在不大于第一夹角、大于第二夹角的这一现象持续了N2帧(N2为大于0的整数，例如，持续了9帧或10帧等)，则确定驾驶员出现了一次视线偏离现象，可以记录一次视线偏离，也可以记录本次视线偏离持续时长。
在一个可选示例中，上述第一朝向、第二朝向、第一夹角、第二夹角、N1以及N2的取值可以根据实际情况设置，本公开不限制取值的大小。
在上述实施例中，可以基于深度学习技术获取眼部状态信息，根据该眼部状态信息确定闭眼程度的参数值、发呆程度的参数值和眨眼程度的参数值，提高了闭眼程度的参数值、发呆程度的参数值和眨眼程度的参数值的准确性，使得基于眼部状态信息确定的用于表征驾驶员状态的指标的参数值更精确，从而有助于提升驾驶员状态的检测结果的准确性。
在上述实施例中，可以基于深度学习技术获取嘴部状态信息，根据该嘴部状态信息确定用于表征打哈欠程度的参数值，提高了打哈欠程度的参数值的准确性，使得基于嘴部状态信息确定的用于表征驾驶员状态的指标的参数值更精确，从而有助于提升驾驶员状态的检测结果的准确性。
在上述实施例中，第六神经网络可以基于深度学习技术，预先利用样本图像训练得到，训练完成后的第六神经网络可以直接针对输入的图像输出视线方向信息，提高视线方向信息的准确性，从而提升驾驶员状态的检测结果的准确性。
其中，可以采用多种方式对第六神经网络进行训练方法，本公开对此不作限制。例如，在其中一种方式中，可以根据拍摄样本图像的摄像头以及该样本图像中的瞳孔确定第一视线方向，该样本图像至少包括眼部图像；经第六神经网络检测样本图像的视线方向，得到第一检测视线方向；根据第一视线方向和第一检测视线方向，训练第六神经网络。再如，在其中一种方式中，确定样本图像中的瞳孔参考点在第一相机坐标系下的第一坐标，以及确定样本图像中的角膜参考点在第一相机坐标系下的第二坐标，该样本图像中至少包括眼部图像；根据第一坐标和第二坐标确定样本图像的第二视线方向；经第六神经网络对样本图像进行视线方向检测，得到第二检测视线方向；根据第二视线方向和所述第二检测视线方向训练第六神经网络。
在其中一些可选示例中，根据头部姿态信息确定驾驶员图像中驾驶员的视线方向，得到视线方向信息，可以包括：根据人脸关键点中的眼睛关键点所定位的眼睛图像确定瞳孔边沿位置，并根据瞳孔边沿位置计算瞳孔中心位置；根据瞳孔中心位置与眼睛中心位置获取头部姿态信息对应头部姿态下的眼珠转角信息；根据头部姿态信息和眼珠转角信息确定驾驶员的视线方向，得到视线方向信息。
其中，根据人脸关键点中的眼睛关键点所定位的眼睛图像确定瞳孔边沿位置，可以包括：基于第七神经网络对根据人脸关键点分割出的图像中的眼睛图像进行瞳孔边沿位置的检测，并根据该第七神经网络输出的信息获取瞳孔边沿位置。
作为一个可选示例，可以从驾驶员图像中剪切并放大的眼睛图像，并将该剪切放大后的眼睛图像提供给用于瞳孔定位的第七神经网络进行瞳孔关键点检测、输出检测到的瞳孔关键点，根据第七神经网络输出的瞳孔关键点获取到瞳孔边沿位置，通过对瞳孔边沿位置进行计算(例如，计算圆心位置)，即可获得瞳孔中心位置。
作为一个可选示例，可以基于上述上眼睑线和下眼睑线获取到眼睛中心位置，例如，将上眼睑线和下眼睑线的所有关键点的坐标信息进行相加，并除以上眼睑线和下眼睑线的所有关键点的数量，将相除后获得的坐标信息作为眼睛中心位置。另外，也可以采用其他方式获取眼睛中心位置，例如，针对检测到的人脸关键点中的眼睛关键点进行计算，从而获得眼睛中心位置；本公开不限制获取眼睛中心位置的实现方式。
本实施例通过在瞳孔关键点检测的基础上来获取瞳孔中心位置，可以获取到更为准确的瞳孔中心位置；通过在眼睑线定位的基础上来获取眼睛中心位置，可以获取到更为准确的眼睛中心位置，从而在利用瞳孔中心位置和眼睛中心位置来确定视线方向时，可以获得较为准确的视线方向信息。另外，通过利用瞳孔关键点检测的方式来定位瞳孔中心位置，并利用瞳孔中心位置和眼睛中心位置来确定视线方向，使确定视线方向的实现方式在具有准确性，还兼具有易于实现的特点。
在一个可选示例中，本公开可以采用现有的神经网络来实现瞳孔边沿位置的检测以及眼睛中心位置的检测。
其中，第七神经网络可以基于深度学习技术预先训练完成。本实施例中，利用第七神经网络进行瞳孔边沿位置的检测，可以实现瞳孔边沿位置的精确检测，从而提高视线方向信息的准确性。
在一个可选示例中，根据一段时间内的视线方向信息获取视线方向偏离程度的参数值时，可以根据一段时间内的视线方向信息相对于参考视线方向的偏离角度，获取视线方向偏离程度的参数值。
其中，参考视线方向可以预先设定；或者，也可以以基于驾驶员图像所在视频中的前N帧驾驶员图像确定的平均视线方向为参考视线方向。其中，N为大于1的整数。
本公开实施例中，在眼睛处于睁眼状态且持续一定时间时，可以认为处于发呆状态。在一个可选示例中，根据一段时间内的眼部状态信息，获取发呆程度的参数值，可以包括：根据上述眼部状态信息，在驾驶员的眼睛处于睁眼状态且持续达到预设发呆时间时，确定驾驶员处于发呆状态；根据一段时间内的眼部状态信息，获取发呆程度的参数值。其中，该一段时间包括上述预设发呆时间。
本公开实施例中，头部由正常头部位置突然向下低头、然后又恢复到正常头部位置(即，头部姿态信息中的俯仰角由正常驾驶状态时的0度在一个预设较短时间内变化到一定角度然后再恢复到0度时)的过程，可以看作一次打盹点头。在一个可选示例中，根据一段时间内的头部位置信息，获取打盹程度的参数值，可以包括：根据头部位置信息，在驾驶员的头部位置相对于预设参考头部位置的偏离程度在第一预设时间内达到预设偏离范围、且在第二预设时间内恢复至预设参考头部位置时，确定驾驶员处于打盹状态；根据一段时间内的头部位置信息，获取打盹程度的参数值；其中，一段时间包括第一预设时间和第二预设时间。
本公开实施例中，嘴巴由闭合状态、到张开状态、再到闭合状态的过程，可以认为完成一次打哈欠动作，一次打哈欠动作所需时长通常大于400ms。在一个可选示例中，根据一段时间内的嘴部状态信息，获取打哈欠程度的参数值，可以包括：根据嘴部状态信息，在驾驶员的嘴巴由闭嘴状态变化到张嘴状态、再恢复到闭嘴状态的时间在预设时间范围内时，确定驾驶员完成一次打哈欠动作；根据一段时间内的嘴部状态信息，获取打哈欠程度的参数值。其中，该一段时间包括上述驾驶员的嘴巴由闭嘴状态变化到张嘴状态、再恢复到闭嘴状态的时间。
在其中一些实施方式中，根据用于表征驾驶员疲劳状态的指标的参数值，确定疲劳状态检测结果，可以包括：在任意一项或多项用于表征驾驶员疲劳状态的指标的参数值满足预定疲劳条件时，确定疲劳状态检测结果为疲劳状态；和/或，在所有用于表征驾驶员疲劳状态的指标的参数值均不满足预定疲劳条件时，确定疲劳状态检测结果为非疲劳状态。
其中，上述预定疲劳条件可以包括多个疲劳等级条件。相应地，上述在任意一项或多项用于表征驾驶员疲劳状态的指标的参数值满足预定疲劳条件时，确定疲劳状态检测结果为疲劳状态，包括：根据用于表征驾驶员疲劳状态的指标的参数值满足的疲劳等级条件，确定疲劳状态等级；将确定的疲劳状态等级作为疲劳状态检测结果。
该实施例中，疲劳状态检测结果表示为疲劳驾驶程度，该疲劳驾驶程度例如可以包括：正常驾驶级别(即非疲劳状态)以及疲劳驾驶级别(即疲劳状态等级)；其中的疲劳驾驶级别可以为一个疲劳状态等级，也可以被划分为多个不同的疲劳状态等级，例如，上述疲劳驾驶级别可以被划分为：提示疲劳级别(也可以称为轻度疲劳级别)和警告疲劳级别(也可以称为重度疲劳级别)。另外，疲劳驾驶程度也可以被划分为更多级别，例如，轻度疲劳级别、中度疲劳级别以及重度疲劳级别等。本公开不限制疲劳驾驶程度所包括的不同疲劳状态等级。
在一个可选示例中，疲劳驾驶程度所包含的每一个疲劳状态等级均对应有疲劳等级条件，可以将用于表征驾驶员疲劳状态的指标的参数值满足的疲劳等级条件所对应的疲劳状态等级，或者将用于表征驾驶员疲劳状态的指标的参数值不满足所有疲劳等级条件的非疲劳状态确定为疲劳驾驶程度。
在一个可选示例中，正常驾驶级别(即非疲劳状态)对应的预设条件(即不满足预定疲劳条件)可以包括：
条件20a、不存在半闭眼以及闭眼现象；
条件20b、不存在打哈欠现象；
在上述条件20a、条件20b均满足的情况下，驾驶员当前处于正常驾驶级别(即非疲劳状态)。
在一个可选示例中，提示疲劳级别对应的疲劳等级条件可以包括：
条件20c、存在半闭眼现象；
条件20d、存在打哈欠现象；
在上述条件20c、条件20d中的任一条件满足的情况下，驾驶员当前处于提示疲劳级别。
在一个可选示例中，警告疲劳级别对应的疲劳等级条件可以包括：
条件20d、存在闭眼现象或者在一段时间内的闭眼次数达到一预设次数或者在一段时间内的闭眼时间达到一预设时间；
条件20e、在一段时间内的打哈欠的次数达到一预设次数；
在上述条件20d、条件20e中的任一条件满足的情况下，驾驶员当前处于警告疲劳级别。
在其中一些实施方式中，根据用于表征驾驶员分心状态的指标的参数值，确定分心状态检测结果，可以包括：在任意一项或多项用于表征驾驶员分心状态的指标的参数值满足预定分心条件时，确定分心状态检测结果为分心状态；和/或，在所有用于表征驾驶员分心状态的指标的参数值均不满足预定分心条件时，确定分心状态检测结果为非分心状态。
其中，上述预定分心条件可以包括多个分心等级条件。相应地，上述在任意一项或多项用于表征驾驶员分心状态的指标的参数值满足预定分心条件时，确定分心状态检测结果为分心状态，包括：根据用于表征驾驶员分心状态的指标的参数值满足的分心等级条件，确定分心状态等级；将确定的分心状态等级作为分心状态检测结果。
该实施例中，分心状态检测结果可以表示为分心驾驶程度，该分心驾驶程度例如可以包括：驾驶员注意力集中(驾驶员注意力未分散，非分心状态)，驾驶员注意力分散(分心状态)。例如，若视线方向偏离角度、人脸朝向偏离角度、头部位置的偏离角度均小于第一预设角度，睁眼持续时长小于第一预设时长，为驾驶员注意力集中(驾驶员注意力未分散，非分心状态)。其中的驾驶员注意力分散级别例如可以包括：驾驶员注意力轻度分散，驾驶员注意力中度分散，驾驶员注意力严重分散等。其中的驾驶员注意力分散级别可以通过用于表征驾驶员分心状态的指标的参数值所满足的分心等级条件确定。例如，若视线方向偏离角度人脸朝向偏离角度和头部位置的偏离角度任一不小于预设角度，且持续时间不大于第一预设时长、且小于第二预设时长，或者睁眼持续时长不大于第一预设时长、且小于第二预设时长，为驾驶员注意力轻度分散；若视线方向偏离角度和人脸朝向偏离角度任一不小于预设角度，且持续时间不大于第二预设时长、且小于第三预设时长，或者睁眼持续时长不大于第二预设时长、且小于第三预设时长，为驾驶员注意力中度分散；若视线方向偏离角度和人脸朝向偏离角度任一不小于预设角度，且持续时间不小于第三预设时长，或者睁眼持续时长不小于第三预设时长，为驾驶员注意力重度分散。
图3为本公开驾驶状态检测方法又一个实施例的流程图。相对于上述图1或图2所示，该实施例的驾驶状态检测方法还包括对驾驶员图像进行预定分心动作检测的相关操作，如图3所示，该对驾驶员图像进行预定分心动作检测的实施例包括：
302，对驾驶员图像进行预定分心动作检测，确定是否出现预定分心动作。
本公开实施例中的预定分心动作，可以是任意可能分散驾驶员的注意力的分心动作，例如抽烟动作、喝水动作、饮食动作、打电话动作、娱乐动作、化妆动作等。其中，饮食动作例如吃水果、零食等食物；娱乐动作例如发信息、玩游戏、K歌等任意借助于电子设备执行的动作。其中的电子设备例如手机终端、掌上电脑、游戏机等。
在一个可选示例中，该操作302可以由处理器调用存储器存储的相应指令执行，也可以由被处理器运行的第二检测模块执行。
若出现预定分心动作，执行操作304。否则，若未出现预定分心动作，不执行本实施例的后续流程。
304，根据一段时间内是否出现预定分心动作的确定结果，获取用于表征驾驶员的分心程度的指标的参数值。
其中，分心程度的参数值例如可以包括但不限于以下任意一项或多项：预定分心动作的次数、预定分心动作的持续时长、预定分心动作的频率，等等。例如，抽烟动作的次数、持续时长、频率；喝水动作的次数、持续时长、频率；打电话动作的次数、持续时长、频率；等等。
在一个可选示例中，该操作304可以由处理器调用存储器存储的相应指令执行，也可以由被处理器运行的第一获取模块执行。
306，根据上述用于表征驾驶员的分心程度的指标的参数值，确定驾驶员预定分心动作的检测结果。
在一个可选示例中，该操作306可以由处理器调用存储器存储的相应指令执行，也可以由被处理器运行的第四确定模块执行。
在其中一些实施方式中，操作302中，对驾驶员图像进行预定分心动作检测，确定是否出现预定分心动作，可以包括：
提取驾驶员图像的特征；
基于特征提取可能包括预定分心动作的多个候选框；
基于多个候选框确定动作目标框，其中，动作目标框包括人脸的局部区域和动作交互物，或者还可以进一步选择性地包括手部区域。其中，人脸的局部区域例如可以包括但不限于以下任意一项或多项：嘴部区域，耳部区域，眼部区域，等等；和/或，动作交互物例如可以包括但不限于以下任意一项或多项：容器、烟、手机、食物、工具、饮料瓶、眼镜、口罩，等等；
基于动作目标框进行预定分心动作的分类检测，确定是否出现预定分心动作。
在另一些实施方式中，操作302中，对驾驶员图像进行预定分心动作检测，确定是否出现预定分心动作，可以包括：对驾驶员图像进行预定分心动作相应的目标对象检测，得到目标对象的检测框；根据目标对象的检测框，确定是否出现预定分心动作。
本实施例提供了对驾驶员进行预定分心动作检测的实现方案，通过检测预定分心动作相应的目标对象、根据检测到的目标对象的检测框确定是否出现预定分心动作，从而判断驾驶员是否分心，有助于获取准确的驾驶员预定分心动作检测的结果，从而有助于提高驾驶状态检测结果的准确性。
例如，预定分心动作为抽烟动作时，上述对驾驶员图像进行预定分心动作检测，确定是否出现预定分心动作可以包括：经第八神经网络对驾驶员图像进行人脸检测，得到人脸检测框，并提取人脸检测框的特征信息；经该第八神经网络根据人脸检测框的特征信息确定是否出现抽烟动作。
又如，预定分心动作为饮食动作/喝水动作/打电话动作/娱乐动作(即：饮食动作和/或喝水动作和/或打电话动作和/或娱乐动作)/化妆动作时，上述对驾驶员图像进行预定分心动作检测，确定是否出现预定分心动作可以包括：经第九神经网络对驾驶员图像进行饮食动作/喝水动作/打电话动作/娱乐动作/化妆动作相应的预设目标对象检测，得到预设目标对象的检测框，其中的预设目标对象包括：手部、嘴部、眼部、动作交互物；动作交互物例如可以包括但不限于以下任意一类或多类：容器、食物、电子设备、化妆品，等等；根据预设目标对象的检测框确定是否出现预定分心动作，是否出 现预定分心动作的确定结果可以包括以下之一：未出现饮食动作/喝水动作/打电话动作/娱乐动作，出现饮食动作，出现喝水动作，出现打电话动作，出现娱乐动作，出现化妆动作。
在一些可选示例中，预定分心动作为饮食动作/喝水动作/打电话动作/娱乐动作(即：饮食动作和/或喝水动作和/或打电话动作和/或娱乐动作)/化妆动作时，根据预设目标对象的检测框，确定是否出现预定分心动作，可以包括：根据是否检测到手部的检测框、嘴部的检测框、眼部的检测框和动作交互物的检测框，以及根据手部的检测框与动作交互物的检测框是否重叠、动作交互物的类型以及动作交互物的检测框与嘴部的检测框或眼部的检测框之间的距离是否满足预设条件，确定是否出现预定分心动作。
可选地，若手部的检测框与动作交互物的检测框重叠，动作交互物的类型为容器或食物、且动作交互物的检测框与嘴部的检测框之间重叠，确定出现饮食动作、喝水动作或化妆动作；和/或，若手部的检测框与动作交互物的检测框重叠，动作交互物的类型为电子设备，且动作交互物的检测框与嘴部的检测框之间的最小距离小于第一预设距离、或者动作交互物的检测框与眼部的检测框之间的最小距离小于第二预设距离，确定出现娱乐动作或打电话动作。
另外，若未同时检测到手部的检测框、嘴部的检测框和任一动作交互物的检测框，且未同时检测到手部的检测框、眼部的检测框和任一动作交互物的检测框，确定是否出现预定分心动作确定结果为未检测到饮食动作、喝水动作、打电话动作、娱乐动作和化妆动作；和/或，若手部的检测框与动作交互物的检测框未重叠，确定是否出现分心动作的确定结果为未检测到饮食动作、喝水动作、打电话动作、娱乐动作和化妆动作；和/或，若动作交互物的类型为容器或食物、且动作交互物的检测框与嘴部的检测框之间未重叠，和/或，动作交互物的类型为电子设备、且动作交互物的检测框与嘴部的检测框之间的最小距离不小于第一预设距离、或者动作交互物的检测框与眼部的检测框之间的最小距离不小于第二预设距离，确定是否预定分心动作的确定结果为未检测到饮食动作、喝水动作、打电话动作、娱乐动作和化妆动作。
在上述示例中，可以由第八神经网络、第九神经网络对驾驶员图像进行预定分心动作检测，并输出出现各种预定分心动作的概率，可以选取大于预设概率阈值且概率值最高的分心动作作为预定分心动作的检测结果，若各种预定分心动作的概率均低于预设概率阈值，可以认为未检测到预定分心动作。
另外，在上述对驾驶员图像进行预定分心动作检测的实施例中，还可以包括：若确定出现预定分心动作，提示检测到的分心动作，例如，检测到抽烟动作时，提示检测到抽烟；检测到喝水动作时，提示检测到喝水；检测到打电话动作时，提示检测到打电话，以提醒驾驶员集中注意力驾驶。
另外，在上述对驾驶员图像进行预定分心动作检测的实施例中，若确定出现预定分心动作，还可以包括：
根据用于表征分心程度的指标的参数值满足的分心动作等级条件，确定分心动作等级；
将确定的分心动作等级作为驾驶员预定分心动作的检测结果。
本实施例中，驾驶员预定分心动作的检测结果可以包括：未出现预定分心动作(即未检测到预定分心动作，也可以称为专注驾驶级别)，出现预定分心动作(即检测到预定分心动作)。另外，上述驾驶员预定分心动作的检测结果也可以为分心动作等级，例如，提示分心级别(也可以称为轻度分心级别)和警告分心级别(也可以称为重度分心级别)。另外，分心动作等级也可以被划分为更多级别，例如：未分心级别，轻度分心级别、中度分心级别以及重度分心级别等。另外，本公开各实施例的分心动作等级也可以按照其他情况划分，不限制为上述级别划分情况。
其中的分心动作等级可以通过用于表征分心程度的指标的参数值所满足的分心动作等级条件确定。例如，驾驶员预定分心动作的检测结果为未出现预定分心动作；若确定出现预定分心动作，检测到预定分心动作的持续时间小于第一预设时长、且频率小于第一预设频率，分心动作等级为轻度分心级别；若检测到预定分心动作的持续时间大于第一预设时长，和/或频率大于第一预设频率，分心动作等级为重度分心级别。
另外，在上述实施例中，还可以包括：响应于预定分心动作的检测结果满足预定报警条件，输出满足预定报警条件的预定分心动作的检测结果相应的报警信息。
在上述各实施例的一些实施方式中，响应于疲劳状态检测结果和分心状态检测结果二者其中之一满足预定报警条件，输出满足预定报警条件的相应检测结果的报警信息，可以包括：响应于疲劳状态检测结果、分心状态检测结果和预定分心动作的检测结果三者其中之一满足预定报警条件，输出满足预定报警条件的相应检测结果的报警信息。
在上述各实施例的一些实施方式中，响应于疲劳状态检测结果和分心状态检测结果二者均满足预定报警条件，输出满足预定报警条件的疲劳状态检测结果的报警信息，可以包括：响应于疲劳状态检测结果、分心状态检测结果和预定分心动作的检测结果均满足预定报警条件，输出满足预定报警条件的疲劳状态检测结果的报警信息；和/或，响应于疲劳状态检测结果、分心状态检测结果和预定分心动作的检测结果三者其中之二满足预定报警条件，按照预设报警策略输出报警信息。
例如，在其中一些可选示例中，响应于疲劳状态检测结果、分心状态检测结果和预定分心动作的检测结果三者其中之二满足预定报警条件，按照预设报警策略输出报警信息，可以包括：
响应于疲劳状态检测结果和预定分心动作的检测结果满足预定报警条件，输出满足预定报警条件的疲劳状态检测结果的报警信息；和/或，
响应于分心状态检测结果和预定分心动作的检测结果三者其中之二满足预定报警条件，输出满足预定报警条件的分心状态检测结果的报警信息，或者，输出满足预定报警条件的预定分心动作的检测结果相应的报警信息。
另外，在进一步实施方式中，可以在在输出满足预定报警条件的预定分心动作的检测结果相应的报警信息后的预设时间内，抑制掉满足预定报警条件的其他检测结果相应的报警信息，从而既可以实现提示驾驶员安全驾驶的目的，又可以避免重复输出多种提示/告警信息干扰驾驶员正常驾驶，从而提升驾驶的安全性。
在其中一些实施方式中，在疲劳状态检测结果为疲劳状态等级时，输出满足预定报警条件的疲劳状态检测结果的报警信息，可以包括：根据疲劳状态等级输出相应的提示或者告警信息；和/或，在分心状态检测结果为分心状态等级时，输出满足预定报警条件的分心状态检测结果相应的提示/告警信息，可以包括：根据分心状态等级输出相应的提示或者告警信息；和/或，在预定分心动作的检测结果为分心动作等级时，输出满足预定报警条件的预定分心动作的检测结果相应的报警信息，可以包括：根据分心动作等级输出相应的提示或者告警信息。
例如，基于上述实施例，在疲劳状态的检测结果为疲劳状态时，输出与疲劳状态的检测结果相应的提示/告警信息， 可以包括：
基于头部位置信息发现驾驶员在做打盹点头动作，确定疲劳状态检测结果为疲劳状态，输出疲劳提示信息。如果驾驶员一直在做打盹点头动作，则驾驶员的疲劳程度会持续加深(即疲劳状态等级提高)，疲劳程度依次为轻度疲劳级别、中度疲劳级别直至重度疲劳级别，每隔X秒输出一次疲劳报警信息，直至疲劳状态检测结果为非疲劳状态，即驾驶员恢复正常驾驶状态，其中，X为大于0的数值；
基于眼部状态信息发现驾驶员的双眼闭眼持续时长达到一定时长或眨眼频率达到一定数值，确定疲劳状态检测结果为疲劳状态，输出疲劳提示信息。如果驾驶员的双眼闭眼持续时长持续增长或眨眼频率一直过高，则驾驶员的疲劳程度会持续加深(即疲劳状态等级提高)，疲劳程度依次为轻度疲劳级别、中度疲劳级别直至重度疲劳级别，每隔X秒输出一次疲劳报警信息，直至疲劳状态检测结果为非疲劳状态，即驾驶员恢复正常驾驶状态；
基于嘴部状态信息，发现驾驶员在做打哈欠动作，确定疲劳状态检测结果为疲劳状态，输出疲劳提示信息。如果驾驶员一直在做打哈欠动作，则驾驶员的疲劳程度会持续加深(即疲劳状态等级提高)，疲劳程度依次为轻度疲劳级别、中度疲劳级别直至重度疲劳级别，每隔X秒输出一次疲劳报警信息，直至疲劳状态检测结果为非疲劳状态，即驾驶员恢复正常驾驶状态；
基于头部位置信息、眼部状态信息、嘴部状态信息，如果发现驾驶员存在打盹点头、双眼闭眼持续时长达到一定时长、眨眼频率达到一定数值、打哈欠等四种行为中两种或两种以上状态，则判定驾驶员处于重度疲劳级别，每隔X秒输出一次疲劳报警信息，直至疲劳状态检测结果为非疲劳状态，即驾驶员恢复正常驾驶状态。
例如，基于上述实施例，在分心状态检测结果为分心状态时，输出与分心状态检测结果相应的提示/告警信息，可以包括：
基于头部位置信息，如果驾驶员头部位置偏离角度超出预设范围，且偏离时间超过Y秒，确定分心状态检测结果为分心状态，输出分心提示信息，其中，Y为大于0的数值。随着驾驶员的头部位置偏离程度超出预设范围的偏离时间持续增加，驾驶员的分心程度会持续加深(即分心状态等级提高)，分心程度依次为驾驶员注意力轻度分散，驾驶员注意力中度分散，驾驶员注意力严重分散等，每隔X秒输出一次分心报警信息，直至分心状态检测结果为非分心状态，即驾驶员恢复正常驾驶状态；
基于头部位置信息和视线方向信息，如果驾驶员头部位置偏离角度未超出预设范围，但视线方向偏离角度超出预设视线安全范围、且持续时间超过Y秒，确定分心状态检测结果为分心状态，输出分心提示信息。随着视线方向偏离角度超出预设视线安全范围的持续时间增加，驾驶员的分心程度会持续加深(即分心状态等级提高)，分心程度依次为驾驶员注意力轻度分散，驾驶员注意力中度分散，驾驶员注意力严重分散等，每隔X秒输出一次分心报警信息，直至分心状态检测结果为非分心状态，即驾驶员恢复正常驾驶状态；
基于头部位置信息、视线方向信息和眼睛睁合状态信息，如果驾驶员头部位置偏离角度未超出预设范围、且视线方向偏离角度未超出预设视线安全范围，但检测到驾驶员处于发呆状态，确定分心状态检测结果为分心状态，输出分心提示信息。驾驶员的分心程度会持续加深(即分心状态等级提高)，分心程度依次为驾驶员注意力轻度分散，驾驶员注意力中度分散，驾驶员注意力严重分散等，每隔X秒输出一次分心报警信息，直至分心状态检测结果为非分心状态，即驾驶员恢复正常驾驶状态。
另外，在本公开上述各实施例中，还可以包括：响应于疲劳状态检测结果、分心状态检测结果和预定分心动作的检测结果其中任意一项或多项满足预定驾驶模式切换条件时，将驾驶模式切换为自动驾驶模式。
在其中一些实施方式中，可以在疲劳状态等级和/或分心状态等级和/或分心动作等级满足预定驾驶模式切换条件时，将驾驶模式切换为自动驾驶模式，以实现安全驾驶，避免发生道路交通事故。
在本实施例中，在满足预定驾驶模式切换条件时将驾驶模式切换为自动驾驶模式，通过自动驾驶模式实现车辆的安全驾驶，以避免发生道路交通事故。
另外，在本公开各实施例中，确定驾驶员状态的检测结果之后，还可以输出驾驶员状态的检测结果，例如，在本地输出驾驶员状态的检测结果和/或远程输出驾驶员状态的检测结果。其中，本地输出驾驶员状态的检测结果即通过驾驶状态检测装置或者驾驶员监控***输出驾驶员状态的检测结果，或者向车辆中的中控***输出驾驶员状态的检测结果，以便车辆基于该驾驶员状态的检测结果对车辆进行智能驾驶控制。远程输出驾驶员状态的检测结果，例如可以是向云服务器或管理节点发送驾驶员状态的检测结果，以便由云服务器或管理节点进行驾驶员状态的检测结果的收集、分析和/或管理，或者基于该驾驶员状态的检测结果对车辆进行远程控制。
进一步地，在上述实施例中，还可以将驾驶状态的检测结果存储在数据库中该驾驶员的用户信息中，对该驾驶员的驾驶状态检测结果进行记录，以便于后续对驾驶员的驾驶状态检测结果进行查阅，或者对驾驶员的驾驶行为习惯进行分析、统计等。
另外，在本公开驾驶状态检测方法的又一个实施例中，还可以包括：通过红外摄像头进行图像采集，例如，通过车辆内至少一个位置部署的红外摄像头进行图像采集，得到驾驶员图像。
本公开实施例中的驾驶员图像通常为通过红外摄像头(包括近红外摄像头等)针对驾驶室摄取到的视频中的图像帧。
其中的红外摄像头的波长可以包括940nm或者850nm。其中的红外摄像头可以设置在车辆驾驶室内任意可以拍摄到驾驶员的位置，例如，可以将红外摄像头部署在以下任意一个或多个位置：仪表盘上方或附近位置，中控台上方或附近位置，A柱或附近位置，后视镜或附近位置。例如，在一些可选示例中，红外摄像头可以设置在仪表盘上方(例如正上方位置)，朝向正前方位置；可以设置在中控台上方(例如正中间位置)，朝向正前方位置；还可以设置在A柱上(例如可以贴在靠近A柱的玻璃上)，朝向驾驶员面部；还可以设置在后视镜上(例如可以贴在后视镜上方的玻璃上)，朝向驾驶员面部。其中，红外摄像头设置于仪表盘上方、中控台上方的位置时，可以根据摄像头的视角和驾驶员的位置确定其可选位置，例如设置于仪表盘上方的位置时，可以是红外摄像头朝向驾驶员，以保证摄像头视角不被方向盘遮挡；设置于中控台上方的位置时，如果摄像头的视角足够大，则可以针对后方，保证驾驶员在摄像机视野内，如果视角不够大，可以朝向驾驶员，以保证驾驶员出现在红外摄像头的视角中。
由于驾驶员所在区域(例如车内或者驾驶室等)的光线往往较复杂，而红外摄像头所摄取的驾驶员图像的质量往往 会优于普通摄像头所摄取的驾驶员图像的质量，尤其是在夜晚或者阴天或者隧道内等外部光线较暗环境下，红外摄像头所摄取到驾驶员图像通常明显优于普通摄像头所摄取的驾驶员图像的质量，从而有利于提高驾驶员分心状态检测和分心动作检测的准确性，进而有利于提高驾驶状态监控的准确性。
本公开实施例中，采用易部署、易使用的红外摄像头实时获取驾驶员图像，摄像头安装位置多样，在车辆的中控台、仪表盘、A柱、内后视镜等均可，利用基于深度学习技术的神经网络实现驾驶员疲劳状态检测和分心状态检测，鲁棒性好，应用范围广，在白天、黑夜、强光、弱光等场景下均能实现较好的驾驶状态检测效果。
可选地，在实际应用中，摄像头获取的原始图像由于受到各种条件的限制和随机干扰，往往不能直接使用，在本公开的一些可选示例中，可以对红外摄像头拍摄的驾驶员图像进行灰度化预处理，将红绿蓝(RGB)3通道的图像转成灰度图像，再进行驾驶员的身份认证、分心状态检测和分心动作检测等操作，以提高身份认证、分心状态检测和分心动作检测的准确性。
在其中一些实施方式中，例如可以在如下场景中，通过红外摄像头进行图像采集，得到驾驶员图像：
在车辆处于行驶状态时通过红外摄像头进行图像采集，获得驾驶员图像；和/或，
在车辆的行驶速度超过预设车速时通过红外摄像头进行图像采集，获得驾驶员图像；和/或，
在检测到车辆点火后通过红外摄像头进行图像采集，获得驾驶员图像；和/或，
在检测到车辆的启动指令时通过红外摄像头进行图像采集，获得驾驶员图像；和/或，
在检测到对车辆或车辆中部件或***的控制指令(例如加速、加速、转向、开关车窗、开关空调、开关娱乐***等等)时通过红外摄像头进行图像采集，获得驾驶员图像。
在其中一些应用场景中，可以在驾驶员启动车辆时、启动驾驶状态监测装置或者驾驶员监控***时，开始启动红外摄像头采集驾驶员图像进行驾驶状态检测。
本公开上述实施例的驾驶状态检测方法，可以由红外(包括近红外)摄像头进行图像采集获得驾驶员图像后，发送给可以加载神经网络的单片机、FPGA、ARM、CPU、GPU、微处理器、智能移动电话、笔记型计算机、平板电脑(PAD)、台式计算机或者服务器等电子设备实现，该电子设备能够运行计算机程序(也可以称为程序代码)，该计算机程序可以存储于闪存、缓存、硬盘或者光盘等计算机可读存储介质中。
本公开实施例提供的任一种驾驶状态分析方法可以由任意适当的具有数据处理能力的设备执行，包括但不限于：终端设备和服务器等。或者，本公开实施例提供的任一种驾驶状态分析方法可以由处理器执行，如处理器通过调用存储器存储的相应指令来执行本公开实施例提及的任一种驾驶状态分析方法。下文不再赘述。
本领域普通技术人员可以理解：实现上述方法实施例的全部或部分步骤可以通过程序指令相关的硬件来完成，前述的程序可以存储于一计算机可读取存储介质中，该程序在执行时，执行包括上述方法实施例的步骤；而前述的存储介质包括：ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。
图4为本公开驾驶状态分析装置一个实施例的结构示意图。该实施例的驾驶状态分析装置可用于实现本公开上述各驾驶状态分析方法实施例。如图4所示，该实施例的驾驶状态分析装置包括：驾驶状态检测模块，用于对驾驶员图像进行驾驶员的疲劳状态检测和分心状态检测，得到疲劳状态检测结果和分心状态检测结果；报警模块，用于响应于疲劳状态检测结果和分心状态检测结果二者其中之一满足预定报警条件，输出满足预定报警条件的相应检测结果的报警信息；和/或，响应于疲劳状态检测结果和分心状态检测结果二者均满足预定报警条件，输出满足预定报警条件的疲劳状态检测结果的报警信息。
在其中一些实施方式中，报警模块响应于疲劳状态检测结果和分心状态检测结果二者其中之一满足预定报警条件，输出满足预定报警条件的相应检测结果的报警信息时，用于在疲劳状态检测结果为疲劳状态时，输出与疲劳状态检测结果相应的提示/告警信息；和/或，在分心状态检测结果为分心状态时，输出与分心状态检测结果相应的提示/告警信息。
基于本公开上述实施例提供的驾驶状态分析装置，可以对驾驶员图像实现驾驶员疲劳状态检测和驾驶员分心状态的共同检测，在疲劳状态检测结果和分心状态检测结果二者其中之一满足预定报警条件时，输出满足预定报警条件的相应检测结果的报警信息；和/或，在疲劳状态检测结果和分心状态检测结果二者均满足预定报警条件时，输出满足预定报警条件的疲劳状态检测结果的报警信息，以便于提醒驾驶员注意，以提高驾驶安全性，降低道路交通事故发生率；并且，在疲劳状态检测结果和分心状态检测结果二者均满足预定报警条件时，只输出满足预定报警条件的疲劳状态检测结果的报警信息，可以避免过多或者过于频繁的报警引起驾驶员的分心和反感，本公开通过优化报警策略，提高了辅助驾驶的安全性和用户体验。
在另一些实施例中，报警模块，还可用于：在输出疲劳状态检测结果相应的报警信息后的预设时间内，抑制掉满足预定报警条件的其他检测结果相应的报警信息；和/或，在输出分心状态检测结果相应的报警信息后的预设时间内，抑制掉满足预定报警条件的其他检测结果相应的报警信息。
本实施例可以避免过多或者过于频繁的报警引起驾驶员的分心和反感，进一步提高了辅助驾驶的安全性和用户体验
图5为本公开驾驶状态分析装置一个实施例的结构示意图。如图5所示，与图4所示的实施例相比，该实施例的驾驶状态分析装置还可以包括：第一确定模块，用于确定驾驶员图像中驾驶员的头部位置的偏离角度是否超出预设范围。相应地，该实施例中，驾驶状态检测模块，用于在驾驶员的头部位置的偏离角度超出预设范围时，对驾驶员图像进行驾驶员的分心状态检测，得到分心状态检测结果；和/或，在驾驶员的头部位置的偏离角度未超出预设范围时，对驾驶员图像进行驾驶员的疲劳状态检测和分心状态检测的操作，得到疲劳状态检测结果和分心状态检测结果。
如图5所示，在其中一些实施方式中，驾驶状态检测模块可以包括：第一检测模块，用于对驾驶员图像进行头部姿态检测、眼部状态检测和/或嘴部状态检测，得到头部姿态信息、眼部状态信息和/或嘴部状态信息；第二确定模块，用于根据头部姿态信息和/或眼部状态信息，确定驾驶员的分心状态检测结果；第三确定模块，用于根据头部姿态信息、眼部状态信息和/或嘴部状态信息，确定驾驶员的疲劳状态检测结果。
其中，在一些可选示例中，第二确定模块可以包括：第一确定单元，用于根据头部姿态信息和/或眼部状态信息，确定用于表征驾驶员的分心状态的指标的参数值；第二确定单元，用于根据用于表征驾驶员的分心状态的指标的参数值，确定驾驶员的分心状态检测结果。
在一些可选示例中，所第三确定模块可以包括：第三确定单元，用于根据头部姿态信息、眼部状态信息和/或嘴部状态信息，确定用于表征驾驶员的疲劳状态的指标的参数值；第四确定单元，用于根据用于表征驾驶员的疲劳状态的指标的参数值，确定驾驶员的疲劳状态检测结果。
在一些可选示例中，第一检测模块可以包括：关键点检测单元，用于对驾驶员图像进行人脸关键点检测；第一获取单元，用于根据检测到的人脸关键点获取头部姿态信息、眼部状态信息和/或嘴部状态信息。
在一些可选示例中，第一获取单元根据检测到的人脸关键点获取头部姿态信息时，用于经第一神经网络基于人脸关键点得到头部姿态信息。
在一些可选示例中，第一获取单元根据检测到的人脸关键点获取眼部状态信息时，用于根据人脸关键点确定驾驶员图像中的眼部区域图像；基于第二神经网络对眼部区域图像进行上眼睑线和下眼睑线的检测；根据上眼睑线和下眼睑线之间的间隔确定驾驶员的眼睛睁合状态信息；其中，眼部状态信息包括：眼睛睁合状态信息。
在一些可选示例中，第一获取单元根据检测到的人脸关键点获取眼部状态信息时，用于根据人脸关键点确定驾驶员图像中的眼部区域图像；基于第三神经网络对眼部区域图像进行睁闭眼的分类处理，得到睁眼或闭眼的分类结果；其中，眼部状态信息包括：睁眼状态或闭眼状态。
在一些可选示例中，第一获取单元根据检测到的人脸关键点获取嘴部状态信息时，用于根据人脸关键点确定驾驶员图像中的嘴部区域图像；基于第四神经网络对嘴部区域图像进行上唇线和下唇线的检测；根据上唇线和下唇线之间的间隔确定驾驶员的嘴巴开合状态信息；其中，嘴部状态信息包括嘴巴开合状态信息。
在一些可选示例中，第一获取单元根据检测到的人脸关键点获取嘴部状态信息时，用于根据人脸关键点确定驾驶员图像中的嘴部区域图像；基于第五神经网络对嘴部区域图像进行张闭嘴的分类处理，得到张嘴或闭嘴的分类结果；其中，嘴部状态信息包括张嘴状态或闭嘴状态。
在一些可选示例中，第三确定单元，用于根据头部姿态信息、眼部状态信息和嘴部状态信息，确定用于表征驾驶员的疲劳状态的指标的参数值。
在一些可选示例中，第一确定单元，用于：根据头部姿态信息确定驾驶员图像中驾驶员的头部位置，得到头部位置信息；根据一段时间内的头部位置信息，获取头部位置偏离程度的参数值；和/或，根据头部姿态信息确定驾驶员图像中驾驶员的人脸朝向，得到人脸朝向信息；根据一段时间内的人脸朝向信息获取人脸朝向偏离程度的参数值；和/或，根据头部姿态信息确定驾驶员图像中驾驶员的视线方向，得到视线方向信息，根据一段时间内的视线方向信息获取视线方向偏离程度的参数值；或者，根据人脸关键点确定驾驶员图像中的眼部区域图像，基于第六神经网络得到眼部区域图像中驾驶员的视线方向信息，根据一段时间内的视线方向信息获取视线方向偏离程度的参数值；和/或，根据一段时间内的眼部状态信息，获取发呆程度的参数值。
在一些可选示例中，第三确定单元，用于：根据头部姿态信息确定驾驶员图像中驾驶员的头部位置，得到头部位置信息；根据一段时间内的头部位置信息，获取打盹程度的参数值；和/或，根据一段时间内的眼部状态信息，获取闭眼程度的参数值；和/或，根据一段时间内的眼部状态信息，获取眨眼程度的参数值；和/或，
根据一段时间内的嘴部状态信息，获取打哈欠程度的参数值。
在其中一些实施方式中，用于表征驾驶员疲劳状态的指标包括以下任意一项或多项：打盹程度、闭眼程度、眨眼程度、打哈欠程度；和/或，用于表征驾驶员分心状态的指标包括以下任意一项或多项：头部位置偏离程度、人脸朝向偏离程度、视线方向偏离程度、发呆程度。
在一些可选示例中，第一确定单元或第三确定单元根据头部姿态信息确定驾驶员图像中驾驶员的头部位置，得到头部位置信息时，用于获取头部姿态信息中的俯仰角作为头部位置；和/或，第一确定单元根据头部姿态信息确定驾驶员图像中驾驶员的人脸朝向，得到人脸朝向信息时，用于获取头部姿态信息中的俯仰角和偏航角作为人脸朝向。
在一些可选示例中，第一确定单元根据头部姿态信息确定驾驶员图像中驾驶员的视线方向，得到视线方向信息时，用于根据人脸关键点中的眼睛关键点所定位的眼睛图像确定瞳孔边沿位置，并根据瞳孔边沿位置计算瞳孔中心位置；根据瞳孔中心位置与眼睛中心位置获取头部姿态信息对应头部姿态下的眼珠转角信息；根据头部姿态信息和眼珠转角信息确定驾驶员的视线方向，得到视线方向信息。
在一些可选示例中，第一确定单元根据人脸关键点中的眼睛关键点所定位的眼睛图像确定瞳孔边沿位置时，用于基于第七神经网络对根据人脸关键点分割出的图像中的眼睛区域图像进行瞳孔边沿位置的检测，并根据第七神经网络输出的信息获取瞳孔边沿位置。
在一些可选示例中，第一确定单元根据一段时间内的视线方向信息获取视线方向偏离程度的参数值时，用于根据一段时间内的视线方向信息相对于参考视线方向的偏离角度，获取视线方向偏离程度的参数值。其中，参考视线方向可以预先设定，或者，可以为基于驾驶员图像所在视频中的前N帧驾驶员图像确定的平均视线方向；其中，N为大于1的整数。
在一些可选示例中，第一确定单元根据一段时间内的眼部状态信息，获取发呆程度的参数值时，用于根据眼部状态信息，在驾驶员的眼睛处于睁眼状态且持续达到预设发呆时间时，确定驾驶员处于发呆状态；根据一段时间内的眼部状态信息，获取发呆程度的参数值；其中，一段时间包括预设发呆时间。
在一些可选示例中，第三确定单元根据一段时间内的头部位置信息，获取打盹程度的参数值时，用于根据头部位置信息，在驾驶员的头部位置相对于预设参考头部位置的偏离程度在第一预设时间内达到预设偏离范围、且在第二预设时间内恢复至预设参考头部位置时，确定驾驶员处于打盹状态；根据一段时间内的头部位置信息，获取打盹程度的参数值；其中，一段时间包括第一预设时间和第二预设时间。
在一些可选示例中，第三确定单元根据一段时间内的嘴部状态信息，获取打哈欠程度的参数值时，用于根据嘴部状态信息，在驾驶员的嘴巴由闭嘴状态变化到张嘴状态、再恢复到闭嘴状态的时间在预设时间范围内时，确定驾驶员完成一次打哈欠动作；根据一段时间内的嘴部状态信息，获取打哈欠程度的参数值；其中，一段时间包括驾驶员的嘴巴由闭嘴状态变化到张嘴状态、再恢复到闭嘴状态的时间。
在一些可选示例中，头部位置偏离程度的参数值包括以下任意一项或多项：头部位置偏离状态，头部位置偏离方向， 头部位置在头部位置偏离方向上的偏离角度，头部位置偏离持续时长，头部位置偏离频率；和/或，人脸朝向偏离程度的参数值包括以下任意一项或多项：转头次数、转头持续时长、转头频率；和/或，视线方向偏离程度的参数值包括以下任意一项或多项：视线方向偏离角度、视线方向偏离时长、视线方向偏离频率；和/或，发呆程度的参数值包括以下任意一项或多项：睁眼幅度、睁眼持续时长、睁眼累计时长占统计时间窗的比值；和/或，打盹程度的参数值包括以下任意一项或多项：打盹点头状态、打盹点头幅度、打盹点头次数、打盹点头频率、打盹点头持续时长；和/或，闭眼程度的参数值包括以下任意一项或多项：闭眼次数、闭眼频率、闭眼持续时长、闭眼幅度、半闭眼次数、半闭眼频率、闭眼累计时长占统计时间窗的比值；和/或，眨眼程度的参数值包括以下任意一项或多项：眨眼次数、眨眼频率、眨眼持续时长、眨眼累计时长占统计时间窗的比值；和/或，打哈欠程度的参数值包括以下任意一项或多项：打哈欠状态、打哈欠次数、打哈欠持续时长、打哈欠频率。
在一些可选示例中，第二确定单元，用于在任意一项或多项用于表征驾驶员的分心状态的指标的参数值满足预定分心条件时，确定驾驶员的分心状态检测结果为分心状态；和/或，在所有用于表征驾驶员的分心状态的指标的参数值均不满足预定分心条件时，确定驾驶员的分心状态检测结果为非分心状态。
在一些可选示例中，预定分心条件包括多个分心等级条件。相应地，第二确定单元在任意一项或多项用于表征驾驶员的分心状态的指标的参数值满足预定分心条件时，确定驾驶员的分心状态检测结果为分心状态时，用于根据用于表征驾驶员的分心状态的指标的参数值满足的分心等级条件，确定分心状态等级；将确定的分心状态等级作为驾驶员的分心状态检测结果。
在一些可选示例中，第四确定单元用于在任意一项或多项用于表征驾驶员的疲劳状态的指标的参数值满足预定疲劳条件时，确定驾驶员的疲劳状态检测结果为疲劳状态；和/或，在所有用于表征驾驶员的疲劳状态的指标的参数值均不满足预定疲劳条件时，确定驾驶员的疲劳状态检测结果为非疲劳状态。
在一些可选示例中，预定疲劳条件包括多个疲劳等级条件。相应地，第四确定单元在任意一项或多项用于表征驾驶员的疲劳状态的指标的参数值满足预定疲劳条件时，确定驾驶员的疲劳状态检测结果为疲劳状态时，用于根据用于表征驾驶员的疲劳状态的指标的参数值满足的疲劳等级条件，确定疲劳状态等级；将确定的疲劳状态等级作为驾驶员的疲劳状态检测结果。
另外，再参见图5，在本公开驾驶状态检测装置的又一个实施例中，还可以包括：第二检测模块，用于对驾驶员图像进行预定分心动作检测，确定是否出现预定分心动作；第一获取模块，用于在出现预定分心动作时，根据一段时间内是否出现预定分心动作的确定结果，获取用于表征驾驶员的分心程度的指标的参数值；第四确定模块，用于根据用于表征驾驶员的分心程度的指标的参数值，确定驾驶员预定分心动作的检测结果。其中，分心程度的参数值可以包括以下任意一项或多项：预定分心动作的次数、预定分心动作的持续时长、预定分心动作的频率，等等。其中的预定分心动作可以包括以下任意一项或多项：抽烟动作，喝水动作，饮食动作，打电话动作，娱乐动作，化妆动作，等等。
在其中一些实施方式中，第二检测模块，用于提取驾驶员图像的特征；基于特征提取可能包括预定分心动作的多个候选框；基于多个候选框确定动作目标框，其中，动作目标框包括人脸的局部区域和动作交互物，或者还可以包括手部区域；基于动作目标框进行预定分心动作的分类检测，确定是否出现预定分心动作。其中，人脸的局部区域可以包括以下任意一项或多项：嘴部区域，耳部区域，眼部区域；和/或，动作交互物可以包括以下任意一项或多项：容器、烟、手机、食物、工具、饮料瓶、眼镜、口罩。
在其中一些实施方式中，第二检测模块，用于经第八神经网络对驾驶员图像进行人脸检测，得到人脸检测框，并提取人脸检测框的特征信息；经第八神经网络根据人脸检测框的特征信息确定是否出现抽烟动作。
在其中一些实施方式中，第二检测模块，用于经第九神经网络对驾驶员图像进行饮食动作/喝水动作/打电话动作/娱乐动作/化妆动作相应的预设目标对象检测，得到预设目标对象的检测框，预设目标对象包括：手部、嘴部、眼部、动作交互物，动作交互物包括以下任意一类或多类：容器、食物、电子设备、化妆品；根据预设目标对象的检测框，确定是否出现预定分心动作；其中，是否出现预定分心动作的确定结果包括以下之一：未出现饮食动作/喝水动作/打电话动作/娱乐动作/化妆动作；出现饮食动作，出现喝水动作，出现打电话动作，出现娱乐动作，出现化妆动作。
在其中一些实施方式中，第二检测模块根据预设目标对象的检测框，确定是否出现预定分心动作时，用于根据是否检测到手部的检测框、嘴部的检测框、眼部的检测框和动作交互物的检测框，以及根据手部的检测框与动作交互物的检测框是否重叠、动作交互物的类型以及动作交互物的检测框与嘴部的检测框或眼部的检测框之间的距离是否满足预设条件，确定是否出现预定分心动作。
在其中一些可选示例中，第二检测模块根据手部的检测框与动作交互物的检测框是否重叠、以及动作交互物的检测框与嘴部的检测框或眼部的检测框之间的位置关系是否满足预设条件，确定是否出现预定分心动作时，用于：若手部的检测框与动作交互物的检测框重叠，动作交互物的类型为容器或食物、且动作交互物的检测框与嘴部的检测框之间重叠，确定出现饮食动作或喝水动作；和/或，若手部的检测框与动作交互物的检测框重叠，动作交互物的类型为电子设备，且动作交互物的检测框与嘴部的检测框之间的最小距离小于第一预设距离、或者动作交互物的检测框与眼部的检测框之间的最小距离小于第二预设距离，确定出现娱乐动作或打电话动作。
在其中一些可选示例中，第二检测模块，还用于：若未同时检测到手部的检测框、嘴部的检测框和任一动作交互物的检测框，且未同时检测到手部的检测框、眼部的检测框和任一动作交互物的检测框，确定是否预定分心动作的确定结果为未检测到饮食动作、喝水动作、打电话动作、娱乐动作和化妆动作；和/或，若手部的检测框与动作交互物的检测框未重叠，确定是否预定分心动作的确定结果为未检测到饮食动作、喝水动作、打电话动作、娱乐动作、化妆动作；和/或，若动作交互物的类型为容器或食物、且动作交互物的检测框与嘴部的检测框之间未重叠，和/或，动作交互物的类型为电子设备、且动作交互物的检测框与嘴部的检测框之间的最小距离不小于第一预设距离、或者动作交互物的检测框与眼部的检测框之间的最小距离不小于第二预设距离，确定是否预定分心动作的确定结果为未检测到饮食动作、喝水动作、打电话动作、娱乐动作和化妆动作。
另外，再参见图5，在本公开驾驶状态检测装置的再一个实施例中，还可以包括：第五确定模块，用于根据用于表征分心程度的指标的参数值满足的分心动作等级条件，确定分心动作等级。相应地，该实施例中，第四确定模块用于将 确定的分心动作等级作为驾驶员预定分心动作的检测结果。
另外，再参见图5，在本公开驾驶状态检测装置的再一个实施例中，还可以包括：报警模块，用于根据驾驶员状态的检测结果，进行报警；和/或，驾驶控制模块，用于根据驾驶员状态的检测结果，进行智能驾驶控制。其中，驾驶员状态的检测结果包括以下任意一项或多项：疲劳状态的检测结果，分心状态检测结果，预定分心动作的检测结果。
在上述实施例中，报警模块，还可用于响应于预定分心动作的检测结果满足预定报警条件，输出满足预定报警条件的预定分心动作的检测结果相应的报警信息。
在其中一些实施方式中，报警模块响应于疲劳状态检测结果和分心状态检测结果二者其中之一满足预定报警条件，输出满足预定报警条件的相应检测结果的报警信息时，用于响应于疲劳状态检测结果、分心状态检测结果和预定分心动作的检测结果三者其中之一满足预定报警条件，输出满足预定报警条件的相应检测结果的报警信息。
在其中一些实施方式中，报警模块响应于疲劳状态检测结果和分心状态检测结果二者均满足预定报警条件，输出满足预定报警条件的疲劳状态检测结果的报警信息时，用于：响应于疲劳状态检测结果、分心状态检测结果和预定分心动作的检测结果均满足预定报警条件，输出满足预定报警条件的疲劳状态检测结果的报警信息；和/或，响应于疲劳状态检测结果、分心状态检测结果和预定分心动作的检测结果三者其中之二满足预定报警条件，按照预设报警策略输出报警信息。
其中，报警模块响应于疲劳状态检测结果、分心状态检测结果和预定分心动作的检测结果三者其中之二满足预定报警条件，按照预设报警策略输出报警信息时，可用于：响应于疲劳状态检测结果和预定分心动作的检测结果满足预定报警条件，输出满足预定报警条件的疲劳状态检测结果的报警信息；和/或，响应于分心状态检测结果和预定分心动作的检测结果三者其中之二满足预定报警条件，输出满足预定报警条件的分心状态检测结果的报警信息，或者，输出满足预定报警条件的预定分心动作的检测结果相应的报警信息。
在进一步实施方式中，报警模块，还可用于在输出满足预定报警条件的预定分心动作的检测结果相应的报警信息后的预设时间内，抑制掉满足预定报警条件的其他检测结果相应的报警信息。
在其中一些实施方式中，报警模块在疲劳状态检测结果为疲劳状态等级时，根据疲劳状态等级输出相应的提示或者告警信息；和/或，在分心状态检测结果为分心状态等级时，根据分心状态等级输出相应的提示或者告警信息；和/或，在预定分心动作的检测结果为分心动作等级时，根据分心动作等级输出相应的提示或者告警信息。
另外，再参见图5，在在本公开驾驶状态检测装置的再一个实施例中，还可以包括：驾驶控制模块，用于响应于疲劳状态检测结果、分心状态检测结果和预定分心动作的检测结果其中任意一项或多项满足预定驾驶模式切换条件时，将驾驶模式切换为自动驾驶模式。
在其中一些实施方式中，驾驶控制模块，用于在疲劳状态等级和/或分心状态等级和/或分心动作等级满足预定驾驶模式切换条件时，将驾驶模式切换为自动驾驶模式。
另外，再参见图5，在本公开上述各实施例中，还可以包括：红外摄像头，用于进行图像采集，得到驾驶员图像。该红外摄像头部署于车辆内的至少一个位置，例如以下任意一个或多个位置：仪表盘上方或附近位置，中控台上方或附近位置，A柱或附近位置，后视镜或附近位置。
在其中一些实施方式中，红外摄像头，用于：在车辆处于行驶状态时进行图像采集，获得驾驶员图像；和/或，在车辆的行驶速度超过预设车速时进行图像采集，获得驾驶员图像；和/或，在检测到车辆点火后进行图像采集，获得驾驶员图像；和/或，在检测到车辆的启动指令时进行图像采集，获得驾驶员图像；和/或，在检测到对车辆或车辆中部件或***的控制指令时进行图像采集，获得驾驶员图像。
图6为本公开驾驶员监控***一个实施例的结构示意图。该实施例的驾驶员监控***可用于实现本公开上述各驾驶状态检测方法实施例。如图6所示，该实施例的驾驶员监控***包括：显示装置，用于显示驾驶员图像；驾驶状态分析装置，用于对驾驶员图像进行驾驶员的疲劳状态检测和分心状态检测；响应于疲劳状态检测结果和分心状态检测结果二者其中之一满足预定报警条件，输出满足预定报警条件的相应检测结果的报警信息；和/或，响应于疲劳状态检测结果和分心状态检测结果二者均满足预定报警条件，输出满足预定报警条件的疲劳状态检测结果的报警信息。
其中的驾驶员状态检测装置可以通过本公开上述任一实施例的驾驶状态检测装置实现。
基于本公开上述实施例提供的驾驶员监控***，可以对驾驶员图像实现驾驶员疲劳状态检测和驾驶员分心状态的共同检测，在疲劳状态检测结果和分心状态检测结果二者其中之一满足预定报警条件时，输出满足预定报警条件的相应检测结果的报警信息；和/或，在疲劳状态检测结果和分心状态检测结果二者均满足预定报警条件时，输出满足预定报警条件的疲劳状态检测结果的报警信息，以便于提醒驾驶员注意，以提高驾驶安全性，降低道路交通事故发生率；并且，在疲劳状态检测结果和分心状态检测结果二者均满足预定报警条件时，只输出满足预定报警条件的疲劳状态检测结果的报警信息，可以避免过多或者过于频繁的报警引起驾驶员的分心和反感，本公开通过优化报警策略，提高了辅助驾驶的安全性和用户体验。
另外，本公开实施例提供的另一种电子设备，包括：
存储器，用于存储计算机程序；
处理器，用于执行存储器中存储的计算机程序，且计算机程序被执行时，实现本公开上述任一实施例的驾驶状态分析方法。
图7为本公开电子设备一个应用实施例的结构示意图。下面参考图7，其示出了适于用来实现本公开实施例的终端设备或服务器的电子设备的结构示意图。如图7所示，该电子设备包括一个或多个处理器、通信部等，一个或多个处理器例如：一个或多个中央处理单元(CPU)，和/或一个或多个图像处理器(GPU)等，处理器可以根据存储在只读存储器(ROM)中的可执行指令或者从存储部分加载到随机访问存储器(RAM)中的可执行指令而执行各种适当的动作和处理。通信部可包括但不限于网卡，网卡可包括但不限于IB(Infiniband)网卡，处理器可与只读存储器和/或随机访问存储器中通信以执行可执行指令，通过总线与通信部相连、并经通信部与其他目标设备通信，从而完成本公开实施例提供的任一方法对应的操作，例如，对驾驶员图像进行驾驶员的疲劳状态检测和分心状态检测疲劳状态检测结果分心状态检测结果；响应于疲劳状态检测结果和分心状态检测结果二者其中之一满足预定报警条件，输出满足预定报警条件的相 应检测结果的报警信息；和/或，响应于疲劳状态检测结果和分心状态检测结果二者均满足预定报警条件，输出满足预定报警条件的疲劳状态检测结果的报警信息。
此外，在RAM中，还可存储有装置操作所需的各种程序和数据。CPU、ROM以及RAM通过总线彼此相连。在有RAM的情况下，ROM为可选模块。RAM存储可执行指令，或在运行时向ROM中写入可执行指令，可执行指令使处理器执行本公开上述任一方法对应的操作。输入/输出(I/O)接口也连接至总线。通信部可以集成设置，也可以设置为具有多个子模块(例如多个IB网卡)，并在总线链接上。
以下部件连接至I/O接口：包括键盘、鼠标等的输入部分；包括诸如阴极射线管(CRT)、液晶显示器(LCD)等以及扬声器等的输出部分；包括硬盘等的存储部分；以及包括诸如LAN卡、调制解调器等的网络接口卡的通信部分。通信部分经由诸如因特网的网络执行通信处理。驱动器也根据需要连接至I/O接口。可拆卸介质，诸如磁盘、光盘、磁光盘、半导体存储器等等，根据需要安装在驱动器上，以便于从其上读出的计算机程序根据需要被安装入存储部分。
需要说明的，如图7所示的架构仅为一种可选实现方式，在具体实践过程中，可根据实际需要对上述图7的部件数量和类型进行选择、删减、增加或替换；在不同功能部件设置上，也可采用分离设置或集成设置等实现方式，例如GPU和CPU可分离设置或者可将GPU集成在CPU上，通信部可分离设置，也可集成设置在CPU或GPU上，等等。这些可替换的实施方式均落入本公开公开的保护范围。
特别地，根据本公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括有形地包含在机器可读介质上的计算机程序，计算机程序包含用于执行流程图所示的方法的程序代码，程序代码可包括对应执行本公开任一实施例提供的方法步骤对应的指令。在这样的实施例中，该计算机程序可以通过通信部分从网络上被下载和安装，和/或从可拆卸介质被安装。在该计算机程序被CPU执行时，执行本公开的方法中限定的上述功能。
另外，本公开实施例还提供了一种计算机程序，包括计算机指令，当计算机指令在设备的处理器中运行时，实现本公开上述任一实施例的驾驶状态分析方法。
另外，本公开实施例还提供了一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时，实现本公开上述任一实施例的驾驶状态分析方法。
图8为本公开车辆一个实施例的结构示意图。如图8所示，该实施例的车辆包括中控***，还包括：本公开上述任一实施例的驾驶状态分析装置或者驾驶员监控***。
基于本公开上述实施例提供的车辆，包括本公开上述任一实施例的驾驶状态分析装置或者驾驶员监控***，可以对驾驶员图像实现驾驶员疲劳状态检测和驾驶员分心状态的共同检测，在疲劳状态检测结果和分心状态检测结果二者其中之一满足预定报警条件时，输出满足预定报警条件的相应检测结果的报警信息；和/或，在疲劳状态检测结果和分心状态检测结果二者均满足预定报警条件时，输出满足预定报警条件的疲劳状态检测结果的报警信息，以便于提醒驾驶员注意，以提高驾驶安全性，降低道路交通事故发生率；并且，在疲劳状态检测结果和分心状态检测结果二者均满足预定报警条件时，只输出满足预定报警条件的疲劳状态检测结果的报警信息，可以避免过多或者过于频繁的报警引起驾驶员的分心和反感，本公开通过优化报警策略，提高了辅助驾驶的安全性和用户体验。
在其中一些实施方式中，中控***用于在驾驶员状态的检测结果满足预定驾驶模式切换条件时，将驾驶模式切换为自动驾驶模式，并在自动驾驶模式下对车辆进行自动驾驶控制。
在另一些实施方式中，中控***还可用于在接收到切换为人工驾驶的驾驶指令时，将驾驶模式切换为人工驾驶模式。
再参见图8，上述实施例的车辆还可以包括：娱乐***，用于根据中控***的控制指令，输出与提示/告警预定条件相应的提示/告警信息；和/或，根据中控***的控制指令，调整提示/告警信息的预警效果、或者娱乐项目的播放效果。
其中的娱乐***例如可以包括扬声器、蜂鸣器、灯光设备等。
再参见图8，上述实施例的车辆还可以包括：至少一个红外摄像头，用于进行图像采集。
在其中一些实施方式中，车辆中的红外摄像头可以部署在车辆内的至少一个位置，例如可以部署在以下任意一个或多个位置：仪表盘上方或附近位置，中控台上方或附近位置，A柱或附近位置，后视镜或附近位置，等。
本说明书中各个实施例均采用递进的方式描述，每个实施例重点说明的都是与其它实施例的不同之处，各个实施例之间相同或相似的部分相互参见即可。对于***实施例而言，由于其与方法实施例基本对应，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。
可能以许多方式来实现本公开的方法和装置、设备。例如，可通过软件、硬件、固件或者软件、硬件、固件的任何组合来实现本公开的方法和装置、设备。用于所述方法的步骤的上述顺序仅是为了进行说明，本公开的方法的步骤不限于以上描述的顺序，除非以其它方式特别说明。此外，在一些实施例中，还可将本公开实施为记录在记录介质中的程序，这些程序包括用于实现根据本公开的方法的机器可读指令。因而，本公开还覆盖存储用于执行根据本公开的方法的程序的记录介质。
本公开的描述是为了示例和描述起见而给出的，而并不是无遗漏的或者将本公开限于所公开的形式。很多修改和变化对于本领域的普通技术人员而言是显然的。选择和描述实施例是为了更好说明本公开的原理和实际应用，并且使本领域的普通技术人员能够理解本公开从而设计适于特定用途的带有各种修改的各种实施例。

Claims (120)

  1. 一种驾驶状态分析方法，其特征在于，包括：
    对驾驶员图像进行驾驶员的疲劳状态检测和分心状态检测，得到疲劳状态检测结果和分心状态检测结果；
    响应于所述疲劳状态检测结果和所述分心状态检测结果二者其中之一满足预定报警条件，输出满足预定报警条件的相应检测结果的报警信息；和/或，响应于所述疲劳状态检测结果和所述分心状态检测结果二者均满足预定报警条件，输出满足预定报警条件的所述疲劳状态检测结果的报警信息。
  2. 根据权利要求1所述的方法，其特征在于，所述响应于所述疲劳状态检测结果和所述分心状态检测结果二者其中之一满足预定报警条件，输出满足预定报警条件的相应检测结果的报警信息，包括：
    在所述疲劳状态检测结果为疲劳状态时，输出与所述疲劳状态检测结果相应的提示/告警信息；和/或，
    在所述分心状态检测结果为分心状态时，输出与所述分心状态检测结果相应的提示/告警信息。
  3. 根据权利要求1或2所述的方法，其特征在于，还包括：
    在输出所述疲劳状态检测结果相应的报警信息后的预设时间内，抑制掉满足预定报警条件的其他检测结果相应的报警信息；和/或，
    在输出所述分心状态检测结果相应的报警信息后的预设时间内，抑制掉满足预定报警条件的其他检测结果相应的报警信息。
  4. 根据权利要求1-3任一所述的方法，其特征在于，所述对驾驶员图像进行驾驶员的疲劳状态和分心状态检测之前，还包括：
    确定所述驾驶员图像中驾驶员的头部位置的偏离角度是否超出预设范围；
    若所述驾驶员的头部位置的偏离角度超出预设范围，对所述驾驶员图像进行驾驶员的分心状态检测，得到所述分心状态检测结果；和/或，
    若所述驾驶员的头部位置的偏离角度未超出预设范围，执行所述对驾驶员图像进行驾驶员的疲劳状态检测和分心状态检测的操作，得到所述疲劳状态检测结果和所述分心状态检测结果。
  5. 根据权利要求4所述的方法，其特征在于，对所述驾驶员图像进行驾驶员的分心状态检测，包括：
    对驾驶员图像进行头部姿态检测和/或眼部状态检测，得到头部姿态信息和/或眼部状态信息；
    根据所述头部姿态信息和/或所述眼部状态信息，确定驾驶员的分心状态检测结果。
  6. 根据权利要求5所述的方法，其特征在于，所述根据所述头部姿态信息和/或所述眼部状态信息，确定驾驶员的分心状态检测结果，包括：
    根据所述头部姿态信息和/或所述眼部状态信息，确定用于表征驾驶员的分心状态的指标的参数值；
    根据用于表征驾驶员的分心状态的指标的参数值，确定驾驶员的分心状态检测结果。
  7. 根据权利要求1-6任一所述的方法，其特征在于，所述对驾驶员图像进行驾驶员的疲劳状态检测和分心状态检测，包括：
    对驾驶员图像进行头部姿态检测、眼部状态检测和/或嘴部状态检测，得到头部姿态信息、眼部状态信息和/或嘴部状态信息；
    根据所述头部姿态信息、所述眼部状态信息和/或嘴部状态信息，确定驾驶员的疲劳状态检测结果和分心状态检测结果。
  8. 根据权利要求7所述的方法，其特征在于，所述根据所述头部姿态信息、所述眼部状态信息和/或嘴部状态信息，确定驾驶员的疲劳状态检测结果和分心状态检测结果，包括:
    根据所述头部姿态信息、所述眼部状态信息和/或嘴部状态信息，确定用于表征驾驶员的疲劳状态的指标的参数值和用于表征驾驶员的分心状态的指标的参数值；
    根据所述用于表征驾驶员的疲劳状态的指标的参数值，确定驾驶员的疲劳状态检测结果，以及根据所述用于表征驾驶员的分心状态的指标的参数值，确定驾驶员的分心状态检测结果。
  9. 根据权利要求5-8任一所述的方法，其特征在于，所述对驾驶员图像进行头部姿态检测、眼部状态检测和/或嘴部状态检测，得到头部姿态信息、眼部状态信息和/或嘴部状态信息，包括：
    对所述驾驶员图像进行人脸关键点检测；
    根据检测到的所述人脸关键点获取头部姿态信息、眼部状态信息和/或嘴部状态信息。
  10. 根据权利要求9所述的方法，其特征在于，所述根据检测到的所述人脸关键点获取头部姿态信息，包括：
    经第一神经网络基于所述人脸关键点得到所述头部姿态信息。
  11. 根据权利要求9或10所述的方法，其特征在于，所述根据检测到的所述人脸关键点获取眼部状态信息，包括：
    根据所述人脸关键点确定所述驾驶员图像中的眼部区域图像；
    基于第二神经网络对所述眼部区域图像进行上眼睑线和下眼睑线的检测；
    根据所述上眼睑线和下眼睑线之间的间隔确定所述驾驶员的眼睛睁合状态信息；其中，所述眼部状态信息包括：所述眼睛睁合状态信息。
  12. 根据权利要求9或10所述的方法，其特征在于，所述根据检测到的所述人脸关键点获取眼部状态信息，包括：
    根据所述人脸关键点确定所述驾驶员图像中的眼部区域图像；
    基于第三神经网络对所述眼部区域图像进行睁闭眼的分类处理，得到睁眼或闭眼的分类结果；其中，所述眼部状态信息包括：睁眼状态或闭眼状态。
  13. 根据权利要求9-12任一所述的方法，其特征在于，所述根据检测到的所述人脸关键点获取嘴部状态信息，包括：
    根据所述人脸关键点确定所述驾驶员图像中的嘴部区域图像；
    基于第四神经网络对所述嘴部区域图像进行上唇线和下唇线的检测；
    根据所述上唇线和下唇线之间的间隔确定所述驾驶员的嘴巴开合状态信息；其中，所述嘴部状态信息包括所述嘴巴开合状态信息。
  14. 根据权利要求9-12任一所述的方法，其特征在于，所述根据检测到的所述人脸关键点获取嘴部状态信息，包括：
    根据所述人脸关键点确定所述驾驶员图像中的嘴部区域图像；
    基于第五神经网络对所述嘴部区域图像进行张闭嘴的分类处理，得到张嘴或闭嘴的分类结果；其中，所述嘴部状态信息包括张嘴状态或闭嘴状态。
  15. 根据权利要求8-14任一所述的方法，其特征在于，所述根据所述头部姿态信息、所述眼部状态信息和/或嘴部状态信息，确定用于表征驾驶员的疲劳状态的指标的参数值，包括：
    根据所述头部姿态信息、所述眼部状态信息和所述嘴部状态信息，确定用于表征驾驶员的疲劳状态的指标的参数值。
  16. 根据权利要求6、8-15任一所述的方法，其特征在于，所述根据所述头部姿态信息和/或所述眼部状态信息，确定用于表征驾驶员的分心状态的指标的参数值，包括：
    根据所述头部姿态信息确定所述驾驶员图像中驾驶员的头部位置，得到头部位置信息；根据一段时间内的所述头部位置信息，获取头部位置偏离程度的参数值；和/或，
    根据所述头部姿态信息确定所述驾驶员图像中驾驶员的人脸朝向，得到人脸朝向信息；根据一段时间内的人脸朝向信息获取人脸朝向偏离程度的参数值；和/或，
    根据所述头部姿态信息确定所述驾驶员图像中驾驶员的视线方向，得到视线方向信息，根据一段时间内的视线方向信息获取视线方向偏离程度的参数值；或者，根据所述人脸关键点确定所述驾驶员图像中的眼部区域图像，基于第六神经网络得到所述眼部区域图像中驾驶员的视线方向信息，根据一段时间内的视线方向信息获取视线方向偏离程度的参数值；和/或，
    根据一段时间内的所述眼部状态信息，获取发呆程度的参数值。
  17. 根据权利要求8-16任一所述的方法，其特征在于，所述根据所述头部姿态信息、所述眼部状态信息和/或嘴部状态信息，确定用于表征驾驶员的疲劳状态的指标的参数值和用于表征驾驶员的分心状态的指标的参数值，包括：
    根据所述头部姿态信息确定所述驾驶员图像中驾驶员的头部位置，得到头部位置信息；根据一段时间内的所述头部位置信息，获取头部位置偏离程度的参数值；和/或，
    根据所述头部姿态信息确定所述驾驶员图像中驾驶员的人脸朝向，得到人脸朝向信息；根据一段时间内的人脸朝向信息获取人脸朝向偏离程度的参数值；和/或，
    根据所述头部姿态信息确定所述驾驶员图像中驾驶员的视线方向，得到视线方向信息，根据一段时间内的视线方向信息获取视线方向偏离程度的参数值；或者，根据所述人脸关键点确定所述驾驶员图像中的眼部区域图像，基于第六神经网络得到所述眼部区域图像中驾驶员的视线方向信息，根据一段时间内的视线方向信息获取视线方向偏离程度的参数值；和/或，
    根据一段时间内的所述眼部状态信息，获取发呆程度的参数值；和/或，
    根据所述头部姿态信息确定所述驾驶员图像中驾驶员的头部位置，得到头部位置信息；根据一段时间内的所述头部位置信息，获取打盹程度的参数值；和/或，
    根据一段时间内的所述眼部状态信息，获取闭眼程度的参数值；和/或，
    根据一段时间内的所述眼部状态信息，获取眨眼程度的参数值；和/或，
    根据一段时间内的所述嘴部状态信息，获取打哈欠程度的参数值。
  18. 根据权利要求16或17所述的方法，其特征在于，所述用于表征驾驶员疲劳状态的指标包括以下任意一项或多项：打盹程度、闭眼程度、眨眼程度、打哈欠程度；和/或，
    所述用于表征驾驶员分心状态的指标包括以下任意一项或多项：头部位置偏离程度、人脸朝向偏离程度、视线方向偏离程度、发呆程度。
  19. 根据权利要求16-18任一所述的方法，其特征在于，所述根据所述头部姿态信息确定所述驾驶员图像中驾驶员的头部位置，包括：获取所述头部姿态信息中的俯仰角作为所述头部位置；和/或，
    所述根据所述头部姿态信息确定所述驾驶员图像中驾驶员的人脸朝向，包括：获取所述头部姿态信息中的俯仰角和偏航角作为所述人脸朝向。
  20. 根据权利要求16-19任一所述的方法，其特征在于，所述根据所述头部姿态信息确定所述驾驶员图像中驾驶员的视线方向，得到视线方向信息，包括：
    根据所述人脸关键点中的眼睛关键点所定位的眼睛图像确定瞳孔边沿位置，并根据所述瞳孔边沿位置计算瞳孔中心位置；
    根据所述瞳孔中心位置与眼睛中心位置获取所述头部姿态信息对应头部姿态下的眼珠转角信息；
    根据所述头部姿态信息和所述眼珠转角信息确定所述驾驶员的视线方向，得到视线方向信息。
  21. 根据权利要求20所述的方法，其特征在于，所述根据所述人脸关键点中的眼睛关键点所定位的眼睛图像确定瞳孔边沿位置，包括：
    基于第七神经网络对根据所述人脸关键点分割出的所述图像中的眼睛区域图像进行瞳孔边沿位置的检测，并根据所述第七神经网络输出的信息获取所述瞳孔边沿位置。
  22. 根据权利要求16-21任一所述的方法，其特征在于，所述根据一段时间内的视线方向信息获取视线方向偏离程度的参数值，包括：
    根据一段时间内的视线方向信息相对于参考视线方向的偏离角度，获取视线方向偏离程度的参数值。
  23. 根据权利要求22所述的方法，其特征在于，还包括：
    预先设定所述参考视线方向；或者，
    以基于所述驾驶员图像所在视频中的前N帧驾驶员图像确定的平均视线方向作为所述参考视线方向；其中，N为大于1的整数。
  24. 根据权利要求16-23任一所述的方法，其特征在于，所述根据一段时间内的所述眼部状态信息，获取发呆程度的参数值，包括：
    根据所述眼部状态信息，在所述驾驶员的眼睛处于睁眼状态且持续达到预设发呆时间时，确定所述驾驶员处于发呆状态；
    根据一段时间内的眼部状态信息，获取发呆程度的参数值；其中，所述一段时间包括所述预设发呆时间。
  25. 根据权利要求17-24任一所述的方法，其特征在于，所述根据一段时间内的所述头部位置信息，获取打盹程度的参数值，包括：
    根据所述头部位置信息，在所述驾驶员的头部位置相对于预设参考头部位置的偏离程度在第一预设时间内达到预设偏离范围、且在第二预设时间内恢复至所述预设参考头部位置时，确定所述驾驶员处于打盹状态；
    根据一段时间内的所述头部位置信息，获取打盹程度的参数值；其中，所述一段时间包括所述第一预设时间和所述第二预设时间。
  26. 根据权利要求17-24任一所述的方法，其特征在于，所述根据一段时间内的所述嘴部状态信息，获取打哈欠程度的参数值，包括：
    根据所述嘴部状态信息，在所述驾驶员的嘴巴由闭嘴状态变化到张嘴状态、再恢复到闭嘴状态的时间在预设时间范围内时，确定所述驾驶员完成一次打哈欠动作；
    根据一段时间内的嘴部状态信息，获取打哈欠程度的参数值；其中，所述一段时间包括所述驾驶员的嘴巴由闭嘴状态变化到张嘴状态、再恢复到闭嘴状态的时间。
  27. 根据权利要求16-26任一所述的方法，其特征在于，所述头部位置偏离程度的参数值包括以下任意一项或多项：头部位置偏离状态，头部位置偏离方向，头部位置在所述头部位置偏离方向上的偏离角度，头部位置偏离持续时长，头部位置偏离频率；和/或，
    所述人脸朝向偏离程度的参数值包括以下任意一项或多项：转头次数、转头持续时长、转头频率；和/或，
    所述视线方向偏离程度的参数值包括以下任意一项或多项：视线方向偏离角度、视线方向偏离时长、视线方向偏离频率；和/或，
    所述发呆程度的参数值包括以下任意一项或多项：睁眼幅度、睁眼持续时长、睁眼累计时长占统计时间窗的比值；和/或，
    所述打盹程度的参数值包括以下任意一项或多项：打盹点头状态、打盹点头幅度、打盹点头次数、打盹点头频率、打盹点头持续时长；和/或，
    所述闭眼程度的参数值包括以下任意一项或多项：闭眼次数、闭眼频率、闭眼持续时长、闭眼幅度、半闭眼次数、半闭眼频率、闭眼累计时长占统计时间窗的比值；和/或，
    所述眨眼程度的参数值包括以下任意一项或多项：眨眼次数、眨眼频率、眨眼持续时长、眨眼累计时长占统计时间窗的比值；和/或，
    所述打哈欠程度的参数值包括以下任意一项或多项：打哈欠状态、打哈欠次数、打哈欠持续时长、打哈欠频率。
  28. 根据权利要求6、8-27任一所述的方法，其特征在于，根据所述用于表征驾驶员的分心状态的指标的参数值，确定驾驶员的分心状态检测结果，包括：
    在任意一项或多项所述用于表征驾驶员的分心状态的指标的参数值满足预定分心条件时，确定驾驶员的分心状态检测结果为分心状态；和/或，
    在所有所述用于表征驾驶员的分心状态的指标的参数值均不满足预定分心条件时，确定驾驶员的分心状态检测结果为非分心状态。
  29. 根据权利要求28所述的方法，其特征在于，所述预定分心条件包括多个分心等级条件；
    所述在任意一项或多项所述用于表征驾驶员的分心状态的指标的参数值满足预定分心条件时，确定驾驶员的分心状态检测结果为分心状态，包括：
    根据所述用于表征驾驶员的分心状态的指标的参数值满足的分心等级条件，确定分心状态等级；
    将确定的分心状态等级作为驾驶员的分心状态检测结果。
  30. 根据权利要求8-29任一所述的方法，其特征在于，根据所述用于表征驾驶员的疲劳状态的指标的参数值，确定驾驶员的疲劳状态的检测结果，包括：
    在任意一项或多项所述用于表征驾驶员的疲劳状态的指标的参数值满足预定疲劳条件时，确定驾驶员的疲劳状态检测结果为疲劳状态；和/或，
    在所有所述用于表征驾驶员的疲劳状态的指标的参数值均不满足预定疲劳条件时，确定驾驶员的疲劳状态检测结果为非疲劳状态。
  31. 根据权利要求30所述的方法，其特征在于，所述预定疲劳条件包括多个疲劳等级条件；
    所述在任意一项或多项所述用于表征驾驶员的疲劳状态的指标的参数值满足预定疲劳条件时，确定驾驶员的疲劳状态检测结果为疲劳状态，包括：
    根据所述用于表征驾驶员的疲劳状态的指标的参数值满足的疲劳等级条件，确定疲劳状态等级；
    将确定的疲劳状态等级作为驾驶员的疲劳状态检测结果。
  32. 根据权利要求1-27任一所述的方法，其特征在于，还包括：
    对所述驾驶员图像进行预定分心动作检测，确定是否出现预定分心动作；
    若出现预定分心动作，获取一段时间内是否出现所述预定分心动作的确定结果，获取用于表征驾驶员的分心程度的指标的参数值；
    根据所述用于表征驾驶员的分心程度的指标的参数值，确定驾驶员预定分心动作的检测结果。
  33. 根据权利要求32所述的方法，其特征在于，所述预定分心动作包括以下任意一项或多项：抽烟动作，喝水动作，饮食动作，打电话动作，娱乐动作，化妆动作。
  34. 根据权利要求33所述的方法，其特征在于，所述对驾驶员图像进行预定分心动作检测，确定是否出现预定分心动作，包括：
    提取所述驾驶员图像的特征；
    基于所述特征提取可能包括预定分心动作的多个候选框；
    基于多个所述候选框确定动作目标框，其中，所述动作目标框包括人脸的局部区域和动作交互物；
    基于所述动作目标框进行预定分心动作的分类检测，确定是否出现所述预定分心动作。
  35. 根据权利要求34所述的方法，其特征在于，所述人脸的局部区域包括以下任意一项或多项：嘴部区域，耳部区域，眼部区域；和/或，
    所述动作交互物包括以下任意一项或多项：容器、烟、手机、食物、工具、饮料瓶、眼镜、口罩。
  36. 根据权利要求34或35所述的方法，其特征在于，所述动作目标框还包括：手部区域。
  37. 根据权利要求33所述的方法，其特征在于，所述对驾驶员图像进行预定分心动作检测，确定是否出现预定分心动作，包括：
    经第八神经网络对所述驾驶员图像进行人脸检测，得到人脸检测框，并提取所述人脸检测框的特征信息；
    经所述第八神经网络根据所述人脸检测框的特征信息确定是否出现抽烟动作。
  38. 根据权利要求33所述的方法，其特征在于，所述对驾驶员图像进行预定分心动作检测，确定是否出现预定分心动作，包括：
    经第九神经网络对所述驾驶员图像进行所述饮食动作/喝水动作/打电话动作/娱乐动作/化妆动作相应的预设目标对象检测，得到预设目标对象的检测框；所述预设目标对象包括：手部、嘴部、眼部、动作交互物；所述动作交互物包括以下任意一类或多类：容器、食物、电子设备、化妆品；
    根据所述预设目标对象的检测框，确定是否出现预定分心动作；其中，是否出现预定分心动作的确定结果包括以下之一：未出现饮食动作/喝水动作/打电话动作/娱乐动作/化妆动作；出现饮食动作，出现喝水动作，出现打电话动作，出现娱乐动作，出现化妆动作。
  39. 根据权利要求38所述的方法，其特征在于，所述根据所述预设目标对象的检测框，确定是否出现预定分心动作，包括：
    根据是否检测到手部的检测框、嘴部的检测框、眼部的检测框和动作交互物的检测框，以及根据手部的检测框与动作交互物的检测框是否重叠、所述动作交互物的类型以及动作交互物的检测框与嘴部的检测框或眼部的检测框之间的距离是否满足预设条件，确定是否出现预定分心动作。
  40. 根据权利要求39所述的方法，其特征在于，所述根据手部的检测框与动作交互物的检测框是否重叠、以及动作交互物的检测框与嘴部的检测框或眼部的检测框之间的位置关系是否满足预设条件，确定是否出现预定分心动作，包括：
    若所述手部的检测框与所述动作交互物的检测框重叠，所述动作交互物的类型为容器或食物、且所述动作交互物的检测框与嘴部的检测框之间重叠，确定出现饮食动作、喝水动作或化妆动作；和/或，
    若所述手部的检测框与所述动作交互物的检测框重叠，所述动作交互物的类型为电子设备，且所述动作交互物的检测框与嘴部的检测框之间的最小距离小于第一预设距离、或者所述动作交互物的检测框与眼部的检测框之间的最小距离小于第二预设距离，确定出现娱乐动作或打电话动作。
  41. 根据权利要求39或40所述的方法，其特征在于，还包括：
    若未同时检测到手部的检测框、嘴部的检测框和任一所述动作交互物的检测框，且未同时检测到手部的检测框、眼部的检测框和任一所述动作交互物的检测框，确定是否预定分心动作的确定结果为未检测到饮食动作、喝水动作、打电话动作、娱乐动作和化妆动作；和/或，
    若手部的检测框与所述动作交互物的检测框未重叠，确定是否预定分心动作的确定结果为未检测到饮食动作、喝水动作、打电话动作、娱乐动作、化妆动作；和/或，
    若所述动作交互物的类型为容器或食物、且所述动作交互物的检测框与嘴部的检测框之间未重叠，和/或，所述动作交互物的类型为电子设备、且所述动作交互物的检测框与嘴部的检测框之间的最小距离不小于第一预设距离、或者所述动作交互物的检测框与眼部的检测框之间的最小距离不小于第二预设距离，确定是否预定分心动作的确定结果为未检测到饮食动作、喝水动作、打电话动作、娱乐动作和化妆动作。
  42. 根据权利要求32-41任一所述的方法，其特征在于，所述分心程度的参数值包括以下任意一项或多项：预定分心动作的次数、预定分心动作的持续时长、预定分心动作的频率。
  43. 根据权利要求32-42任一所述的方法，其特征在于，若确定出现预定分心动作，还包括：
    根据所述用于表征分心程度的指标的参数值满足的分心动作等级条件，确定分心动作等级；
    将确定的分心动作等级作为驾驶员预定分心动作的检测结果。
  44. 根据权利要求32-43任一所述的方法，其特征在于，还包括：
    响应于预定分心动作的检测结果满足预定报警条件，输出满足预定报警条件的预定分心动作的检测结果相应的报警信息。
  45. 根据权利要求1-44任一所述的方法，其特征在于，所述响应于所述疲劳状态检测结果和所述分心状态检测结果二者其中之一满足预定报警条件，输出满足预定报警条件的相应检测结果的报警信息，包括：响应于所述疲劳状态检测结果、所述分心状态检测结果和所述预定分心动作的检测结果三者其中之一满足预定报警条件，输出满足预定报警条件的相应检测结果的报警信息。
  46. 根据权利要求1-45任一所述的方法，其特征在于，所述响应于所述疲劳状态检测结果和所述分心状态检测结果二者均满足预定报警条件，输出满足预定报警条件的所述疲劳状态检测结果的报警信息，包括：
    响应于所述疲劳状态检测结果、所述分心状态检测结果和所述预定分心动作的检测结果均满足预定报警条件，输出满足预定报警条件的所述疲劳状态检测结果的报警信息；和/或，
    响应于所述疲劳状态检测结果、所述分心状态检测结果和所述预定分心动作的检测结果三者其中之二满足预定报警条件，按照预设报警策略输出报警信息。
  47. 根据权利要求46所述的方法，其特征在于，所述响应于所述疲劳状态检测结果、所述分心状态检测结果和所述预定分心动作的检测结果三者其中之二满足预定报警条件，按照预设报警策略输出报警信息，包括：
    响应于所述疲劳状态检测结果和所述预定分心动作的检测结果满足预定报警条件，输出满足预定报警条件的所述疲劳状态检测结果的报警信息；和/或，
    响应于所述分心状态检测结果和所述预定分心动作的检测结果三者其中之二满足预定报警条件，输出满足预定报警条件的所述分心状态检测结果的报警信息，或者，输出满足预定报警条件的所述预定分心动作的检测结果相应的报警信息。
  48. 根据权利要求44-47任一所述的方法，其特征在于，还包括：
    在输出满足预定报警条件的所述预定分心动作的检测结果相应的报警信息后的预设时间内，抑制掉满足预定报警条件的其他检测结果相应的报警信息。
  49. 根据权利要求44-48任一所述的方法，其特征在于，在所述疲劳状态检测结果为疲劳状态等级时，所述输出满足预定报警条件的所述疲劳状态检测结果的报警信息，包括：根据所述疲劳状态等级输出相应的提示或者告警信息；和/或，
    在所述分心状态检测结果为分心状态等级时，所述输出满足预定报警条件的所述分心状态检测结果相应的提示/告警信息，包括：根据所述分心状态等级输出相应的提示或者告警信息；和/或，
    在预定分心动作的检测结果为分心动作等级时，所述输出满足预定报警条件的预定分心动作的检测结果相应的报警信息，包括：根据所述分心动作等级输出相应的提示或者告警信息。
  50. 根据权利要求49所述的方法，其特征在于，还包括：
    响应于所述疲劳状态检测结果、所述分心状态检测结果和所述预定分心动作的检测结果其中任意一项或多项满足预定驾驶模式切换条件时，将驾驶模式切换为自动驾驶模式。
  51. 根据权利要求50所述的方法，其特征在于，所述响应于所述疲劳状态检测结果、所述分心状态检测结果和所述预定分心动作的检测结果其中任意一项或多项满足预定驾驶模式切换条件时，将驾驶模式切换为自动驾驶模式，包括：
    在所述疲劳状态等级和/或所述分心状态等级和/或所述分心动作等级满足预定驾驶模式切换条件时，将驾驶模式切换为自动驾驶模式。
  52. 根据权利要求1-51任一所述的方法，其特征在于，还包括：
    通过红外摄像头进行图像采集，得到所述驾驶员图像。
  53. 根据权利要求52所述的方法，其特征在于，所述通过红外摄像头进行图像采集，包括：
    通过车辆内至少一个位置部署的所述红外摄像头进行图像采集。
  54. 根据权利要求53所述的方法，其特征在于，所述至少一个位置包括以下任意一个或多个位置：仪表盘上方或附近位置，中控台上方或附近位置，A柱或附近位置，后视镜或附近位置。
  55. 根据权利要求52-54任一所述的方法，其特征在于，所述通过红外摄像头进行图像采集，得到所述驾驶员图像，包括：
    在所述车辆处于行驶状态时通过红外摄像头进行图像采集，获得所述驾驶员图像；和/或，
    在所述车辆的行驶速度超过预设车速时通过红外摄像头进行图像采集，获得所述驾驶员图像；和/或，
    在检测到所述车辆点火后通过红外摄像头进行图像采集，获得所述驾驶员图像；和/或，
    在检测到所述车辆的启动指令时通过红外摄像头进行图像采集，获得所述驾驶员图像；和/或，
    在检测到对所述车辆或所述车辆中部件或***的控制指令时通过红外摄像头进行图像采集，获得所述驾驶员图像。
  56. 一种驾驶状态分析装置，其特征在于，包括：
    驾驶状态检测模块，用于对驾驶员图像进行驾驶员的疲劳状态检测和分心状态检测，得到疲劳状态检测结果和分心状态检测结果；
    报警模块，用于响应于所述疲劳状态检测结果和所述分心状态检测结果二者其中之一满足预定报警条件，输出满足预定报警条件的相应检测结果的报警信息；和/或，响应于所述疲劳状态检测结果和所述分心状态检测结果二者均满足预定报警条件，输出满足预定报警条件的所述疲劳状态检测结果的报警信息。
  57. 根据权利要求56所述的装置，其特征在于，所述报警模块响应于疲劳状态检测结果和分心状态检测结果二者其中之一满足预定报警条件，输出满足预定报警条件的相应检测结果的报警信息时，用于在所述疲劳状态检测结果为疲劳状态时，输出与所述疲劳状态检测结果相应的提示/告警信息；和/或，在所述分心状态检测结果为分心状态时，输出与所述分心状态检测结果相应的提示/告警信息。
  58. 根据权利要求56或57所述的装置，其特征在于，所述报警模块，还用于：在输出所述疲劳状态检测结果相应的报警信息后的预设时间内，抑制掉满足预定报警条件的其他检测结果相应的报警信息；和/或，在输出所述分心状态检测结果相应的报警信息后的预设时间内，抑制掉满足预定报警条件的其他检测结果相应的报警信息。
  59. 根据权利要求56-58任一所述的装置，其特征在于，还包括：
    第一确定模块，用于确定所述驾驶员图像中驾驶员的头部位置的偏离角度是否超出预设范围；
    所述驾驶状态检测模块，用于在所述驾驶员的头部位置的偏离角度超出预设范围时，对所述驾驶员图像进行驾驶员的分心状态检测，得到所述分心状态检测结果；和/或，在所述驾驶员的头部位置的偏离角度未超出预设范围时，对驾驶员图像进行驾驶员的疲劳状态检测和分心状态检测的操作，得到所述疲劳状态检测结果和所述分心状态检测结果。
  60. 根据权利要求59所述的装置，其特征在于，所述驾驶状态检测模块包括：
    第一检测模块，用于对驾驶员图像进行头部姿态检测、眼部状态检测和/或嘴部状态检测，得到头部姿态信息、眼部状态信息和/或嘴部状态信息；
    第二确定模块，用于根据所述头部姿态信息和/或所述眼部状态信息，确定驾驶员的分心状态检测结果；
    第三确定模块，用于根据所述头部姿态信息、所述眼部状态信息和/或嘴部状态信息，确定驾驶员的疲劳状态检测结果。
  61. 根据权利要求60所述的装置，其特征在于，所述第二确定模块包括：
    第一确定单元，用于所述根据所述头部姿态信息和/或所述眼部状态信息，确定用于表征驾驶员的分心状态的指标的参数值；
    第二确定单元，用于根据用于表征驾驶员的分心状态的指标的参数值，确定驾驶员的分心状态检测结果。
  62. 根据权利要求60或61所述的装置，其特征在于，所第三确定模块包括:
    第三确定单元，用于根据所述头部姿态信息、所述眼部状态信息和/或嘴部状态信息，确定用于表征驾驶员的疲劳状态的指标的参数值；
    第四确定单元，用于根据所述用于表征驾驶员的疲劳状态的指标的参数值，确定驾驶员的疲劳状态检测结果。
  63. 根据权利要求60-62任一所述的装置，其特征在于，所述第一检测模块包括：
    关键点检测单元，用于对所述驾驶员图像进行人脸关键点检测；
    第一获取单元，用于根据检测到的所述人脸关键点获取头部姿态信息、眼部状态信息和/或嘴部状态信息。
  64. 根据权利要求63所述的装置，其特征在于，所述第一获取单元根据检测到的所述人脸关键点获取头部姿态信息时，用于经第一神经网络基于所述人脸关键点得到所述头部姿态信息。
  65. 根据权利要求63或64所述的装置，其特征在于，所述第一获取单元根据检测到的所述人脸关键点获取眼部状态信息时，用于根据所述人脸关键点确定所述驾驶员图像中的眼部区域图像；基于第二神经网络对所述眼部区域图像进行上眼睑线和下眼睑线的检测；根据所述上眼睑线和下眼睑线之间的间隔确定所述驾驶员的眼睛睁合状态信息；其中，所述眼部状态信息包括：所述眼睛睁合状态信息。
  66. 根据权利要求63或64所述的装置，其特征在于，所述第一获取单元根据检测到的所述人脸关键点获取眼部状态信息时，用于根据所述人脸关键点确定所述驾驶员图像中的眼部区域图像；基于第三神经网络对所述眼部区域图像进行睁闭眼的分类处理，得到睁眼或闭眼的分类结果；其中，所述眼部状态信息包括：睁眼状态或闭眼状态。
  67. 根据权利要求63-66任一所述的装置，其特征在于，所述第一获取单元根据检测到的所述人脸关键点获取嘴部状态信息时，用于根据所述人脸关键点确定所述驾驶员图像中的嘴部区域图像；基于第四神经网络对所述嘴部区域图像进行上唇线和下唇线的检测；根据所述上唇线和下唇线之间的间隔确定所述驾驶员的嘴巴开合状态信息；其中，所述嘴部状态信息包括所述嘴巴开合状态信息。
  68. 根据权利要求63-66任一所述的装置，其特征在于，所述第一获取单元根据检测到的所述人脸关键点获取嘴部状态信息时，用于根据所述人脸关键点确定所述驾驶员图像中的嘴部区域图像；基于第五神经网络对所述嘴部区域图像进行张闭嘴的分类处理，得到张嘴或闭嘴的分类结果；其中，所述嘴部状态信息包括张嘴状态或闭嘴状态。
  69. 根据权利要求62-68任一所述的装置，其特征在于，所述第三确定单元，用于所述根据所述头部姿态信息、所述眼部状态信息和所述嘴部状态信息，确定用于表征驾驶员的疲劳状态的指标的参数值。
  70. 根据权利要求61-69任一所述的装置，其特征在于，所述第一确定单元，用于：
    根据所述头部姿态信息确定所述驾驶员图像中驾驶员的头部位置，得到头部位置信息；根据一段时间内的所述头部位置信息，获取头部位置偏离程度的参数值；和/或，
    根据所述头部姿态信息确定所述驾驶员图像中驾驶员的人脸朝向，得到人脸朝向信息；根据一段时间内的人脸朝向信息获取人脸朝向偏离程度的参数值；和/或，
    根据所述头部姿态信息确定所述驾驶员图像中驾驶员的视线方向，得到视线方向信息，根据一段时间内的视线方向信息获取视线方向偏离程度的参数值；或者，根据所述人脸关键点确定所述驾驶员图像中的眼部区域图像，基于第六神经网络得到所述眼部区域图像中驾驶员的视线方向信息，根据一段时间内的视线方向信息获取视线方向偏离程度的参数值；和/或，
    根据一段时间内的所述眼部状态信息，获取发呆程度的参数值。
  71. 根据权利要求62-70任一所述的装置，其特征在于，所述第三确定单元，用于：根据所述头部姿态信息确定所述驾驶员图像中驾驶员的头部位置，得到头部位置信息；根据一段时间内的所述头部位置信息，获取打盹程度的参数值；和/或，
    根据一段时间内的所述眼部状态信息，获取闭眼程度的参数值；和/或，
    根据一段时间内的所述眼部状态信息，获取眨眼程度的参数值；和/或，
    根据一段时间内的所述嘴部状态信息，获取打哈欠程度的参数值。
  72. 根据权利要求61-71任一所述的装置，其特征在于，所述用于表征驾驶员疲劳状态的指标包括以下任意一项或多项：打盹程度、闭眼程度、眨眼程度、打哈欠程度；和/或，
    所述用于表征驾驶员分心状态的指标包括以下任意一项或多项：头部位置偏离程度、人脸朝向偏离程度、视线方向偏离程度、发呆程度。
  73. 根据权利要求70-72任一所述的装置，其特征在于，所述第一确定单元或所述第三确定单元根据所述头部姿态信息确定所述驾驶员图像中驾驶员的头部位置时，用于获取所述头部姿态信息中的俯仰角作为所述头部位置；和/或，
    所述第一确定单元根据所述头部姿态信息确定所述驾驶员图像中驾驶员的人脸朝向时，用于获取所述头部姿态信息中的俯仰角和偏航角作为所述人脸朝向。
  74. 根据权利要求70-73任一所述的装置，其特征在于，所述第一确定单元根据所述头部姿态信息确定所述驾驶员图像中驾驶员的视线方向，得到视线方向信息时，用于根据所述人脸关键点中的眼睛关键点所定位的眼睛图像确定瞳孔边沿位置，并根据所述瞳孔边沿位置计算瞳孔中心位置；根据所述瞳孔中心位置与眼睛中心位置获取所述头部姿态信息对应头部姿态下的眼珠转角信息；根据所述头部姿态信息和所述眼珠转角信息确定所述驾驶员的视线方向，得到视线方向信息。
  75. 根据权利要求74所述的装置，其特征在于，所述第一确定单元根据所述人脸关键点中的眼睛关键点所定位的 眼睛图像确定瞳孔边沿位置时，用于基于第七神经网络对根据所述人脸关键点分割出的所述图像中的眼睛区域图像进行瞳孔边沿位置的检测，并根据所述第七神经网络输出的信息获取瞳孔边沿位置。
  76. 根据权利要求70-75任一所述的装置，其特征在于，所述第一确定单元根据一段时间内的视线方向信息获取视线方向偏离程度的参数值时，用于根据一段时间内的视线方向信息相对于参考视线方向的偏离角度，获取视线方向偏离程度的参数值。
  77. 根据权利要求76所述的装置，其特征在于，所述参考视线方向预先设定，或者，所述参考视线方向为基于所述驾驶员图像所在视频中的前N帧驾驶员图像确定的平均视线方向；其中，N为大于1的整数。
  78. 根据权利要求70-77任一所述的装置，其特征在于，所述第一确定单元根据一段时间内的所述眼部状态信息，获取发呆程度的参数值时，用于根据所述眼部状态信息，在所述驾驶员的眼睛处于睁眼状态且持续达到预设发呆时间时，确定所述驾驶员处于发呆状态；根据一段时间内的眼部状态信息，获取发呆程度的参数值；其中，所述一段时间包括所述预设发呆时间。
  79. 根据权利要求71-78任一所述的装置，其特征在于，所述第三确定单元根据一段时间内的所述头部位置信息，获取打盹程度的参数值时，用于根据所述头部位置信息，在所述驾驶员的头部位置相对于预设参考头部位置的偏离程度在第一预设时间内达到预设偏离范围、且在第二预设时间内恢复至所述预设参考头部位置时，确定所述驾驶员处于打盹状态；根据一段时间内的所述头部位置信息，获取打盹程度的参数值；其中，所述一段时间包括所述第一预设时间和所述第二预设时间。
  80. 根据权利要求71-79任一所述的装置，其特征在于，所述第三确定单元根据一段时间内的所述嘴部状态信息，获取打哈欠程度的参数值时，用于根据所述嘴部状态信息，在所述驾驶员的嘴巴由闭嘴状态变化到张嘴状态、再恢复到闭嘴状态的时间在预设时间范围内时，确定所述驾驶员完成一次打哈欠动作；根据一段时间内的嘴部状态信息，获取打哈欠程度的参数值；其中，所述一段时间包括所述驾驶员的嘴巴由闭嘴状态变化到张嘴状态、再恢复到闭嘴状态的时间。
  81. 根据权利要求70-80任一所述的装置，其特征在于，所述头部位置偏离程度的参数值包括以下任意一项或多项：头部位置偏离状态，头部位置偏离方向，头部位置在所述头部位置偏离方向上的偏离角度，头部位置偏离持续时长，头部位置偏离频率；和/或，
    所述人脸朝向偏离程度的参数值包括以下任意一项或多项：转头次数、转头持续时长、转头频率；和/或，
    所述视线方向偏离程度的参数值包括以下任意一项或多项：视线方向偏离角度、视线方向偏离时长、视线方向偏离频率；和/或，
    所述发呆程度的参数值包括以下任意一项或多项：睁眼幅度、睁眼持续时长、睁眼累计时长占统计时间窗的比值；和/或，
    所述打盹程度的参数值包括以下任意一项或多项：打盹点头状态、打盹点头幅度、打盹点头次数、打盹点头频率、打盹点头持续时长；和/或，
    所述闭眼程度的参数值包括以下任意一项或多项：闭眼次数、闭眼频率、闭眼持续时长、闭眼幅度、半闭眼次数、半闭眼频率、闭眼累计时长占统计时间窗的比值；和/或，
    所述眨眼程度的参数值包括以下任意一项或多项：眨眼次数、眨眼频率、眨眼持续时长、眨眼累计时长占统计时间窗的比值；和/或，
    所述打哈欠程度的参数值包括以下任意一项或多项：打哈欠状态、打哈欠次数、打哈欠持续时长、打哈欠频率。
  82. 根据权利要求61-81任一所述的装置，其特征在于，所述第二确定单元，用于在任意一项或多项所述用于表征驾驶员的分心状态的指标的参数值满足预定分心条件时，确定驾驶员的分心状态检测结果为分心状态；和/或，在所有所述用于表征驾驶员的分心状态的指标的参数值均不满足预定分心条件时，确定驾驶员的分心状态检测结果为非分心状态。
  83. 根据权利要求82所述的装置，其特征在于，所述预定分心条件包括多个分心等级条件；
    所述第二确定单元在任意一项或多项所述用于表征驾驶员的分心状态的指标的参数值满足预定分心条件时，确定驾驶员的分心状态检测结果为分心状态时，用于根据所述用于表征驾驶员的分心状态的指标的参数值满足的分心等级条件，确定分心状态等级；将确定的分心状态等级作为驾驶员的分心状态检测结果。
  84. 根据权利要求62-83任一所述的装置，其特征在于，所述第四确定单元用于在任意一项或多项所述用于表征驾驶员的疲劳状态的指标的参数值满足预定疲劳条件时，确定驾驶员的疲劳状态检测结果为疲劳状态；和/或，在所有所述用于表征驾驶员的疲劳状态的指标的参数值均不满足预定疲劳条件时，确定驾驶员的疲劳状态检测结果为非疲劳状态。
  85. 根据权利要求84所述的装置，其特征在于，所述预定疲劳条件包括多个疲劳等级条件；
    所述第四确定单元在任意一项或多项所述用于表征驾驶员的疲劳状态的指标的参数值满足预定疲劳条件时，确定驾驶员的疲劳状态检测结果为疲劳状态时，用于根据所述用于表征驾驶员的疲劳状态的指标的参数值满足的疲劳等级条件，确定疲劳状态等级；将确定的疲劳状态等级作为驾驶员的疲劳状态检测结果。
  86. 根据权利要求56-85任一所述的装置，其特征在于，还包括：
    第二检测模块，用于对所述驾驶员图像进行预定分心动作检测，确定是否出现预定分心动作；
    第一获取模块，用于在出现预定分心动作时，根据一段时间内是否出现所述预定分心动作的确定结果，获取用于表征驾驶员的分心程度的指标的参数值；
    第四确定模块，用于根据所述用于表征驾驶员的分心程度的指标的参数值，确定驾驶员预定分心动作的检测结果。
  87. 根据权利要求86所述的装置，其特征在于，所述预定分心动作包括以下任意一项或多项：抽烟动作，喝水动作，饮食动作，打电话动作，娱乐动作，化妆动作。
  88. 根据权利要求87所述的装置，其特征在于，所述第二检测模块，用于提取所述驾驶员图像的特征；基于所述特征提取可能包括预定分心动作的多个候选框；基于多个所述候选框确定动作目标框，其中，所述动作目标框包括人脸的局部区域和动作交互物；基于所述动作目标框进行预定分心动作的分类检测，确定是否出现所述预定分心动作。
  89. 根据权利要求88所述的装置，其特征在于，所述人脸的局部区域包括以下任意一项或多项：嘴部区域，耳部区域，眼部区域；和/或，
    所述动作交互物包括以下任意一项或多项：容器、烟、手机、食物、工具、饮料瓶、眼镜、口罩。
  90. 根据权利要求88或89所述的装置，其特征在于，所述动作目标框还包括：手部区域。
  91. 根据权利要求87所述的装置，其特征在于，所述第二检测模块，用于经第八神经网络对所述驾驶员图像进行人脸检测，得到人脸检测框，并提取所述人脸检测框的特征信息；经所述第八神经网络根据所述人脸检测框的特征信息确定是否出现抽烟动作。
  92. 根据权利要求87所述的装置，其特征在于，所述第二检测模块，用于经第九神经网络对所述驾驶员图像进行所述饮食动作/喝水动作/打电话动作/娱乐动作/化妆动作相应的预设目标对象检测，得到预设目标对象的检测框，所述预设目标对象包括：手部、嘴部、眼部、动作交互物，所述动作交互物包括以下任意一类或多类：容器、食物、电子设备、化妆品；根据所述预设目标对象的检测框，确定是否出现预定分心动作；其中，是否出现预定分心动作的确定结果包括以下之一：未出现饮食动作/喝水动作/打电话动作/娱乐动作/化妆动作；出现饮食动作，出现喝水动作，出现打电话动作，出现娱乐动作，出现化妆动作。
  93. 根据权利要求92所述的装置，其特征在于，所述第二检测模块根据所述预设目标对象的检测框，确定是否出现预定分心动作时，用于根据是否检测到手部的检测框、嘴部的检测框、眼部的检测框和动作交互物的检测框，以及根据手部的检测框与动作交互物的检测框是否重叠、所述动作交互物的类型以及动作交互物的检测框与嘴部的检测框或眼部的检测框之间的距离是否满足预设条件，确定是否出现预定分心动作。
  94. 根据权利要求93所述的装置，其特征在于，所述第二检测模块根据手部的检测框与动作交互物的检测框是否重叠、以及动作交互物的检测框与嘴部的检测框或眼部的检测框之间的位置关系是否满足预设条件，确定是否出现预定分心动作时，用于：若所述手部的检测框与所述动作交互物的检测框重叠，所述动作交互物的类型为容器或食物、且所述动作交互物的检测框与嘴部的检测框之间重叠，确定出现饮食动作、喝水动作或化妆动作；和/或，若所述手部的检测框与所述动作交互物的检测框重叠，所述动作交互物的类型为电子设备，且所述动作交互物的检测框与嘴部的检测框之间的最小距离小于第一预设距离、或者所述动作交互物的检测框与眼部的检测框之间的最小距离小于第二预设距离，确定出现娱乐动作或打电话动作。
  95. 根据权利要求93或94所述的装置，其特征在于，所述第二检测模块，还用于：若未同时检测到手部的检测框、嘴部的检测框和任一所述动作交互物的检测框，且未同时检测到手部的检测框、眼部的检测框和任一所述动作交互物的检测框，确定是否预定分心动作的确定结果为未检测到饮食动作、喝水动作、打电话动作、娱乐动作和化妆动作；和/或，若手部的检测框与所述动作交互物的检测框未重叠，确定是否预定分心动作的确定结果为未检测到饮食动作、喝水动作、打电话动作、娱乐动作、化妆动作；和/或，若所述动作交互物的类型为容器或食物、且所述动作交互物的检测框与嘴部的检测框之间未重叠，和/或，所述动作交互物的类型为电子设备、且所述动作交互物的检测框与嘴部的检测框之间的最小距离不小于第一预设距离、或者所述动作交互物的检测框与眼部的检测框之间的最小距离不小于第二预设距离，确定是否预定分心动作的确定结果为未检测到饮食动作、喝水动作、打电话动作、娱乐动作和化妆动作。
  96. 根据权利要求86-95任一所述的装置，其特征在于，所述分心程度的参数值包括以下任意一项或多项：预定分心动作的次数、预定分心动作的持续时长、预定分心动作的频率。
  97. 根据权利要求86-96任一所述的装置，其特征在于，还包括：
    第五确定模块，用于根据所述用于表征分心程度的指标的参数值满足的分心动作等级条件，确定分心动作等级；
    所述第四确定模块，用于将确定的分心动作等级作为驾驶员预定分心动作的检测结果。
  98. 根据权利要求86-97任一所述的装置，其特征在于，所述报警模块，还用于响应于预定分心动作的检测结果满足预定报警条件，输出满足预定报警条件的预定分心动作的检测结果相应的报警信息。
  99. 根据权利要求56-98任一所述的装置，其特征在于，所述报警模块响应于所述疲劳状态检测结果和所述分心状态检测结果二者其中之一满足预定报警条件，输出满足预定报警条件的相应检测结果的报警信息时，用于响应于所述疲劳状态检测结果、所述分心状态检测结果和所述预定分心动作的检测结果三者其中之一满足预定报警条件，输出满足预定报警条件的相应检测结果的报警信息。
  100. 根据权利要求56-99任一所述的装置，其特征在于，所述报警模块响应于疲劳状态检测结果和分心状态检测结果二者均满足预定报警条件，输出满足预定报警条件的疲劳状态检测结果的报警信息时，用于：响应于所述疲劳状态检测结果、所述分心状态检测结果和所述预定分心动作的检测结果均满足预定报警条件，输出满足预定报警条件的所述疲劳状态检测结果的报警信息；和/或，响应于所述疲劳状态检测结果、所述分心状态检测结果和所述预定分心动作的检测结果三者其中之二满足预定报警条件，按照预设报警策略输出报警信息。
  101. 根据权利要求100所述的装置，其特征在于，所述报警模块响应于疲劳状态检测结果、分心状态检测结果和预定分心动作的检测结果三者其中之二满足预定报警条件，按照预设报警策略输出报警信息时，用于：响应于所述疲劳状态检测结果和所述预定分心动作的检测结果满足预定报警条件，输出满足预定报警条件的所述疲劳状态检测结果的报警信息；和/或，响应于所述分心状态检测结果和所述预定分心动作的检测结果三者其中之二满足预定报警条件，输出满足预定报警条件的所述分心状态检测结果的报警信息，或者，输出满足预定报警条件的所述预定分心动作的检测结果相应的报警信息。
  102. 根据权利要求98-101任一所述的装置，其特征在于，所述报警模块，还用于在输出满足预定报警条件的预定分心动作的检测结果相应的报警信息后的预设时间内，抑制掉满足预定报警条件的其他检测结果相应的报警信息。
  103. 根据权利要求98-102任一所述的装置，其特征在于，所述报警模块在所述疲劳状态检测结果为疲劳状态等级时，根据所述疲劳状态等级输出相应的提示或者告警信息；和/或，在所述分心状态检测结果为分心状态等级时，根据所述分心状态等级输出相应的提示或者告警信息；和/或，在所述预定分心动作的检测结果为分心动作等级时，根据所述分心动作等级输出相应的提示或者告警信息。
  104. 根据权利要求103所述的装置，其特征在于，还包括：
    驾驶控制模块，用于响应于所述疲劳状态检测结果、所述分心状态检测结果和所述预定分心动作的检测结果其中任意一项或多项满足预定驾驶模式切换条件时，将驾驶模式切换为自动驾驶模式。
  105. 根据权利要求104所述的装置，其特征在于，所述驾驶控制模块，用于在所述疲劳状态等级和/或所述分心状态等级和/或所述分心动作等级满足预定驾驶模式切换条件时，将驾驶模式切换为自动驾驶模式。
  106. 根据权利要求56-105任一所述的装置，其特征在于，还包括：
    红外摄像头，用于进行图像采集，得到所述驾驶员图像。
  107. 根据权利要求106所述的装置，其特征在于，所述红外摄像头部署于车辆内的至少一个位置。
  108. 根据权利要求107所述的装置，其特征在于，所述至少一个位置包括以下任意一个或多个位置：仪表盘上方或附近位置，中控台上方或附近位置，A柱或附近位置，后视镜或附近位置。
  109. 根据权利要求106-108任一所述的装置，其特征在于，所述红外摄像头，用于：在所述车辆处于行驶状态时进行图像采集，获得所述驾驶员图像；和/或，在所述车辆的行驶速度超过预设车速时进行图像采集，获得所述驾驶员图像；和/或，在检测到所述车辆点火后进行图像采集，获得所述驾驶员图像；和/或，在检测到所述车辆的启动指令时进行图像采集，获得所述驾驶员图像；和/或，在检测到对所述车辆或所述车辆中部件或***的控制指令时进行图像采集，获得所述驾驶员图像。
  110. 一种驾驶员监控***，其特征在于，包括：
    显示装置，用于显示驾驶员图像；
    驾驶状态分析装置，用于对驾驶员图像进行驾驶员的疲劳状态检测和分心状态检测，得到疲劳状态检测结果和分心状态检测结果；响应于所述疲劳状态检测结果和所述分心状态检测结果二者其中之一满足预定报警条件，输出满足预定报警条件的相应检测结果的报警信息；和/或，响应于所述疲劳状态检测结果和所述分心状态检测结果二者均满足预定报警条件，输出满足预定报警条件的所述疲劳状态检测结果的报警信息。
  111. 根据权利要求110所述的***，其特征在于，所述驾驶员状态检测装置包括权利要求56-109任一所述的驾驶状态分析装置。
  112. 一种电子设备，其特征在于，包括：
    存储器，用于存储计算机程序；
    处理器，用于执行所述存储器中存储的计算机程序，且所述计算机程序被执行时，实现上述权利要求1-55任一所述的方法。
  113. 一种计算机可读存储介质，其上存储有计算机程序，其特征在于，该计算机程序被处理器执行时，实现上述权利要求1-55任一所述的方法。
  114. 一种车辆，包括中控***，其特征在于，还包括：权利要求56-107任一所述的驾驶状态分析装置，或者权利要求108-109任一所述的驾驶员监控***。
  115. 根据权利要求114所述的车辆，其特征在于，所述中控***，用于在驾驶员状态的检测结果满足预定驾驶模式切换条件时，将驾驶模式切换为自动驾驶模式，并在自动驾驶模式下对所述车辆进行自动驾驶控制。
  116. 根据权利要求114或115所述的车辆，其特征在于，所述中控***，还用于根据所述驾驶状态分析装置或者所述驾驶员监控***输出的手势检测的结果生成的控制指令，对所述车辆进行相应的控制。
  117. 根据权利要求114-116任一所述的车辆，其特征在于，所述中控***，还用于在接收到切换为人工驾驶的驾驶指令时，将驾驶模式切换为人工驾驶模式。
  118. 根据权利要求114-117任一所述的车辆，其特征在于，还包括：
    娱乐***，用于根据所述中控***的控制指令进行报警；和/或，根据所述中控***的控制指令，调整报警的预警效果、或者娱乐项目的播放效果。
  119. 根据权利要求114-118任一所述的车辆，其特征在于，还包括：
    至少一个红外摄像头，用于进行图像采集。
  120. 根据权利要求119所述的车辆，其特征在于，所述红外摄像头部署在所述车辆内的至少一个位置，所述至少一个位置包括以下任意一个或多个位置：仪表盘上方或附近位置，中控台上方或附近位置，A柱或附近位置，后视镜或附近位置。
PCT/CN2019/111932 2018-10-19 2019-10-18 驾驶状态分析方法和装置、驾驶员监控***、车辆 WO2020078465A1 (zh)

Priority Applications (4)

Application Number Priority Date Filing Date Title
KR1020207029610A KR102469234B1 (ko) 2018-10-19 2019-10-18 운전 상태 분석 방법 및 장치, 운전자 모니터링 시스템 및 차량
JP2020551931A JP2021516829A (ja) 2018-10-19 2019-10-18 運転状態分析方法及び装置、運転者監視システム並びに車両
SG11202009420RA SG11202009420RA (en) 2018-10-19 2019-10-18 Driving state analysis method and apparatus, driver monitoring system and vehicle
US17/031,030 US11386679B2 (en) 2018-10-19 2020-09-24 Driving state analysis method and apparatus, driver monitoring system and vehicle

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811224316.3A CN111079476B (zh) 2018-10-19 2018-10-19 驾驶状态分析方法和装置、驾驶员监控***、车辆
CN201811224316.3 2018-10-19

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/031,030 Continuation US11386679B2 (en) 2018-10-19 2020-09-24 Driving state analysis method and apparatus, driver monitoring system and vehicle

Publications (1)

Publication Number Publication Date
WO2020078465A1 true WO2020078465A1 (zh) 2020-04-23

Family

ID=70282873

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/111932 WO2020078465A1 (zh) 2018-10-19 2019-10-18 驾驶状态分析方法和装置、驾驶员监控***、车辆

Country Status (6)

Country Link
US (1) US11386679B2 (zh)
JP (1) JP2021516829A (zh)
KR (1) KR102469234B1 (zh)
CN (1) CN111079476B (zh)
SG (1) SG11202009420RA (zh)
WO (1) WO2020078465A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113642522A (zh) * 2021-09-01 2021-11-12 中国科学院自动化研究所 基于音视频的疲劳状态检测方法和装置
CN114495069A (zh) * 2020-10-27 2022-05-13 中车株洲电力机车研究所有限公司 一种监测司机驾驶状态的方法及***

Families Citing this family (28)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018033137A1 (zh) * 2016-08-19 2018-02-22 北京市商汤科技开发有限公司 在视频图像中展示业务对象的方法、装置和电子设备
CN111626082A (zh) * 2019-02-28 2020-09-04 佳能株式会社 检测装置和方法及图像处理装置和***
KR20210009596A (ko) * 2019-07-17 2021-01-27 엘지전자 주식회사 지능적 음성 인식 방법, 음성 인식 장치 및 지능형 컴퓨팅 디바이스
JP7047821B2 (ja) * 2019-07-18 2022-04-05 トヨタ自動車株式会社 運転支援装置
JP7410681B2 (ja) * 2019-09-25 2024-01-10 株式会社Subaru 車両制御システム
TWI805925B (zh) * 2020-06-05 2023-06-21 財團法人車輛研究測試中心 駕駛者狀態監控器測試方法及其測試系統
CN111695510A (zh) * 2020-06-12 2020-09-22 浙江工业大学 一种基于图像的电脑操作员的疲劳检测方法
CN111950371B (zh) * 2020-07-10 2023-05-19 上海淇毓信息科技有限公司 疲劳驾驶预警方法及其装置、电子设备、存储介质
CN112070927A (zh) * 2020-08-28 2020-12-11 浙江省机电设计研究院有限公司 一种高速公路车辆微观驾驶行为分析***与分析方法
CN112356839A (zh) * 2020-11-06 2021-02-12 广州小鹏自动驾驶科技有限公司 一种驾驶状态监测方法、***及汽车
CN112754498B (zh) * 2021-01-11 2023-05-26 一汽解放汽车有限公司 驾驶员的疲劳检测方法、装置、设备及存储介质
FR3119145B1 (fr) * 2021-01-22 2023-10-27 Renault Sas Procédé de détermination d’un niveau de distraction d’un conducteur de véhicule
CN115082978A (zh) * 2021-03-10 2022-09-20 佳能株式会社 面部姿态的检测装置、方法、图像处理***及存储介质
CN113436413A (zh) * 2021-05-25 2021-09-24 东风电驱动***有限公司 一种疲劳驾驶预警***及方法
CN113313019A (zh) * 2021-05-27 2021-08-27 展讯通信(天津)有限公司 一种分神驾驶检测方法、***及相关设备
CN113298041B (zh) * 2021-06-21 2024-07-02 黑芝麻智能科技(上海)有限公司 用于标定驾驶员分心参考方向的方法及***
CN113536967B (zh) * 2021-06-25 2023-05-30 武汉极目智能技术有限公司 基于驾驶员头部运动姿态与人眼开合度的驾驶员状态辨识方法、装置、电子设备
CN113558621B (zh) * 2021-07-23 2023-08-01 福州数据技术研究院有限公司 一种驾驶员疲劳检测与提醒的方法和***
CN113537115A (zh) * 2021-07-26 2021-10-22 东软睿驰汽车技术（沈阳）有限公司 驾驶员的驾驶状态获取方法、装置及电子设备
US11830259B2 (en) * 2021-08-24 2023-11-28 Nvidia Corporation Robust state estimation
CN113795069B (zh) * 2021-11-18 2022-02-11 深圳市奥新科技有限公司 一种隧道照明控制方法及隧道照明***
WO2023108364A1 (zh) * 2021-12-13 2023-06-22 华为技术有限公司 驾驶员状态检测方法、装置及存储介质
CN114179831B (zh) * 2021-12-29 2023-02-17 吉林大学 一种基于驾驶员分心判断的人机转向切换控制方法
CN114373280A (zh) * 2022-01-20 2022-04-19 一汽解放汽车有限公司 驾驶行为监测***及监测方法
CN115311819B (zh) * 2022-10-10 2023-01-20 南京智慧交通信息股份有限公司 一种智慧公交智能车载实时告警***及其方法
CN116311181B (zh) * 2023-03-21 2023-09-12 重庆利龙中宝智能技术有限公司 一种异常驾驶的快速检测方法及***
CN116469085B (zh) * 2023-03-30 2024-04-02 万联易达物流科技有限公司 一种风险驾驶行为的监控方法及***
CN116749988A (zh) * 2023-06-20 2023-09-15 中国第一汽车股份有限公司 一种驾驶员疲劳预警方法、装置、电子设备及存储介质

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102930693A (zh) * 2012-11-06 2013-02-13 深圳市艾大机器人有限公司 安全驾驶预警***及方法
CN102975721A (zh) * 2011-09-02 2013-03-20 沃尔沃汽车公司 用于改善车辆驾驶员的表现估计的***和方法
CN108022451A (zh) * 2017-12-06 2018-05-11 驾玉科技（上海）有限公司 一种基于云端的驾驶员状态预警上报及分发***

Family Cites Families (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE502005002674D1 (de) * 2005-08-02 2008-03-13 Delphi Tech Inc Verfahren zur Steuerung eines Fahrerassistenzsystems und dazugehörige Vorrichtung
JP4704174B2 (ja) * 2005-09-30 2011-06-15 富士フイルム株式会社 状態識別装置、プログラムおよび方法
JP2007265377A (ja) * 2006-03-01 2007-10-11 Toyota Central Res & Dev Lab Inc 運転者状態判定装置及び運転支援装置
JP2009012608A (ja) * 2007-07-04 2009-01-22 Denso Corp 車両用照明装置
JP6210287B2 (ja) * 2012-11-16 2017-10-11 豊田合成株式会社 運転者用警告装置
KR101386823B1 (ko) * 2013-10-29 2014-04-17 김재철 동작, 안면, 눈, 입모양 인지를 통한 2단계 졸음운전 방지 장치
CN104085396A (zh) * 2014-07-03 2014-10-08 上海纵目科技有限公司 一种全景车道偏离预警方法及***
JP6235425B2 (ja) * 2014-07-07 2017-11-22 株式会社デンソーアイティーラボラトリ 安全確認判定装置、及び運転支援装置
CN104599443B (zh) * 2015-01-12 2017-05-31 中设设计集团股份有限公司 一种基于信息融合的驾驶行为预警车载终端及其预警方法
CN105574487A (zh) 2015-11-26 2016-05-11 中国第一汽车股份有限公司 基于面部特征的驾驶人注意力状态检测方法
FR3048544B1 (fr) 2016-03-01 2021-04-02 Valeo Comfort & Driving Assistance Dispositif et methode de surveillance d'un conducteur d'un vehicule automobile
WO2017208529A1 (ja) * 2016-06-02 2017-12-07 オムロン株式会社 運転者状態推定装置、運転者状態推定システム、運転者状態推定方法、運転者状態推定プログラム、対象者状態推定装置、対象者状態推定方法、対象者状態推定プログラム、および記録媒体
JP2018133007A (ja) * 2017-02-16 2018-08-23 いすゞ自動車株式会社 警報装置
US10948911B2 (en) * 2017-10-31 2021-03-16 Denso International America, Inc. Co-pilot
CN108021875A (zh) * 2017-11-27 2018-05-11 上海灵至科技有限公司 一种车辆驾驶员个性化疲劳监测及预警方法
CN108437999B (zh) 2018-03-20 2020-07-28 中国计量大学 一种注意力辅助***

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102975721A (zh) * 2011-09-02 2013-03-20 沃尔沃汽车公司 用于改善车辆驾驶员的表现估计的***和方法
CN102930693A (zh) * 2012-11-06 2013-02-13 深圳市艾大机器人有限公司 安全驾驶预警***及方法
CN108022451A (zh) * 2017-12-06 2018-05-11 驾玉科技（上海）有限公司 一种基于云端的驾驶员状态预警上报及分发***

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114495069A (zh) * 2020-10-27 2022-05-13 中车株洲电力机车研究所有限公司 一种监测司机驾驶状态的方法及***
CN113642522A (zh) * 2021-09-01 2021-11-12 中国科学院自动化研究所 基于音视频的疲劳状态检测方法和装置
CN113642522B (zh) * 2021-09-01 2022-02-08 中国科学院自动化研究所 基于音视频的疲劳状态检测方法和装置

Also Published As

Publication number Publication date
KR20200131883A (ko) 2020-11-24
CN111079476A (zh) 2020-04-28
US20210004619A1 (en) 2021-01-07
JP2021516829A (ja) 2021-07-08
US11386679B2 (en) 2022-07-12
KR102469234B1 (ko) 2022-11-18
CN111079476B (zh) 2024-03-26
SG11202009420RA (en) 2020-10-29

Similar Documents

Publication Publication Date Title
WO2020078465A1 (zh) 驾驶状态分析方法和装置、驾驶员监控***、车辆
WO2020078464A1 (zh) 驾驶状态检测方法和装置、驾驶员监控***、车辆
KR102391279B1 (ko) 운전 상태 모니터링 방법 및 장치, 운전자 모니터링 시스템 및 차량
WO2019232972A1 (zh) 驾驶管理方法和***、车载智能***、电子设备、介质
TWI741512B (zh) 駕駛員注意力監測方法和裝置及電子設備
KR101386823B1 (ko) 동작, 안면, 눈, 입모양 인지를 통한 2단계 졸음운전 방지 장치
CN113056390A (zh) 情境驾驶员监控***
Hossain et al. IOT based real-time drowsy driving detection system for the prevention of road accidents
JPWO2019198179A1 (ja) 搭乗者状態判定装置、警告出力制御装置及び搭乗者状態判定方法
Gupta et al. Real time driver drowsiness detecion using transfer learning
Ujir et al. Real-time driver’s monitoring mobile application through head pose, drowsiness and angry detection
Amanullah et al. Accident prevention by eye-gaze tracking using imaging Constraints
Shostak et al. Using Internet of Things Technologies to Ensure Cargo Transportation Safety
Ashwini et al. Deep Learning Based Drowsiness Detection With Alert System Using Raspberry Pi Pico
P Mathai A New Proposal for Smartphone-Based Drowsiness Detection and Warning System for Automotive Drivers
Trinh Effective driver distraction warning system incorporating fast image recognition methods.
CN118155192A (zh) 疲劳驾驶检测方法、车辆、设备、存储介质及程序产品
TW202222619A (zh) 車內駕駛監測系統
CN114529887A (zh) 一种驾驶行为分析方法及装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19872570

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2020551931

Country of ref document: JP

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 20207029610

Country of ref document: KR

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19872570

Country of ref document: EP

Kind code of ref document: A1