CN110096957B - Fatigue driving monitoring method and system based on facial recognition and behavior recognition fusion - Google Patents
Fatigue driving monitoring method and system based on facial recognition and behavior recognition fusion Download PDFInfo
- Publication number
- CN110096957B CN110096957B CN201910236282.8A CN201910236282A CN110096957B CN 110096957 B CN110096957 B CN 110096957B CN 201910236282 A CN201910236282 A CN 201910236282A CN 110096957 B CN110096957 B CN 110096957B
- Authority
- CN
- China
- Prior art keywords
- state
- recognition
- face
- facial
- fatigue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/254—Fusion techniques of classification results, e.g. of results related to same input data
- G06F18/256—Fusion techniques of classification results, e.g. of results related to same input data of results relating to different input data, e.g. multimodal recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/172—Classification, e.g. identification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Human Computer Interaction (AREA)
- General Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Computational Biology (AREA)
- Social Psychology (AREA)
- Psychiatry (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a fatigue driving monitoring method based on fusion of facial recognition and behavior recognition, which comprises the following steps: detecting the position of a human body on the collected behavior image; performing skeleton detection on the detected positions of the human body to obtain the positions of all parts of the human body in the image and the corresponding confidence, predicting the association vector fields among all the parts, and representing the connection relation among all the parts through the association vector fields to obtain a human skeleton model; comparing the skeleton model of the predefined fatigue driving state with the obtained skeleton model to obtain a behavior state identification result; processing the collected facial images to obtain facial expression state characteristics, judging whether to drive fatigue according to the facial expression state characteristics, and obtaining a facial state recognition result; and fusing the face state recognition result and the behavior state recognition result to obtain a final detection result. The face recognition result and the behavior recognition result are dynamically fused, so that the fatigue driving state can be accurately judged, and the accuracy is higher.
Description
Technical Field
The invention relates to the technical field of fatigue driving detection, in particular to a fatigue driving monitoring method and system based on fusion of facial recognition and behavior recognition.
Background
There are many methods for detecting fatigue states of a driver, and methods based on physiological signals of a driver, operation behaviors of a driver, vehicle state information, physiological response characteristics of a driver, and the like are roughly classified according to the types of detection.
The accuracy of judging fatigue driving based on physiological signals (brain electrical signals, electrocardiosignals and the like) is higher, physiological signals are not very different for all healthy drivers, and the physiological signals have commonality, but the traditional physiological signal acquisition mode needs to adopt contact measurement, so that a lot of inconvenience and limitation are brought to the practical application of fatigue detection of the drivers.
The operation behavior of the driver is affected by personal habits, traveling speed, road environment, and operation skills, in addition to the fatigue state, and thus many disturbance factors to be considered affect the accuracy of determining the fatigue driving based on the operation behavior of the driver (such as steering wheel operation).
The fatigue state of the driver can be estimated by using the vehicle running state information such as the vehicle running track change and the lane departure, but the running state of the vehicle is also related to many environmental factors such as the vehicle characteristics and the road, and the driving experience and the driving habit of the driver are highly relevant, so that the disturbance factors to be considered in determining the fatigue driving based on the vehicle state information are also more.
The fatigue driving judging method based on the physiological response characteristics of the driver is to infer the fatigue state of the driver by utilizing the eye characteristics, mouth movement characteristics and the like of the driver, the information is considered to be important characteristics reflecting fatigue, and the blink amplitude, blink frequency, average closing time, yawning actions and the like can be directly used for detecting fatigue, but the robustness of judging the state of the driver through a single facial expression characteristic is not high enough due to certain differences among habits and characteristics of different drivers.
However, the detection means of a single detection index has limitations due to individual differences of drivers, mainly manifested as low accuracy, easy occurrence of deviation, and the like.
Therefore, how to introduce the multi-sensor information fusion technology into the driving fatigue detection technology to improve the accuracy and the real-time performance of the driver fatigue detection becomes a problem to be solved urgently.
Disclosure of Invention
In order to solve the technical problems, the invention provides a fatigue driving monitoring system and a fatigue driving monitoring method based on facial recognition and behavior recognition fusion, which dynamically fuses a facial recognition result and a behavior recognition result, can accurately judge the fatigue driving state and has higher precision.
The technical scheme adopted by the invention is as follows:
a fatigue driving monitoring method based on fusion of facial recognition and behavior recognition comprises the following steps:
s01: detecting the position of a human body by adopting a first depth convolution neural network on the acquired behavior image;
s02: performing skeleton detection on the detected human body position by adopting a second deep convolutional neural network to obtain the position of each part of the human body in the image and the corresponding confidence coefficient, simultaneously predicting the association vector field among the parts, representing the connection relation among the parts by the association vector field, and obtaining a human body skeleton model according to the obtained position of each part and the association vector field among the parts;
s03: comparing the skeleton model of the predefined fatigue driving state with the obtained skeleton model to obtain a behavior state identification result p 1 (x)；
S04: processing the collected facial images of the driver to obtain facial expression state characteristics of the driver, judging whether to fatigue driving according to the facial expression state characteristics, and obtaining a facial state recognition result p 2 (x)；
S05: result p of face status recognition 2 (x) Behavior state recognition result p 1 (x) Fusion is carried out to obtain a final detection result p (x), namely: p (x) =w 1 ·p 1 (x)+w 2 ·p 2 (x)；
Wherein w is 1 ,w 2 Weights for feature fusion.
In a preferred technical solution, the step S05 further includes dividing a fatigue grade, where the fatigue grade includes a stage I, light fatigue; stage II, moderate fatigue; stage III, severe fatigue; and when the fatigue driving is judged, adopting different alarm modes for alarming the fatigue of different grades.
In a preferred embodiment, w in the identification process 1 ,w 2 According to p 1 P 2 Values of (2)And (3) dynamically adjusting, wherein the corresponding relation formula is as follows: w (w) 1 ＝p 1 *p 1 ，w 2 ＝p 2 *p 2 。
In a preferred embodiment, the step of detecting the position of the human body in the step S01 includes:
1) Determining a plurality of candidate boxes in the image;
2) Obtaining a multidimensional feature vector diagram according to the first deep convolutional neural network;
3) Obtaining a plurality of possible candidate frames for each position of the image;
4) Judging whether the multidimensional features extracted from the candidate frames belong to a specific class or not by using a classifier;
5) And for candidate frames belonging to a certain class, adjusting the positions of the candidate frames by using a regressor to obtain a target position with the minimum loss function, namely a target frame Rect (x, y, width, height), wherein x, y are central point coordinates, width is width, and height is height.
In a preferred technical solution, the specific step S04 includes:
s41: processing the acquired face image of the driver by adopting a progressive calibration network to calibrate the face, wherein the progressive calibration network comprises three layers of networks, and a layer 1 network is used for calculating the face deflection angle and calibrating the face; the layer 2 network is used for further calculating the face deflection angle and further calibrating the face; the layer 3 network is used for calculating the face deflection angle, calibrating the face, and outputting face classification, deflection angle and boundary regression;
s42: positioning facial features, wherein the facial features comprise a mouth, eyes and a head, separating out eye images, inputting the eye images into a third depth convolutional neural network, classifying to obtain eye states, and judging whether the eye states are eye-closed states or not; separating the mouth region image, inputting the mouth region image into a fourth depth network, classifying to obtain a state of a mouth, and judging whether the state is a yawning state or not; analyzing to obtain the head movement state by counting the head area track, and judging whether the head is in a relatively static state or not;
s43: counting and judging the state characteristics of the mouth, eyes and head in a period of timeWhether the broken eyes generate a closed eye state, whether the mouth generates a yawning state and whether the head is in a relatively static state are cumulatively judged to obtain a face state recognition result p 2 (x)。
The invention also discloses a fatigue driving monitoring system based on the fusion of facial recognition and behavior recognition, which comprises:
the human body position detection module is used for detecting the position of a human body by adopting a first depth convolution neural network to the acquired behavior image;
the human skeleton detection module is used for carrying out skeleton detection on the detected human body position by adopting a second deep convolutional neural network to obtain the position of each part of the human body in the image and the corresponding confidence coefficient, and simultaneously predicting the association vector field among the parts, representing the connection relation among the parts by the association vector field, and obtaining a human skeleton model according to the obtained position of each part and the association vector field among the parts;
the behavior state recognition module is used for comparing the skeleton model of the predefined fatigue driving state with the obtained skeleton model to obtain a behavior state recognition result p 1 (x)；
Facial image processing and recognition module: processing the collected facial images of the driver to obtain facial expression state characteristics of the driver, judging whether to fatigue driving according to the facial expression state characteristics, and obtaining a facial state recognition result p 2 (x)；
And a fusion module: result p of face status recognition 2 (x) Behavior state recognition result p 1 (x) Fusion is carried out to obtain a final detection result p (x), namely: p (x) =w 1 ·p 1 (x)+w 2 ·p 2 (x)；
Wherein w is 1 ,w 2 Weights for feature fusion.
In the preferred technical scheme, the device also comprises a judging and alarming module which is used for dividing the fatigue grade, wherein the fatigue grade comprises an I stage and mild fatigue; stage II, moderate fatigue; stage III, severe fatigue; and when the fatigue driving is judged, adopting different alarm modes for alarming the fatigue of different grades.
In a preferred technical solution, the fusion module, w, during the identification process 1 ,w 2 According to p 1 P 2 The value of (2) is dynamically adjusted, and the corresponding relation formula is as follows: w (w) 1 ＝p 1 *p 1 ，w 2 ＝p 2 *p 2 。
In a preferred technical solution, the human body position detection step in the human body position detection module includes:
1) Determining a plurality of candidate boxes in the image;
2) Obtaining a multidimensional feature vector diagram according to the first deep convolutional neural network;
3) Obtaining a plurality of possible candidate frames for each position of the image;
4) Judging whether the multidimensional features extracted from the candidate frames belong to a specific class or not by using a classifier;
5) And for candidate frames belonging to a certain class, adjusting the positions of the candidate frames by using a regressor to obtain a target position with the minimum loss function, namely a target frame Rect (x, y, width, height), wherein x, y are central point coordinates, width is width, and height is height.
In a preferred technical solution, the specific processing steps of the facial image processing and identifying module include:
s41: processing the acquired face image of the driver by adopting a progressive calibration network to calibrate the face, wherein the progressive calibration network comprises three layers of networks, and a layer 1 network is used for calculating the face deflection angle and calibrating the face; the layer 2 network is used for further calculating the face deflection angle and further calibrating the face; the layer 3 network is used for calculating the face deflection angle, calibrating the face, and outputting face classification, deflection angle and boundary regression;
s42: positioning facial features, wherein the facial features comprise a mouth, eyes and a head, separating out eye images, inputting the eye images into a third depth convolutional neural network, classifying to obtain eye states, and judging whether the eye states are eye-closed states or not; separating the mouth region image, inputting the mouth region image into a fourth depth network, classifying to obtain a state of a mouth, and judging whether the state is a yawning state or not; analyzing to obtain the head movement state by counting the head area track, and judging whether the head is in a relatively static state or not;
s43: counting the state characteristics of the mouth, eyes and head in a period of time, judging whether the eyes generate the eye closing state, judging whether the mouth generates the yawning state and judging whether the head is in a relatively static state in an accumulated way, and obtaining a face state recognition result p 2 (x)。
Compared with the prior art, the invention has the beneficial effects that:
1. according to the invention, the face recognition result and the behavior recognition result are dynamically fused, so that the fatigue driving state can be accurately judged, and the grading early warning reminding can be performed in real time, so that the accuracy is higher.
2. The invention uses the human skeleton model state to represent the posture of the human body, can more accurately analyze and predict the behavior state of the driver, and monitors the behavior of the driver.
Drawings
The invention is further described below with reference to the accompanying drawings and examples:
FIG. 1 is a flow chart of a fatigue driving monitoring method based on fusion of facial recognition and behavior recognition;
FIG. 2 is a schematic diagram showing the detection effect of the human body target according to the present invention;
FIG. 3 is a schematic diagram of the detection effect of the human skeleton model of the present invention;
fig. 4 is a schematic diagram of the rotating face detection of the present invention.
Detailed Description
The objects, technical solutions and advantages of the present invention will become more apparent by the following detailed description of the present invention with reference to the accompanying drawings. It should be understood that the description is only illustrative and is not intended to limit the scope of the invention. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the present invention.
Examples
As shown in fig. 1, the fatigue driving monitoring method based on the fusion of facial recognition and behavior recognition of the invention comprises the following steps:
the invention adopts a double-camera acquisition device to acquire an image S101, mainly comprises a face camera 1, acquires the image aiming at the face of a driver, and inputs the image to a face analysis module. The face is captured by the camera and input to the algorithm module for recognition and processing. The camera 1 adopts an MIPI digital camera, and the image resolution is 1920×1080. Supporting automatic exposure, automatic focusing and automatic white balance. The image output per second is not less than 30 frames.
The behavior camera 2 is downwards from the front of the driver by about 45 degrees, is aimed at the body part of the driver, collects the behavior image of the driver and inputs the behavior image into the behavior analysis module. The camera 2 adopts an AHD analog camera, and the image resolution is 1280 x 720. Supporting automatic exposure, automatic focusing and automatic white balance. The image output per second is not less than 30 frames.
And the human body positioning detection module is used for determining the position S102 of the human body through the behavior image acquired by the behavior camera. Based on the deep learning technology, a lightweight convolutional neural network is designed, and an end-to-end single-stage training method is used. The feature images of different convolution layers are detected respectively, and finally, the non-maximum suppression method is used for integrating the results to detect the human body, so that the real-time performance of the algorithm is greatly improved compared with the method of using an image pyramid detection in the traditional algorithm. As shown in fig. 2, the human body is located, and the target position Rect (x, y, width, height) is returned. The end-to-end human body detection method based on deep learning comprises the following steps:
1) About 1000-2000 candidate boxes are determined in the image (using selective searching).
2) And inputting the whole picture into a convolutional neural network to obtain 128-dimensional feature vector diagrams related to human body gradients, edges and the like.
3) 256-channel images at scale 51 x 39, consider 9 possible candidate windows for each position of the image.
4) And judging whether the 128-dimensional features extracted from the candidate frames belong to a specific class or not by using a classifier.
5) And for a candidate frame belonging to a certain characteristic, adjusting the position of the candidate frame by using a regressive to obtain a target position Rect (x, y, width, height) with the minimum loss function, wherein x, y, a center point x, y coordinates, a width of the height object are the same as the target position Rect.
A human skeleton S103 is obtained. The human body framework state is obtained by carrying out framework detection on the human body framework, and the current action and behavior of the human body are represented, so that the current behavior action of the human body can be deduced as long as the state of the human body framework can be detected, and the driving behavior of a driver can be monitored through the human body framework detection model.
The human skeleton model utilizes a deep convolutional neural network to process an input image, outputs the positions of all parts of a human body in a picture and the corresponding confidence degrees, predicts the association vector fields among all the parts to represent the connection relation among all the parts, and finally utilizes a greedy algorithm to infer the output positions and the association vector fields among the parts to obtain the human skeleton model.
And obtaining a behavior state recognition result S104 through the classification result, as shown in fig. 3, comparing the obtained skeleton model with the skeleton model when the fatigue state of the driver is defined through training in advance, and calculating the distance between vectors through a support vector machine algorithm to obtain the similarity. Namely, the behavior recognition state result P1.
The face and the five sense organs are located through the image collected by the facial camera S105, and the face and the five sense organs can be located through the collected facial image, and the expression characteristic can be obtained by processing the face and the five sense organs by the method described in the patent document of 201310731567.1 application to judge the fatigue driving. But the present application proposes a new solution.
The present solution proposes to progressively calibrate the network detector. Given an image, all candidate faces are obtained according to the sliding window and image pyramid principles, and each candidate window steps through the detector. In each stage of the progressive calibration network, the detector simultaneously rejects most candidates with low face confidence, performs a boundary regression on the remaining face candidates, and calibrates the plane rotation direction of the face candidates. After each stage, non-maximal suppression (NMS) is used to merge those candidates that are highly overlapping.
Layer 1 network: firstly, a deflection angle is slightly judged, the face is calibrated, and the deflection angle range of the face is reduced.
Layer 2 network: in the same way, the face deflection angle range is further calibrated to be reduced.
Layer 3 network: and (3) accurately calculating the deflection angle, and directly outputting face classification, deflection angle and boundary regression by using a third layer network after calibration based on the first two steps. The face angle is calibrated progressively, the deflection angle of the face is reduced gradually, the face rotating at any angle can be processed by the method, and the result is shown in figure 4.
Modeling and classifying the mouth, eyes and head, S106, positioning the mouth, eyes and head parts, separating eye images, inputting the eye images into a CNN network, and classifying to obtain the eye state, and whether the eye state is a closed eye state or not. And separating the mouth region image, inputting the mouth region image into a CNN network, and classifying to obtain states of the mouth, such as states of yawning or the like. And (3) analyzing and obtaining the head movement state by counting the head area track, and judging whether the head of the driver is in a relatively static state or not.
The facial state recognition result S107 is obtained through the classification result, and the facial state recognition result P2 is obtained through statistics of the mouth and eye state characteristics in a period of time, namely, through accumulated judgment of whether the eyes generate the eye closing state, whether the mouth generates the yawning state and whether the head is in the relatively static state.
The fusion judgment strategy S108 fuses the facial recognition state and the behavior recognition state to obtain a final detection result, namely:
p(x)＝w 1 ·p 1 (x)+w 2 ·p 2 (x)
wherein w is 1 ,w 2 Weights fused to features; initial weight w 1 ＝0.5，w 2 =0.5, w in the identification process 1 ,w 2 According to p 1 P 2 The corresponding relation formula is as follows: w (w) 1 ＝p 1 *p 1 ，w 2 ＝p 2 *p 2 。
And after the fatigue driving is judged, the fatigue grade is divided, and different forms of alarm modes are adopted for the fatigue of different grades. The fatigue grade is divided into stage I and mild fatigue; stage II, moderate fatigue; stage III, severe fatigue. The behavior is shown in the following table. The alarm prompting sound in the stage I is gentle, the prompting sound in the stage II is rapid and high in volume, the prompting sound in the stage III is rapid and high in volume, and a vibration seat can be additionally arranged.
It is to be understood that the above-described embodiments of the present invention are merely illustrative of or explanation of the principles of the present invention and are in no way limiting of the invention. Accordingly, any modification, equivalent replacement, improvement, etc. made without departing from the spirit and scope of the present invention should be included in the scope of the present invention. Furthermore, the appended claims are intended to cover all such changes and modifications that fall within the scope and boundary of the appended claims, or equivalents of such scope and boundary.
Claims (8)
1. The fatigue driving monitoring method based on the fusion of facial recognition and behavior recognition is characterized by comprising the following steps of:
s01: detecting the position of a human body by adopting a first depth convolution neural network on the acquired behavior image;
s02: performing skeleton detection on the detected human body position by adopting a second deep convolutional neural network to obtain the position of each part of the human body in the image and the corresponding confidence coefficient, simultaneously predicting the association vector field among the parts, representing the connection relation among the parts by the association vector field, and obtaining a human body skeleton model according to the obtained position of each part and the association vector field among the parts;
s03: a skeleton model of a predefined fatigue driving state and the obtained skeletonComparing the models to obtain a behavior state recognition result p 1 (x)；
S04: processing the collected facial images of the driver to obtain facial expression state characteristics of the driver, judging whether to fatigue driving according to the facial expression state characteristics, and obtaining a facial state recognition result p 2 (x)；
S05: result p of face status recognition 2 (x) Behavior state recognition result p 1 (x) Fusion is carried out to obtain a final detection result p (x), namely: p (x) =w 1 ·p 1 (x)+w 2 ·p 2 (x)；
Wherein w is 1 ，w 2 Weights fused to features;
the step S05 further comprises the steps of classifying fatigue grades, wherein the fatigue grades comprise I phases and mild fatigue; stage II, moderate fatigue; stage III, severe fatigue; and when the fatigue driving is judged, adopting different alarm modes for alarming the fatigue of different grades.
2. The fatigue driving monitoring method based on the fusion of facial recognition and behavior recognition according to claim 1, wherein in the recognition process, w 1 ，w 2 According to p 1 P 2 The value of (2) is dynamically adjusted, and the corresponding relation formula is as follows: w (w) 1 ＝p 1 *p 1 ，w 2 ＝p 2 *p 2 。
3. The fatigue driving monitoring method based on the fusion of facial recognition and behavior recognition according to claim 1, wherein the human body position detection step in step S01 includes:
1) Determining a plurality of candidate boxes in the image;
2) Obtaining a multidimensional feature vector diagram according to the first deep convolutional neural network;
3) Obtaining a plurality of possible candidate frames for each position of the image;
4) Judging whether the multidimensional features extracted from the candidate frames belong to a specific class or not by using a classifier;
5) And for candidate frames belonging to a certain class, adjusting the positions of the candidate frames by using a regressor to obtain a target position with the minimum loss function, namely a target frame Rect (x, y, width, height), wherein x, y are central point coordinates, width is width, and height is height.
4. The fatigue driving monitoring method based on the fusion of facial recognition and behavior recognition according to claim 1, wherein the specific step of step S04 includes:
s41: processing the acquired face image of the driver by adopting a progressive calibration network to calibrate the face, wherein the progressive calibration network comprises three layers of networks, and a layer 1 network is used for calculating the face deflection angle and calibrating the face; the layer 2 network is used for further calculating the face deflection angle and further calibrating the face; the layer 3 network is used for calculating the face deflection angle, calibrating the face, and outputting face classification, deflection angle and boundary regression;
s42: positioning facial features, wherein the facial features comprise a mouth, eyes and a head, separating out eye images, inputting the eye images into a third depth convolutional neural network, classifying to obtain eye states, and judging whether the eye states are eye-closed states or not; separating the mouth region image, inputting the mouth region image into a fourth depth network, classifying to obtain a state of a mouth, and judging whether the state is a yawning state or not; analyzing to obtain the head movement state by counting the head area track, and judging whether the head is in a relatively static state or not;
s43: counting the state characteristics of the mouth, eyes and head in a period of time, judging whether the eyes generate the eye closing state, judging whether the mouth generates the yawning state and judging whether the head is in a relatively static state in an accumulated way, and obtaining a face state recognition result p 2 (x)。
5. A fatigue driving monitoring system based on fusion of facial recognition and behavior recognition, comprising:
the human body position detection module is used for detecting the position of a human body by adopting a first depth convolution neural network to the acquired behavior image;
the human skeleton detection module is used for carrying out skeleton detection on the detected human body position by adopting a second deep convolutional neural network to obtain the position of each part of the human body in the image and the corresponding confidence coefficient, and simultaneously predicting the association vector field among the parts, representing the connection relation among the parts by the association vector field, and obtaining a human skeleton model according to the obtained position of each part and the association vector field among the parts;
the behavior state recognition module is used for comparing the skeleton model of the predefined fatigue driving state with the obtained skeleton model to obtain a behavior state recognition result p 1 (x)；
Facial image processing and recognition module: processing the collected facial images of the driver to obtain facial expression state characteristics of the driver, judging whether to fatigue driving according to the facial expression state characteristics, and obtaining a facial state recognition result p 2 (x)；
And a fusion module: result p of face status recognition 2 (x) Behavior state recognition result p 1 (x) Fusion is carried out to obtain a final detection result p (x), namely: p (x) =w 1 ·p 1 (x)+w 2 ·p 2 (x)；
Wherein w is 1 ，w 2 Weights fused to features;
the fatigue grade detection device also comprises a judging and alarming module which is used for dividing the fatigue grade, wherein the fatigue grade comprises a stage I and mild fatigue; stage II, moderate fatigue; stage III, severe fatigue; and when the fatigue driving is judged, adopting different alarm modes for alarming the fatigue of different grades.
6. The fatigue driving monitoring system based on fusion of facial recognition and behavior recognition according to claim 5, wherein the fusion module, during recognition, w 1 ，w 2 According to p 1 P 2 The value of (2) is dynamically adjusted, and the corresponding relation formula is as follows: w (w) 1 ＝p 1 *p 1 ，w 2 ＝p 2 *p 2 。
7. The fatigue driving monitoring system based on fusion of facial recognition and behavior recognition according to claim 5, wherein the human body position detection step in the human body position detection module comprises:
1) Determining a plurality of candidate boxes in the image;
2) Obtaining a multidimensional feature vector diagram according to the first deep convolutional neural network;
3) Obtaining a plurality of possible candidate frames for each position of the image;
4) Judging whether the multidimensional features extracted from the candidate frames belong to a specific class or not by using a classifier;
5) And for candidate frames belonging to a certain class, adjusting the positions of the candidate frames by using a regressor to obtain a target position with the minimum loss function, namely a target frame Rect (x, y, width, height), wherein x, y are central point coordinates, width is width, and height is height.
8. The fatigue driving monitoring system based on the fusion of facial recognition and behavior recognition according to claim 5, wherein the specific processing steps of the facial image processing and recognition module include:
s41: processing the acquired face image of the driver by adopting a progressive calibration network to calibrate the face, wherein the progressive calibration network comprises three layers of networks, and a layer 1 network is used for calculating the face deflection angle and calibrating the face; the layer 2 network is used for further calculating the face deflection angle and further calibrating the face; the layer 3 network is used for calculating the face deflection angle, calibrating the face, and outputting face classification, deflection angle and boundary regression;
s42: positioning facial features, wherein the facial features comprise a mouth, eyes and a head, separating out eye images, inputting the eye images into a third depth convolutional neural network, classifying to obtain eye states, and judging whether the eye states are eye-closed states or not; separating the mouth region image, inputting the mouth region image into a fourth depth network, classifying to obtain a state of a mouth, and judging whether the state is a yawning state or not; analyzing to obtain the head movement state by counting the head area track, and judging whether the head is in a relatively static state or not;
s43: counting the state characteristics of the mouth, eyes and head in a period of time, judging whether the eyes generate the eye closing state, judging whether the mouth generates the yawning state and judging whether the head is in a relatively static state in an accumulated way, and obtaining a face state recognition result p 2 (x)。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910236282.8A CN110096957B (en) | 2019-03-27 | 2019-03-27 | Fatigue driving monitoring method and system based on facial recognition and behavior recognition fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910236282.8A CN110096957B (en) | 2019-03-27 | 2019-03-27 | Fatigue driving monitoring method and system based on facial recognition and behavior recognition fusion |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110096957A CN110096957A (en) | 2019-08-06 |
| CN110096957B true CN110096957B (en) | 2023-08-08 |
Family
ID=67443092
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910236282.8A Active CN110096957B (en) | 2019-03-27 | 2019-03-27 | Fatigue driving monitoring method and system based on facial recognition and behavior recognition fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110096957B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110852190B (en) * | 2019-10-23 | 2022-05-20 | 华中科技大学 | Driving behavior recognition method and system integrating target detection and gesture recognition |
| CN110781872A (en) * | 2019-12-31 | 2020-02-11 | 南斗六星***集成有限公司 | Driver fatigue grade recognition system with bimodal feature fusion |
| CN111695510A (en) * | 2020-06-12 | 2020-09-22 | 浙江工业大学 | Image-based fatigue detection method for computer operator |
| CN111882827A (en) * | 2020-07-27 | 2020-11-03 | 复旦大学 | Fatigue driving monitoring method, system and device and readable storage medium |
| CN111616718B (en) * | 2020-07-30 | 2020-11-10 | 苏州清研微视电子科技有限公司 | Method and system for detecting fatigue state of driver based on attitude characteristics |
| CN113524182B (en) * | 2021-07-13 | 2023-05-16 | 东北石油大学 | Device and method for intelligently adjusting distance between person and screen |
| CN113971216B (en) * | 2021-10-22 | 2023-02-03 | 北京百度网讯科技有限公司 | Data processing method and device, electronic equipment and memory |
| CN115641570B (en) * | 2022-12-26 | 2023-06-23 | 中国汽车技术研究中心有限公司 | Driving behavior determination method, driving behavior determination device, electronic equipment and storage medium |
| CN117152670A (en) * | 2023-10-31 | 2023-12-01 | 江西拓世智能科技股份有限公司 | Behavior recognition method and system based on artificial intelligence |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2011065952A1 (en) * | 2009-11-30 | 2011-06-03 | Hewlett-Packard Development Company, L.P. | Face recognition apparatus and methods |
| CN103714660A (en) * | 2013-12-26 | 2014-04-09 | 苏州清研微视电子科技有限公司 | System for achieving fatigue driving judgment on basis of image processing and fusion between heart rate characteristic and expression characteristic |
| CN106218405A (en) * | 2016-08-12 | 2016-12-14 | 深圳市元征科技股份有限公司 | Fatigue driving monitoring method and cloud server |
| WO2017193272A1 (en) * | 2016-05-10 | 2017-11-16 | 深圳市赛亿科技开发有限公司 | Vehicle-mounted fatigue pre-warning system based on human face recognition and pre-warning method |
| CN108038453A (en) * | 2017-12-15 | 2018-05-15 | 罗派智能控制技术（上海）有限公司 | A kind of driver's state-detection and identifying system based on RGBD |
| CN108089708A (en) * | 2017-12-22 | 2018-05-29 | 西安交通大学 | A kind of hanging gesture interaction method for improving gesture fatigue |
| CN108216252A (en) * | 2017-12-29 | 2018-06-29 | 中车工业研究院有限公司 | A kind of subway driver vehicle carried driving behavior analysis method, car-mounted terminal and system |
| CN108446678A (en) * | 2018-05-07 | 2018-08-24 | 同济大学 | A kind of dangerous driving behavior recognition methods based on skeleton character |
| CN109145868A (en) * | 2018-09-11 | 2019-01-04 | 广州杰赛科技股份有限公司 | A kind of Activity recognition method and apparatus assisting running training |
-
2019
- 2019-03-27 CN CN201910236282.8A patent/CN110096957B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2011065952A1 (en) * | 2009-11-30 | 2011-06-03 | Hewlett-Packard Development Company, L.P. | Face recognition apparatus and methods |
| CN103714660A (en) * | 2013-12-26 | 2014-04-09 | 苏州清研微视电子科技有限公司 | System for achieving fatigue driving judgment on basis of image processing and fusion between heart rate characteristic and expression characteristic |
| WO2017193272A1 (en) * | 2016-05-10 | 2017-11-16 | 深圳市赛亿科技开发有限公司 | Vehicle-mounted fatigue pre-warning system based on human face recognition and pre-warning method |
| CN106218405A (en) * | 2016-08-12 | 2016-12-14 | 深圳市元征科技股份有限公司 | Fatigue driving monitoring method and cloud server |
| CN108038453A (en) * | 2017-12-15 | 2018-05-15 | 罗派智能控制技术（上海）有限公司 | A kind of driver's state-detection and identifying system based on RGBD |
| CN108089708A (en) * | 2017-12-22 | 2018-05-29 | 西安交通大学 | A kind of hanging gesture interaction method for improving gesture fatigue |
| CN108216252A (en) * | 2017-12-29 | 2018-06-29 | 中车工业研究院有限公司 | A kind of subway driver vehicle carried driving behavior analysis method, car-mounted terminal and system |
| CN108446678A (en) * | 2018-05-07 | 2018-08-24 | 同济大学 | A kind of dangerous driving behavior recognition methods based on skeleton character |
| CN109145868A (en) * | 2018-09-11 | 2019-01-04 | 广州杰赛科技股份有限公司 | A kind of Activity recognition method and apparatus assisting running training |
Non-Patent Citations (1)
| Title |
|---|
| "疲劳驾驶检测方法研究";葛宇等;《科学技术创新》;20180505;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110096957A (en) | 2019-08-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110096957B (en) | Fatigue driving monitoring method and system based on facial recognition and behavior recognition fusion | |
| Saleh et al. | Driving behavior classification based on sensor data fusion using LSTM recurrent neural networks | |
| EP3223196B1 (en) | A method and a device for generating a confidence measure for an estimation derived from images captured by a camera mounted on a vehicle | |
| KR101607224B1 (en) | Dynamic object classification | |
| McCall et al. | Lane change intent analysis using robust operators and sparse bayesian learning | |
| JP6144656B2 (en) | System and method for warning a driver that visual recognition of a pedestrian may be difficult | |
| CN112700470A (en) | Target detection and track extraction method based on traffic video stream | |
| CN106687037A (en) | Device, method, and computer program for detecting momentary sleep | |
| JP2007122362A (en) | State estimation method using neural network and state estimation apparatus using neural network | |
| CN110516622A (en) | A kind of gender of occupant, age and emotional intelligence recognition methods and system | |
| CN111274862A (en) | Device and method for generating a label object of a surroundings of a vehicle | |
| CN112949345A (en) | Fatigue monitoring method and system, automobile data recorder and intelligent cabin | |
| Al Redhaei et al. | Realtime driver drowsiness detection using machine learning | |
| CN108520528A (en) | Based on the mobile vehicle tracking for improving differential threshold and displacement field match model | |
| CN105046285B (en) | A kind of abnormal behaviour discrimination method based on kinematic constraint | |
| CN116665155B (en) | Underground trackless man-vehicle getting-off early warning safety management system based on image processing technology | |
| CN117115752A (en) | Expressway video monitoring method and system | |
| US20230227044A1 (en) | Apparatus, method, and computer program for monitoring driver | |
| CN112926364A (en) | Head posture recognition method and system, automobile data recorder and intelligent cabin | |
| US20220301282A1 (en) | Method for Analyzing Image Information Using Assigned Scalar Values | |
| Brisimi et al. | Sensing and classifying roadway obstacles: The street bump anomaly detection and decision support system | |
| US20220284718A1 (en) | Driving analysis device and driving analysis method | |
| JP2020177557A (en) | Information processor, information processing method, and program | |
| WO2023108364A1 (en) | Method and apparatus for detecting driver state, and storage medium | |
| KR20230079606A (en) | Deep Learning-based Road Condition Monitoring System and Method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |