CN115227247B - Fatigue driving detection method, system and storage medium based on multisource information fusion - Google Patents
Fatigue driving detection method, system and storage medium based on multisource information fusion Download PDFInfo
- Publication number
- CN115227247B CN115227247B CN202210862524.6A CN202210862524A CN115227247B CN 115227247 B CN115227247 B CN 115227247B CN 202210862524 A CN202210862524 A CN 202210862524A CN 115227247 B CN115227247 B CN 115227247B
- Authority
- CN
- China
- Prior art keywords
- fatigue
- driver
- physiological
- visual
- signal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 25
- 238000001514 detection method Methods 0.000 title claims abstract description 23
- 230000000007 visual effect Effects 0.000 claims abstract description 43
- 238000000034 method Methods 0.000 claims abstract description 16
- 208000003464 asthenopia Diseases 0.000 claims description 41
- 238000004364 calculation method Methods 0.000 claims description 35
- 238000013528 artificial neural network Methods 0.000 claims description 14
- 230000005611 electricity Effects 0.000 claims description 14
- 230000001815 facial effect Effects 0.000 claims description 14
- 238000012545 processing Methods 0.000 claims description 11
- 238000006243 chemical reaction Methods 0.000 claims description 8
- 238000000605 extraction Methods 0.000 claims description 8
- 210000003128 head Anatomy 0.000 claims description 8
- 238000010606 normalization Methods 0.000 claims description 7
- 210000000744 eyelid Anatomy 0.000 claims description 4
- 238000002372 labelling Methods 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 3
- 238000001914 filtration Methods 0.000 claims description 3
- 238000012544 monitoring process Methods 0.000 abstract description 10
- 206010039203 Road traffic accident Diseases 0.000 abstract description 6
- 230000006399 behavior Effects 0.000 abstract description 3
- 238000007500 overflow downdraw method Methods 0.000 abstract description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 238000009434 installation Methods 0.000 description 2
- 230000034994 death Effects 0.000 description 1
- 231100000517 death Toxicity 0.000 description 1
- 230000006866 deterioration Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 210000000887 face Anatomy 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/165—Evaluating the state of mind, e.g. depression, anxiety
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
- A61B5/0205—Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/18—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state for vehicle drivers or machine operators
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/725—Details of waveform analysis using specific filters therefor, e.g. Kalman or adaptive filters
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
- G08B21/02—Alarms for ensuring the safety of persons
- G08B21/06—Alarms for ensuring the safety of persons indicating a condition of sleep, e.g. anti-dozing alarms
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B2503/00—Evaluating a particular growth phase or type of persons or animals
- A61B2503/20—Workers
- A61B2503/22—Motor vehicles operators, e.g. drivers, pilots, captains
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
- A61B5/024—Detecting, measuring or recording pulse rate or heart rate
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/05—Detecting, measuring or recording for diagnosis by means of electric currents or magnetic fields; Measuring using microwaves or radio waves
- A61B5/053—Measuring electrical impedance or conductance of a portion of the body
- A61B5/0531—Measuring skin impedance
- A61B5/0533—Measuring galvanic skin response
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Surgery (AREA)
- Public Health (AREA)
- General Health & Medical Sciences (AREA)
- Animal Behavior & Ethology (AREA)
- Molecular Biology (AREA)
- Veterinary Medicine (AREA)
- Psychiatry (AREA)
- Biophysics (AREA)
- Pathology (AREA)
- Biomedical Technology (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Physiology (AREA)
- Artificial Intelligence (AREA)
- Signal Processing (AREA)
- Psychology (AREA)
- Cardiology (AREA)
- Developmental Disabilities (AREA)
- Social Psychology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Child & Adolescent Psychology (AREA)
- Hospice & Palliative Care (AREA)
- Educational Technology (AREA)
- Mathematical Physics (AREA)
- Fuzzy Systems (AREA)
- Evolutionary Computation (AREA)
- Pulmonology (AREA)
- Business, Economics & Management (AREA)
- Emergency Management (AREA)
- General Physics & Mathematics (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
Abstract
The invention relates to the technical field of safe driving, and discloses a fatigue driving detection method, a system and a storage medium based on multi-source information fusion, wherein the method combines visual signals and physiological signals of a driver, adopts the multi-source information fusion method to judge the fatigue state of the driver, and has higher monitoring precision; considering the problems that acquired data is distorted and cannot be used and the like possibly caused in a complex driving environment, when one signal source fails, the two signal sources can independently judge the fatigue state of a driver, and the method has good robustness; under the condition of realizing real-time fatigue monitoring of a driver, normal driving behaviors cannot be disturbed, the suggested continuous driving duration of the driver can be obtained, the driver is helped to reasonably plan the route arrangement, the practical value is higher, the robustness is high, and various traffic accidents caused by fatigue driving can be effectively reduced.
Description
Technical Field
The invention relates to the technical field of safe driving, in particular to a fatigue driving detection method, a system and a storage medium based on multi-source information fusion.
Background
Fatigue driving is a main cause of traffic accidents, the number of deaths caused by road traffic accidents is up to 120 ten thousand people each year worldwide, and about tens of thousands of people die from various traffic accidents caused by fatigue driving. In addition, rail traffic is also facing the threat of fatigue driving. Related researches show that fatigue driving of a train driver is also a main cause of train accidents. It can be seen that fatigue driving has seriously threatened the trip safety of people. Especially, the problems of urban traffic environment deterioration, long commute time and the like occur, and fatigue of a driver is more easily caused, so that the fatigue monitoring of the driver has a vital meaning for guaranteeing the trip safety of people.
At present, most fatigue driving monitoring systems are based on a single information source, fatigue driving is a complex physiological and psychological phenomenon, fatigue indexes of a single channel are difficult to accurately reflect fatigue conditions of drivers, and meanwhile, the extraction process of fatigue characteristics is easily influenced by complex external environments, such as strong illumination, bumpy road surfaces, driving habits of drivers and the like and individual differences. Therefore, the existing fatigue driving monitoring system has the problems of high false detection rate, low detection precision, poor robustness and the like, and has poor practicability.
Disclosure of Invention
The invention provides a fatigue driving detection method, a system and a storage medium based on multi-source information fusion, which are used for solving the problems in the prior art.
In order to achieve the above object, the present invention is realized by the following technical scheme:
in a first aspect, the present invention provides a fatigue driving detection method based on multi-source information fusion, including:
the method comprises the steps of collecting physiological signals and visual signals of a driver, and obtaining personal body information before the driver starts driving;
judging the validity of the physiological signal and judging the validity of the visual signal;
calculating physiological fatigue according to the physiological signal when the physiological signal is effective, and calculating visual fatigue according to the visual signal when the visual signal is effective;
calculating comprehensive fatigue according to the physiological fatigue and the visual fatigue, and determining a recommended driving duration according to the personal body information and the comprehensive fatigue;
if the physiological fatigue calculation fails and the visual fatigue calculation is successful, determining a recommended driving duration according to the visual fatigue and the personal body information;
if the visual fatigue calculation fails and the physiological fatigue calculation is successful, determining the recommended driving duration according to the physiological fatigue and the personal body information.
Optionally, the physiological signal includes a heart rate and skin electricity of the driver;
the determining the validity of the physiological signal includes:
and calculating the heart rate average value and the skin electricity average value of the driver in a preset time window, and judging that the physiological signal is effective if the difference between the heart rate average value and a preset heart rate threshold value is less than 30 percent and the difference between the skin electricity average value and the preset skin electricity threshold value is less than 30 percent.
Optionally, the visual signal includes facial image information of the driver;
the judging the validity of the visual signal comprises the following steps:
if the face image information is detected to have a face, judging that the signal is valid, otherwise, judging that the signal is invalid.
Optionally, the calculating the physiological fatigue according to the physiological signal includes:
carrying out Kalman filtering processing on the collected physiological signals of the driver;
in the calculation time window, the filtered data are arranged and converted into a physiological signal data sequence with fixed length in time sequence;
respectively carrying out feature extraction on the physiological signal data sequence on a time domain and a frequency domain, and carrying out normalization processing on an extraction result;
and inputting the normalized characteristics into a first circulating neural network containing a self-attention mechanism, and acquiring a first preset physiological fatigue degree output by the circulating neural network containing the self-attention mechanism.
Optionally, the visual signal is a facial image of a driver, and the calculating the visual fatigue according to the visual signal includes:
carrying out graying treatment on the collected facial image;
in a calculation time window, the processed gray level images are arranged and converted into an image sequence with fixed length in time sequence;
labeling the interested area of each image in the image sequence by adopting a face recognition model based on a cascade regression tree;
detecting the region of interest by adopting a preset face key point detector to locate key points to obtain key point coordinates, wherein the key points comprise eyes and a mouth;
matching the two-dimensional face key point coordinates with a preset standard 3D face model to obtain a conversion relation, and calculating the head gesture of the driver according to the conversion relation;
extracting the opening degree, eyelid closure degree and head posture pitch angle of a driver in each image, and carrying out normalization treatment;
inputting the normalized characteristics into a second preset circulating neural network containing a self-attention mechanism, and obtaining the visual fatigue degree of the second preset circulating neural network containing the self-attention mechanism.
Optionally, after the determining the recommended driving duration, the method further includes:
displaying the fatigue degree of the current driver and the suggested continuous driving duration in real time; and if the comprehensive fatigue degree of the driver exceeds a set threshold value, controlling an alarm to give an alarm.
In a second aspect, the present application provides a fatigue driving detection system based on multi-source information fusion, including:
the physiological signal acquisition device is used for acquiring physiological signals of a driver;
the visual signal acquisition device is used for acquiring visual signals of a driver;
data processing means for:
calculating physiological fatigue according to the physiological signal when the physiological signal is effective, and calculating visual fatigue according to the visual signal when the visual signal is effective;
calculating comprehensive fatigue according to the physiological fatigue and the visual fatigue, and determining a recommended driving duration according to the personal body information and the comprehensive fatigue;
if the physiological fatigue calculation fails and the visual fatigue calculation is successful, determining a recommended driving duration according to the visual fatigue and the personal body information;
if the visual fatigue calculation fails and the physiological fatigue calculation is successful, determining a recommended driving duration according to the physiological fatigue and the personal body information;
and the output device is used for displaying the fatigue degree and the recommended driving duration.
In a third aspect, the present application provides a computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method steps according to the first aspect.
The beneficial effects are that:
the fatigue driving detection method based on the multi-source information fusion combines the visual signals and the physiological signals of the driver, adopts the multi-source information fusion method to judge the fatigue state of the driver, and has higher monitoring precision; considering the problems that acquired data is distorted and cannot be used possibly caused in a complex driving environment, when one signal source fails, the two signal sources can independently judge the fatigue state of a driver, and the method has good robustness; under the condition of realizing real-time fatigue monitoring of a driver, normal driving behaviors cannot be disturbed, the suggested continuous driving duration of the driver can be obtained, the driver is helped to reasonably plan the route arrangement, the practical value is higher, the robustness is high, and various traffic accidents caused by fatigue driving can be effectively reduced.
Drawings
FIG. 1 is one of the flowcharts of a fatigue driving detection method based on multi-source information fusion according to the preferred embodiment of the present invention;
FIG. 2 is a second flowchart of a fatigue driving detection method based on multi-source information fusion according to the preferred embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a fatigue driving detection system based on multi-source information fusion according to a preferred embodiment of the present invention;
fig. 4 is a schematic device installation diagram of a fatigue driving detection system based on multi-source information fusion according to a preferred embodiment of the present invention.
Reference numerals:
1. a computer; 2. an alarm; 3. a display screen; 4. a camera; 5. an intelligent bracelet.
Detailed Description
The following description of the present invention will be made clearly and fully, and it is apparent that the embodiments described are only some, but not all, of the embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Unless defined otherwise, technical or scientific terms used herein should be given the ordinary meaning as understood by one of ordinary skill in the art to which this invention belongs. The terms "first," "second," and the like, as used herein, do not denote any order, quantity, or importance, but rather are used to distinguish one element from another. Likewise, the terms "a" or "an" and the like do not denote a limitation of quantity, but rather denote the presence of at least one. The terms "connected" or "connected," and the like, are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "upper", "lower", "left", "right", etc. are used merely to indicate a relative positional relationship, which changes accordingly when the absolute position of the object to be described changes.
Referring to fig. 1-2, the fatigue driving detection method based on multi-source information fusion provided by the present application includes:
the method comprises the steps of collecting physiological signals and visual signals of a driver, and obtaining personal body information before the driver starts driving;
judging the validity of the physiological signal and judging the validity of the visual signal;
under the condition that the physiological signal is effective, calculating the physiological fatigue according to the physiological signal, and under the condition that the visual signal is effective, calculating the visual fatigue according to the visual signal;
calculating comprehensive fatigue according to the physiological fatigue and the visual fatigue, and determining a recommended driving duration according to personal body information and the comprehensive fatigue;
if the physiological fatigue calculation fails and the visual fatigue calculation is successful, determining the recommended driving duration according to the visual fatigue and the personal body information;
if the visual fatigue calculation fails and the physiological fatigue calculation is successful, determining the recommended driving duration according to the physiological fatigue and the personal body information.
In this embodiment, the physiological signal may be a heart rate and skin electricity of the driver, and the visual signal may be facial image information of the driver, which is merely exemplary and not limiting. The physiological signal can be collected by a physiological signal collecting device, and the physiological signal collecting device can be an intelligent bracelet 5; the visual signal may be collected by an image collecting device, which may be a camera 4, disposed on the front surface of the driver to collect facial image information of the driver.
In an example, the comprehensive fatigue degree is calculated according to the physiological fatigue degree and the visual fatigue degree, and the comprehensive fatigue degree is calculated by carrying out multi-source information fusion according to the physiological fatigue degree and the visual fatigue degree and by a D-S evidence theory. This is by way of example only and is not limiting.
In this embodiment, the calculation failure refers to that the corresponding data is invalid, so the calculation fails.
The fatigue driving detection method based on the multi-source information fusion combines the visual signals and the physiological signals of the driver, adopts the multi-source information fusion method to judge the fatigue state of the driver, and has higher monitoring precision; considering the problems that acquired data is distorted and cannot be used and the like possibly caused in a complex driving environment, when one signal source fails, the two signal sources can independently judge the fatigue state of a driver, and the method has good robustness; under the condition of realizing real-time fatigue monitoring of a driver, normal driving behaviors cannot be disturbed, the suggested continuous driving duration of the driver can be obtained, the driver is helped to reasonably plan the route arrangement, the practical value is higher, the robustness is high, and various traffic accidents caused by fatigue driving can be effectively reduced.
It should be understood that, due to the complex driving environment, distortion of the acquired data easily occurs, for example, the smart band 5 cannot be worn correctly or physiological signals such as heart rate, skin electricity and the like caused by noise interference are low or high, and cannot be used, for example, under the conditions of strong variation illumination or possible shielding, the face cannot be detected, so that feature extraction and the like on facial images are performed, and the fatigue monitoring system has the problems of false alarm, incapacity of working normally and the like, so that the effectiveness of distinguishing the data can improve the accuracy of detection.
Optionally, the physiological signal includes a heart rate and skin electricity of the driver;
the determining the validity of the physiological signal includes:
and calculating the heart rate average value and the skin electricity average value of the driver in a preset time window, and judging that the physiological signal is effective if the difference between the heart rate average value and a preset heart rate threshold value is less than 30 percent and the difference between the skin electricity average value and the preset skin electricity threshold value is less than 30 percent.
In one example, the determination of the validity of the physiological signal specifically includes the following steps:
calculating the data average value in the time window, and judging that the signal is valid if the average value is not more than 30% of the range of normal fluctuation of the adult; otherwise, judging that the signal is invalid, and sending the calculated signal to the information analysis fusion module to avoid the next single fatigue degree calculation. If the average value of the heart rate is within the range of 60-100 times/minute, the average value of the heart rate is not more than 30% of the range of normal fluctuation of the adult, namely the average value of the heart rate is within the range of 48-112 times/minute, the signal is valid, and if the average value is out of the range, the data is invalid.
Optionally, the visual signal includes facial image information of the driver;
the judging the validity of the visual signal comprises the following steps:
if the face image information is detected to have a face, judging that the signal is valid, otherwise, judging that the signal is invalid.
In one example, faces in facial image information may be detected using a trained face recognition model based on a cascading regression tree. If the face exists in the image, judging that the signal is valid; otherwise, judging that the signal is invalid, and sending the calculated signal to the information analysis fusion module without further single fatigue degree calculation.
Optionally, the calculating the physiological fatigue according to the physiological signal includes:
carrying out Kalman filtering processing on the collected physiological signals of the driver;
in the calculation time window, the filtered data are arranged and converted into a physiological signal data sequence with fixed length in time sequence;
respectively carrying out feature extraction on the physiological signal data sequence on a time domain and a frequency domain, and carrying out normalization processing on an extraction result;
and inputting the normalized characteristics into a first circulating neural network containing a self-attention mechanism, and acquiring a first preset physiological fatigue degree output by the circulating neural network containing the self-attention mechanism.
Optionally, the calculating the visual fatigue according to the visual signal includes:
carrying out graying treatment on the collected facial image;
in a calculation time window, the processed gray level images are arranged and converted into an image sequence with fixed length in time sequence;
labeling the interested area of each image in the image sequence by adopting a face recognition model based on a cascade regression tree;
detecting the region of interest by adopting a preset face key point detector to locate key points to obtain key point coordinates, wherein the key points comprise eyes and a mouth;
matching the two-dimensional face key point coordinates with a preset standard 3D face model to obtain a conversion relation, and calculating the head gesture of the driver according to the conversion relation;
extracting the opening degree, eyelid closure degree and head posture pitch angle of a driver in each image, and carrying out normalization treatment;
inputting the normalized characteristics into a second preset circulating neural network containing a self-attention mechanism, and obtaining the visual fatigue degree of the second preset circulating neural network containing the self-attention mechanism.
Wherein after the determining the recommended driving duration, the method further comprises:
displaying the fatigue degree of the current driver and the suggested continuous driving duration in real time; and if the comprehensive fatigue degree of the driver exceeds a set threshold value, controlling an alarm to give an alarm. In this embodiment, the fatigue degree of the current driver includes physiological fatigue degree, visual fatigue degree or comprehensive fatigue degree, so that the driver can be alerted to have a rest in time through the prompt of the alarm 2.
Referring to fig. 3, the present application further provides a fatigue driving detection system based on multi-source information fusion, including:
the physiological signal acquisition device is used for acquiring physiological signals of a driver;
the visual signal acquisition device is used for acquiring visual signals of a driver;
the data processing device may specifically include:
a physiological signal processing module for calculating physiological fatigue according to the physiological signal when the physiological signal is valid;
the image information processing module is used for calculating visual fatigue according to the visual signal under the condition that the visual signal is effective;
the comprehensive information analysis module calculates comprehensive fatigue according to the physiological fatigue and the visual fatigue, and determines recommended driving duration according to the personal body information and the comprehensive fatigue;
if the physiological fatigue calculation fails and the visual fatigue calculation is successful, determining a recommended driving duration according to the visual fatigue and the personal body information;
if the visual fatigue calculation fails and the physiological fatigue calculation is successful, determining a recommended driving duration according to the physiological fatigue and the personal body information;
and the output device is used for displaying the fatigue degree and the recommended driving duration.
In this embodiment, the data processing device may be a computer 1, the output device may be a display 3, the physiological signal acquisition device may be an intelligent bracelet 5, and the visual signal acquisition device may be a camera 4.
In one example, the installation positions of the above devices are shown in fig. 4.
The fatigue driving detection system based on the multi-source information fusion can realize the embodiments of the fatigue driving detection method based on the multi-source information fusion, and can achieve the same beneficial effects, and the description is omitted here.
The present application also provides a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the above-mentioned method steps. The readable storage medium can implement the embodiments of the method described above and achieve the same advantageous effects, and will not be described here in detail.
The foregoing describes in detail preferred embodiments of the present invention. It should be understood that numerous modifications and variations can be made in accordance with the concepts of the invention by one of ordinary skill in the art without undue burden. Therefore, all technical solutions which can be obtained by logic analysis, reasoning or limited experiments based on the prior art by the person skilled in the art according to the inventive concept shall be within the scope of protection defined by the claims.
Claims (6)
1. The fatigue driving detection method based on multi-source information fusion is characterized by comprising the following steps of:
the method comprises the steps of collecting physiological signals and visual signals of a driver, and obtaining personal body information before the driver starts driving;
judging the validity of the physiological signal and judging the validity of the visual signal;
calculating physiological fatigue according to the physiological signal when the physiological signal is effective, and calculating visual fatigue according to the visual signal when the visual signal is effective;
calculating comprehensive fatigue according to the physiological fatigue and the visual fatigue, and determining a recommended driving duration according to the personal body information and the comprehensive fatigue;
if the physiological fatigue calculation fails and the visual fatigue calculation is successful, determining a recommended driving duration according to the visual fatigue and the personal body information;
if the visual fatigue calculation fails and the physiological fatigue calculation is successful, determining a recommended driving duration according to the physiological fatigue and the personal body information;
the physiological signal includes a heart rate and skin electricity of the driver;
the determining the validity of the physiological signal includes:
calculating the heart rate average value and the skin electricity average value of the driver in a preset time window, and judging that the physiological signal is effective if the difference between the heart rate average value and a preset heart rate threshold value is less than 30% and the difference between the skin electricity average value and the preset skin electricity threshold value is less than 30%;
the visual signal is a facial image of a driver, and the calculating the visual fatigue according to the visual signal comprises the following steps:
carrying out graying treatment on the collected facial image;
in a calculation time window, the processed gray level images are arranged and converted into an image sequence with fixed length in time sequence;
labeling the interested area of each image in the image sequence by adopting a face recognition model based on a cascade regression tree;
detecting the region of interest by adopting a preset face key point detector to locate key points to obtain key point coordinates, wherein the key points comprise eyes and a mouth;
matching the two-dimensional face key point coordinates with a preset standard 3D face model to obtain a conversion relation, and calculating the head gesture of the driver according to the conversion relation;
extracting the opening degree, eyelid closure degree and head posture pitch angle of a driver in each image, and carrying out normalization treatment;
inputting the normalized characteristics into a second preset circulating neural network containing a self-attention mechanism, and obtaining the visual fatigue degree of the second preset circulating neural network containing the self-attention mechanism.
2. The multi-source information fusion-based fatigue driving detection method according to claim 1, wherein the visual signal includes facial image information of a driver;
the judging the validity of the visual signal comprises the following steps:
if the face image information is detected to have the face, judging that the signal is valid, otherwise, judging that the signal is invalid.
3. The method for detecting fatigue driving based on multi-source information fusion according to claim 1, wherein the calculating physiological fatigue according to the physiological signal comprises:
carrying out Kalman filtering processing on the collected physiological signals of the driver;
in the calculation time window, the filtered data are arranged and converted into a physiological signal data sequence with fixed length in time sequence;
respectively carrying out feature extraction on the physiological signal data sequence on a time domain and a frequency domain, and carrying out normalization processing on an extraction result;
and inputting the normalized characteristics into a first circulating neural network containing a self-attention mechanism, and acquiring a first preset physiological fatigue degree output by the circulating neural network containing the self-attention mechanism.
4. The method for detecting fatigue driving based on multi-source information fusion according to claim 1, wherein after the determination of the recommended driving duration, the method further comprises:
displaying the fatigue degree of the current driver and the suggested continuous driving duration in real time; and if the comprehensive fatigue degree of the driver exceeds a set threshold value, controlling an alarm to give an alarm.
5. A fatigue driving detection system based on multisource information fusion is characterized by comprising:
the physiological signal acquisition device is used for acquiring physiological signals of a driver;
the visual signal acquisition device is used for acquiring visual signals of a driver;
data processing means for:
calculating physiological fatigue according to the physiological signal when the physiological signal is effective, and calculating visual fatigue according to the visual signal when the visual signal is effective;
calculating comprehensive fatigue according to the physiological fatigue and the visual fatigue, and determining a recommended driving duration according to personal body information and the comprehensive fatigue;
if the physiological fatigue calculation fails and the visual fatigue calculation is successful, determining a recommended driving duration according to the visual fatigue and the personal body information;
if the visual fatigue calculation fails and the physiological fatigue calculation is successful, determining a recommended driving duration according to the physiological fatigue and the personal body information;
the output device is used for displaying the fatigue degree and the suggested driving duration;
the visual signal is a facial image of a driver, and the calculating the visual fatigue according to the visual signal comprises the following steps:
carrying out graying treatment on the collected facial image;
in a calculation time window, the processed gray level images are arranged and converted into an image sequence with fixed length in time sequence;
labeling the interested area of each image in the image sequence by adopting a face recognition model based on a cascade regression tree;
detecting the region of interest by adopting a preset face key point detector to locate key points to obtain key point coordinates, wherein the key points comprise eyes and a mouth;
matching the two-dimensional face key point coordinates with a preset standard 3D face model to obtain a conversion relation, and calculating the head gesture of the driver according to the conversion relation;
extracting the opening degree, eyelid closure degree and head posture pitch angle of a driver in each image, and carrying out normalization treatment;
inputting the normalized characteristics into a second preset circulating neural network containing a self-attention mechanism, and obtaining the visual fatigue degree of the second preset circulating neural network containing the self-attention mechanism.
6. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, carries out the method steps according to any of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210862524.6A CN115227247B (en) | 2022-07-20 | 2022-07-20 | Fatigue driving detection method, system and storage medium based on multisource information fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210862524.6A CN115227247B (en) | 2022-07-20 | 2022-07-20 | Fatigue driving detection method, system and storage medium based on multisource information fusion |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115227247A CN115227247A (en) | 2022-10-25 |
| CN115227247B true CN115227247B (en) | 2023-12-26 |
Family
ID=83675748
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210862524.6A Active CN115227247B (en) | 2022-07-20 | 2022-07-20 | Fatigue driving detection method, system and storage medium based on multisource information fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115227247B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117935231B (en) * | 2024-03-20 | 2024-06-07 | 杭州臻稀生物科技有限公司 | Non-inductive fatigue driving monitoring and intervention method |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012113477A (en) * | 2010-11-24 | 2012-06-14 | Denso Corp | Driving fatigue degree determination device |
| WO2014020465A1 (en) * | 2012-08-01 | 2014-02-06 | Koninklijke Philips N.V. | Estimation of remaining safe driving time or distance |
| CN103714660A (en) * | 2013-12-26 | 2014-04-09 | 苏州清研微视电子科技有限公司 | System for achieving fatigue driving judgment on basis of image processing and fusion between heart rate characteristic and expression characteristic |
| CN104801549A (en) * | 2015-04-27 | 2015-07-29 | 燕山大学 | Cold rolling strip steel plate shape instrument signal distortion channel data processing method |
| CN110770107A (en) * | 2017-09-25 | 2020-02-07 | 宝马股份公司 | Method and device for assessing the fatigue of an occupant of a vehicle |
| CN111179552A (en) * | 2019-12-31 | 2020-05-19 | 苏州清研微视电子科技有限公司 | Driver state monitoring method and system based on multi-sensor fusion |
| CN111354161A (en) * | 2018-12-20 | 2020-06-30 | 奥迪股份公司 | Fatigue driving reminding method and device, computer equipment and storage medium |
| CN111583585A (en) * | 2020-05-26 | 2020-08-25 | 苏州智华汽车电子有限公司 | Information fusion fatigue driving early warning method, system, device and medium |
| CN112220480A (en) * | 2020-10-21 | 2021-01-15 | 合肥工业大学 | Driver state detection system and vehicle based on millimeter wave radar and camera fusion |
| CN112754498A (en) * | 2021-01-11 | 2021-05-07 | 一汽解放汽车有限公司 | Driver fatigue detection method, device, equipment and storage medium |
| CN114492656A (en) * | 2022-02-09 | 2022-05-13 | 河北科技大学 | Fatigue degree monitoring system based on computer vision and sensor |
| CN114735010A (en) * | 2022-05-17 | 2022-07-12 | 中南大学 | Intelligent vehicle driving control method and system based on emotion recognition and storage medium |
-
2022
- 2022-07-20 CN CN202210862524.6A patent/CN115227247B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012113477A (en) * | 2010-11-24 | 2012-06-14 | Denso Corp | Driving fatigue degree determination device |
| WO2014020465A1 (en) * | 2012-08-01 | 2014-02-06 | Koninklijke Philips N.V. | Estimation of remaining safe driving time or distance |
| CN103714660A (en) * | 2013-12-26 | 2014-04-09 | 苏州清研微视电子科技有限公司 | System for achieving fatigue driving judgment on basis of image processing and fusion between heart rate characteristic and expression characteristic |
| CN104801549A (en) * | 2015-04-27 | 2015-07-29 | 燕山大学 | Cold rolling strip steel plate shape instrument signal distortion channel data processing method |
| CN110770107A (en) * | 2017-09-25 | 2020-02-07 | 宝马股份公司 | Method and device for assessing the fatigue of an occupant of a vehicle |
| CN111354161A (en) * | 2018-12-20 | 2020-06-30 | 奥迪股份公司 | Fatigue driving reminding method and device, computer equipment and storage medium |
| CN111179552A (en) * | 2019-12-31 | 2020-05-19 | 苏州清研微视电子科技有限公司 | Driver state monitoring method and system based on multi-sensor fusion |
| CN111583585A (en) * | 2020-05-26 | 2020-08-25 | 苏州智华汽车电子有限公司 | Information fusion fatigue driving early warning method, system, device and medium |
| CN112220480A (en) * | 2020-10-21 | 2021-01-15 | 合肥工业大学 | Driver state detection system and vehicle based on millimeter wave radar and camera fusion |
| CN112754498A (en) * | 2021-01-11 | 2021-05-07 | 一汽解放汽车有限公司 | Driver fatigue detection method, device, equipment and storage medium |
| CN114492656A (en) * | 2022-02-09 | 2022-05-13 | 河北科技大学 | Fatigue degree monitoring system based on computer vision and sensor |
| CN114735010A (en) * | 2022-05-17 | 2022-07-12 | 中南大学 | Intelligent vehicle driving control method and system based on emotion recognition and storage medium |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117935231B (en) * | 2024-03-20 | 2024-06-07 | 杭州臻稀生物科技有限公司 | Non-inductive fatigue driving monitoring and intervention method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115227247A (en) | 2022-10-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104013414B (en) | A kind of Study in Driver Fatigue State Surveillance System based on intelligent movable mobile phone | |
| Rahman et al. | Real time drowsiness detection using eye blink monitoring | |
| Dong et al. | Driver inattention monitoring system for intelligent vehicles: A review | |
| Assari et al. | Driver drowsiness detection using face expression recognition | |
| CN101593425B (en) | Machine vision based fatigue driving monitoring method and system | |
| CN104224204B (en) | A kind of Study in Driver Fatigue State Surveillance System based on infrared detection technology | |
| Junaedi et al. | Driver drowsiness detection based on face feature and PERCLOS | |
| Lee et al. | Real-time physiological and vision monitoring of vehicle driver for non-intrusive drowsiness detection | |
| CN112241658A (en) | Fatigue driving early warning system and method based on depth camera | |
| CN103824420A (en) | Fatigue driving identification system based on heart rate variability non-contact measuring | |
| CN104318237A (en) | Fatigue driving warning method based on face identification | |
| CN115227247B (en) | Fatigue driving detection method, system and storage medium based on multisource information fusion | |
| Ahmed et al. | Robust driver fatigue recognition using image processing | |
| CN104123549A (en) | Eye positioning method for real-time monitoring of fatigue driving | |
| CN107563346A (en) | One kind realizes that driver fatigue sentences method for distinguishing based on eye image processing | |
| CN111179552A (en) | Driver state monitoring method and system based on multi-sensor fusion | |
| CN112464782A (en) | Pedestrian identification method and system | |
| Islam et al. | A study on tiredness assessment by using eye blink detection | |
| Tayibnapis et al. | A novel driver fatigue monitoring using optical imaging of face on safe driving system | |
| Alam et al. | Vision-based driver’s attention monitoring system for smart vehicles | |
| Alam et al. | Active vision-based attention monitoring system for non-distracted driving | |
| US20220036056A1 (en) | Image processing apparatus and method for recognizing state of subject | |
| Shilaskar et al. | Driver Safety System using Microcontroller and Image Processing | |
| CN203885510U (en) | Driver fatigue detection system based on infrared detection technology | |
| Diddi et al. | Head pose and eye state monitoring (HEM) for driver drowsiness detection: Overview |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |