CN115227247A - Fatigue driving detection method and system based on multi-source information fusion and storage medium - Google Patents
Fatigue driving detection method and system based on multi-source information fusion and storage medium Download PDFInfo
- Publication number
- CN115227247A CN115227247A CN202210862524.6A CN202210862524A CN115227247A CN 115227247 A CN115227247 A CN 115227247A CN 202210862524 A CN202210862524 A CN 202210862524A CN 115227247 A CN115227247 A CN 115227247A
- Authority
- CN
- China
- Prior art keywords
- fatigue
- physiological
- driver
- visual
- signal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 27
- 238000001514 detection method Methods 0.000 title claims abstract description 25
- 230000000007 visual effect Effects 0.000 claims abstract description 41
- 238000000034 method Methods 0.000 claims abstract description 12
- 208000003464 asthenopia Diseases 0.000 claims description 39
- 238000004364 calculation method Methods 0.000 claims description 26
- 238000012545 processing Methods 0.000 claims description 17
- 238000013528 artificial neural network Methods 0.000 claims description 12
- 125000004122 cyclic group Chemical group 0.000 claims description 12
- 230000005611 electricity Effects 0.000 claims description 11
- 230000001815 facial effect Effects 0.000 claims description 8
- 238000006243 chemical reaction Methods 0.000 claims description 6
- 238000001914 filtration Methods 0.000 claims description 6
- 210000003128 head Anatomy 0.000 claims description 6
- 238000000605 extraction Methods 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 3
- 210000000744 eyelid Anatomy 0.000 claims description 3
- 238000010606 normalization Methods 0.000 claims description 3
- 238000012544 monitoring process Methods 0.000 abstract description 10
- 206010039203 Road traffic accident Diseases 0.000 abstract description 6
- 230000006399 behavior Effects 0.000 abstract description 3
- 238000007500 overflow downdraw method Methods 0.000 abstract description 3
- 230000009286 beneficial effect Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 238000009434 installation Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000034994 death Effects 0.000 description 1
- 231100000517 death Toxicity 0.000 description 1
- 230000006866 deterioration Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 210000000887 face Anatomy 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/165—Evaluating the state of mind, e.g. depression, anxiety
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
- A61B5/0205—Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/18—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state for vehicle drivers or machine operators
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/725—Details of waveform analysis using specific filters therefor, e.g. Kalman or adaptive filters
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
- G08B21/02—Alarms for ensuring the safety of persons
- G08B21/06—Alarms for ensuring the safety of persons indicating a condition of sleep, e.g. anti-dozing alarms
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B2503/00—Evaluating a particular growth phase or type of persons or animals
- A61B2503/20—Workers
- A61B2503/22—Motor vehicles operators, e.g. drivers, pilots, captains
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
- A61B5/024—Detecting, measuring or recording pulse rate or heart rate
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/05—Detecting, measuring or recording for diagnosis by means of electric currents or magnetic fields; Measuring using microwaves or radio waves
- A61B5/053—Measuring electrical impedance or conductance of a portion of the body
- A61B5/0531—Measuring skin impedance
- A61B5/0533—Measuring galvanic skin response
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Surgery (AREA)
- Public Health (AREA)
- General Health & Medical Sciences (AREA)
- Animal Behavior & Ethology (AREA)
- Molecular Biology (AREA)
- Veterinary Medicine (AREA)
- Psychiatry (AREA)
- Biophysics (AREA)
- Pathology (AREA)
- Biomedical Technology (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Physiology (AREA)
- Artificial Intelligence (AREA)
- Signal Processing (AREA)
- Psychology (AREA)
- Cardiology (AREA)
- Developmental Disabilities (AREA)
- Social Psychology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Child & Adolescent Psychology (AREA)
- Hospice & Palliative Care (AREA)
- Educational Technology (AREA)
- Mathematical Physics (AREA)
- Fuzzy Systems (AREA)
- Evolutionary Computation (AREA)
- Pulmonology (AREA)
- Business, Economics & Management (AREA)
- Emergency Management (AREA)
- General Physics & Mathematics (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
Abstract
The invention relates to the technical field of safe driving, and discloses a fatigue driving detection method, a system and a storage medium based on multi-source information fusion, wherein the method combines visual signals and physiological signals of a driver, adopts the multi-source information fusion method to judge the fatigue state of the driver, and has higher monitoring precision; the problems of data acquisition distortion, incapability of use and the like possibly caused in a complex driving environment are considered, when one signal source fails, the fatigue state of a driver can be independently judged by the two signal sources, and the method has good robustness; under the condition of realizing the real-time fatigue monitoring of the driver, normal driving behaviors can not be interfered, the driver can be helped to reasonably plan the travel arrangement when the driver continues to drive according to the suggestion, the practical value is higher, the robustness is high, the practicability is high, and various traffic accidents caused by fatigue driving can be effectively reduced.
Description
Technical Field
The invention relates to the technical field of safe driving, in particular to a fatigue driving detection method and system based on multi-source information fusion and a storage medium.
Background
Fatigue driving is a leading cause of traffic accidents, and the number of deaths caused by road traffic accidents is as high as 120 thousands of people every year in the world, wherein about tens of thousands of people die from various traffic accidents caused by fatigue driving. In addition, rail transit is also facing the threat of fatigue driving. Related research shows that fatigue driving of train drivers is also a main cause of train accidents. Therefore, fatigue driving has seriously threatened the safety of people in trip. Particularly, the problems of urban traffic environment deterioration, long commuting time and the like occur, and the fatigue of the driver is more easily caused, so that the fatigue monitoring of the driver has a vital significance for guaranteeing the safety of people going out.
Most of the existing fatigue driving monitoring systems are based on a single information source, fatigue driving is taken as a complex physiological and psychological phenomenon, fatigue indexes of a single channel are difficult to accurately reflect the fatigue condition of a driver, and meanwhile, the extraction process of fatigue characteristics is easily influenced by complex external environments such as strong illumination, bumpy road surfaces, driving habits of the driver and the like and individual differences. Therefore, the existing fatigue driving monitoring system has the problems of high false detection rate, low detection precision, poor robustness and the like, and is poor in practicability.
Disclosure of Invention
The invention provides a fatigue driving detection method, a system and a storage medium based on multi-source information fusion, which aim to solve the problems in the prior art.
In order to achieve the purpose, the invention is realized by the following technical scheme:
in a first aspect, the invention provides a fatigue driving detection method based on multi-source information fusion, which comprises the following steps:
acquiring physiological signals and visual signals of a driver, and acquiring personal body information of the driver before the driver starts driving;
judging the validity of the physiological signal and judging the validity of the visual signal;
under the condition that the physiological signal is effective, calculating physiological fatigue according to the physiological signal, and under the condition that the visual signal is effective, calculating visual fatigue according to the visual signal;
calculating comprehensive fatigue according to the physiological fatigue and the visual fatigue, and determining recommended driving duration according to the personal body information and the comprehensive fatigue;
if the physiological fatigue degree is calculated unsuccessfully and the visual fatigue degree is calculated successfully, determining a recommended driving time according to the visual fatigue degree and the personal body information;
and if the visual fatigue calculation fails and the physiological fatigue calculation succeeds, determining the recommended driving time according to the physiological fatigue and the personal body information.
Optionally, the physiological signal comprises a heart rate and a skin charge of the driver;
the judging the effectiveness of the physiological signal comprises the following steps:
and calculating the heart rate mean value and the skin electricity mean value of the driver in a preset time window, and judging that the physiological signal is effective if the difference value between the heart rate mean value and a preset heart rate threshold value is less than 30% and the difference value between the skin electricity mean value and the preset skin electricity threshold value is less than 30%.
Optionally, the visual signal comprises facial image information of the driver;
the judging the effectiveness of the visual signal comprises:
and if the face image information is detected to have the face, judging that the signal is valid, otherwise, judging that the signal is invalid.
Optionally, the calculating physiological fatigue according to the physiological signal includes:
performing Kalman filtering processing on the acquired physiological signals of the driver;
in a calculation time window, the data after filtering processing is arranged and converted into a physiological signal data sequence with fixed length according to the time sequence;
respectively extracting the characteristics of the physiological signal data sequence in a time domain and a frequency domain, and normalizing the extraction result;
and inputting the normalized features into a first cyclic neural network including a self-attention mechanism, and acquiring the physiological fatigue output by the first preset cyclic neural network including the self-attention mechanism.
Optionally, the visual signal is a facial image of a driver, and the calculating the visual fatigue degree according to the visual signal includes:
carrying out gray processing on the collected face image;
in a calculation time window, arranging and converting the processed gray level images into image sequences with fixed length according to a time sequence;
marking out an interested region of each image in the image sequence by adopting a face recognition model based on a cascade regression tree;
detecting the region of interest by adopting a preset human face key point detector to position key points to obtain key point coordinates, wherein the key points comprise eyes and a mouth;
matching the two-dimensional face key point coordinates with a preset standard 3D face model to obtain a conversion relation, and calculating the head posture of the driver according to the conversion relation;
extracting the opening degree of the mouth angle, the closing degree of the eyelid and the head posture pitch angle of the driver in each image, and performing normalization processing;
and inputting the normalized features into a second preset cyclic neural network containing a self-attention mechanism, and acquiring the visual fatigue of the second preset cyclic neural network containing the self-attention mechanism.
Optionally, after the determining the recommended driving duration, the method further comprises:
displaying the fatigue of the current driver and the duration of continuous driving recommendation in real time; and if the comprehensive fatigue degree of the driver exceeds a set threshold value, controlling an alarm to give an alarm.
In a second aspect, the present application provides a fatigue driving detection system based on multi-source information fusion, including:
the physiological signal acquisition device is used for acquiring a physiological signal of a driver;
the visual signal acquisition device is used for acquiring visual signals of a driver;
data processing apparatus for:
under the condition that the physiological signal is effective, calculating physiological fatigue according to the physiological signal, and under the condition that the visual signal is effective, calculating visual fatigue according to the visual signal;
calculating comprehensive fatigue according to the physiological fatigue and the visual fatigue, and determining recommended driving duration according to the personal body information and the comprehensive fatigue;
if the physiological fatigue degree is calculated unsuccessfully and the visual fatigue degree is calculated successfully, determining suggested driving time according to the visual fatigue degree and the personal body information;
if the visual fatigue calculation fails and the physiological fatigue calculation succeeds, determining a recommended driving duration according to the physiological fatigue and the personal body information;
and the output device is used for displaying the fatigue degree and the recommended driving time.
In a third aspect, the present application provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the method steps as set forth in the first aspect.
Has the beneficial effects that:
the fatigue driving detection method based on the multi-source information fusion provided by the invention combines the visual signal and the physiological signal of the driver, adopts the multi-source information fusion method to judge the fatigue state of the driver, and has higher monitoring precision; the problems of data acquisition distortion and incapability of use possibly caused in a complex driving environment are considered, when one signal source fails, the fatigue state of a driver can be independently judged by the two signal sources, and the robustness is good; under the condition of realizing the real-time fatigue monitoring of the driver, normal driving behaviors can not be interfered, the driver can be helped to reasonably plan the travel arrangement when the driver continues to drive according to the suggestion, the practical value is higher, the robustness is high, the practicability is high, and various traffic accidents caused by fatigue driving can be effectively reduced.
Drawings
FIG. 1 is a flowchart of a fatigue driving detection method based on multi-source information fusion according to a preferred embodiment of the present invention;
FIG. 2 is a second flowchart of a fatigue driving detection method based on multi-source information fusion according to the preferred embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a fatigue driving detection system based on multi-source information fusion according to a preferred embodiment of the present invention;
fig. 4 is a schematic device installation diagram of a fatigue driving detection system based on multi-source information fusion according to a preferred embodiment of the present invention.
Reference numerals:
1. a computer; 2. an alarm; 3. a display screen; 4. a camera; 5. intelligent bracelet.
Detailed Description
The technical solutions of the present invention are described clearly and completely below, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Unless defined otherwise, technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention belongs. The use of "first," "second," and similar terms in the present application do not denote any order, quantity, or importance, but rather the terms are used to distinguish one element from another. Also, the use of the terms "a" or "an" and the like do not denote a limitation of quantity, but rather denote the presence of at least one. The terms "connected" or "coupled" and the like are not restricted to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "upper", "lower", "left", "right", and the like are used merely to indicate relative positional relationships, and when the absolute position of the object being described is changed, the relative positional relationships are changed accordingly.
Referring to fig. 1-2, the fatigue driving detection method based on multi-source information fusion provided by the present application includes:
acquiring physiological signals and visual signals of a driver, and acquiring personal body information of the driver before the driver starts driving;
judging the validity of the physiological signal and judging the validity of the visual signal;
under the condition that the physiological signal is effective, calculating the physiological fatigue according to the physiological signal, and under the condition that the visual signal is effective, calculating the visual fatigue according to the visual signal;
calculating comprehensive fatigue according to the physiological fatigue and the visual fatigue, and determining recommended driving duration according to the personal physical information and the comprehensive fatigue;
if the physiological fatigue calculation fails and the visual fatigue calculation succeeds, determining the recommended driving duration according to the visual fatigue and the personal body information;
and if the calculation of the visual fatigue fails and the calculation of the physiological fatigue succeeds, determining the recommended driving duration according to the physiological fatigue and the personal body information.
In this embodiment, the physiological signal may be a heart rate and a skin current of the driver, and the visual signal may be facial image information of the driver, which is only an example and is not limited herein. The physiological signal can be collected by a physiological signal collecting device, and the physiological signal collecting device can be an intelligent bracelet 5; the visual signal can be collected by an image collecting device which can be a camera 4 and is arranged on the front of the driver so as to collect the facial image information of the driver.
In an example, the calculating of the comprehensive fatigue degree according to the physiological fatigue degree and the visual fatigue degree may be that according to the physiological fatigue degree and the visual fatigue degree, multi-source information fusion is performed by a D-S evidence theory, and the comprehensive fatigue degree is calculated. This is by way of example only and not by way of limitation.
In this embodiment, the calculation failure means that the corresponding data is invalid, so the calculation fails.
The fatigue driving detection method based on the multi-source information fusion combines the visual signal and the physiological signal of the driver, adopts the multi-source information fusion method to judge the fatigue state of the driver, and has higher monitoring precision; the problems of data acquisition distortion, incapability of use and the like possibly caused in a complex driving environment are considered, when one signal source fails, the fatigue state of a driver can be independently judged by the two signal sources, and the method has good robustness; under the condition of realizing the real-time fatigue monitoring of the driver, normal driving behaviors can not be interfered, the driver can be helped to reasonably plan the travel arrangement when the driver continues to drive according to the suggestion, the practical value is higher, the robustness is high, the practicability is high, and various traffic accidents caused by fatigue driving can be effectively reduced.
It should be understood that, due to the complex driving environment, the acquired data is easy to be distorted, for example, the smart band 5 is not worn correctly or is interfered by noise, the physiological signals such as heart rate, skin electricity and the like are low or high and cannot be used, for example, under the strong change of illumination or possible shielding, the face cannot be detected, so that the face image is subjected to feature extraction and the like, and the fatigue monitoring system has the problems of false alarm, missing report, abnormal work and the like, so that the validity of the judgment data can improve the detection accuracy.
Optionally, the physiological signal comprises a heart rate and a skin charge of the driver;
the judging the effectiveness of the physiological signal comprises the following steps:
and calculating the heart rate mean value and the skin electricity mean value of the driver in a preset time window, and judging that the physiological signal is effective if the difference value between the heart rate mean value and a preset heart rate threshold value is less than 30% and the difference value between the skin electricity mean value and the preset skin electricity threshold value is less than 30%.
In one example, the validity determination of the physiological signal specifically includes the following steps:
the mean value of the data within a set time window is calculated, if the average value is not more than 30% of the normal fluctuation range of the adult, the judgment signal is effective; otherwise, the judgment signal is invalid, the sending of the calculation signal fails to the information analysis and fusion module, and the next single fatigue degree calculation is not carried out. If the normal adult is in the range of 60-100 times/minute, the average value is not more than 30% of the extreme difference of the normal fluctuation range of the adult, namely the heart rate average value is in the range of 48-112 times/minute, the signal is judged to be valid, and if the average value is out of the range, the data is judged to be invalid.
Optionally, the visual signal comprises facial image information of the driver;
the judging the effectiveness of the visual signal comprises:
and if the face image information is detected to have the face, judging that the signal is valid, otherwise, judging that the signal is invalid.
In an example, a trained cascading regression tree based face recognition model may be used to detect faces in facial image information. If the human face is detected to exist in the image, judging that the signal is effective; otherwise, the signal is judged to be invalid, the calculation signal is failed to be sent to the information analysis and fusion module, and the next single fatigue degree calculation is not carried out.
Optionally, the calculating physiological fatigue according to the physiological signal includes:
performing Kalman filtering processing on the acquired physiological signals of the driver;
in a calculation time window, the data after filtering processing is arranged and converted into a physiological signal data sequence with fixed length according to the time sequence;
respectively extracting the characteristics of the physiological signal data sequence in a time domain and a frequency domain, and normalizing the extraction result;
and inputting the normalized features into a first cyclic neural network including a self-attention mechanism, and acquiring the physiological fatigue output by the first preset cyclic neural network including the self-attention mechanism.
Optionally, the calculating the visual fatigue according to the visual signal includes:
carrying out gray processing on the collected face image;
in a calculation time window, arranging and converting the processed gray level images into image sequences with fixed lengths according to a time sequence;
marking out the interested region of each image in the image sequence by adopting a face recognition model based on a cascade regression tree;
detecting the region of interest by adopting a preset human face key point detector to position key points to obtain key point coordinates, wherein the key points comprise eyes and a mouth;
matching the two-dimensional face key point coordinates with a preset standard 3D face model to obtain a conversion relation, and calculating the head posture of the driver according to the conversion relation;
extracting the opening degree of the mouth angle, the closing degree of the eyelid and the head posture pitch angle of the driver in each image, and performing normalization processing;
and inputting the normalized features into a second preset cyclic neural network containing a self-attention mechanism, and acquiring the visual fatigue of the second preset cyclic neural network containing the self-attention mechanism.
Wherein after the determining the recommended driving duration, the method further comprises:
displaying the fatigue degree of the current driver and the recommended duration of continuous driving in real time; and if the comprehensive fatigue of the driver exceeds a set threshold, controlling an alarm to give an alarm. In this embodiment, the fatigue degree of the current driver includes physiological fatigue degree, visual fatigue degree or comprehensive fatigue degree, and thus, the driver can be alerted to have a rest in time through the prompt of the alarm 2.
Referring to fig. 3, the present application further provides a fatigue driving detection system based on multi-source information fusion, including:
the physiological signal acquisition device is used for acquiring a physiological signal of a driver;
the visual signal acquisition device is used for acquiring visual signals of a driver;
the data processing apparatus may specifically include:
the physiological signal processing module is used for calculating physiological fatigue according to the physiological signal under the condition that the physiological signal is effective;
the image information processing module is used for calculating the visual fatigue degree according to the visual signal under the condition that the visual signal is effective;
the comprehensive information analysis module is used for calculating comprehensive fatigue according to the physiological fatigue and the visual fatigue and determining the recommended driving time according to the personal body information and the comprehensive fatigue;
if the physiological fatigue degree is calculated unsuccessfully and the visual fatigue degree is calculated successfully, determining a recommended driving time according to the visual fatigue degree and the personal body information;
if the visual fatigue calculation fails and the physiological fatigue calculation succeeds, determining a recommended driving duration according to the physiological fatigue and the personal body information;
and the output device is used for displaying the fatigue degree and the recommended driving time.
In this embodiment, the data processing device may be a computer 1, the output device may be a display screen 3, the physiological signal collecting device may be an intelligent bracelet 5, and the visual signal collecting device may be a camera 4.
In one example, the installation positions of the above devices are shown in fig. 4.
The fatigue driving detection system based on the multi-source information fusion can realize each embodiment of the fatigue driving detection method based on the multi-source information fusion, and can achieve the same beneficial effects, and the detailed description is omitted here.
The present application also provides a computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the above-mentioned method steps. The readable storage medium can implement the embodiments of the method described above, and can achieve the same beneficial effects, which are not described herein again.
The foregoing detailed description of the preferred embodiments of the invention has been presented. It should be understood that numerous modifications and variations could be devised by those skilled in the art in light of the present teachings without departing from the inventive concepts. Therefore, the technical solutions that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the prior art according to the concepts of the present invention should be within the scope of protection determined by the claims.
Claims (8)
1. A fatigue driving detection method based on multi-source information fusion is characterized by comprising the following steps:
acquiring physiological signals and visual signals of a driver, and acquiring personal body information of the driver before the driver starts driving;
judging the validity of the physiological signal and judging the validity of the visual signal;
under the condition that the physiological signal is effective, calculating physiological fatigue according to the physiological signal, and under the condition that the visual signal is effective, calculating visual fatigue according to the visual signal;
calculating comprehensive fatigue according to the physiological fatigue and the visual fatigue, and determining recommended driving duration according to the personal body information and the comprehensive fatigue;
if the physiological fatigue degree is calculated unsuccessfully and the visual fatigue degree is calculated successfully, determining a recommended driving time according to the visual fatigue degree and the personal body information;
and if the visual fatigue calculation fails and the physiological fatigue calculation succeeds, determining the recommended driving time according to the physiological fatigue and the personal body information.
2. The multi-source information fusion-based fatigue driving detection method according to claim 1, wherein the physiological signals comprise heart rate and skin electricity of a driver;
the judging the effectiveness of the physiological signal comprises the following steps:
and calculating the heart rate mean value and the skin electricity mean value of the driver in a preset time window, and judging that the physiological signal is effective if the difference value between the heart rate mean value and a preset heart rate threshold value is less than 30% and the difference value between the skin electricity mean value and the preset skin electricity threshold value is less than 30%.
3. The multi-source information fusion-based fatigue driving detection method according to claim 1, wherein the visual signal includes facial image information of a driver;
the judging the effectiveness of the visual signal comprises:
and if the face image information is detected to have the face, judging that the signal is valid, otherwise, judging that the signal is invalid.
4. The method for detecting fatigue driving based on multi-source information fusion according to claim 1, wherein the calculating physiological fatigue degree according to the physiological signal comprises:
performing Kalman filtering processing on the acquired physiological signals of the driver;
in a calculation time window, the data after filtering processing is arranged and converted into a physiological signal data sequence with fixed length according to the time sequence;
respectively extracting the characteristics of the physiological signal data sequence in a time domain and a frequency domain, and normalizing the extraction result;
and inputting the normalized features into a first cyclic neural network including a self-attention mechanism, and acquiring the physiological fatigue output by the first preset cyclic neural network including the self-attention mechanism.
5. The multi-source information fusion-based fatigue driving detection method according to claim 1, wherein the visual signal is a facial image of a driver, and the calculating the visual fatigue degree according to the visual signal comprises:
carrying out gray processing on the collected face image;
in a calculation time window, arranging and converting the processed gray level images into image sequences with fixed lengths according to a time sequence;
marking out the interested region of each image in the image sequence by adopting a face recognition model based on a cascade regression tree;
detecting the region of interest by adopting a preset human face key point detector to position key points to obtain key point coordinates, wherein the key points comprise eyes and a mouth;
matching the two-dimensional face key point coordinates with a preset standard 3D face model to obtain a conversion relation, and calculating the head posture of the driver according to the conversion relation;
extracting the opening degree of the mouth angle, the closing degree of the eyelid and the head posture pitch angle of the driver in each image, and performing normalization processing;
and inputting the normalized features into a second preset cyclic neural network containing a self-attention mechanism, and acquiring the visual fatigue of the second preset cyclic neural network containing the self-attention mechanism.
6. The method for detecting fatigue driving based on multi-source information fusion according to claim 1, wherein after determining the recommended driving duration, the method further comprises:
displaying the fatigue degree of the current driver and the recommended duration of continuous driving in real time; and if the comprehensive fatigue of the driver exceeds a set threshold, controlling an alarm to give an alarm.
7. A fatigue driving detection system based on multi-source information fusion is characterized by comprising:
the physiological signal acquisition device is used for acquiring a physiological signal of a driver;
the visual signal acquisition device is used for acquiring visual signals of a driver;
data processing apparatus for:
under the condition that the physiological signal is effective, calculating physiological fatigue according to the physiological signal, and under the condition that the visual signal is effective, calculating visual fatigue according to the visual signal;
calculating comprehensive fatigue according to the physiological fatigue and the visual fatigue, and determining recommended driving duration according to the personal body information and the comprehensive fatigue;
if the physiological fatigue degree is calculated unsuccessfully and the visual fatigue degree is calculated successfully, determining suggested driving time according to the visual fatigue degree and the personal body information;
if the visual fatigue calculation fails and the physiological fatigue calculation succeeds, determining a recommended driving duration according to the physiological fatigue and the personal body information;
and the output device is used for displaying the fatigue degree and the recommended driving time.
8. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method steps of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210862524.6A CN115227247B (en) | 2022-07-20 | 2022-07-20 | Fatigue driving detection method, system and storage medium based on multisource information fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210862524.6A CN115227247B (en) | 2022-07-20 | 2022-07-20 | Fatigue driving detection method, system and storage medium based on multisource information fusion |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115227247A true CN115227247A (en) | 2022-10-25 |
| CN115227247B CN115227247B (en) | 2023-12-26 |
Family
ID=83675748
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210862524.6A Active CN115227247B (en) | 2022-07-20 | 2022-07-20 | Fatigue driving detection method, system and storage medium based on multisource information fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115227247B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117935231A (en) * | 2024-03-20 | 2024-04-26 | 杭州臻稀生物科技有限公司 | Non-inductive fatigue driving monitoring and intervention method |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012113477A (en) * | 2010-11-24 | 2012-06-14 | Denso Corp | Driving fatigue degree determination device |
| WO2014020465A1 (en) * | 2012-08-01 | 2014-02-06 | Koninklijke Philips N.V. | Estimation of remaining safe driving time or distance |
| CN103714660A (en) * | 2013-12-26 | 2014-04-09 | 苏州清研微视电子科技有限公司 | System for achieving fatigue driving judgment on basis of image processing and fusion between heart rate characteristic and expression characteristic |
| CN104801549A (en) * | 2015-04-27 | 2015-07-29 | 燕山大学 | Cold rolling strip steel plate shape instrument signal distortion channel data processing method |
| CN110770107A (en) * | 2017-09-25 | 2020-02-07 | 宝马股份公司 | Method and device for assessing the fatigue of an occupant of a vehicle |
| CN111179552A (en) * | 2019-12-31 | 2020-05-19 | 苏州清研微视电子科技有限公司 | Driver state monitoring method and system based on multi-sensor fusion |
| CN111354161A (en) * | 2018-12-20 | 2020-06-30 | 奥迪股份公司 | Fatigue driving reminding method and device, computer equipment and storage medium |
| CN111583585A (en) * | 2020-05-26 | 2020-08-25 | 苏州智华汽车电子有限公司 | Information fusion fatigue driving early warning method, system, device and medium |
| CN112220480A (en) * | 2020-10-21 | 2021-01-15 | 合肥工业大学 | Driver state detection system and vehicle based on millimeter wave radar and camera fusion |
| CN112754498A (en) * | 2021-01-11 | 2021-05-07 | 一汽解放汽车有限公司 | Driver fatigue detection method, device, equipment and storage medium |
| CN114492656A (en) * | 2022-02-09 | 2022-05-13 | 河北科技大学 | Fatigue degree monitoring system based on computer vision and sensor |
| CN114735010A (en) * | 2022-05-17 | 2022-07-12 | 中南大学 | Intelligent vehicle driving control method and system based on emotion recognition and storage medium |
-
2022
- 2022-07-20 CN CN202210862524.6A patent/CN115227247B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012113477A (en) * | 2010-11-24 | 2012-06-14 | Denso Corp | Driving fatigue degree determination device |
| WO2014020465A1 (en) * | 2012-08-01 | 2014-02-06 | Koninklijke Philips N.V. | Estimation of remaining safe driving time or distance |
| CN103714660A (en) * | 2013-12-26 | 2014-04-09 | 苏州清研微视电子科技有限公司 | System for achieving fatigue driving judgment on basis of image processing and fusion between heart rate characteristic and expression characteristic |
| CN104801549A (en) * | 2015-04-27 | 2015-07-29 | 燕山大学 | Cold rolling strip steel plate shape instrument signal distortion channel data processing method |
| CN110770107A (en) * | 2017-09-25 | 2020-02-07 | 宝马股份公司 | Method and device for assessing the fatigue of an occupant of a vehicle |
| CN111354161A (en) * | 2018-12-20 | 2020-06-30 | 奥迪股份公司 | Fatigue driving reminding method and device, computer equipment and storage medium |
| CN111179552A (en) * | 2019-12-31 | 2020-05-19 | 苏州清研微视电子科技有限公司 | Driver state monitoring method and system based on multi-sensor fusion |
| CN111583585A (en) * | 2020-05-26 | 2020-08-25 | 苏州智华汽车电子有限公司 | Information fusion fatigue driving early warning method, system, device and medium |
| CN112220480A (en) * | 2020-10-21 | 2021-01-15 | 合肥工业大学 | Driver state detection system and vehicle based on millimeter wave radar and camera fusion |
| CN112754498A (en) * | 2021-01-11 | 2021-05-07 | 一汽解放汽车有限公司 | Driver fatigue detection method, device, equipment and storage medium |
| CN114492656A (en) * | 2022-02-09 | 2022-05-13 | 河北科技大学 | Fatigue degree monitoring system based on computer vision and sensor |
| CN114735010A (en) * | 2022-05-17 | 2022-07-12 | 中南大学 | Intelligent vehicle driving control method and system based on emotion recognition and storage medium |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117935231A (en) * | 2024-03-20 | 2024-04-26 | 杭州臻稀生物科技有限公司 | Non-inductive fatigue driving monitoring and intervention method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115227247B (en) | 2023-12-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104013414B (en) | A kind of Study in Driver Fatigue State Surveillance System based on intelligent movable mobile phone | |
| Assari et al. | Driver drowsiness detection using face expression recognition | |
| CN104224204B (en) | A kind of Study in Driver Fatigue State Surveillance System based on infrared detection technology | |
| CN1225375C (en) | Method for detecting fatigue driving based on multiple characteristic fusion | |
| CN112396658B (en) | Indoor personnel positioning method and system based on video | |
| CN202257856U (en) | Driver fatigue-driving monitoring device | |
| CN103824420A (en) | Fatigue driving identification system based on heart rate variability non-contact measuring | |
| CN112241658A (en) | Fatigue driving early warning system and method based on depth camera | |
| CN202130312U (en) | Driver fatigue driving monitoring device | |
| CN103942850A (en) | Medical staff on-duty monitoring method based on video analysis and RFID (radio frequency identification) technology | |
| CN104240446A (en) | Fatigue driving warning system on basis of human face recognition | |
| CN101673464A (en) | Intelligent management system of fatigue driving | |
| CN208498370U (en) | Fatigue driving based on steering wheel detects prior-warning device | |
| CN104068868A (en) | Method and device for monitoring driver fatigue on basis of machine vision | |
| CN109367539A (en) | A kind of intelligence system detecting fatigue driving | |
| CN104123549A (en) | Eye positioning method for real-time monitoring of fatigue driving | |
| CN107563346A (en) | One kind realizes that driver fatigue sentences method for distinguishing based on eye image processing | |
| Choi et al. | Driver drowsiness detection based on multimodal using fusion of visual-feature and bio-signal | |
| CN109977930A (en) | Method for detecting fatigue driving and device | |
| CN103700220A (en) | Fatigue driving monitoring device | |
| CN102104771A (en) | Multi-channel people stream rate monitoring system based on wireless monitoring | |
| Ma et al. | Real time drowsiness detection based on lateral distance using wavelet transform and neural network | |
| CN115227247B (en) | Fatigue driving detection method, system and storage medium based on multisource information fusion | |
| CN110781872A (en) | Driver fatigue grade recognition system with bimodal feature fusion | |
| CN104299363A (en) | Fatigue driving pre-warning system based on multi-feature fusion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |