CN116807478A - Method, device and equipment for detecting sleepiness starting state of driver - Google Patents

Method, device and equipment for detecting sleepiness starting state of driver Download PDF

Info

Publication number
CN116807478A
CN116807478A CN202310772607.0A CN202310772607A CN116807478A CN 116807478 A CN116807478 A CN 116807478A CN 202310772607 A CN202310772607 A CN 202310772607A CN 116807478 A CN116807478 A CN 116807478A
Authority
CN
China
Prior art keywords
data sample
window
rising edge
driver
drowsiness
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202310772607.0A
Other languages
Chinese (zh)
Other versions
CN116807478B (en
Inventor
焦影影
焦竹青
陈娟
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Changzhou University
Original Assignee
Changzhou University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Changzhou University filed Critical Changzhou University
Priority to CN202310772607.0A priority Critical patent/CN116807478B/en
Publication of CN116807478A publication Critical patent/CN116807478A/en
Application granted granted Critical
Publication of CN116807478B publication Critical patent/CN116807478B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)

Abstract

The invention relates to a method for detecting the start state of sleepiness of a driver, which comprises the following steps: based on an electroencephalogram signal including a dozing stateA vertical electro-oculogram signal is led to obtain a pre-trained one-dimensional convolutional neural network, and a k-means clustering algorithm is utilized to obtain a rising edge centroid C o Centroid C of non-rising edge i I=1, 2, …, k; sliding on the electroencephalogram data by utilizing a first sliding window to obtain a current window electroencephalogram data sample O c Inputting the obtained result into a one-dimensional convolutional neural network trained in advance to obtain a classification prediction result; when the classification prediction result is that alpha wave exists, recording O c Time range t of (2) 0 ,t 0 +l 1 ]The method comprises the steps of carrying out a first treatment on the surface of the From t using wavelet energy distribution method 0 ,t 0 +l 1 ]In the method, the starting time t of the occurrence of the alpha wave is obtained α The method comprises the steps of carrying out a first treatment on the surface of the Based on t α Acquiring a preset time region, and sliding on the vertical electro-oculogram signal by using a second sliding window until the current window electro-oculogram data sample V c The ending point of (2) reaches the ending point of the preset time zone; calculate V acquired at each sliding c And C o C (C) i Correlation coefficients between i=1, 2, …, k; if V is present c And C o The maximum correlation coefficient detects the driver's drowsiness start state.

Description

Method, device and equipment for detecting sleepiness starting state of driver
Technical Field
The invention relates to the technical field of biomedical engineering and machine learning intersection, in particular to a method, a device and equipment for detecting the start state of drowsiness of a driver.
Background
Drivers are reported to have almost all of their dozing experiences during driving. Fatigue driving can cause the reduction of the functions of drivers, so that serious traffic accidents are easy to cause, and huge economic loss and mental injury are brought to society and families; since human sleep is affected by biological rhythms, drowsiness is more likely to occur in afternoon and midnight, and when sleep pressure is accumulated to a certain extent, the sleep state is unintentionally entered, thereby causing traffic accidents. In addition, even under unmanned conditions, the driver is liable to enter a sleep state in a long-time monotonous vehicle-mounted environment because of lack of external stimulus. Modern technology society relies on 24-hour camping or shift work in the transportation, medical, aviation, and many public services industries, which can cause significant disturbance to human sleep and circadian rhythms, thereby inducing fatigue. Therefore, the method realizes the accurate detection and timely early warning of the fatigue or the sleepiness of the driver, is an urgent need for preventing road traffic accidents caused by the fatigue of the driver, and has important research value for industries such as high-speed rail, aviation, education and the like.
During driving, the drowsiness behavior of the driver is extremely dangerous for road safety. The research on the change rule of the electroencephalogram and the electrooculogram signals in the dozing process and the proposal of a corresponding detection method are of great practical significance. Bioelectric signals such as electroencephalogram signals and electrooculogram signals are considered to be the most likely mechanism for generating fatigue of a driver. The existing methods for detecting driver fatigue based on bioelectric signals can be classified into methods based on statistical analysis and methods based on machine learning. Both methods require first performing fatigue classification using data collected by conventional fatigue metric methods. However, the fatigue measurement method, such as a subjective scale method, a video marking method or other methods according to the driving behavior of the vehicle, such as lane departure value, has the problems of subjectivity and poor reliability, so that the reliability of the detection result of the fatigue driving detection method based on statistical analysis and machine learning is not high and the practicability is limited.
And the drowsiness process is a short time to fall asleep, the drowsiness behavior starts with an unintentional eye-closing behavior first, and frequent drowsiness means frequent eye-closing behavior. Previous studies have found that drowsiness occurs with changes in brain alpha waves: the blocking phenomenon of the brain electricity alpha wave, namely, the brain electricity alpha wave continuously appears in the whole sleepy eye-closing period; the attenuation-disappearance phenomenon of the brain electricity alpha wave, namely, the brain electricity alpha wave briefly appears when the sleepiness happens, and then the attenuation disappears. Meanwhile, eye closure behavior generated when drowsiness occurs causes a rising edge waveform to be generated on the vertical electro-oculogram signal. The existing technology for detecting the brain wave alpha mainly adopts a digital signal processing method, such as Fourier transform, wavelet transform and the like to calculate the brain wave alpha frequency band energy, judges whether the brain wave alpha frequency band energy is the alpha wave according to a preset threshold value, needs to manually preset the threshold value, and has the problem of low detection accuracy caused by improper threshold value setting. For the detection of rising edge waveforms in vertical electro-oculogram signals, the prior art concentrates on the research of horizontal electro-oculogram signals, and the detection of vertical electro-oculogram signals is a way of marking by collecting facial information videos, so that the problems of low marking efficiency and high labor cost exist; and in the existing detection of the vertical electro-oculogram signal, the rising edge waveform is judged by combining a signal difference method with a threshold value, so that the detection efficiency is low.
Disclosure of Invention
Therefore, the invention aims to solve the technical problems of low alpha wave detection precision and low vertical electro-oculogram signal rising edge waveform detection efficiency in the prior art.
In order to solve the technical problems, the invention provides a method for detecting the start state of drowsiness of a driver, which comprises the following steps:
acquiring a window data sample set A containing alpha waves based on an electroencephalogram signal containing a doze state + And window data sample set A not containing alpha wave - As a training set, training to obtain a pre-trained one-dimensional convolutional neural network;
based on the first-order vertical electro-oculogram signal containing sleepiness state, a rising edge window data sample set R is obtained + Window data sample set R with non-rising edges - The method comprises the steps of carrying out a first treatment on the surface of the Acquiring a rising edge window data sample set R by using a k-means clustering algorithm + A rising edge centroid C of (a) o Window data sample set R with non-rising edges - Center of mass C of k non-rising edges in (C) i ,i＝1,2,…,k；
Acquiring a brain conduction electrical measurement data and a corresponding vertical electro-oculogram signal of a driver to be detected;
sliding on a brain wave conductive test data of a driver to be detected by utilizing a first sliding window with preset duration and step length to obtain a current window brain wave data sample O sliding each time c Inputting the pre-trained one-dimensional convolutional neural network to obtain a classification prediction result;
when the classification prediction result is that alpha waves exist, recording a current window brain electrical data sample O c Time range t of (2) 0 ,t 0 +l 1 ]The method comprises the steps of carrying out a first treatment on the surface of the And utilizing wavelet energy distribution method to obtain brain electrical data sample O from the current window c Time range t of (2) 0 ,t 0 +l 1 ]In the method, the starting time t of the occurrence of the alpha wave is obtained α ；
Based on the starting time t α Acquiring a preset time region, and utilizing a second sliding window with preset duration and step length to detect a driver to be detectedIs slid on the vertical electro-oculogram signal until the current window electro-oculogram data sample V c The ending point of the (c) reaches the ending point of the preset time zone;
calculating a current window electrooculogram data sample V acquired during each sliding c Correlation coefficients with a rising edge centroid and a plurality of non-rising edge centroids;
if there is present current window electrooculogram data sample V c Detecting the drowsiness starting state of the driver if the distance between the current window eye electric data sample V and the ascending edge centroid is minimum or the correlation coefficient is maximum c Start time t of (1) v The start state of drowsiness of the driver is the occurrence time.
In one embodiment of the invention, the window data sample set A containing alpha waves is obtained based on an electroencephalogram signal containing a doze state + And window data sample set A not containing alpha wave - As a training set, training to obtain a pre-trained one-dimensional convolutional neural network, including:
collecting an electroencephalogram signal containing a doze state, and preprocessing to obtain an alpha wave plate segment and a non-alpha wave segment;
sliding on the alpha wave segment by using a first sliding window with preset duration and step length to obtain a window data sample set A containing alpha waves + ；
Sliding on the non-alpha wave segment by using a first sliding window with preset duration and step length to obtain a window data sample set A which does not contain alpha waves - ；
Constructing a one-dimensional convolutional neural network by utilizing a window data sample set A containing alpha waves + And window data sample set A not containing alpha wave - As a training set, the cross entropy loss function is used as a loss function of network training to train, and the Adam optimization algorithm is utilized to update network parameters, so that a pre-trained one-dimensional convolutional neural network is obtained.
In one embodiment of the present invention, the one-dimensional convolutional neural network sequentially includes, in series along the positive propagation direction:
the input module is used for inputting the brain electrical data sample of the current window;
the front convolution module comprises two serially connected convolution units, wherein each convolution unit comprises a front convolution layer, a batch standardization layer and an activation function layer which are serially connected in sequence; the front convolution layer is provided with convolution kernels with preset number and preset length;
The maximum pooling layer is used for setting the size of a pooling window and the moving step length and carrying out maximum pooling operation on the output of the front convolution module;
the post-convolution module comprises two serially connected convolution units, each convolution unit comprises a post-convolution layer, a batch standardization layer and an activation function layer which are serially connected in sequence, and the output of the maximum pooling layer is convolved;
the global maximum pooling layer is used for setting the size and the moving step length of a global pooling window and carrying out global maximum pooling operation on the output of the post convolution module;
the multi-layer perceptron is sequentially connected with a preset number of full-connection layers and an output layer in series along the forward propagation direction, and is used for carrying out weighted summation output on the output of the global maximum pooling layer by utilizing a preset number of neurons, mapping the output to a linear separable space, carrying out classified prediction and judging whether an input current window electroencephalogram data sample contains alpha waves.
In one embodiment of the present invention, the rising edge window data sample set R is obtained based on a vertical electro-oculogram signal containing a doze state + Window data sample set R with non-rising edges - The method comprises the steps of carrying out a first treatment on the surface of the Acquiring a rising edge window data sample set R by using a k-means clustering algorithm + A rising edge centroid C of (a) o Window data sample set R with non-rising edges - Center of mass C of k non-rising edges in (C) i I=1, 2, …, k, comprising:
collecting a vertical electro-oculogram signal containing a doze state, and preprocessing to obtain a rising edge segment and a non-rising edge segment;
processing the rising edge segment into a window data segment with preset duration and centered rising edge waveform, and obtaining a rising edge window data sample set R + ；
A second sliding window using preset duration and step lengthSliding on the non-rising edge segment to obtain a window data sample set R with non-rising edge - ；
Acquiring a rising edge window data sample set R by using a k-means clustering algorithm + A rising edge centroid C of (a) o ；
Acquiring a window data sample set R with non-rising edges by using a k-means clustering algorithm - Center of mass C of k non-rising edges in (C) i ,i＝1,2,…,k。
In one embodiment of the present invention, after the inputting the pre-trained one-dimensional convolutional neural network and obtaining the classification prediction result, the method further includes:
and when the classification prediction result is that no alpha wave exists, predicting the current window electroencephalogram data sample acquired by the next first sliding window by using a pre-trained one-dimensional convolutional neural network.
In one embodiment of the present invention, the brain electrical data sample O is obtained from the current window using a wavelet energy distribution method c Time range t of (2) 0 ,t 0 +l 1 ]In the method, the starting time t of the occurrence of the alpha wave is obtained α Comprising:
the complex Morlet wavelet is used as a mother wavelet, and the brain electric data sample O of the current window is obtained c Performing continuous wavelet transformation with the scale range of [1, z ]]Obtaining a complex-valued continuous wavelet coefficient matrix:
wherein y is the sampling point number of the current window electroencephalogram data sample, and z is the maximum value 1024 of the scale range;
an absolute value matrix of the continuous wavelet coefficient matrix is obtained:
adding all columns of the absolute value matrix B to obtain a one-dimensional matrixCol＝[c 11 …c 1y], wherein ,
dividing the one-dimensional matrix Col into 3 parts averagely to obtain three arrays: col 1 ＝[c 11 …c 1u ]，Col 2 ＝[c 1(u+1) …c 1v ]，Col 3 ＝[c 1(v+1) …c 1y]； wherein ,
calculate three arrays Col respectively 1 、Col 2 、Col 3 And: sum (Sum) 1 ＝sum(Col 1 )，Sum 2 ＝sum(Col 2) and Sum3 ＝sum(Col 3 )；
If Col 1 If the sum is maximum, then consider the starting time t of the alpha wave α ＝t 0 ；
If Col 2 Maximum sum, then the onset of alpha wave occurrence
If Col 3 Maximum sum, then the onset of alpha wave occurrence
Wherein Fs is a preset data sampling rate; INT (·) is a rounding function.
In one embodiment of the invention, the calculation of the current window electrooculogram data sample V taken at each swipe c Correlation coefficients with a rising edge centroid and a plurality of non-rising edge centroids, comprising:
If there is no current window electrooculogram data sample V c The correlation coefficient between the non-rising edge centroids is greater than the current window electrooculogram data sample V c The correlation coefficient with the centroid of the rising edge detects the drowsiness starting state of the driver and records the current window eyeElectrical data sample V c Start time t of (1) v The time when the drowsiness starting state of the driver occurs is the time;
if there is present current window electrooculogram data sample V c The correlation coefficient between the non-rising edge centroids is greater than the current window electrooculogram data sample V c And calculating the correlation coefficient between the current window electrooculogram data sample acquired by the next second sliding window and the rising edge centroid and the plurality of non-rising edge centroids.
The embodiment of the invention also provides a device for detecting the sleepiness starting state of the driver, which comprises the following steps:
the model building module is used for acquiring a window data sample set A containing alpha waves based on an electroencephalogram signal containing a doze state + And window data sample set A not containing alpha wave - As a training set, training to obtain a pre-trained one-dimensional convolutional neural network; based on the first-order vertical electro-oculogram signal containing sleepiness state, a rising edge window data sample set R is obtained + Window data sample set R with non-rising edges - The method comprises the steps of carrying out a first treatment on the surface of the Acquiring a rising edge window data sample set R by using a k-means clustering algorithm + A rising edge centroid C of (a) o Window data sample set R with non-rising edges - Center of mass C of k non-rising edges in (C) i ,i＝1,2,…,k；
The signal acquisition module is used for acquiring a brain conduction test data of a driver to be detected and a corresponding vertical electro-oculogram signal;
the electroencephalogram signal detection module is used for sliding on electroencephalogram data of a pilot of a driver to be detected by utilizing a first sliding window with preset duration and step length to obtain a current window electroencephalogram data sample O sliding each time c Inputting the pre-trained one-dimensional convolutional neural network to obtain a classification prediction result; when the classification prediction result is that alpha waves exist, recording a current window brain electrical data sample O c Time range t of (2) 0 ,t 0 +l 1 ]The method comprises the steps of carrying out a first treatment on the surface of the And utilizing wavelet energy distribution method to obtain brain electrical data sample O from the current window c Time range t of (2) 0 ,t 0 +l 1 ]In the method, the starting time t of the occurrence of the alpha wave is obtained α ；
An electro-oculogram signal detection module for detecting an electro-oculogram signal based on a start time t α Acquiring a preset time region, and sliding on a vertical electro-oculogram signal of a driver to be detected by utilizing a second sliding window with preset duration and step length until a current window electro-oculogram data sample V c The ending point of the (c) reaches the ending point of the preset time zone; calculating a current window electrooculogram data sample V acquired during each sliding c Correlation coefficients with a rising edge centroid and a plurality of non-rising edge centroids;
the detection result acquisition module is used for acquiring the current window electro-oculogram data sample V if the current window electro-oculogram data sample V exists c Detecting the drowsiness starting state of the driver if the distance between the current window eye electric data sample V and the ascending edge centroid is minimum or the correlation coefficient is maximum c Start time t of (1) v The start state of drowsiness of the driver is the occurrence time.
The embodiment of the invention also provides a device for detecting the sleepiness starting state of the driver, which comprises the following components:
the head-mounted electroencephalogram signal acquisition device is used for acquiring electroencephalogram signal data of a driver;
the electro-oculogram signal acquisition device is used for acquiring a vertical electro-oculogram signal of a driver;
the upper computer is in communication connection with the head-mounted electroencephalogram signal acquisition device and the electro-oculogram signal acquisition device and is used for acquiring electro-brain measurement data and a vertical electro-oculogram signal of a driver, executing the method for detecting the sleepiness starting state of the driver and acquiring a detection result;
and the display device is in communication connection with the upper computer and is used for displaying the detection result.
In one embodiment of the invention, the vehicle-mounted monitoring system further comprises an early warning device which is in communication connection with the upper computer and is used for sending out early warning when the detection result obtained by the upper computer is that the drowsiness starting state of the driver is detected.
Compared with the prior art, the technical scheme of the invention has the following advantages:
the method for detecting the drowsiness starting state of the driver utilizes the mode of combining the alpha wave of the brain electric signal with the rising edge waveform of the vertical eye electric signal to detect the drowsiness starting state; the acquired current window electroencephalogram data samples with the alpha waves are classified and predicted by utilizing a one-dimensional convolutional neural network, so that the method is suitable for analyzing time sequence data, can automatically learn characteristics, does not need manual design characteristics, and can accurately classify the alpha waves and the non-alpha waves; using a wavelet energy distribution method, describing time-frequency characteristics of an electroencephalogram alpha wave based on continuous wavelet transformation of Mor et mother wavelet, and accurately acquiring the starting time of the alpha wave; in a time region containing the alpha wave starting time, calculating the obtained rising edge centroid and non-rising edge centroid and the correlation coefficient of the current window electro-oculogram data sample, judging whether a rising edge waveform exists or not, and further judging whether a sleepiness starting state exists or not; the invention can effectively identify the sleepiness starting state of the driver by combining the detection of the brain electric signal alpha wave and the vertical eye electric signal rising edge waveform, and avoid the road traffic safety problem caused by fatigue driving.
Drawings
In order that the invention may be more readily understood, a more particular description of the invention will be rendered by reference to specific embodiments thereof that are illustrated in the appended drawings, in which
FIG. 1 is a flowchart showing the steps of a method for detecting a drowsiness start state of a driver according to the present invention;
FIG. 2 is a schematic diagram showing signal processing of a method for detecting a drowsiness start state of a driver according to the present invention;
FIG. 3 is a schematic diagram of the position of the electroencephalogram electrode O2;
fig. 4 is a schematic diagram of the position of a two-conductor vertical electro-oculogram electrode.
Detailed Description
The present invention will be further described with reference to the accompanying drawings and specific examples, which are not intended to be limiting, so that those skilled in the art will better understand the invention and practice it.
In terms of analysis processing of bioelectric signals, a digital signal processing method and a machine learning method have been widely used. Recently, deep learning methods have demonstrated performance advantages over traditional machine learning methods. The EEG signal is a time sequence signal, and numerous researches show that the frequency band energy of delta wave (delta, 0.3Hz-3.5 Hz), theta wave (theta, 4Hz-7.5 Hz), alpha wave (alpha, 8Hz-13 Hz) and beta wave (beta, 14Hz-30 Hz) in the EEG signal has a close relation with driving fatigue. The alpha wave is similar to a sine signal and has obvious frequency oscillation characteristics, so that the sleeping state is indicated; the beta wave indicates a relative alert state; theta waves are associated with extreme drowsiness; delta waves are associated with deep sleep. And, (α+θ)/β is considered as the most reliable index indicating driving fatigue. Recent fatigue driving related researches find that there are two kinds of brain electricity alpha wave change rules in the sleepy process: the blocking phenomenon of the brain electricity alpha wave, namely, the brain electricity alpha wave continuously appears in the whole sleepy eye-closing period; attenuation-disappearance of brain electrical alpha wave-brain electrical alpha wave appears in the early stage of eye closure but can suddenly attenuate and disappear until the eye closure is finished. Moreover, the blocking phenomenon of the brain wave and the attenuation-disappearance phenomenon distribution of the brain wave represent two different drowsiness degrees: relaxation wakes up and sleep begins. Meanwhile, the change rule of the eye electric signal in the dozing process is that the eye closing behavior generated when the dozing occurs leads to the generation of rising edge waveforms on the vertical eye electric signal; the vertical electro-oculogram signal exhibits a gentle waveform in a drowsiness sustaining state; at the end of the drowsiness, the re-eye opening action causes a falling edge waveform of the vertical eye electrical signal. Therefore, the invention realizes the detection of the drowsiness starting state of the driver by utilizing the one-dimensional convolutional neural network model and the k-means clustering based on the alpha wave of the brain electric signal and the rising edge waveform of the eye electric signal.
Referring to fig. 1, the method for detecting the start state of drowsiness of the driver according to the present invention comprises the following specific steps:
s101: acquiring a window data sample set A containing alpha waves based on an electroencephalogram signal containing a doze state + And window data sample set A not containing alpha wave - As a training set, training to obtain a pre-trained one-dimensional convolutional neural network;
s102: based on a vertical electro-oculogram signal containing a doze state, obtainingTaking a rising edge window data sample set R + Window data sample set R with non-rising edges - The method comprises the steps of carrying out a first treatment on the surface of the Acquiring a rising edge window data sample set R by using a k-means clustering algorithm + A rising edge centroid C of (a) o Window data sample set R with non-rising edges - Center of mass C of k non-rising edges in (C) i ,i＝1,2,…,k；
S103: acquiring a brain conduction electrical measurement data and a corresponding vertical electro-oculogram signal of a driver to be detected;
s104: sliding on a brain wave conductive test data of a driver to be detected by utilizing a first sliding window with preset duration and step length to obtain a current window brain wave data sample O sliding each time c Inputting the pre-trained one-dimensional convolutional neural network to obtain a classification prediction result;
s105: when the classification prediction result is that alpha waves exist, recording a current window brain electrical data sample O c Time range t of (2) 0 ,t 0 +l 1 ]The method comprises the steps of carrying out a first treatment on the surface of the And utilizing wavelet energy distribution method to obtain brain electrical data sample O from the current window c Time range t of (2) 0 ,t 0 +l 1 ]In the method, the starting time t of the occurrence of the alpha wave is obtained α ；
When the classification prediction result is that no alpha wave exists, predicting a current window electroencephalogram data sample acquired by a next first sliding window by utilizing a pre-trained one-dimensional convolutional neural network;
s106: based on the starting time t α Acquiring a preset time region, and sliding on a vertical electro-oculogram signal of a driver to be detected by utilizing a second sliding window with preset duration and step length until a current window electro-oculogram data sample V c The ending point of the (c) reaches the ending point of the preset time zone;
s107: calculating a current window electrooculogram data sample V acquired during each sliding c Correlation coefficients with a rising edge centroid and a plurality of non-rising edge centroids;
s108: if there is present current window electrooculogram data sample V c If the distance from the center of mass of the rising edge is minimum or the correlation coefficient is maximum, driving is detectedA doze starting state of the person, recording the current window eye electric data sample V c Start time t of (1) v The time when the drowsiness starting state of the driver occurs is the time;
i.e. if there is no current window electro-oculogram data sample V c The correlation coefficient between the non-rising edge centroids is greater than the current window electrooculogram data sample V c Detecting the drowsiness start state of the driver, recording the current window eye electric data sample V c Start time t of (1) v The time when the drowsiness starting state of the driver occurs is the time; if there is present current window electrooculogram data sample V c The correlation coefficient between the non-rising edge centroids is greater than the current window electrooculogram data sample V c And calculating the correlation coefficient between the current window electrooculogram data sample acquired by the next second sliding window and the rising edge centroid and the plurality of non-rising edge centroids.
Specifically, in step S101, it includes:
s101-1: collecting an electroencephalogram signal containing a doze state, and preprocessing to obtain an alpha wave plate segment and a non-alpha wave segment;
s101-2: sliding on the alpha wave segment by using a first sliding window with preset duration and step length to obtain a window data sample set A containing alpha waves + ；
S101-3: sliding on the non-alpha wave segment by using a first sliding window with preset duration and step length to obtain a window data sample set A which does not contain alpha waves - ；
S101-4: constructing a one-dimensional convolutional neural network by utilizing a window data sample set A containing alpha waves + And window data sample set A not containing alpha wave - As a training set, the cross entropy loss function is used as a loss function of network training to train, and the Adam optimization algorithm is utilized to update network parameters, so that a pre-trained one-dimensional convolutional neural network is obtained.
The one-dimensional convolutional neural network constructed in step S101 sequentially includes in series in the forward propagation direction:
the input module is used for inputting the brain electrical data sample of the current window;
the front convolution module comprises two serially connected convolution units, wherein each convolution unit comprises a front convolution layer, a batch standardization layer and an activation function layer which are serially connected in sequence; the front convolution layer is provided with convolution kernels with preset number and preset length;
the maximum pooling layer is used for setting the size of a pooling window and the moving step length and carrying out maximum pooling operation on the output of the front convolution module;
the post-convolution module comprises two serially connected convolution units, each convolution unit comprises a post-convolution layer, a batch standardization layer and an activation function layer which are serially connected in sequence, and the output of the maximum pooling layer is convolved;
the global maximum pooling layer is used for setting the size and the moving step length of a global pooling window and carrying out global maximum pooling operation on the output of the post convolution module;
The multi-layer perceptron is sequentially connected with a preset number of full-connection layers and an output layer in series along the forward propagation direction, and is used for carrying out weighted summation output on the output of the global maximum pooling layer by utilizing a preset number of neurons, mapping the output to a linear separable space, carrying out classified prediction and judging whether an input current window electroencephalogram data sample contains alpha waves.
Specifically, in the embodiment of the present invention, the number m of convolution kernels of the pre-convolution layer 1 Satisfy 100 m 1 Convolution kernel length l less than or equal to 300 3 Satisfy 10.ltoreq.l 3 Less than or equal to 50; the convolution kernel step size is 1, and the convolution layer is not filled. The pooling window size l w And the moving step length l p All satisfy 3.ltoreq.l w ＝l p Less than or equal to 5; the number m of convolution kernels of the post-convolution layer 2 Satisfy 50 m or less 2 Convolution kernel length l less than or equal to 100 c Satisfy 10.ltoreq.l c Less than or equal to 20; the preset number of neurons of the full-connection layer is within 50-100; the number of neurons of the output layer is 2, and two types of labels are output.
Specifically, in step S102, it includes:
s102-1: collecting a vertical electro-oculogram signal containing a doze state, and preprocessing to obtain a rising edge segment and a non-rising edge segment;
s102-2: processing the rising edge segment into a window data segment with preset duration and centered rising edge waveform, and obtaining a rising edge window data sample set R + ；
S102-3: sliding on the non-rising edge segment by utilizing a second sliding window with preset duration and step length to obtain a window data sample set R with a non-rising edge - ；
S102-4: acquiring a rising edge window data sample set R by using a k-means clustering algorithm + A rising edge centroid C of (a) o ；
S102-5: acquiring a window data sample set R with non-rising edges by using a k-means clustering algorithm - Center of mass C of k non-rising edges in (C) i ,i＝1,2,…,k。
Specifically, in the embodiment of the present invention, the preset duration of the first sliding window of the preset duration and the step length is a fixed duration l 1 ，1s≤l 1 The sliding step length s is preset and is less than or equal to 3s 1 ，0.1s≤s 1 Less than or equal to 0.3s; the preset duration of the second sliding window with the preset duration and the step length is a fixed duration l 2 ，1s≤l 2 The sliding step length s is preset and is less than or equal to 2s 2 ，0.1s≤s 2 ≤0.3s。
Specifically, in step S105, it includes:
s105-1: the complex Morlet wavelet is used as a mother wavelet, and the brain electric data sample O of the current window is obtained c Performing continuous wavelet transformation with the scale range of [1, z ]]Obtaining a complex-valued continuous wavelet coefficient matrix:
wherein y is the sampling point number of the current window electroencephalogram data sample, and z is the maximum value 1024 of the scale range;
s105-2: an absolute value matrix of the continuous wavelet coefficient matrix is obtained:
adding all columns of the absolute value matrix B to obtain a one-dimensional matrix Col= [ c ] 11 …c 1y], wherein ,
s105-3: dividing the one-dimensional matrix Col into 3 parts averagely to obtain three arrays: col 1 ＝[c 11 …c 1u ]，Col 2 ＝[c 1(u+1) …c 1v ]，Col 3 ＝[c 1(v+1) …c 1y]； wherein ,
s105-4: calculate three arrays Col respectively 1 、Col 2 、Col 3 And: sum (Sum) 1 ＝sum(Col 1 )，Sum 2 ＝sum(Col 2) and Sum3 ＝sum(Col 3 )；
If Col 1 If the sum is maximum, then consider the starting time t of the alpha wave α ＝t 0 ；
If Col 2 Maximum sum, then the onset of alpha wave occurrence
If Col 3 Maximum sum, then the onset of alpha wave occurrence
Wherein Fs is a preset data sampling rate; INT (·) is a rounding function.
In this embodiment, the one-dimensional convolutional neural network is very suitable for analyzing time series data, and features are directly learned from original data through an end-to-end learning mode, so that simple modes in electroencephalogram signal data can be well identified. The waveform of the vertical electro-oculogram signal is generated by simple eyeballs and eyelid movements, and the eyeballs and the eyelid movements respectively have specific modes which are easy to distinguish by naked eyes; using k-means clustering, samples of the same pattern are automatically clustered into a class, thereby identifying rising edge waveforms generated by eyelid closure behavior at the onset of drowsiness.
Based on the above embodiment, the embodiment of the present invention further provides a device for detecting a drowsiness start state of a driver, including:
The model building module 100 is used for acquiring a window data sample set A containing alpha waves based on an electroencephalogram signal containing a doze state + And window data sample set A not containing alpha wave - As a training set, training to obtain a pre-trained one-dimensional convolutional neural network; based on the first-order vertical electro-oculogram signal containing sleepiness state, a rising edge window data sample set R is obtained + Window data sample set R with non-rising edges - The method comprises the steps of carrying out a first treatment on the surface of the Acquiring a rising edge window data sample set R by using a k-means clustering algorithm + A rising edge centroid C of (a) o Window data sample set R with non-rising edges - Center of mass C of k non-rising edges in (C) i ,i＝1,2,…,k；
The signal acquisition module 200 is used for acquiring a brain conduction test data of a driver to be detected and a corresponding vertical electro-oculogram signal;
the electroencephalogram signal detection module 300 is configured to slide on electroencephalogram data of a driver to be detected by using a first sliding window with a preset duration and step length, and obtain a current window electroencephalogram data sample O of each sliding c Inputting the pre-trained one-dimensional convolutional neural network to obtain a classification prediction result; when the classification prediction result is that alpha waves exist, recording a current window brain electrical data sample O c Time range t of (2) 0 ,t 0 +l 1 ]The method comprises the steps of carrying out a first treatment on the surface of the And utilizing wavelet energy distribution method to obtain brain electrical data sample O from the current window c Time range t of (2) 0 ,t 0 +l 1 ]In the method, the starting time t of the occurrence of the alpha wave is obtained α ；
An electrooculogram signal detection module 400 for detecting an electrooculogram signal based on a start time t α Acquiring a preset timeThe inter-area slides on a vertical electro-oculogram signal of a driver to be detected by utilizing a second sliding window with preset duration and step length until the current window electro-oculogram data sample V c The ending point of the (c) reaches the ending point of the preset time zone; calculating a current window electrooculogram data sample V acquired during each sliding c Correlation coefficients with a rising edge centroid and a plurality of non-rising edge centroids;
detection result obtaining module 500, if there is a current window electro-oculogram data sample V c Detecting the drowsiness starting state of the driver if the distance between the current window eye electric data sample V and the ascending edge centroid is minimum or the correlation coefficient is maximum c Start time t of (1) v The start state of drowsiness of the driver is the occurrence time.
The driver ' S drowsiness start state detection device of the present embodiment is used to implement the aforementioned driver ' S drowsiness start state detection method, and thus the embodiment of the driver ' S drowsiness start state detection device can be seen from the example portions of the driver ' S drowsiness start state detection method in the foregoing, for example, the model building module 100 is used to implement steps S101 and S102 in the aforementioned driver ' S drowsiness start state detection method; a signal acquisition module 200 for implementing step S103 in the above-mentioned driver sleepiness starting state detection method; an electroencephalogram signal detection module 300 for implementing steps S104 and S105 in the above-described driver sleepiness start state detection method; an electro-oculogram signal detection module 400 for implementing steps S106 and S107 in the above-described driver drowsiness start state detection method; the detection result obtaining module 500 is configured to implement step S108 in the above-mentioned method for detecting a drowsiness start state of a driver; therefore, the specific embodiments thereof may refer to the descriptions of the corresponding examples of the respective parts, and will not be repeated herein.
Specifically, referring to fig. 2, the method for detecting the start state of drowsiness of the driver according to the present invention detects the start state of drowsiness by combining the alpha wave of the electroencephalogram signal with the rising edge waveform of the vertical electro-oculogram signal; the acquired current window electroencephalogram data samples with the alpha waves are classified and predicted by utilizing a one-dimensional convolutional neural network, so that the method is suitable for analyzing time sequence data, can automatically learn characteristics without artificial design characteristics, and can realize accurate classification of the alpha waves and the non-alpha waves; using a wavelet energy distribution method, describing time-frequency characteristics of an electroencephalogram alpha wave based on continuous wavelet transformation of Morlet mother wavelet, and accurately acquiring the starting time of the alpha wave; and in the time region containing the alpha wave starting time, calculating the rising edge centroid and the non-rising edge centroid which are obtained by k-means clustering, and the correlation coefficient of the rising edge centroid and the current window electrooculogram data sample, judging whether a rising edge waveform exists or not, and further judging whether a sleepiness starting state exists or not.
Based on the above embodiment, in the present embodiment, a simulated driving experiment is set to recruit drivers with the afternoon nap habit, the experimental object is the driver with the afternoon nap habit, 15 drivers participate in the experiment, and all the drivers have the afternoon nap habit lasting up to 1 year and are required to finish an Epworth sleepiness scale, the scale values are all larger than 9, and the daytime sleepiness degree with higher degree is indicated; the method for detecting the drowsiness starting state of the driver comprises the following specific steps of:
S201: simulating driving experiment setting;
the driving experiment environment is simulated, a driver sits in a real vehicle, and the LCD large screen erected in front of the vehicle can display virtual driving road conditions; the driver can operate the virtual vehicle to run in the virtual driving road condition through the steering wheel, the accelerator and the brake pedal;
s202: an ESI NeuralScan electroencephalogram acquisition system is adopted to acquire a guide pillow electroencephalogram and a guide vertical eye electricity;
referring to fig. 3, according to the international electrode configuration method (10-20 system electrode configuration method), a conductive brain electricity at the occipital region O2 is collected;
referring to fig. 4, two electrode signals Vu and Vd at the upper and lower positions of the eye at the conventional vertical electro-oculogram position are collected, and a vertical electro-oculogram signal VEOG is obtained by subtracting the two electrode signals Vu and Vd;
the signal sampling rate is preset to 1000Hz in the ESI NeuralScan electroencephalogram acquisition system, and the filtering range is set to be within the range of 0-40 Hz;
s203: the acquired electroencephalogram O2 signal and the vertical electro-oculogram signal VEOG are subjected to noise reduction treatment by adopting a median filtering method;
s204: marking an alpha wave segment and a non-alpha wave segment on the electroencephalogram and marking a rising edge segment and a non-rising edge segment on the vertical electrooculogram by adopting a traditional visual method in the electroencephalogram according to the frequency and amplitude characteristics of the electroencephalogram alpha wave;
S205: sliding in all calibrated alpha wave fragments and non-alpha wave fragments respectively by adopting a first sliding window with fixed duration of 1 second and a sliding step length with specific duration of 0.1 second so as to obtain window data sample sets containing alpha waves respectivelyAnd window data sample set not containing alpha wave +.>Since the data sampling rate is 1000Hz +.> and />1000 data points for each window data sample;
s206: processing the marked rising edge segments into a window segment of a fixed duration of 1.5 seconds centered in the rising edge waveform to obtain a rising edge window sample setAnd a window with the same time length of 1.5 seconds is adopted for all non-rising edge fragments and sliding is carried out with a sliding step length with the time length of 0.1 seconds, so as to obtain a window sample set +.>Since the data sampling rate is 1000Hz +.> and />Each window data sample is 1500 data points;
s207: constructing a one-dimensional convolutional neural network, and collecting two window data sample sets containing alpha waves and not containing alpha waves and />Training to obtain a trained one-dimensional convolutional neural network model M 1 ；
S208: for a rising edge window sample setWindow sample set with non-rising edge +.>Downsampling is carried out, the original 1500 sample points are downsampled to 100 points, and corresponding downsampled sample sets are obtained respectively > and />
S209: calculating the distance from each sample to the mass center by adopting a k-Means clustering algorithm and adopting the Euclidean distance, and collecting the sample set of the rising edge windowFind 1 centroid +.>And sample set for non-rising edge window +.>Find k=6 centroids: />i=1, 2,3, …, k; setting k=6 is the number of modes distinguishable by visual observation on the vertical electro-ocular signal;
s210: for continuous electroencephalogram test data at a lead O2 position, a first sliding window with a fixed duration of 1 second is adopted and sliding is carried out on a sliding step length with a duration of 0.1 second, so that a current window sample p during each sliding is obtained c ；
S211: using a trained one-dimensional convolutional neural network M 1 For the current window data sample p c Predicting the category of the window to be 0 or 1, namely indicating that the current window contains alpha waves or does not contain alpha waves;
s212: when one-dimensional convolutional neural network M 1 When the category of the data of the current test window is predicted to be changed from 0 to 1, the current window is indicated to have the brain wave alpha, and at the moment, the data w of the current window is recorded * Time range t of (2) * ,t * +1]Second, wherein the second is;
s213: for the recorded time range [ t ] * ,t * +1]Second current window data w * The method based on wavelet energy distribution is adopted to further accurately position the starting time point of the occurrence of the alpha wave
S214: once the starting time point of the alpha wave appearance is locatedThe steering time domain range isSecond vertical electro-oculogram signal and fixed duration +.>Second sliding window of seconds and with a duration of +.>The sliding step length of seconds slides on the vertical electro-oculogram signal until the end point of the second sliding window reaches the momentSecond, wherein the second is;
s215: a point in time corresponding to each slide in step S214Sample the current window +.>(the time range is +.>Second), calculate k+1 centroids +.>i=0, 1,2,3, …, k, if the current window sample +.>And centroid C o If the correlation coefficient of (1) is the largest or the distance is the smallest, the rising edge waveform is considered to be detected, and the time point +.>Otherwise, repeating step S213;
s216: when the alpha wave startsAnd rising edge waveform occurrence time point +.>When it is detected that the detection of a certain occurrence of a defect is detected,the drowsiness start state of the driver is detected.
Specifically, in step S207, it includes:
s207-1: sample set of original two types of window data and />Down-sampling from original 1000 data points to 500 points as training data of a one-dimensional convolutional neural network model, so that the length of input one-dimensional data of the one-dimensional convolutional neural network is 500;
S207-2: setting two front convolution layers, wherein each layer has 150 convolution kernels and the convolution kernels have a length of 50; setting the step length of the convolution kernel as 1, wherein the convolution layer is not filled; batch normalization (batch normalization) of the data before each convolutional layer uses the activation function; the activation function used by each convolution layer is the Relu function;
s207-3: setting a maximum pooling layer, wherein the size of a pooling window is 3, and the moving step length of pooling operation is 3;
s207-4: two post-convolution layers are arranged, each layer has 100 convolution kernels, and the convolution kernels are 10 in length; batch normalization (batch normalization) of the data before each convolutional layer uses the activation function; the activation function used by each convolution layer is the Relu function;
s207-5: setting a global maximum pooling layer;
s207-6: setting two full-connection layers, wherein the number of each layer is 80 and 50 in sequence, and the activation function of each layer adopts a Relu function;
s207-7: setting an output layer with the number of neurons being 2, and outputting two types of data labels; the activating function of the output layer adopts a Sigmoid function;
s207-8: and adopting the cross loss function as a loss function of model training, and adopting an Adam optimization algorithm to update network parameters.
Specifically, in step S213, it includes:
s213-1: the complex Morlet wavelet is used as a mother wavelet for window data w * Performing continuous wavelet transformation, setting the scale range of the wavelet transformation to be 1-1024, and obtaining a complex-valued continuous wavelet coefficient matrix:wherein 100 is the sampling point number 1 of the window data, 1024 is the maximum of the scale range, and the absolute value matrix is obtained: /> wherein />
S213-2: matrix B of the absolute values * Is added to all columns of a one-dimensional matrix wherein />Dividing the one-dimensional matrix into 3 parts averagely, and obtaining three arrays: /> wherein ,
s213-3: respectively calculate three groupsAnd:
and />
If it isIf the value of (2) is maximum, then the starting time point of the alpha wave is regarded as +.>
If it isIf the value of (2) is maximum, then the start time point of alpha wave is considered +.>
If it isMaximum value of (2), then the start time point of alpha wave +.>
Based on the above embodiments, the embodiments of the present invention further provide a driver drowsiness start state detection device, including:
the head-mounted electroencephalogram signal acquisition device is used for acquiring electroencephalogram signals of a driver;
the eye electric signal acquisition device is used for acquiring eye electric signals of a driver;
the upper computer is in communication connection with the head-mounted electroencephalogram signal acquisition device and the electro-oculogram signal acquisition device and is used for acquiring the electroencephalogram signal and the electro-oculogram signal of a driver, executing the method for detecting the drowsiness starting state of the driver and acquiring a detection result;
And the display device is in communication connection with the upper computer and is used for displaying the detection result.
Specifically, the upper computer performs classification prediction on the electroencephalogram signals by using a one-dimensional convolutional neural network to obtain window data with alpha waves, and obtains the starting time of the alpha waves in a time region of the window data; and constructing a time region based on the starting time, detecting a rising edge waveform in the vertical electro-oculogram signal in the time region, calculating correlation coefficients between the rising edge waveform and the rising edge centroid and between the rising edge centroid and the non-rising edge centroid, and judging whether a sleepiness starting state exists.
Based on the above embodiment, in the embodiment of the present invention, the driver drowsiness start state detection device further includes an early warning device, which is communicatively connected to the upper computer, and is configured to send an early warning to remind the driver not to enter the sleep state when the detection result obtained by the upper computer is that the driver drowsiness start state is detected, so as to ensure driving safety.
The method for detecting the drowsiness starting state of the driver utilizes the mode of combining the alpha wave of the brain electric signal with the rising edge waveform of the vertical eye electric signal to detect the drowsiness starting state; the acquired current window electroencephalogram data samples with the alpha waves are classified and predicted by utilizing a one-dimensional convolutional neural network, the method is suitable for analyzing time sequence data, and features can be automatically learned without artificial design features, so that the accurate classification of the alpha waves and the non-alpha waves is realized; using a wavelet energy distribution method, describing time-frequency characteristics of an electroencephalogram alpha wave based on continuous wavelet transformation of Mor et mother wavelet, and accurately acquiring the starting time of the alpha wave; in a time region containing the alpha wave starting time, calculating the rising edge centroid and the non-rising edge centroid which are obtained by k-means clustering, and the correlation coefficient of the rising edge centroid and the current window electrooculogram data sample, judging whether a rising edge waveform exists or not, and further judging whether a sleepiness starting state exists or not; the invention can effectively identify the sleepiness starting state of the driver by combining the detection of the brain electric signal alpha wave and the vertical eye electric signal rising edge waveform, and avoid the road traffic safety problem caused by fatigue driving.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It is apparent that the above examples are given by way of illustration only and are not limiting of the embodiments. Other variations and modifications of the present invention will be apparent to those of ordinary skill in the art in light of the foregoing description. It is not necessary here nor is it exhaustive of all embodiments. While still being apparent from variations or modifications that may be made by those skilled in the art are within the scope of the invention.

Claims (10)

1. A method for detecting a drowsiness start state of a driver, comprising:
acquiring a window data sample set A containing alpha waves based on an electroencephalogram signal containing a doze state + And window data sample set A not containing alpha wave - As a training set, training to obtain a pre-trained one-dimensional convolutional neural network;
based on the first-order vertical electro-oculogram signal containing sleepiness state, a rising edge window data sample set R is obtained + Window data sample set R with non-rising edges - The method comprises the steps of carrying out a first treatment on the surface of the Acquiring a rising edge window data sample set R by using a k-means clustering algorithm + A rising edge centroid C of (a) o Window data sample set R with non-rising edges - Center of mass C of k non-rising edges in (C) i ,i＝1,2,…,k；
Acquiring a brain conduction electrical measurement data and a corresponding vertical electro-oculogram signal of a driver to be detected;
sliding on a brain wave conductive test data of a driver to be detected by utilizing a first sliding window with preset duration and step length to obtain a current window brain wave data sample O sliding each time c Inputting the pre-trained one-dimensional convolutional neural network to obtain a classification prediction result;
when the classification prediction result is that alpha waves exist, recording a current window brain electrical data sample O c Time range t of (2) 0 ,t 0 +l 1 ]The method comprises the steps of carrying out a first treatment on the surface of the And utilizing wavelet energy distribution method to obtain brain electrical data sample O from the current window c Time range t of (2) 0 ,t 0 +l 1 ]In the method, the starting time t of the occurrence of the alpha wave is obtained α ；
Based on the starting time t α Acquiring a preset time region, and sliding on a vertical electro-oculogram signal of a driver to be detected by utilizing a second sliding window with preset duration and step length until a current window electro-oculogram data sample V c The ending point of the (c) reaches the ending point of the preset time zone;
calculating the current window eye acquired during each slidingElectrical data sample V c Correlation coefficients with a rising edge centroid and a plurality of non-rising edge centroids;
if there is present current window electrooculogram data sample V c Detecting the drowsiness starting state of the driver if the distance between the current window eye electric data sample V and the ascending edge centroid is minimum or the correlation coefficient is maximum c Start time t of (1) v The start state of drowsiness of the driver is the occurrence time.
2. The method for detecting a drowsiness start state of a driver according to claim 1, wherein the window data sample set A including alpha waves is obtained based on an electroencephalogram signal including a drowsiness state + And window data sample set A not containing alpha wave - As a training set, training to obtain a pre-trained one-dimensional convolutional neural network, including:
Collecting an electroencephalogram signal containing a doze state, and preprocessing to obtain an alpha wave plate segment and a non-alpha wave segment;
sliding on the alpha wave segment by using a first sliding window with preset duration and step length to obtain a window data sample set A containing alpha waves + ；
Sliding on the non-alpha wave segment by using a first sliding window with preset duration and step length to obtain a window data sample set A which does not contain alpha waves - ；
Constructing a one-dimensional convolutional neural network by utilizing a window data sample set A containing alpha waves + And window data sample set A not containing alpha wave - As a training set, the cross entropy loss function is used as a loss function of network training to train, and the Adam optimization algorithm is utilized to update network parameters, so that a pre-trained one-dimensional convolutional neural network is obtained.
3. The method for detecting a drowsiness start state of a driver according to claim 2, wherein the one-dimensional convolutional neural network sequentially includes in series in a forward propagation direction:
the input module is used for inputting the brain electrical data sample of the current window;
the front convolution module comprises two serially connected convolution units, wherein each convolution unit comprises a front convolution layer, a batch standardization layer and an activation function layer which are serially connected in sequence; the front convolution layer is provided with convolution kernels with preset number and preset length;
The maximum pooling layer is used for setting the size of a pooling window and the moving step length and carrying out maximum pooling operation on the output of the front convolution module;
the post-convolution module comprises two serially connected convolution units, each convolution unit comprises a post-convolution layer, a batch standardization layer and an activation function layer which are serially connected in sequence, and the output of the maximum pooling layer is convolved;
the global maximum pooling layer is used for setting the size and the moving step length of a global pooling window and carrying out global maximum pooling operation on the output of the post convolution module;
the multi-layer perceptron is sequentially connected with a preset number of full-connection layers and an output layer in series along the forward propagation direction, and is used for carrying out weighted summation output on the output of the global maximum pooling layer by utilizing a preset number of neurons, mapping the output to a linear separable space, carrying out classified prediction and judging whether an input current window electroencephalogram data sample contains alpha waves.
4. The method for detecting a drowsiness start state of a driver according to claim 1, wherein the rising edge window data sample set R is obtained based on a vertical electro-oculogram signal including a drowsiness state + Window data sample set R with non-rising edges - The method comprises the steps of carrying out a first treatment on the surface of the Acquiring a rising edge window data sample set R by using a k-means clustering algorithm + A rising edge centroid C of (a) o Window data sample set R with non-rising edges - Center of mass C of k non-rising edges in (C) i I=1, 2, …, k, comprising:
collecting a vertical electro-oculogram signal containing a doze state, and preprocessing to obtain a rising edge segment and a non-rising edge segment;
processing the rising edge segment into a window data segment with preset duration and centered rising edge waveform, and obtaining a rising edge window data sample set R + ；
Sliding on the non-rising edge segment by utilizing a second sliding window with preset duration and step length to obtain a window data sample set R with a non-rising edge - ；
Acquiring a rising edge window data sample set R by using a k-means clustering algorithm + A rising edge centroid C of (a) o ；
Acquiring a window data sample set R with non-rising edges by using a k-means clustering algorithm - Center of mass C of k non-rising edges in (C) i ,i＝1,2,…,k。
5. The method for detecting a drowsiness start state of a driver according to claim 1, wherein the inputting the pretrained one-dimensional convolutional neural network, after obtaining the classification prediction result, further comprises:
and when the classification prediction result is that no alpha wave exists, predicting the current window electroencephalogram data sample acquired by the next first sliding window by using a pre-trained one-dimensional convolutional neural network.
6. The method for detecting a drowsiness start state of a driver according to claim 1, wherein the brain wave energy distribution method is used to obtain brain wave data samples O from the current window c Time range t of (2) 0 ,t 0 +l 1 ]In the method, the starting time t of the occurrence of the alpha wave is obtained α Comprising:
adopting complex Morlet wavelet as mother wavelet for brain electric data sample O of current window c Performing continuous wavelet transformation with the scale range of [1, z ]]Obtaining a complex-valued continuous wavelet coefficient matrix:
wherein y is the sampling point number of the current window electroencephalogram data sample, and z is the maximum value 1024 of the scale range;
an absolute value matrix of the continuous wavelet coefficient matrix is obtained:
adding all columns of the absolute value matrix B to obtain a one-dimensional matrix Col= [ c ] 11 …c 1y, wherein ,
dividing the one-dimensional matrix Col into 3 parts averagely to obtain three arrays: col 1 ＝[c 11 …c 1u ]，Col 2 ＝[c 1(u+1) …c 1v ，Col 3 ＝[c 1(v+1) …c 1y； wherein ,
calculate three arrays Col respectively 1 、Col 2 、Col 3 And: sum (Sum) 1 ＝sum(Col 1 )，Sum 2 ＝sum(Col 2) and Sum3 ＝sum(Col 3 )；
If Col 1 If the sum is maximum, then consider the starting time t of the alpha wave α ＝t 0 ；
If Col 2 Maximum sum, then the onset of alpha wave occurrence
If Col 3 Maximum sum, then the onset of alpha wave occurrence
Wherein Fs is a preset data sampling rate; INT (·) is a rounding function.
7. The driver's drowsiness start state detection method according to claim 1, characterized in that the calculation is performed each timeCurrent window electrooculogram data sample V acquired while sliding c Correlation coefficients with a rising edge centroid and a plurality of non-rising edge centroids, comprising:
if there is no current window electrooculogram data sample V c The correlation coefficient between the non-rising edge centroids is greater than the current window electrooculogram data sample V c Detecting the drowsiness start state of the driver, recording the current window eye electric data sample V c Start time t of (1) v The time when the drowsiness starting state of the driver occurs is the time;
if there is present current window electrooculogram data sample V c The correlation coefficient between the non-rising edge centroids is greater than the current window electrooculogram data sample V c And calculating the correlation coefficient between the current window electrooculogram data sample acquired by the next second sliding window and the rising edge centroid and the plurality of non-rising edge centroids.
8. A driver's drowsiness start state detection device, characterized by comprising:
the model building module is used for acquiring a window data sample set A containing alpha waves based on an electroencephalogram signal containing a doze state + And window data sample set A not containing alpha wave - As a training set, training to obtain a pre-trained one-dimensional convolutional neural network; based on the first-order vertical electro-oculogram signal containing sleepiness state, a rising edge window data sample set R is obtained + Window data sample set R with non-rising edges - The method comprises the steps of carrying out a first treatment on the surface of the Acquiring a rising edge window data sample set R by using a k-means clustering algorithm + A rising edge centroid C of (a) o Window data sample set R with non-rising edges - Center of mass C of k non-rising edges in (C) i ,i＝1,2,…,k；
The signal acquisition module is used for acquiring a brain conduction test data of a driver to be detected and a corresponding vertical electro-oculogram signal;
the electroencephalogram signal detection module is used for guiding electroencephalogram of the driver to be detected by utilizing a first sliding window with preset duration and step lengthSliding on the test data to obtain a current window electroencephalogram data sample O sliding each time c Inputting the pre-trained one-dimensional convolutional neural network to obtain a classification prediction result; when the classification prediction result is that alpha waves exist, recording a current window brain electrical data sample O c Time range t of (2) 0 ,t 0 +l 1 ]The method comprises the steps of carrying out a first treatment on the surface of the And utilizing wavelet energy distribution method to obtain brain electrical data sample O from the current window c Time range t of (2) 0 ,t 0 +l 1 ]In the method, the starting time t of the occurrence of the alpha wave is obtained α ；
An electro-oculogram signal detection module for detecting an electro-oculogram signal based on a start time t α Acquiring a preset time region, and sliding on a vertical electro-oculogram signal of a driver to be detected by utilizing a second sliding window with preset duration and step length until a current window electro-oculogram data sample V c The ending point of the (c) reaches the ending point of the preset time zone; calculating a current window electrooculogram data sample V acquired during each sliding c Correlation coefficients with a rising edge centroid and a plurality of non-rising edge centroids;
the detection result acquisition module is used for acquiring the current window electro-oculogram data sample V if the current window electro-oculogram data sample V exists c Detecting the drowsiness starting state of the driver if the distance between the current window eye electric data sample V and the ascending edge centroid is minimum or the correlation coefficient is maximum c Start time t of (1) v The start state of drowsiness of the driver is the occurrence time.
9. A driver's drowsiness start state detection device characterized by comprising:
the head-mounted electroencephalogram signal acquisition device is used for acquiring electroencephalogram signal data of a driver;
the electro-oculogram signal acquisition device is used for acquiring a vertical electro-oculogram signal of a driver;
the upper computer is in communication connection with the head-mounted electroencephalogram signal acquisition device and the electro-oculogram signal acquisition device and is used for acquiring electro-brain measurement data and a vertical electro-oculogram signal of a driver, executing the method for detecting the sleepiness starting state of the driver according to any one of claims 1 to 7 and acquiring a detection result;
And the display device is in communication connection with the upper computer and is used for displaying the detection result.
10. The apparatus for detecting a drowsiness start state of a driver according to claim 9, further comprising an early warning device communicatively connected to the host computer for giving an early warning when the detection result obtained by the host computer is that the drowsiness start state of the driver is detected.
CN202310772607.0A 2023-06-27 2023-06-27 Method, device and equipment for detecting sleepiness starting state of driver Active CN116807478B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310772607.0A CN116807478B (en) 2023-06-27 2023-06-27 Method, device and equipment for detecting sleepiness starting state of driver

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310772607.0A CN116807478B (en) 2023-06-27 2023-06-27 Method, device and equipment for detecting sleepiness starting state of driver

Publications (2)

Publication Number Publication Date
CN116807478A true CN116807478A (en) 2023-09-29
CN116807478B CN116807478B (en) 2024-07-12

Family

ID=88117934

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310772607.0A Active CN116807478B (en) 2023-06-27 2023-06-27 Method, device and equipment for detecting sleepiness starting state of driver

Country Status (1)

Country Link
CN (1) CN116807478B (en)

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100217145A1 (en) * 2006-06-09 2010-08-26 Bracco Spa Method of processing multichannel and multivariate signals and method of classifying sources of multichannel and multivariate signals operating according to such processing method
CN108926349A (en) * 2017-05-24 2018-12-04 上海交通大学 Daily sleep based on brain electricity alpha wave starts period detection method
CN109124625A (en) * 2018-09-04 2019-01-04 大连理工大学 A kind of driver fatigue state horizontal mipmap method
CN110367967A (en) * 2019-07-19 2019-10-25 南京邮电大学 A kind of pocket lightweight human brain condition detection method based on data fusion
CN111671420A (en) * 2020-06-17 2020-09-18 河北省科学院应用数学研究所 Method for extracting features from resting electroencephalogram data and terminal equipment
US20210197834A1 (en) * 2016-11-21 2021-07-01 George Shaker System and method for sensing with millimeter waves for sleep position detection, vital signs monitoring and/or driver detection
US20220023584A1 (en) * 2018-12-07 2022-01-27 Ewha University - Industry Collaboration Foundation Artificial intelligence-based non-invasive neural circuit control treatment system and method for improving sleep
CN113974653A (en) * 2021-11-30 2022-01-28 杭州妞诺霄云大数据科技有限公司 Optimized spike detection method and device based on Joyston index, storage medium and terminal
CN114916937A (en) * 2022-06-10 2022-08-19 长春理工大学 BDPCA clustering algorithm-based driver electroencephalogram fatigue grade division method
WO2023075161A1 (en) * 2021-10-29 2023-05-04 전남대학교 산학협력단 Vehicular apparatus for determining driver's condition by using artificial intelligence and control method therefor
CN116269212A (en) * 2022-12-23 2023-06-23 浙江工业大学 Multi-mode sleep stage prediction method based on deep learning

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100217145A1 (en) * 2006-06-09 2010-08-26 Bracco Spa Method of processing multichannel and multivariate signals and method of classifying sources of multichannel and multivariate signals operating according to such processing method
US20210197834A1 (en) * 2016-11-21 2021-07-01 George Shaker System and method for sensing with millimeter waves for sleep position detection, vital signs monitoring and/or driver detection
CN108926349A (en) * 2017-05-24 2018-12-04 上海交通大学 Daily sleep based on brain electricity alpha wave starts period detection method
CN109124625A (en) * 2018-09-04 2019-01-04 大连理工大学 A kind of driver fatigue state horizontal mipmap method
US20220023584A1 (en) * 2018-12-07 2022-01-27 Ewha University - Industry Collaboration Foundation Artificial intelligence-based non-invasive neural circuit control treatment system and method for improving sleep
CN110367967A (en) * 2019-07-19 2019-10-25 南京邮电大学 A kind of pocket lightweight human brain condition detection method based on data fusion
CN111671420A (en) * 2020-06-17 2020-09-18 河北省科学院应用数学研究所 Method for extracting features from resting electroencephalogram data and terminal equipment
WO2023075161A1 (en) * 2021-10-29 2023-05-04 전남대학교 산학협력단 Vehicular apparatus for determining driver's condition by using artificial intelligence and control method therefor
CN113974653A (en) * 2021-11-30 2022-01-28 杭州妞诺霄云大数据科技有限公司 Optimized spike detection method and device based on Joyston index, storage medium and terminal
CN114916937A (en) * 2022-06-10 2022-08-19 长春理工大学 BDPCA clustering algorithm-based driver electroencephalogram fatigue grade division method
CN116269212A (en) * 2022-12-23 2023-06-23 浙江工业大学 Multi-mode sleep stage prediction method based on deep learning

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
JIAO, YINGYING: "Driver sleepiness detection from EEG and EOG signals using GAN and LSTM networks", 《NEUROCOMPUTING》, 31 December 2020 (2020-12-31), pages 1 - 5 *
VAN HESE: "Automatic detection of sleep stages using the EEG", 《IEEE》, 31 December 2001 (2001-12-31), pages 1 - 5 *
曹海婷: "基于多特征组合的动态手势识别", 《计算机工程与设计》, 31 December 2018 (2018-12-31), pages 1 - 60 *
李栋: "基于FPGA的驾驶员疲劳检测***的设计与实现", 《中国优秀硕士学位论文全文数据库》, 31 December 2009 (2009-12-31), pages 1 - 60 *

Also Published As

Publication number Publication date
CN116807478B (en) 2024-07-12

Similar Documents

Publication Publication Date Title
Murugan et al. Detection and analysis: Driver state with electrocardiogram (ECG)
EP2281506B1 (en) Method and system for determining an individual's state of attention
Gurudath et al. Drowsy driving detection by EEG analysis using wavelet transform and K-means clustering
Karuppusamy et al. Multimodal system to detect driver fatigue using EEG, gyroscope, and image processing
Hooda et al. A comprehensive review of approaches to detect fatigue using machine learning techniques
WO2006000166A1 (en) Method and device for detecting operator fatigue or quality
CN106913333B (en) Method for selecting sensitivity characteristic index of continuous attention level
Bittner et al. Detecting of fatigue states of a car driver
He et al. Driver fatigue evaluation model with integration of multi‐indicators based on dynamic Bayesian network
Koay et al. Detecting and recognizing driver distraction through various data modality using machine learning: A review, recent advances, simplified framework and open challenges (2014–2021)
Panagopoulos et al. Forecasting markers of habitual driving behaviors associated with crash risk
Hanosh et al. Real-time epileptic seizure detection during sleep using passive infrared sensors
CN109106384A (en) A kind of psychological pressure condition predicting method and system
CN116807478B (en) Method, device and equipment for detecting sleepiness starting state of driver
Rosipal et al. EEG-based drivers’ drowsiness monitoring using a hierarchical Gaussian mixture model
Ng et al. Automated identification of epileptic and alcoholic EEG signals using recurrence quantification analysis
Hassan et al. Sigmoid wake probability model for high-resolution detection of drowsiness using electroencephalogram
Josephin et al. A review on the measures and techniques adapted for the detection of driver drowsiness
Ribeiro et al. EEG-based drowsiness detection platform to compare different methodologies
Murugappan et al. Subtractive fuzzy classifier based driver drowsiness levels classification using EEG
Zhong et al. Localized energy study for analyzing driver fatigue state based on wavelet analysis
Yulita et al. Comparing classification via regression and random committee for automatic sleep stage classification in autism patients
Jiao et al. Detecting slow eye movements with bimodal-LSTM for recognizing drivers’ sleep onset period
Bundele et al. An svm classifier for fatigue-detection using skin conductance for use in the bits-lifeguard wearable computing system
Begawan et al. Sleep stage identification based on eeg signals using parallel convolutional neural network and recurrent neural network

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant