CN114504329A

CN114504329A - Human brain fatigue state autonomous identification system based on 40-lead electroencephalogram acquisition equipment

Info

Publication number: CN114504329A
Application number: CN202210114275.2A
Authority: CN
Inventors: 高忠科; 王贺; 马超; 马文庆
Original assignee: Tianjin Furuilong Metal Products Co ltd; Tianjin University
Current assignee: Tianjin Furuilong Metal Products Co ltd; Tianjin University
Priority date: 2022-01-30
Filing date: 2022-01-30
Publication date: 2022-05-17
Anticipated expiration: 2042-01-30
Also published as: CN114504329B

Abstract

The human brain fatigue state autonomous identification system based on the 40-lead electroencephalogram acquisition equipment can realize accurate acquisition, effective identification and correct classification of EEG electroencephalograms of a driver, identifies the brain state of the driver through an optimal machine learning model, effectively warns the fatigue state of the driver, and prevents driving accidents. The design of the simulation driving experiment enables the collection process of the fatigue-based EEG electroencephalogram signal to be more reasonable and effective, and avoids social risks caused by collection in the actual driving process. The introduction of the optimal machine learning model greatly reduces the workload of manually designing the deep learning model, and can make targeted model design according to the specific characteristics of the collected EEG electroencephalogram signals, thereby improving the final human brain fatigue state identification accuracy of the model.

Description

Human brain fatigue state autonomous identification system based on 40-lead electroencephalogram acquisition equipment

Technical Field

The invention relates to brain fatigue state identification equipment. In particular to a human brain fatigue state autonomous identification system based on 40-lead electroencephalogram acquisition equipment.

Background

EEG electroencephalography, a physiological monitoring method for recording brain electrical activity, continuously collects changes in the electrical potential of the brain via electrodes disposed near the cerebral cortex. In recent years, the development of digital technology creates great convenience for the collection and analysis of electroencephalogram signals, and the time resolution of the signals can be accurate to millisecond level or even higher. In contrast, except for electroencephalography, only cranial Magnetic Resonance Spectroscopy (MRS) and Magnetoencephalography (MEG) in noninvasive cognitive neuroscience techniques are able to acquire data at this sampling rate. In addition, because EEG does not cause noise, EEG can more accurately reflect the stimulation of human brain to sound.

Fatigue (drowsiness), a common state of the brain, is ubiquitous in human life, but when fatigue occurs in actual production activities, it is often a great hazard. A series of changes such as weakened brain activities, decreased reaction capability, prolonged reaction time and the like caused by fatigue can lead an operator to be unable to make judgment and action correctly in time as normal. In particular, under the situations of driving, production operation, research and study, the influence caused by fatigue can lead to the reduction of efficiency, misjudgment and even serious accidents, namely, fatigue driving is a main cause of traffic accidents for many years. Therefore, it is necessary to find a method capable of effectively discriminating the fatigue state. Using EEG electroencephalography, the most reliable and effective measure of human physiological state, fatigue state detection is one of the current possible directions. By monitoring electroencephalogram (EEG) signals and assessing the level of drowsiness of the driver during driving, driving safety can be greatly improved.

The existing fatigue EEG electroencephalogram signal identification model based on the deep learning technology often inherits the difficulty of structural design of the deep learning model, needs a large amount of design experience and priori knowledge, and depends on the problem of long-time manual debugging, and the problems hinder the realization and popularization of a fatigue detection device in the true sense.

In the field of EEG electroencephalogram signal research, in order to overcome signal coupling brought by a complex system of human brain, researchers provide a plurality of corresponding methods for identifying, classifying and predicting EEG electroencephalogram signals. Such as methods for extracting the classical features of the EEG, time-frequency domain analysis techniques for EEG electroencephalograms, bayesian methods, and deep learning methods that are gradually gaining wide attention in recent years. How to extract different features in EEG electroencephalogram signals and effectively fuse collected tested multi-element state information, so that more accurate judgment of the current state of a tested object is a very challenging task. Creating a method that can have good identification performance on different subjects will promote the application and popularization of the fatigue detection system in the actual scene, so it is a topic with great research significance and development prospect.

Disclosure of Invention

The invention aims to solve the technical problem of overcoming the defects of the prior art and provides a human brain fatigue state autonomous identification system based on 40-lead electroencephalogram acquisition equipment, which can realize accurate acquisition of EEG (electroencephalogram) signals of a driver, identify the brain state of the driver through an optimal machine learning model and effectively early warn the fatigue state of the driver.

The technical scheme adopted by the invention is as follows: a human brain fatigue state autonomous identification system based on 40-lead electroencephalogram acquisition equipment comprises: the device comprises a driving simulation system, 40-lead electroencephalogram acquisition equipment, a physiological state learning module and a trained optimal machine learning model which are sequentially connected in series, wherein the optimal machine learning model is also connected with the output of the 40-lead electroencephalogram acquisition equipment, and the physiological state learning module and the trained optimal machine learning model are installed on an upper computer PC in the 40-lead electroencephalogram acquisition equipment; the method comprises the following steps that a testee enters brain states with different fatigue degrees by means of a simulation driving system, 40-lead electroencephalogram acquisition equipment acquires electroencephalogram signals of the testee with different fatigue degrees and sends the electroencephalogram signals to a physiological state learning module, the physiological state learning module receives the electroencephalogram signals, and a machine learning model for intelligently recognizing the brain fatigue state of the current testee is constructed and trained by combining an automatic machine learning technology to generate a trained optimal machine learning model; the 40-lead electroencephalogram acquisition equipment acquires electroencephalogram signals of a user in a normal driving state, sends the electroencephalogram signals into the trained optimal machine learning model, and outputs information of the brain fatigue state of the user in the normal driving state through identification of the trained optimal machine learning model.

The simulated driving system comprises: the driving hardware system, the simulation driving software, the driving scene display screen, the camera and the sound box are respectively connected with an upper computer PC in the 40-lead electroencephalogram acquisition equipment; wherein,

the driving hardware system consists of a steering wheel, a gear shifter and a pedal plate which are used for simulating a real driving environment;

the simulated driving software comprises:

(2.1) the interface simulates daytime driving on a two-lane highway, requiring the subject to cruise the vehicle in the center of the lane. The process simulates the situation of a non-ideal road surface, i.e. causes the vehicle to deviate to the right or left of the lane with the same probability, each lane departure event comprising four signs of a baseline period, the start of the deviation, the start of the response and the end of the event;

(2.2) a baseline period, which is a normal driving phase before the occurrence of the shift; starting the deviation, namely simulating a lane deviation event randomly triggered by driving software to make the automobile deviate to the left side or the right side from an original cruising lane; the response is started by requiring the testee to quickly recover the influence caused by the interference by operating the steering wheel; when the event is finished, the automobile moves back to the original cruise channel;

(2.3) synchronously recording electroencephalogram signals by 40-lead electroencephalogram acquisition equipment in the process of adjusting the posture of the automobile to recover stably by the testee from the deviation of the automobile, and storing the electroencephalogram signals by an upper computer PC; the next event occurs within a 5-10 second interval after completion of the current event, during which the subject must turn the car back to the centerline of the lane, and if the subject falls asleep during the data collection process, no feedback is provided to alert the subject.

The 40 lead electroencephalogram acquisition equipment comprises:

the EEG signal acquisition circuit comprises an EEG electrode cap and a patch cord thereof which are sequentially connected and used for acquiring EEG electroencephalograms, an EEG electroencephalogram signal acquisition circuit used for amplifying and converting the EEG electroencephalograms, an isolation circuit used for distinguishing and isolating analog signals from digital signals and ensuring the purity of the EEG electroencephalograms, an STM32 processor used for controlling the acquisition mode, parameters and working state of the EEG electroencephalogram signal acquisition circuit and reading and transmitting the EEG electroencephalograms acquired by the EEG electroencephalogram signal acquisition circuit, a USB communication circuit used for transmitting the EEG electroencephalograms to an upper computer PC, and a power supply circuit respectively supplying power to the EEG electroencephalogram signal acquisition circuit and the STM32 processor; the EEG electroencephalogram acquisition circuit comprises an EEG cap, a switching wire, an EEG signal acquisition circuit, a power supply circuit and a power supply circuit, wherein the EEG cap and the EEG cap in the switching wire thereof are used for acquiring EEG electroencephalograms of different brain areas of a testee or a user, and are connected with the EEG electroencephalogram acquisition circuit sequentially through the switching wire and Y2 series circular electric connector interfaces and used for acquiring and transmitting the EEG electroencephalograms;

the electroencephalogram cap and the patch cord thereof are used for acquiring EEG signals of a testee or a user corresponding to FP1, FP2, AF7, AF3, AFz, AF4, AF8, F3, F7, Fz, F4, F8, FT7, FC3, FCz, FC4, FT8, T7, C3, Cz, C4, T8, TP7, CP3, CPZ, CP4, TP8, P7, P3, Pz, P4, P8, PO7, PO3, POz, PO4, PO8, O1, O2, Oz and forty electrodes; the electrode distribution of the brain electrode cap conforms to 10/20 international standard leads;

the EEG electroencephalogram signal acquisition circuit is formed by a plurality of EEG electroencephalogram signal acquisition chips with the types of ADS1299 in parallel, and each EEG electroencephalogram signal acquisition chip integrates a high common-mode rejection ratio analog input module for receiving EEG electroencephalogram signals acquired by an electroencephalogram cap, a low-noise programmable gain amplifier for amplifying biological voltage signals and a high-resolution synchronous sampling analog-to-digital converter for converting analog signals into digital signals;

the STM32 processor also controls the transmission mode and transmission speed of the USB communication circuit;

the output of the USB communication circuit is respectively connected with the physiological state learning module and the input end of the optimal machine learning model, so that the communication with an upper computer PC is realized, and high-speed transmission equipment with the highest transmission rate of 480Mbps is set on the basis of the engineering of the HID equipment. The physiological state learning module adopts an automatic machine learning analysis method, fully-automatically extracts the brain state characteristics of the current testee according to EEG electroencephalogram signals induced and acquired by the simulated driving system and 40-lead electroencephalogram acquisition equipment, constructs an optimal machine learning model structure and model parameters, obtains an optimal brain state identification effect, obtains an initially-trained optimal machine learning model, directly searches the obtained result by using the automatic machine learning analysis method, trains the initially-trained optimal machine learning model, and finally obtains the trained optimal machine learning model to judge the brain state of the user. The automatic machine learning analysis method comprises a preprocessing process and a feature extraction process which are sequentially carried out, and searching is carried out by utilizing data extracted by the features.

The preprocessing process in the automatic machine learning analysis method comprises the following steps:

1) firstly, preprocessing EEG electroencephalogram signals of a tested person stored at a PC (personal computer) end of an upper computer, and reducing the sampling frequency from 1000Hz to 250Hz so as to reduce the data volume scale and improve the analysis efficiency; then, a band-pass filter of an EEGLAb toolbox in Matlab is called to reserve signal data of 0.2-50Hz, and other frequency band signals are removed to reduce noise; eliminating flicker artifacts and electro-ocular signal interference in EEG electroencephalogram signals by utilizing independent component analysis;

2) taking 3s EEG signals obtained by a driving simulation system before the deviation of the driving of the testee as a judgment basis of the brain state of the testee before the deviation, extracting the response time from the deviation starting to the response deviation of the testee as a fatigue state judgment index, and judging the brain fatigue state of the testee, wherein the method specifically comprises the following steps:

(2.1) representing the short-term fatigue state of the testee by using the extracted reaction time in the current deviation event, namely the local reaction time;

(2.2) acquiring the average reaction time of all deviation events of the testee within 90s before the current event occurs, wherein the average reaction time is called as the global reaction time and represents the long-term fatigue state of the testee;

(2.3) sequencing all the local reaction times obtained in the whole 90-minute experiment from small to large, and obtaining the local reaction time with the small 5% as the specific alert time of the subject at this time, wherein the specific alert time represents the reaction speed which the subject should have in the waking state;

(2.4) when the local reaction time and the global reaction time in one excursion event are both less than 1.5 times the alert time, considering that the current testee is in a waking state; when the local reaction time and the global reaction time in the primary deviation event are both more than 2.5 times of the warning time, the current testee is considered to be in a fatigue state; labeling EEG signal segments according to the EEG signal segments, and using the labeled EEG signal segments as a reference standard for experimental verification;

(2.5) migration events between awake and fatigue states are not classified.

The feature extraction process in the automatic machine learning analysis method comprises the following steps:

1) respectively extracting the characteristics of the preprocessed EEG electroencephalograms with the labels, firstly, carrying out fast Fourier transform on the EEG electroencephalograms with the labels, and then, respectively extracting theta, alpha and beta frequency bands in frequency spectrums of the EEG electroencephalograms with the labels as EEG electroencephalogram characteristics for analysis;

2) carrying out difference on frequency spectrum data of EEG electroencephalogram signal characteristics by using the electrode space position of 40-lead EEG acquisition equipment, and mapping the measured value of each electrode channel into a two-dimensional EEG signal image; the method specifically comprises the following steps:

(2.1) projecting the positions of the 40 lead electrodes of the subject's head from the 3-dimensional space onto the 2-dimensional plane using a spatial projection mechanism; the transformation keeps the relative spatial position between the adjacent electrodes unchanged; the specific method comprises the following steps:

in the isometric direction projection method, a two-dimensional polar coordinate system is established firstly, and a position on the head of a tested person is selected as a central point of a projection plane, namely a coordinate origin of a polar coordinate; calculating the distance rho of other points of the head of the testee relative to the origin of coordinates and the angle theta relative to the origin of coordinates; converting the current polar coordinate system into a Cartesian coordinate system to obtain two-dimensional projection of the head of the testee;

(2.2) after the two-dimensional plane projection of the 40-lead electroencephalogram collecting electrode on the head of the testee is obtained, normalizing the frequency spectrum power value corresponding to each electrode, and matching the frequency spectrum power value with the electrode position to obtain a discrete image, wherein the normalization formula is as follows:

where t represents the spectral power value, P_tNamely the result after normalization; through the normalization processing of the formula, the sizes of the corresponding pixel values are kept in the range from 0 to 255 when the corresponding pixel values are generated by utilizing the spectral power values;

(2.3) interpolating the spectral power value of the EEG signal characteristic by using a Clough-Tocher interpolation algorithm to finally obtain a 32 x 32 two-dimensional gray-scale EEG signal image;

3) repeating the image mapping process of the step 2) for the theta, alpha and beta different frequency bands of the fatigue state to obtain 32 multiplied by 32 two-dimensional gray-scale electroencephalogram signal images of the theta, alpha and beta frequency bands; two-dimensional gray electroencephalogram signal images of theta, alpha and beta frequency bands are regarded as three primary color channels and are converted into two-dimensional color electroencephalogram signal images together.

The automatic machine learning analysis method for searching by using the data extracted by the features comprises the following steps:

1) constructing a network structure search space suitable for EEG electroencephalogram signal analysis, wherein the network structure is composed of sequentially connected modular units, and the internal connection structure of each unit and a characteristic transformation mode automatically search through gradient descent;

2) taking the obtained two-dimensional color electroencephalogram signal image as input data of automatic machine learning, matching with a label of an EEG electroencephalogram signal, starting to construct an optimal machine learning model structure and model parameters, setting a learning rate to be 0.1, carrying out 200-period cyclic training for the total time, and obtaining an initially trained optimal machine learning model with a Batchsize of 128;

3) a retraining stage, namely a parameter fine tuning stage, which comprises the steps of directly using a searched network model structure, sequentially sending two-dimensional color electroencephalogram images of each testee into an initially trained optimal machine learning model, carrying out full-supervision training on the initially trained optimal machine learning model by using Pythrch, setting the model learning rate to be 0.01, carrying out 200-period circular training altogether, and obtaining a trained optimal machine learning model, wherein the Batchsize is 128; the output of the trained optimal machine learning model is the brain fatigue state of the testee;

4) and in the using stage, the trained optimal machine learning model is used for rapidly judging the brain state of the user, and the output of the trained optimal machine learning model is the brain fatigue state information of the user.

The construction of the network structure search space suitable for EEG electroencephalogram signal analysis in the step 1) is characterized in that on the basis of a PC-DARTS algorithm, aiming at the frequency domain and time domain characteristics of EEG electroencephalogram signals, the search space of the PC-DARTS is expanded into two different input streams: the EEG signal processing device comprises a frequency domain stream and a time domain stream, wherein the frequency domain stream and the time domain stream are used for inputting EEG electroencephalograms of two different forms; wherein,

(1) the frequency domain flow takes the generated two-dimensional color electroencephalogram signal image as the input of the frequency domain flow, and extracts the space and frequency characteristics of the two-dimensional color electroencephalogram signal image by using expansion convolution, separable convolution, maximum pooling and average pooling; the frequency domain stream inherits two special structures of a common unit and a reduction unit in the PC-DARTS algorithm, and the stacking number of the common unit and the reduction unit is increased to 11 on the basis of reserving the stacking rule of the common unit and the reduction unit in the PC-DARTS algorithm; the frequency domain stream retains all candidate operations in the PC-DARTS algorithm; the number of convolution kernels for all candidate operations in the frequency domain stream is set to 16; the input of the frequency domain stream is a two-dimensional color electroencephalogram signal image;

(2) time domain stream: obtaining time units on the basis of common units in a PC-DARTS algorithm, namely increasing the number of nodes to 11 on the basis of keeping connection rules among nodes in the common units; simultaneously replacing the dilated convolution, separable convolution, maximum pooling and average pooling inside the normal unit with time domain convolution of 1 × 1, 1 × 3, 1 × 5, 1 × 7, 1 × 9, 1 × 11 size and channel averaging pooling of 1 × 1, 1 × 3, 1 × 5, 1 × 7, 1 × 9, 1 × 11 size; two 1 multiplied by 1 convolutions are added at the internal start of the time unit to operate the input time sequence signal, two inputs are generated for the first node in the time unit, a total channel average pooling node is also added at the internal tail of the time unit, and the output of the total channel average pooling node is the output of the time unit; the number of all convolution kernels inside the time cell is set to 32; the time domain stream only contains one time unit; inputting the time domain stream into the preprocessed EEG electroencephalogram signal;

(3) and the output ends of the frequency domain stream and the time domain stream are sequentially connected with two shared full-connection layers to complete the identification of the electroencephalogram signals.

The internal connection structure and the characteristic transformation mode of each unit in the step 1) are automatically searched through gradient descent, aiming at the particularity of EEG electroencephalogram signals, a layer number self-adaptive mechanism and an early-stopping mechanism are introduced in the training of a PC-DARTS algorithm; wherein,

(1) after the network structure searching process is completed, if the operand selected as the skip connection in any unit reaches 25%, the searching is considered to be collapsed, the network automatically reduces the complexity of the PC-DARTS algorithm, namely reduces the number of frequency domain stream overlapping units or the number of nodes in a time unit, and starts a new round of searching; if the training precision of the PC-DARTS algorithm can not be stabilized above 85% when the training is stopped, the complexity of the PC-DARTS algorithm is increased, namely the number of frequency domain stream overlapping units or the number of nodes in a time unit is increased;

(2) the early-stopping mechanism is to stop the PC-DARTS algorithm when the operation weight is kept stable in 10 training periods in the network structure searching process.

Compared with the traditional electroencephalogram acquisition instrument of a medical institution, the electroencephalogram acquisition part of the system for autonomously identifying the fatigue state of the human brain based on the 40-lead electroencephalogram acquisition equipment has smaller volume and quality and low working condition requirement under the condition of ensuring acquisition precision and meeting the acquisition speed requirement, further reduces the cost and is widely applied to brain-computer interface equipment. The core chip of the brain electricity collecting part comprises a control chip and an A/D conversion chip. STM32 is one type of control chip, an ARM designed specifically for high-performance, low-cost, low-power embedded applications. Due to the characteristic of low power consumption, the 40-lead electroencephalogram acquisition equipment can normally work only through one USB (universal serial bus) device connected with the PC (personal computer) end. Meanwhile, the high performance characteristic of the system enables the system to be completely competent for controlling the A/D conversion chip and transmitting information.

The human brain fatigue state autonomous identification system based on the 40-lead electroencephalogram acquisition equipment can accurately acquire, effectively identify and correctly classify EEG electroencephalogram signals of a driver, identifies the brain state of the driver through an optimal machine learning model, effectively warns the fatigue state of the driver, and prevents driving accidents. The design of the simulation driving experiment enables the collection process of the fatigue-based EEG electroencephalogram signal to be more reasonable and effective, and avoids social risks caused by collection in the actual driving process. The introduction of the optimal machine learning model greatly reduces the workload of manually designing the deep learning model, and can make targeted model design according to the specific characteristics of the collected EEG electroencephalogram signals, thereby improving the final human brain fatigue state identification accuracy of the model.

Drawings

FIG. 1 is a block diagram of the brain fatigue state autonomous identification system based on 40-lead electroencephalogram acquisition equipment according to the present invention;

FIG. 2 is a block diagram of a driving simulation system according to the present invention;

FIG. 3a is a representation of a simulated driving system of the present invention;

FIG. 3b is a block diagram of the flow of EEG electroencephalogram acquisition in the present invention;

FIG. 4 is a block diagram of a 40-lead electroencephalogram acquisition device according to the present invention;

FIG. 5 is a diagram of the minimum interpolation unit for the Clough-Tocher interpolation in the present invention;

FIG. 6 is the final algorithm framework based on the PC-DARTS algorithm improvement in the present invention.

Detailed Description

The invention discloses a human brain fatigue state automatic identification system based on 40-lead electroencephalogram acquisition equipment, which is described in detail by combining an embodiment and the accompanying drawings.

As shown in fig. 1, the system for autonomously identifying the fatigue state of the human brain based on the 40-lead electroencephalogram acquisition device of the present invention comprises: the device comprises a driving simulation system 1, 40-lead electroencephalogram acquisition equipment 2, a physiological state learning module 3 and a trained optimal machine learning model 4 which are sequentially connected in series, wherein the optimal machine learning model 4 is also connected with the output of the 40-lead electroencephalogram acquisition equipment 2, and the physiological state learning module 3 and the trained optimal machine learning model 4 are installed on an upper computer PC in the 40-lead electroencephalogram acquisition equipment 2 together; a testee enters brain states with different fatigue degrees by means of a simulation driving system 1, the 40-lead electroencephalogram acquisition equipment 2 acquires electroencephalogram signals with different fatigue degrees of the testee and sends the electroencephalogram signals to a physiological state learning module 3, the physiological state learning module 3 receives the electroencephalogram signals, and a machine learning model for intelligently identifying the brain fatigue state of the current testee is constructed and trained by combining an automatic machine learning technology to generate a trained optimal machine learning model 4; the 40-lead electroencephalogram acquisition equipment 2 acquires electroencephalogram signals of a user in a normal driving state, sends the electroencephalogram signals into the trained optimal machine learning model 4, and outputs information of the brain fatigue state of the user in the normal driving state through identification of the trained optimal machine learning model 4.

As shown in fig. 2, the driving simulation system 1 includes: the driving hardware system 11, the simulation driving software 12, the driving scene display screen 13, the camera 14 and the sound equipment 15 are respectively connected with an upper computer PC in the 40-lead electroencephalogram acquisition equipment 2; wherein,

1) the driving hardware system 11 consists of a steering wheel, a gear and a pedal plate for simulating a real driving environment;

2) the simulated driving software 12 includes:

(2.1) As shown in FIG. 3a, the interface simulates daytime driving on a two-lane highway, requiring the subject to cruise the car in the center of the lane. The process simulates the situation of a non-ideal road surface, i.e. causes the vehicle to deviate to the right or left of the lane with the same probability, each lane departure event comprising four signs of a baseline period, the start of the deviation, the start of the response and the end of the event;

(2.2) as shown in FIG. 3b, the baseline period, is the normal driving phase before the shift occurs; the deviation begins, namely a lane departure event is randomly triggered by the simulation driving software 12, so that the automobile deviates from an original cruising lane to the left side or the right side; the response is started by requiring the testee to quickly recover the influence caused by the interference by operating the steering wheel; when the event is finished, the automobile moves back to the original cruise channel;

(2.3) synchronously recording the electroencephalogram signals by the 40-lead electroencephalogram acquisition equipment 2 in the process from the deviation of the automobile to the adjustment of the automobile posture of the testee to recover stably, and storing the electroencephalogram signals by the upper computer PC; in fig. 3b the corresponding direction of turning the steering wheel is shown, with a left offset 251 and a right offset 252. The next event occurs within a 5-10 second interval after completion of the current event, during which the subject must turn the car back to the centerline of the lane, and if the subject falls asleep during the data collection process, no feedback is provided to alert the subject.

As shown in fig. 4, the 40-lead brain electrical acquisition device 2 comprises:

1) the EEG signal acquisition circuit comprises an EEG electrode cap and a patch cord 21 thereof which are connected in sequence and used for acquiring EEG electroencephalograms, an EEG electroencephalogram signal acquisition circuit 22 used for amplifying and converting the EEG electroencephalograms, an isolation circuit 23 used for distinguishing and isolating analog signals from digital signals and ensuring the purity of the EEG electroencephalograms, an STM32 processor 24 used for controlling the acquisition mode, parameters and working state of the EEG electroencephalogram signal acquisition circuit 22 and reading and transmitting the EEG electroencephalograms acquired by the EEG electroencephalogram signal acquisition circuit 22, a USB communication circuit 25 used for transmitting the EEG electroencephalograms to an upper computer PC27, and a power supply circuit 26 respectively supplying power to the EEG electroencephalogram signal acquisition circuit 22 and the STM32 processor 24; the EEG electroencephalogram acquisition circuit is characterized in that the EEG cap and the EEG cap in the patch cord 21 of the EEG cap are used for acquiring EEG electroencephalograms of different brain areas of a testee or a user, and are connected with the EEG electroencephalogram acquisition circuit 22 sequentially through the patch cord and Y2 series circular electric connector interfaces and used for acquiring and transmitting the EEG electroencephalograms;

2) the electroencephalogram cap and the patch cord 21 thereof are used for acquiring EEG signals of a subject or a user corresponding to FP1, FP2, AF7, AF3, AFz, AF4, AF8, F3, F7, Fz, F4, F8, FT7, FC3, FCz, FC4, FT8, T7, C3, Cz, C4, T8, TP7, CP3, CPZ, CP4, TP8, P7, P3, Pz, P4, P8, PO7, PO3, POz, PO4, PO8, O1, O2, Oz and forty electrodes; the electrode distribution of the brain electrode cap conforms to 10/20 international standard leads;

3) the EEG electroencephalogram signal acquisition circuit 22 is formed by a plurality of EEG electroencephalogram signal acquisition chips with the model number of ADS1299 in parallel, and each EEG electroencephalogram signal acquisition chip integrates a high common mode rejection ratio analog input module for receiving EEG electroencephalogram signals acquired by an electroencephalogram cap, a low-noise programmable gain amplifier PGA for amplifying biological voltage signals and a high-resolution synchronous sampling analog-to-digital converter ADC for converting analog signals into digital signals;

4) the STM32 processor 24 is a processor adopting STM32H743IIT6, or STM32H743VIT6 or STM32H743ZGT 6; the STM32 processor 24 also controls the transmission mode and transmission speed of the USB communication circuit 25;

5) the output of the USB communication circuit 25 is respectively connected with the input ends of the physiological state learning module 3 and the optimal machine learning model 4, the USB communication circuit 25 is constructed by utilizing a USB3300 chip, the communication with an upper computer PC27 is realized, and high-speed transmission equipment with the highest transmission rate of 480Mbps is set on the basis of the engineering of HID equipment.

The physiological state learning module 3 adopts an automatic machine learning analysis method, does not need to preset a machine learning model in advance, adopts an automatic machine learning technology, carries out full-automatic extraction on the brain state characteristics of the current testee according to EEG electroencephalogram signals induced and collected by the simulation driving system 1 and the 40-lead electroencephalogram collection equipment 2, constructs an optimal machine learning model structure and model parameters, obtains an optimal brain state identification effect, obtains an initially trained optimal machine learning model, directly searches the obtained result by using the automatic machine learning analysis method, trains the initially trained optimal machine learning model, and finally obtains a trained optimal machine learning model 4 to distinguish the brain state of the user. The automatic machine learning analysis method comprises a preprocessing process and a feature extraction process which are sequentially carried out, and searching is carried out by utilizing data extracted by the features.

1) firstly, preprocessing EEG signals of a tested person stored at a PC (personal computer) end 27 of an upper computer, and reducing the sampling frequency from 1000Hz to 250Hz so as to reduce the data volume scale and improve the analysis efficiency; then, a band-pass filter of an EEGLAb toolbox in Matlab is called to reserve signal data of 0.2-50Hz, and other frequency band signals are removed to reduce noise; eliminating flicker artifacts (Blink artifacts) and electro-ocular signal interference in EEG brain electrical signals using Independent Component Analysis (ICA);

2) the 3s EEG signals obtained by the simulation driving system 1 before the deviation of the driving of the testee is started are used as the judgment basis of the brain state of the testee before the deviation, and the testee is gradually fatigued in the process of acquiring the EEG signals, so that the testee cannot be directly judged whether the testee is in the fatigue state at present. The Response Time (RT) from the start of the shift to the Response shift of the subject is taken as the fatigue state evaluation index, and in fig. 3b, the Response Time of each event is the Time interval between the tag 253 and the tag 251 or 252. Judging the brain fatigue state of the testee, which comprises the following steps:

(2.1) representing the short-term fatigue state of the subject by using the extracted reaction time in the current deviation event, namely local reaction time (local RT);

(2.2) acquiring the average reaction time of all deviation events of the testee within 90s before the current event occurs, and referring to the average reaction time as global RT (global RT), wherein the average reaction time represents the long-term fatigue state of the testee;

(2.3) sequencing all local reaction time obtained in the whole 90-minute experiment from small to large, and obtaining the local reaction time with the small 5% as the specific alert time (alert RT) of the current testee, wherein the alert time represents the reaction speed which the testee should have in the waking state;

(2.5) migration events between awake and fatigue states are not classified. As this may be due to other unknown processes, such as: the subject is making a wandering thought. In addition, excluding a partial excursion event between awake and fatigue states helps to improve the accuracy of the identification.

1) and respectively extracting the characteristics of the preprocessed EEG electroencephalogram signals with the labels, wherein when a testee or a user enters a fatigue state, the frequency domain theta, alpha and beta frequency bands of the EEG electroencephalogram signals of the brain contain characteristic information related to the brain state at most. Wherein, the fatigue degree of the driver is in positive correlation with the spectral power amplitude of the theta frequency band and the alpha frequency band. The beta band is highly correlated with kinesthetic stimuli and thus affects the speed of response of the driver. Based on the above, firstly, fast Fourier transform is carried out on the EEG signals with the labels, and then theta, alpha and beta frequency bands in the frequency spectrum of the EEG signals with the labels are respectively extracted to be used as EEG signal characteristics for analysis;

2) EEG time-series contain only information on the temporal state of the brain, ignoring the interplay and synergy between different regions of the brain. Aiming at the problem, the spatial position of an electrode of 40-lead electroencephalogram acquisition equipment 2 is utilized to carry out difference on frequency spectrum data of EEG electroencephalogram signal characteristics, and the measured value of each electrode channel is mapped into a two-dimensional electroencephalogram signal image together; the method specifically comprises the following steps:

(2.1) projecting the positions of the 40 lead electrodes of the subject's head from the 3-dimensional space onto the 2-dimensional plane using a spatial projection mechanism; this transformation keeps the relative spatial positions between adjacent electrodes unchanged, a mechanism that uses the isometric orientation Projection (AEP) method that is borrowed from mapping applications; the specific method comprises the following steps:

in the isometric direction projection method, a two-dimensional polar coordinate system is established firstly, and a position on the head of a tested person is selected as a central point of a projection plane, namely a coordinate origin of a polar coordinate; calculating the distance rho (namely the arc length on the sphere) of other points of the head of the testee relative to the origin of coordinates and the angle theta relative to the origin of coordinates; converting the current polar coordinate system into a Cartesian coordinate system to obtain two-dimensional projection of the head of the testee;

(2.3) as shown in fig. 5, interpolating the spectral power value of the EEG signal characteristic by using a Clough-Tocher interpolation algorithm, and finally obtaining a 32 x 32 two-dimensional gray-scale EEG signal image; before interpolation, the Clough-Tocher interpolation algorithm triangulates points of an electrode channel in a two-dimensional coordinate system to form a temporary irregular triangular network, a binary polynomial is defined on each triangle to create a curved surface consisting of a series of triangular Clough-Tocher curved surface sheets, a connecting line of a triangle vertex and a centroid divides the triangle into three sub-triangles, and the defined binary cubic polynomial is as follows:

wherein ,c_ijFrom the function f and the first partial derivative f of each vertex_x，f_yAnd normal derivative of the trilateral midpoint

These 12 parameters were determined.

as shown in fig. 6, constructing a network structure search space suitable for EEG electroencephalogram analysis in step 1) is based on a PC-DARTS algorithm, and expanding the search space of the PC-DARTS into two different input streams for the frequency domain and time domain characteristics of the EEG electroencephalogram signal itself: a frequency domain Stream (Spectral Stream) and a time domain Stream (Temporal Stream) for input of EEG brain electrical signals for two different modalities; wherein,

on the left side of fig. 6, the detailed structure and internal connections of the frequency domain stream are shown, and in particular, the internal structure of a Normal Cell (Normal Cell) which is an important part of the frequency domain stream is shown, where N is 4.

(2) Time domain Stream (Temporal Stream): obtaining time units on the basis of common units in a PC-DARTS algorithm, namely increasing the number of nodes to 11 on the basis of keeping connection rules among nodes in the common units; simultaneously replacing the dilated convolution, separable convolution, maximum pooling and average pooling inside the normal unit with time domain convolution of 1 × 1, 1 × 3, 1 × 5, 1 × 7, 1 × 9, 1 × 11 size and channel averaging pooling of 1 × 1, 1 × 3, 1 × 5, 1 × 7, 1 × 9, 1 × 11 size; compared with a common unit, the time-sequence signal input by the time unit is operated by adding two 1 multiplied by 1 convolutions at the internal start of the time unit, two inputs are generated for the first node in the time unit, the total channel average pooling node is also added at the internal tail of the time unit, and the output of the total channel average pooling node is the output of the time unit; the number of all convolution kernels inside the time cell is set to 32; the time domain stream only contains one time unit; inputting the time domain stream into the preprocessed EEG electroencephalogram signal;

On the right side of fig. 6, there are the detailed structure and internal connections of the time domain stream, and in particular, the internal structure of the time Cell (Temporal Cell) which is an important component of the time domain stream is shown, where n is 11.

The internal connection structure and the feature transformation mode of each unit in the step 1) are automatically searched through gradient descent, and although edge regularization is introduced by the PC-DARTS aiming at a possible collapse phenomenon in the NAS process, the existence of the edge regularization sometimes cannot ensure that the final search result of the network cannot collapse (particularly on a electroencephalogram data set with a small data volume). Therefore, aiming at the particularity of EEG electroencephalogram signals, a layer number self-adaptive mechanism and an early-stopping mechanism are introduced in the training of a PC-DARTS algorithm; wherein,

(2) the early-stopping mechanism is that in the network structure searching process, when the operation weight is kept stable in 10 training periods, the PC-DARTS algorithm is stopped;

2) the obtained two-dimensional color electroencephalogram signal image is used as input data of automatic machine learning, the structure and model parameters of an optimal machine learning model are constructed by matching with a label of an EEG electroencephalogram signal, because the model search space is huge, large-scale search and trial are needed, the learning rate is set to be 0.1, 200 cycles of cyclic training are carried out totally, and the size of Batchsize is 128, so that the initially trained optimal machine learning model is obtained;

According to the characteristics of the frequency domain and the time domain of an EEG electroencephalogram, a network structure suitable for EEG electroencephalogram analysis is constructed into a double-flow structure of the frequency domain and the time domain, a basic framework and a search space of the double-flow structure are respectively designed, and the method belongs to the specific description of searching by utilizing feature extraction data in an automatic machine learning analysis method.

Claims

1. The utility model provides a human brain fatigue state is from identification system based on 40 lead brain electricity collection equipment which characterized in that includes: the multifunctional electroencephalograph comprises a simulated driving system (1), 40-lead electroencephalograph acquisition equipment (2), a physiological state learning module (3) and a trained optimal machine learning model (4) which are sequentially connected in series, wherein the optimal machine learning model (4) is also connected with the output of the 40-lead electroencephalograph acquisition equipment (2), and the physiological state learning module (3) and the trained optimal machine learning model (4) are installed on an upper computer PC (personal computer) in the 40-lead electroencephalograph acquisition equipment (2) together; a testee enters brain states with different fatigue degrees by means of a simulation driving system (1), the 40-lead electroencephalogram acquisition equipment (2) acquires electroencephalogram signals with different fatigue degrees of the testee and sends the electroencephalogram signals to a physiological state learning module (3), the physiological state learning module (3) receives the electroencephalogram signals, and a machine learning model for intelligently identifying the brain fatigue state of the current testee is constructed and trained by combining an automatic machine learning technology to generate a trained optimal machine learning model (4); the 40-lead electroencephalogram acquisition equipment (2) acquires electroencephalogram signals of a user in a normal driving state, sends the electroencephalogram signals into the trained optimal machine learning model (4), and outputs information of the brain fatigue state of the user in the normal driving state through identification of the trained optimal machine learning model (4).

2. The human brain fatigue state self-identification system based on 40-lead electroencephalogram acquisition equipment according to claim 1, characterized in that the simulated driving system (1) comprises: the driving hardware system (11), the simulation driving software (12), the driving scene display screen (13), the camera (14) and the sound (15) are respectively connected with an upper computer PC in the 40-lead electroencephalogram acquisition equipment (2); wherein,

(1) the driving hardware system (11) consists of a steering wheel, a gear shifter and a pedal plate which are used for simulating a real driving environment;

(2) the simulated driving software (12) comprises:

(2.2) a baseline period, which is a normal driving phase before the occurrence of the shift; the deviation is started, a lane departure event is randomly triggered by the simulation driving software (12), so that the automobile deviates from an original cruising lane to the left side or the right side; the response is started by requiring the testee to quickly recover the influence caused by the interference by operating the steering wheel; when the event is finished, the automobile moves back to the original cruise channel;

(2.3) synchronously recording the electroencephalogram signals by the 40-lead electroencephalogram acquisition equipment (2) in the process from the deviation of the automobile to the stabilization of the posture adjustment of the automobile by a human subject, and storing the electroencephalogram signals by the upper computer PC; the next event occurs within a 5-10 second interval after completion of the current event, during which the subject must turn the car back to the centerline of the lane, and if the subject falls asleep during the data collection process, no feedback is provided to alert the subject.

3. The human brain fatigue state autonomous identification system based on 40-lead electroencephalogram acquisition equipment according to claim 1, characterized in that the 40-lead electroencephalogram acquisition equipment (2) comprises:

(1) the EEG signal acquisition circuit comprises sequentially connected EEG electrode caps and a patch cord (21) thereof for acquiring EEG electroencephalograms, an EEG electroencephalogram signal acquisition circuit (22) for amplifying and converting the EEG electroencephalograms, an isolation circuit (23) for distinguishing and isolating analog signals from digital signals and ensuring the purity of the EEG electroencephalograms, an STM32 processor (24) for controlling the acquisition mode, parameters and working state of the EEG electroencephalogram signal acquisition circuit (22) and reading and transmitting the EEG electroencephalograms acquired by the EEG electroencephalogram signal acquisition circuit (22), a USB communication circuit (25) for transmitting the EEG electroencephalograms to an upper computer PC (27), and a power supply circuit (26) for respectively supplying power to the EEG signal acquisition circuit (22) and the STM32 processor (24); the EEG electroencephalogram acquisition circuit comprises an EEG electroencephalogram acquisition circuit (22), a switching wire, a Y2 series circular electric connector interface, a Y-shaped switching wire and a power supply, wherein the EEG cap in the EEG cap and the switching wire (21) thereof acquires EEG electroencephalograms of different brain areas of a testee or a user, and is used for acquiring and transmitting the EEG electroencephalograms;

(2) the electroencephalogram cap and the patch cord (21) thereof are used for acquiring electroencephalogram signals of a testee or a user corresponding to FP1, FP2, AF7, AF3, AFz, AF4, AF8, F3, F7, Fz, F4, F8, FT7, FC3, FCz, FC4, FT8, T7, C3, Cz, C4, T8, TP7, CP3, CPZ, CP4, TP8, P7, P3, Pz, P4, P8, PO7, PO3, POz, PO4, PO8, O1, O2 and Oz of the electroencephalogram cap with forty electrodes; the electrode distribution of the brain electrode cap conforms to 10/20 international standard leads;

(3) the EEG electroencephalogram signal acquisition circuit (22) is formed by a plurality of EEG electroencephalogram signal acquisition chips with the models of ADS1299 in parallel, and each EEG electroencephalogram signal acquisition chip integrates a high common mode rejection ratio analog input module for receiving EEG electroencephalogram signals acquired by a brain electrode cap, a low-noise programmable gain amplifier for amplifying biological voltage signals and a high-resolution synchronous sampling analog-to-digital converter for converting analog signals into digital signals;

(4) the STM32 processor (24) also controls the transmission mode and the transmission speed of the USB communication circuit (25);

(5) the output of the USB communication circuit (25) is respectively connected with the physiological state learning module (3) and the input end of the optimal machine learning model (4), so that the communication with an upper computer PC (27) is realized, and high-speed transmission equipment with the highest transmission rate of 480Mbps is set on the basis of the engineering of the HID equipment.

4. The human brain fatigue state automatic identification system based on 40-lead electroencephalogram acquisition equipment as claimed in claim 1, wherein the physiological state learning module (3) adopts an automatic machine learning analysis method, carries out full-automatic extraction of the brain state characteristics of the current subject according to EEG electroencephalogram signals induced and acquired by the simulated driving system (1) and the 40-lead electroencephalogram acquisition equipment (2), constructs the optimal machine learning model structure and model parameters, obtains the optimal brain state identification effect, obtains the optimal initially trained machine learning model, directly uses the results obtained by searching with the automatic machine learning analysis method, trains the optimal initially trained machine learning model, and finally obtains the optimal trained machine learning model (4) to distinguish the brain state of the user. The automatic machine learning analysis method comprises a preprocessing process and a feature extraction process which are sequentially carried out, and searching is carried out by utilizing data extracted by the features.

5. The system for autonomously identifying the fatigue state of the human brain based on the 40-lead electroencephalogram acquisition device according to claim 4, wherein the preprocessing process in the automatic machine learning analysis method comprises the following steps:

1) firstly, preprocessing EEG signals of a tested person stored at a PC (personal computer) end (27) of an upper computer, and reducing the sampling frequency from 1000Hz to 250Hz so as to reduce the data volume scale and improve the analysis efficiency; then, a band-pass filter of an EEGLAb toolbox in Matlab is called to reserve signal data of 0.2-50Hz, and other frequency band signals are removed to reduce noise; eliminating flicker artifacts and electro-ocular signal interference in EEG electroencephalogram signals by utilizing independent component analysis;

2) taking 3s EEG electroencephalogram signals obtained by a driving simulation system (1) before the deviation of the driving of the testee as a judgment basis of the brain state of the testee before the deviation, extracting the response time from the deviation starting to the responding deviation of the testee as a fatigue state judgment index, and judging the brain fatigue state of the testee, wherein the judgment basis is as follows:

(2.5) migration events between awake and fatigue states are not classified.

6. The system for autonomously identifying the fatigue state of the human brain based on the 40-lead electroencephalogram acquisition equipment according to claim 4, wherein the characteristic extraction process in the automatic machine learning analysis method comprises the following steps:

1) respectively extracting the characteristics of the preprocessed EEG signals with the labels, firstly carrying out fast Fourier transform on the EEG signals with the labels, and then respectively extracting theta, alpha and beta frequency bands in frequency spectrums of the EEG signals with the labels as EEG signal characteristics for analysis;

2) the spatial position of an electrode of 40-lead electroencephalogram acquisition equipment (2) is utilized to carry out difference on the frequency spectrum data of EEG electroencephalogram signal characteristics, and the measured value of each electrode channel is mapped into a two-dimensional electroencephalogram signal image together; the method specifically comprises the following steps:

7. The system for autonomously identifying the fatigue state of the human brain based on the 40-lead electroencephalogram acquisition device according to claim 4, wherein the automatic machine learning analysis method for searching by using the data extracted by the features comprises the following steps:

8. The system for autonomously identifying the fatigue state of the human brain based on 40-lead electroencephalogram acquisition equipment according to claim 7, wherein the construction of the network structure search space suitable for EEG electroencephalogram signal analysis in the step 1) is to expand the search space of PC-DARTS into two different input streams according to the frequency domain and time domain characteristics of EEG electroencephalogram signals on the basis of the PC-DARTS algorithm: the EEG signal processing device comprises a frequency domain stream and a time domain stream, wherein the frequency domain stream and the time domain stream are used for inputting EEG electroencephalograms of two different forms; wherein,

(2) time domain streaming: obtaining time units on the basis of common units in a PC-DARTS algorithm, namely increasing the number of nodes to 11 on the basis of keeping connection rules among nodes in the common units; simultaneously replacing the dilated convolution, separable convolution, maximum pooling and average pooling inside the normal unit with time domain convolution of 1 × 1, 1 × 3, 1 × 5, 1 × 7, 1 × 9, 1 × 11 size and channel averaging pooling of 1 × 1, 1 × 3, 1 × 5, 1 × 7, 1 × 9, 1 × 11 size; two 1 multiplied by 1 convolutions are added at the internal start of the time unit to operate the input time sequence signal, two inputs are generated for the first node in the time unit, a total channel average pooling node is also added at the internal tail of the time unit, and the output of the total channel average pooling node is the output of the time unit; the number of all convolution kernels inside the time cell is set to 32; the time domain stream only contains one time unit; inputting the time domain stream into the preprocessed EEG electroencephalogram signal;

9. The system for autonomously identifying the fatigue state of the human brain based on 40-lead electroencephalogram acquisition equipment according to claim 8, characterized in that the internal connection structure and the characteristic transformation mode of each unit in the step 1) are automatically searched through gradient descent, aiming at the particularity of EEG electroencephalogram signals, a layer number self-adaptive mechanism and an early-stopping mechanism are introduced in the training of a PC-DARTS algorithm; wherein,