CN112990008B - Emotion recognition method and system based on three-dimensional characteristic diagram and convolutional neural network - Google Patents
Emotion recognition method and system based on three-dimensional characteristic diagram and convolutional neural network Download PDFInfo
- Publication number
- CN112990008B CN112990008B CN202110272735.XA CN202110272735A CN112990008B CN 112990008 B CN112990008 B CN 112990008B CN 202110272735 A CN202110272735 A CN 202110272735A CN 112990008 B CN112990008 B CN 112990008B
- Authority
- CN
- China
- Prior art keywords
- electroencephalogram
- wavelet
- emotion recognition
- signal
- emotional state
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 230000008909 emotion recognition Effects 0.000 title claims abstract description 51
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000013527 convolutional neural network Methods 0.000 title claims abstract description 33
- 238000010586 diagram Methods 0.000 title claims abstract description 18
- 230000002996 emotional effect Effects 0.000 claims abstract description 56
- 230000009466 transformation Effects 0.000 claims abstract description 12
- 230000006870 function Effects 0.000 claims description 20
- 239000013598 vector Substances 0.000 claims description 19
- 230000008030 elimination Effects 0.000 claims description 7
- 238000003379 elimination reaction Methods 0.000 claims description 7
- 238000003860 storage Methods 0.000 claims description 6
- GNFTZDOKVXKIBK-UHFFFAOYSA-N 3-(2-methoxyethoxy)benzohydrazide Chemical compound COCCOC1=CC=CC(C(=O)NN)=C1 GNFTZDOKVXKIBK-UHFFFAOYSA-N 0.000 claims 2
- FGUUSXIOTUKUDN-IBGZPJMESA-N C1(=CC=CC=C1)N1C2=C(NC([C@H](C1)NC=1OC(=NN=1)C1=CC=CC=C1)=O)C=CC=C2 Chemical compound C1(=CC=CC=C1)N1C2=C(NC([C@H](C1)NC=1OC(=NN=1)C1=CC=CC=C1)=O)C=CC=C2 FGUUSXIOTUKUDN-IBGZPJMESA-N 0.000 claims 2
- 238000000537 electroencephalography Methods 0.000 description 13
- 238000004364 calculation method Methods 0.000 description 10
- 230000008451 emotion Effects 0.000 description 8
- 210000002569 neuron Anatomy 0.000 description 8
- 238000004590 computer program Methods 0.000 description 7
- 238000012545 processing Methods 0.000 description 7
- 230000004913 activation Effects 0.000 description 5
- 238000009826 distribution Methods 0.000 description 4
- 238000001914 filtration Methods 0.000 description 4
- 239000011159 matrix material Substances 0.000 description 4
- 210000004556 brain Anatomy 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000002474 experimental method Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 238000001228 spectrum Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000013145 classification model Methods 0.000 description 2
- 238000000354 decomposition reaction Methods 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007177 brain activity Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000019771 cognition Effects 0.000 description 1
- 230000001054 cortical effect Effects 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000004070 electrodeposition Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000008921 facial expression Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000001671 psychotherapy Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 210000004761 scalp Anatomy 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 230000000946 synaptic effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/08—Feature extraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/015—Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/12—Classification; Matching
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Biophysics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Signal Processing (AREA)
- Dermatology (AREA)
- Neurology (AREA)
- Neurosurgery (AREA)
- Human Computer Interaction (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
Abstract
The emotion recognition method and system based on the three-dimensional characteristic diagram and the convolutional neural network disclosed by the disclosure comprise the following steps: acquiring an electroencephalogram signal to be identified; extracting an electroencephalogram signal without a basic emotional state from the electroencephalogram signal to be identified; decomposing and reconstructing the electroencephalogram signals without the basic emotional state by wavelet packet transformation to obtain a plurality of frequency band signals and obtain a wavelet energy ratio and a wavelet entropy of each frequency band signal; acquiring the complexity of each channel electroencephalogram signal in the electroencephalogram signal without the basic emotional state, and forming electroencephalogram characteristics by the wavelet energy ratio and the wavelet entropy of each frequency band signal; arranging the electroencephalogram features to form a feature cube; and inputting the feature cube into the trained CNN model for emotion recognition. The accuracy of emotion recognition is improved.
Description
Technical Field
The invention relates to the technical field of emotion state recognition, in particular to an emotion recognition method and system based on a three-dimensional characteristic diagram and a convolutional neural network.
Background
The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art.
Emotion is a state that integrates human feelings, ideas, and behaviors, and is very important in human decision processing, interaction, and cognition. Currently, most emotional state classification studies use electroencephalography (EEG) and facial expressions to classify emotional states. EEG is a signal that records cortical surface activity and is the result of synaptic activation of neurons in the brain. In recent years, it has been shown that EEG is a suitable signal for biometric authentication and has an important function. Therefore, emotional state recognition based on EEG comes to mind, namely, the EEG is used for emotional state recognition, so that online psychotherapy and medical diagnosis are realized. The relationship between emotional state and brain activity is recorded in the electroencephalogram signal, and very subtle changes in emotional state are reflected with high time resolution. However, the electroencephalogram signal has the disadvantages of time asymmetry and instability, low signal-to-noise ratio, incapability of directly determining brain region reaction and the like. Therefore, EEG-based emotional state recognition remains a difficult task. Many researchers have proposed methods for emotion state recognition based on EEG, such as those based on convolutional neural networks, deep belief networks, convolutional neural networks, and the like.
In recent years, CNN is gradually adopted in the field, but there is a problem in how to convert a one-dimensional electroencephalogram signal into a form of a 3D feature map and to effectively combine the one-dimensional electroencephalogram signal with CNN, thereby achieving accurate recognition of human emotion.
The following technical problems exist in the prior art:
the single-class electroencephalogram feature emotion recognition accuracy is low, and the traditional classification model is easy to suffer from dimensional disasters and low in emotion recognition accuracy. Compared with a neural network, the extracted features determine the classification performance of the traditional classification model. The convolutional neural network in the neural network has the advantages of weight sharing, insusceptibility to influence of dimensional disasters, self-selection of features and the like.
The convolutional neural network has the capability of learning a local connection structure and developing a multi-scale hierarchical mode, and the efficiency of image processing, video processing and voice recognition tasks is improved. Electroencephalograms are composed of one-dimensional signals generated by each channel, and cannot provide spatial information for emotion recognition. At present, CNN is applied to the field of emotion recognition based on electroencephalogram, but the recognition accuracy is relatively low, and one reason for the CNN is that the information of the spatial relative position between electroencephalogram channels cannot be provided; the data volume of the two data is relatively small, and the overfitting probability of the model is increased; thirdly, the extracted electroencephalogram features have small contribution to emotion recognition.
Disclosure of Invention
In order to solve the problems, the emotion recognition method and system based on the three-dimensional feature map and the convolutional neural network are provided, the feature cube capable of reflecting electroencephalogram signal space information is obtained, emotion recognition is carried out through the feature cube, and the accuracy of emotion recognition is improved.
In order to achieve the purpose, the following technical scheme is adopted in the disclosure:
in a first aspect, a method for emotion recognition based on a three-dimensional feature map and a convolutional neural network is provided, which includes:
acquiring an electroencephalogram signal to be identified;
extracting an electroencephalogram signal without a basic emotional state from the electroencephalogram signal to be identified;
decomposing and reconstructing the electroencephalogram signals without the basic emotional state by wavelet packet transformation to obtain a plurality of frequency band signals, and obtaining a wavelet energy ratio and a wavelet entropy of each frequency band signal;
acquiring the complexity of each channel electroencephalogram signal in the electroencephalogram signals without the basic emotional state, and forming electroencephalogram characteristics with the wavelet energy ratio and the wavelet entropy of each frequency band signal;
arranging the electroencephalogram features to form a feature cube;
and inputting the feature cube into the trained CNN model for emotion recognition.
In a second aspect, a emotion recognition system based on a three-dimensional feature map and a convolutional neural network is provided, which includes:
the data acquisition module is used for acquiring an electroencephalogram signal to be identified;
the basic emotional state elimination module is used for extracting the electroencephalogram signals without the basic emotional state from the electroencephalogram signals to be identified;
the electroencephalogram characteristic acquisition module is used for decomposing and reconstructing electroencephalogram signals without a basic emotional state by adopting wavelet packet transformation, acquiring a plurality of frequency band signals, acquiring the wavelet energy ratio and the wavelet entropy of each frequency band signal, acquiring the complexity of each channel electroencephalogram signal in the electroencephalogram signals without the basic emotional state, and forming electroencephalogram characteristics with the wavelet energy ratio and the wavelet entropy of each frequency band signal;
the characteristic cube acquisition module is used for arranging the acquired electroencephalogram characteristics to form a characteristic cube;
and the emotion recognition module is used for inputting the feature cube into the trained CNN model for emotion recognition.
In a third aspect, an electronic device is provided, which includes a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein the computer instructions, when executed by the processor, perform the steps of the emotion recognition method based on the three-dimensional feature map and the convolutional neural network.
In a fourth aspect, a computer-readable storage medium is provided for storing computer instructions, which when executed by a processor, perform the steps of the emotion recognition method based on a three-dimensional feature map and a convolutional neural network.
Compared with the prior art, the beneficial effect of this disclosure is:
1. the electroencephalogram signal is subjected to wavelet packet transformation, the wavelet energy ratio and the wavelet entropy of each frequency band signal are extracted, the wavelet energy ratio can reflect the energy of each frequency band in the electroencephalogram signal, the wavelet entropy can reflect the order or disorder of signal spectrum energy distribution in each space, the approximate entropy can reflect the complexity of a time sequence and is used for measuring the complexity of the electroencephalogram signal, and a feature cube consisting of the approximate entropy, the wavelet energy ratio and the wavelet entropy provides space information for emotion recognition based on electroencephalogram, so that the emotion recognition accuracy is improved.
2. The method extracts the electroencephalogram signals without the basic emotional states from the electroencephalogram signals to be recognized, the recognition effect is better when the emotional states are recognized based on the electroencephalogram signals without the basic emotional states, and the accuracy of emotional recognition is guaranteed when the emotion recognition is performed by using the feature cube in the feature cube further acquired.
Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, are included to provide a further understanding of the application, and the description of the exemplary embodiments and illustrations of the application are intended to explain the application and are not intended to limit the application.
FIG. 1 is a flow chart of a method disclosed in example 1 of the present disclosure;
FIG. 2 is a diagram showing an arrangement of features disclosed in example 1 of the present disclosure;
fig. 3 is a diagram of a structure of a CNN model disclosed in embodiment 1 of the present disclosure.
The specific implementation mode is as follows:
the present disclosure is further described with reference to the following drawings and examples.
It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the disclosure. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of exemplary embodiments according to the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.
In the present disclosure, terms such as "upper", "lower", "left", "right", "front", "rear", "vertical", "horizontal", "side", "bottom", and the like indicate orientations or positional relationships based on orientations or positional relationships shown in the drawings, are only terms of relationships determined for convenience in describing structural relationships of the components or elements of the present disclosure, do not refer to any components or elements of the present disclosure, and are not to be construed as limiting the present disclosure.
In the present disclosure, terms such as "fixedly connected", "connected", and the like are to be understood in a broad sense, and mean either a fixed connection or an integrally connected or detachable connection; may be directly connected or indirectly connected through an intermediate. The specific meanings of the above terms in the present disclosure can be determined on a case-by-case basis by persons skilled in the relevant art or technicians, and are not to be construed as limitations of the present disclosure.
Example 1
In this embodiment, an emotion recognition method based on a three-dimensional feature map and a convolutional neural network is disclosed, which includes:
acquiring an electroencephalogram signal to be identified;
extracting an electroencephalogram signal without a basic emotional state from the electroencephalogram signal to be identified;
decomposing and reconstructing the electroencephalogram signals without the basic emotional state by wavelet packet transformation to obtain a plurality of frequency band signals, and obtaining a wavelet energy ratio and a wavelet entropy of each frequency band signal;
acquiring the complexity of each channel electroencephalogram signal in the electroencephalogram signals without the basic emotional state, and forming electroencephalogram characteristics with the wavelet energy ratio and the wavelet entropy of each frequency band signal;
arranging the electroencephalogram features to form a feature cube;
and inputting the feature cube into the trained CNN model for emotion recognition.
Further, a method for eliminating the basic emotional state is adopted to extract the electroencephalogram signals without the basic emotional state from the electroencephalogram signals to be identified.
Further, extracting a wavelet coefficient of each frequency band signal;
and calculating the wavelet characteristics of each frequency band signal according to the wavelet coefficient of each frequency band signal.
Furthermore, the complexity of each channel electroencephalogram signal in the electroencephalogram signals without the basic emotional state is calculated by adopting an approximate entropy algorithm.
Further, the CNN model includes an input layer, a convolutional layer, a reconstruction layer, a full link layer, and an output layer.
Further, a cross entropy function is adopted to define a loss function, and the CNN model is trained.
Further, the acquired electroencephalogram features are arranged according to the positions of the corresponding channels on the 2D plane graph to form a feature cube.
The emotion recognition method based on the three-dimensional characteristic diagram and the convolutional neural network disclosed by the embodiment is explained in detail, and the emotion recognition method based on the three-dimensional characteristic diagram and the convolutional neural network disclosed by the embodiment weakens the dependence of an electroencephalogram experiment on a test by using a basic emotion state elimination method, so that the experiment system has higher universality; the wavelet energy ratio may reflect the energy of each band in the EEG signal; the wavelet entropy is an extension of the wavelet energy ratio and can reflect the order or disorder of the signal spectrum energy distribution in each space; the approximate entropy can reflect the complexity of the time series and is used to measure the complexity of the EEG signal; the feature cube provides spatial information for electroencephalogram-based emotion recognition, and specifically comprises the following steps as shown in fig. 1:
s1: acquiring an electroencephalogram signal to be recognized, and acquiring an electroencephalogram signal without a basic emotional state by adopting a basic emotional state elimination method for the electroencephalogram signal to be recognized.
In specific implementation, a basic emotion state elimination method is adopted for the electroencephalogram signal to be recognized, and the specific process of acquiring the electroencephalogram signal without the basic emotion state is as follows:
initializing an input original electroencephalogram data set D, and for each tested brain of the original electroencephalogram signals to be identified in the data set DElectric signal SiIntercepting an experimental trial number i;
eliminating the basic emotional state for each trial i, and extracting a calm electroencephalogram signal SciAnd the EEG signal S after the experimenttiFrom StiMinus SciObtaining the electroencephalogram signal S without the basic emotional state after emotional stimulationsi。
S2: decomposing and reconstructing the electroencephalogram signals without the basic emotional state by wavelet packet transformation to obtain a plurality of frequency band signals, and obtaining the wavelet energy ratio and the wavelet entropy of each frequency band signal.
In specific implementation, wavelet packet transformation is adopted for the electroencephalogram signals without the basic emotional state to be divided into five frequency bands of Delta, Theta, Alpha, Beta and Gamma, and frequency band signals are obtained; and acquiring the wavelet coefficient of each frequency band signal, and calculating the wavelet characteristics of each frequency band signal according to the wavelet coefficient. The method specifically comprises the following steps:
method for decomposing non-basic emotional state electroencephalogram signal S by wavelet packetsiDecomposing and reconstructing to obtain 2 by decomposing j-layer wavelet packet of the EEG signaliA wavelet node;
reconstructing wavelet packet coefficients of nodes of the ith layer to obtain a reconstructed signal S of each nodej,mWherein m is 1, 2, …, 2jExtracting wavelet coefficient C from five frequency bands of Delta, Theta, Alpha, Beta and Gammai；
By CiAnd calculating the wavelet energy ratio and the wavelet entropy of each frequency band signal in each channel.
The wavelet energy ratio and the wavelet entropy are calculated as follows:
hypothesis CiExtracting wavelet coefficients after wavelet packet transformation, wherein i ═ Delta, Theta, Alpha, Beta and Gamma; the calculation of the total energy of the wavelet coefficients is defined as follows:
wavelet energy ratio eta of ith frequency bandiThe calculation is defined as follows:
the wavelet entropy calculation method of the ith frequency band is defined as follows:
Entropyi＝-ηiln(ηi) (3)
in this embodiment, wavelet packet decomposition is employed as db6 wavelet basis wavelet packet decomposition.
S3: the complexity of each channel electroencephalogram signal in the electroencephalogram signals without the basic emotional state is obtained, and the electroencephalogram characteristics are formed by the wavelet energy ratio and the wavelet entropy of each frequency band signal.
In specific implementation, an approximate entropy algorithm is adopted to have the complexity of each channel electroencephalogram signal in the basic emotional state, and the approximate entropy is specifically calculated by the following steps:
a) set a time sequence S in which
b) Determining parameters m and r, wherein m is an integer and represents the length of a vector to be reconstructed, and r is a non-negative real number and represents a similar threshold value between the reconstructed vectors;
c) reconstructing a time series
Wherein Xi＝[si,si+1,…,si+m-1]；
d) Calculating distance between arbitrary reconstruction vectors
Wherein the value range of i, j is 1 to N-m + 1;
e) counting the number of similar vectors of each reconstructed vector
Wherein the value range of i, j is 1 to N-m + 1;
f) order to
g) According to the steps a) to f), and then calculating phim+1；
h)ApEn＝Φm-Φm+1。
And (4) forming the electroencephalogram characteristics by the acquired complexity of each channel electroencephalogram signal, the wavelet energy ratio and the wavelet entropy of each frequency band signal.
S4: and arranging the electroencephalogram characteristics to form a characteristic cube.
In specific implementation, the extracted electroencephalogram features are subjected to feature arrangement according to a feature arrangement rule to form a feature cube; the electroencephalogram electrode is formed by selecting 32 electrode positions according to an international 10-20 system, namely Fp1, AF3, F3, F7, FC5, FC1, C3, T7, CP5, CP1, P3, P7, PO3, O1, Oz, Pz, Fp2, AF4, Fz, F4, F8, FC6, FC, Cz, C4, T8, Cp6, Cp2, P4, P8, PO4 and O2.
The extracted electroencephalogram features are arranged, namely the positions of the electrode channels are mapped to the positions on the 2D plane graph, and then the extracted electroencephalogram features are arranged according to the positions of the 2D plane graph to form a 9 x 11 feature cube.
S5: and inputting the feature cube into the trained CNN model for emotion recognition.
In specific implementation, a CNN model for feature fusion and classification recognition is constructed for a feature cube, and as shown in fig. 3, 1 input layer, 3 convolutional layers, 1 reconstruction layer, 2 full-link layers, and 1 output layer are designed in the model. The input layer is used for receiving the feature cube; the convolutional layer is used for extracting spatial information among electroencephalogram channels and performing weighted summation on characteristics among specific channels to form classification characteristics which are easy to identify by a model; the reconstruction layer is used for arranging the calculated results of the adjacent convolution layers to form a row vector or column vector form and forming a classifier on the full-connection layer and the output layer so as to achieve emotion recognition; the full connection layer is used for carrying out data dimension transformation on the data of the reconstruction layer to provide high-dimension identification information for emotion identification and classification; the output layer is used for judging the calculation result of the full connection layer and outputting the emotion recognition result.
Designing (setting) each layer function of CNN model
Is nth layer two-dimensional feature map size and number):
(1) input layer (L1): the input for each sample is l1A three-dimensional matrix of W H FN, where W H is the arrangement of the simulation electrodes on the scalp and FN is the number of features extracted from each channel.
(2) Convolutional layer (L2): the main function of this layer is to perform spatial filtering and fusion on the input element graph, so the connection between this layer and the input layer is a local connection. The size and the number of the filters are set as
W is the width of convolution kernel, H is the height of convolution kernel, Z is the number of characteristic maps in the previous layer, FN is the number of two-dimensional characteristic maps (or called as the number of filters) to be output, and the size of each obtained characteristic map is
The number of the two-dimensional characteristic graphs is
The reason for the convolution kernels being arranged as a matrix rather than a vector is that it is necessary to fuse spatial information to form abstract features that are easily recognized by the models designed by this disclosure.
(3) Convolutional layer (L3): the main function of this layer is to integrate the new features of L2 and achieve the dimensionality reduction effect. The size and the number of the filters of the third layer are set as
The size of each feature map obtained is
The number of the characteristic graphs is
(4) Convolutional layer (L4): the main function of this layer is the same as L3. The size and the number of the filters of the third layer are set as
The size of each feature map obtained is
The number of the characteristic graphs is
(5) Reconstituted layer (L5): this layer will receive the new feature computed by L4, setting the number of neurons to
Layers 6 to 7 constitute the classifier.
(6) Full tie layer (L6): this layer upscales the features of L5 to provide a high dimensional space for the classification of the last layer. The present disclosure sets the neurons of this layer to
(7) Output layer (L7): this layer outputs the emotional state using the calculation results of the L6 layer.
In this embodiment, the calculation modes of each layer of the model are designed:
is provided with
Is the output of the jth neuron in the nth feature map of the ith layer,
then it is the input of the jth neuron in the nth feature map of the ith layer, and the relationship between the two is shown below.
Where σ (-) is the activation function.
(1) Input layer (L1): l1WXHFN is a three-dimensional feature map and is also an input to the present model.
(2) Convolutional layer (L2): by using
A filter pair l1A filtering operation is performed and then a new feature map is obtained by calculation of the activation function.
Wherein is the operation of convolution,
is of size
The filter of (2) is preferably a filter,
is an offset.
(3) Convolutional layer (L3): by using
A filter pair l2A filtering operation is performed and then a new feature map is obtained by calculation of the activation function.
Wherein is the operation of convolution, and the operation of convolution,
is sizeIs composed of
The filter of (2) is set to be,
is an offset.
(4) Convolutional layer (L4): by using
A filter pair l3A filtering operation is performed and then a new feature map is obtained by calculation of the activation function.
Wherein is the operation of convolution,
is of the size
The filter of (2) is preferably a filter,
is an offset.
(5) Reconstituted layer (L5): the output elements of L4 are reconstructed as a column vector, i.e. o5(j)。
(6) Full tie layer (L6): all neurons in this layer are fully connected to the element of L5.
o6(j)＝σ(o5(j)W6+b6(j)) (8)
Wherein W6Is a fully connected matrix between the L5 and L6 layers,
is an offset.
(7) Output layer (L7): all neurons in this layer were fully connected to L6 neurons.
p＝σ(o6(j)W7+b7(j)) (9)
Wherein W7Is a fully connected matrix between the L6 and L7 layers,
is an offset.
The loss function of the CNN model constructed in this embodiment is defined as follows:
Loss＝cross_entropy(p,l)+α||W||2 (10)
wherein p is the output value of the model, W represents all the parameters to be trained in the model, and alpha is a regular term coefficient.
In order to accelerate the network convergence speed, the weights and deviations of all layers of the network are randomized into normal distribution, the mean value is 0, and the variance is 1/NinputIn which N isinputIs the number of upper features. Besides the L5 and L7 layers, the Batch-Norm technology is added to all other layers, so that the training time can be saved, and the overfitting possibility can be effectively reduced. Since the classification label is a one-hot encoding mode, a cross entropy function is used to define a loss function. Meanwhile, Adam optimizer and gradient descent algorithm are used in TensorFlow to adjust connection weights and bias. The maximum number of iterations is set to 10000 and the loss threshold is set to 0.01.
The method disclosed by the embodiment comprises 5 parts: acquiring an electroencephalogram signal to be identified, eliminating a basic emotional state, acquiring a wavelet energy ratio, a wavelet entropy, acquiring the complexity of the electroencephalogram signal, arranging characteristics and identifying an emotional state. By analysis, it was found that the wavelet energy ratio can reflect the energy of each band in the EEG signal; the wavelet entropy is an extension of the wavelet energy ratio and can reflect the order or disorder of the signal spectrum energy distribution in each space; the approximate entropy can reflect the complexity of the time series and is used to measure the complexity of the EEG signal; the feature cube (3D feature map) provides spatial information for electroencephalogram-based emotion recognition; the method comprises the steps of obtaining the electroencephalogram without the basic emotional state by adopting a basic emotional state elimination method, reconstructing and extracting wavelet coefficients of electroencephalogram frequency bands, obtaining corresponding wavelet characteristics by adopting wavelet energy ratio and wavelet entropy calculation modes according to the wavelet coefficients of five frequency bands of the electroencephalogram eliminated by the basic emotional state, estimating the overall complexity of the electroencephalogram by using approximate entropy, performing characteristic arrangement on the extracted characteristics to build characteristic cubes, and finally inputting the characteristic cubes into a CNN (computer network node) model for classification and prediction to further identify emotional states.
Example 2
In this embodiment, an emotion recognition system based on a three-dimensional feature map and a convolutional neural network is disclosed, including:
the data acquisition module is used for acquiring an electroencephalogram signal to be identified;
the basic emotional state elimination module is used for extracting the electroencephalogram signals without the basic emotional state from the electroencephalogram signals to be identified;
the electroencephalogram characteristic acquisition module is used for decomposing and reconstructing electroencephalogram signals without a basic emotional state by adopting wavelet packet transformation, acquiring a plurality of frequency band signals, acquiring the wavelet energy ratio and the wavelet entropy of each frequency band signal, acquiring the complexity of each channel electroencephalogram signal in the electroencephalogram signals without the basic emotional state, and forming electroencephalogram characteristics with the wavelet energy ratio and the wavelet entropy of each frequency band signal;
the characteristic cube acquisition module is used for arranging the acquired electroencephalogram characteristics to form a characteristic cube;
and the emotion recognition module is used for inputting the feature cube into the trained CNN model for emotion recognition.
Example 3
In this embodiment, an electronic device is disclosed, which comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein the computer instructions, when executed by the processor, perform the steps of the emotion recognition method based on the three-dimensional feature map and the convolutional neural network disclosed in embodiment 1.
Example 4
In this embodiment, a computer readable storage medium is disclosed for storing computer instructions which, when executed by a processor, perform the steps described in the emotion recognition method based on a three-dimensional feature map and a convolutional neural network disclosed in embodiment 1.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting the same, and although the present invention is described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: modifications and equivalents may be made to the embodiments of the invention without departing from the spirit and scope of the invention, which is to be covered by the claims.
Claims (9)
1. The emotion recognition method based on the three-dimensional characteristic diagram and the convolutional neural network is characterized by comprising the following steps of:
acquiring an electroencephalogram signal to be identified;
extracting an electroencephalogram signal without a basic emotional state from the electroencephalogram signal to be identified;
decomposing and reconstructing the electroencephalogram signals without the basic emotional state by wavelet packet transformation to obtain a plurality of frequency band signals, and obtaining a wavelet energy ratio and a wavelet entropy of each frequency band signal;
calculating the approximate entropy of each channel electroencephalogram signal in the electroencephalogram signals without the basic emotional state, and forming electroencephalogram characteristics with the wavelet energy ratio and the wavelet entropy of each frequency band signal;
arranging the electroencephalogram characteristics to form a characteristic cube;
inputting the feature cube into the trained CNN model for emotion recognition;
the approximate entropy is calculated specifically as follows:
step a: setting a time sequence S, wherein S1，s2，…，sN∈S，
Step b: determining parameters m and r, wherein m is an integer and represents the length of a vector to be reconstructed, and r is a non-negative real number and represents a similar threshold value between the reconstructed vectors;
step c: reconstructing a time series X1，X2，…，
Wherein Xi＝[si,si+1,…,si+m-1]；
Step d: calculating distance between arbitrary reconstruction vectors
Wherein the value range of i, j is 1 to N-m + 1;
step e: counting the number of similar vectors of each reconstructed vector
Wherein the value range of i, j is 1 to N-m + 1;
step f: order to
Step g: according to the steps a to f, calculating phim+1；
Step h: ApEn ═ Φm-Φm+1。
2. The emotion recognition method based on a three-dimensional feature map and a convolutional neural network as claimed in claim 1, wherein a method for eliminating a basic emotional state is adopted to extract an electroencephalogram signal without a basic emotional state from an electroencephalogram signal to be recognized.
3. The emotion recognition method based on a three-dimensional feature map and a convolutional neural network as claimed in claim 1, wherein wavelet coefficients of each band signal are extracted;
and calculating the wavelet characteristics of each frequency band signal according to the wavelet coefficient of each frequency band signal.
4. The emotion recognition method based on a three-dimensional feature map and a convolutional neural network of claim 1, wherein the CNN model includes an input layer, a convolutional layer, a reconstruction layer, a fully-connected layer, and an output layer.
5. The emotion recognition method based on a three-dimensional feature map and a convolutional neural network of claim 1, wherein the CNN model is trained by defining a loss function using a cross entropy function.
6. The emotion recognition method based on a three-dimensional feature map and a convolutional neural network as claimed in claim 1, wherein the acquired electroencephalogram features are arranged according to the positions of the corresponding channels on the 2D plane map to form a feature cube.
7. The emotion recognition system based on the three-dimensional characteristic diagram and the convolutional neural network is characterized by comprising:
the data acquisition module is used for acquiring an electroencephalogram signal to be identified;
the basic emotional state elimination module is used for extracting the electroencephalogram signals without the basic emotional state from the electroencephalogram signals to be identified;
the electroencephalogram characteristic acquisition module is used for decomposing and reconstructing the electroencephalogram signals without the basic emotional state by adopting wavelet packet transformation, acquiring a plurality of frequency band signals, acquiring the wavelet energy ratio and the wavelet entropy of each frequency band signal, calculating the approximate entropy of each channel electroencephalogram signal in the electroencephalogram signals without the basic emotional state, and forming electroencephalogram characteristics with the wavelet energy ratio and the wavelet entropy of each frequency band signal;
the characteristic cube acquisition module is used for arranging the acquired electroencephalogram characteristics to form a characteristic cube;
the emotion recognition module is used for inputting the feature cube into the trained CNN model for emotion recognition;
the approximate entropy is calculated specifically as follows:
step a: setting a time sequence S, wherein S1，s2，…，sN∈S，
Step b: determining parameters m and r, wherein m is an integer and represents the length of a vector to be reconstructed, and r is a non-negative real number and represents a similar threshold value between the reconstructed vectors;
step c: reconstructing a time series X1，X2，…，
Wherein Xi＝[si,si+1,…,si+m-1]；
Step d: calculating distance between arbitrary reconstruction vectors
Wherein the value range of i, j is 1 to N-m + 1;
step e: counting the number of similar vectors of each reconstructed vector
Wherein the value range of i, j is 1 to N-m + 1;
step f: order to
Step g: according to the steps a to f, calculating phim+1；
Step h: ApEn ═ Φm-Φm+1。
8. An electronic device comprising a memory and a processor, and computer instructions stored on the memory and executed on the processor, wherein the computer instructions, when executed by the processor, perform the steps of the method for emotion recognition based on a three-dimensional feature map and a convolutional neural network of any of claims 1-6.
9. A computer-readable storage medium storing computer instructions which, when executed by a processor, perform the steps of the method for emotion recognition based on a three-dimensional feature map and a convolutional neural network according to any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110272735.XA CN112990008B (en) | 2021-03-13 | 2021-03-13 | Emotion recognition method and system based on three-dimensional characteristic diagram and convolutional neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110272735.XA CN112990008B (en) | 2021-03-13 | 2021-03-13 | Emotion recognition method and system based on three-dimensional characteristic diagram and convolutional neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112990008A CN112990008A (en) | 2021-06-18 |
| CN112990008B true CN112990008B (en) | 2022-06-17 |
Family
ID=76335410
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110272735.XA Expired - Fee Related CN112990008B (en) | 2021-03-13 | 2021-03-13 | Emotion recognition method and system based on three-dimensional characteristic diagram and convolutional neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112990008B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113780134B (en) * | 2021-08-31 | 2023-05-02 | 昆明理工大学 | Motor imagery brain electrolysis code method based on SheffleNetV 2 network |
| CN113974627B (en) * | 2021-10-26 | 2023-04-07 | 杭州电子科技大学 | Emotion recognition method based on brain-computer generated confrontation |
| CN116269386B (en) * | 2023-03-13 | 2024-06-11 | 中国矿业大学 | Multichannel physiological time sequence emotion recognition method based on ordinal division network |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104538027A (en) * | 2014-12-12 | 2015-04-22 | 复旦大学 | Method and system for calculating emotion spreading of voice social contact media |
| CN111079465A (en) * | 2018-10-18 | 2020-04-28 | 北京入思技术有限公司 | Emotional state comprehensive judgment method based on three-dimensional imaging analysis |
| CN111616721A (en) * | 2020-05-31 | 2020-09-04 | 天津大学 | Emotion recognition system based on deep learning and brain-computer interface and application |
| CN112294341A (en) * | 2020-10-29 | 2021-02-02 | 武汉大学 | Sleep electroencephalogram spindle wave identification method and system based on light convolutional neural network |
-
2021
- 2021-03-13 CN CN202110272735.XA patent/CN112990008B/en not_active Expired - Fee Related
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104538027A (en) * | 2014-12-12 | 2015-04-22 | 复旦大学 | Method and system for calculating emotion spreading of voice social contact media |
| CN111079465A (en) * | 2018-10-18 | 2020-04-28 | 北京入思技术有限公司 | Emotional state comprehensive judgment method based on three-dimensional imaging analysis |
| CN111616721A (en) * | 2020-05-31 | 2020-09-04 | 天津大学 | Emotion recognition system based on deep learning and brain-computer interface and application |
| CN112294341A (en) * | 2020-10-29 | 2021-02-02 | 武汉大学 | Sleep electroencephalogram spindle wave identification method and system based on light convolutional neural network |
Non-Patent Citations (1)
| Title |
|---|
| "EmotioNet: A 3-D Convolutional Neural Network for EEG-based Emotion Recognition";Yi Wang.etc;《2018 International Joint Conference on Neural Networks (IJCNN)》;20181231;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112990008A (en) | 2021-06-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112990008B (en) | Emotion recognition method and system based on three-dimensional characteristic diagram and convolutional neural network | |
| Wang et al. | LSTM-based EEG classification in motor imagery tasks | |
| CN112667080B (en) | Intelligent control method for electroencephalogram signal unmanned platform based on deep convolution countermeasure network | |
| CN113693613B (en) | Electroencephalogram signal classification method, electroencephalogram signal classification device, computer equipment and storage medium | |
| CN114266276B (en) | Motor imagery electroencephalogram signal classification method based on channel attention and multi-scale time domain convolution | |
| CN108960299B (en) | Method for identifying multi-class motor imagery electroencephalogram signals | |
| Sun et al. | Graph convolution neural network based end-to-end channel selection and classification for motor imagery brain–computer interfaces | |
| WO2022183966A1 (en) | Electroencephalogram signal classification method and apparatus, device, storage medium and program product | |
| CN114533086A (en) | Motor imagery electroencephalogram decoding method based on spatial domain characteristic time-frequency transformation | |
| CN112450885B (en) | Epileptic electroencephalogram-oriented identification method | |
| CN115919330A (en) | EEG Emotional State Classification Method Based on Multi-level SE Attention and Graph Convolution | |
| CN113712571A (en) | Abnormal electroencephalogram signal detection method based on Rinyi phase transfer entropy and lightweight convolutional neural network | |
| CN112155549A (en) | ADHD disease diagnosis aid decision-making system based on deep convolution pulse neural network | |
| Ramakrishnan et al. | Epileptic eeg signal classification using multi-class convolutional neural network | |
| Gómez-Tapia et al. | On the minimal amount of EEG data required for learning distinctive human features for task-dependent biometric applications | |
| CN112259228B (en) | Depression screening method by dynamic attention network non-negative matrix factorization | |
| CN117574059A (en) | High-resolution brain-electrical-signal deep neural network compression method and brain-computer interface system | |
| CN114027786A (en) | Sleep disordered breathing detection method and system based on self-supervision memory network | |
| Huang et al. | Modeling task fMRI data via mixture of deep expert networks | |
| Sridhar et al. | A Neural Network Approach for EEG classification in BCI | |
| CN107256408B (en) | Method for searching key path of brain function network | |
| Khalkhali et al. | Low latency real-time seizure detection using transfer deep learning | |
| CN114428555B (en) | Electroencephalogram movement intention recognition method and system based on cortex source signals | |
| Immanuel et al. | Analysis of different emotions with bio-signals (EEG) using deep CNN | |
| CN115721323A (en) | Brain-computer interface signal identification method and system and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20220617 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |