CN117562542A

CN117562542A - Emotion recognition method based on electroencephalogram signals, computer equipment and storage medium

Info

Publication number: CN117562542A
Application number: CN202410063298.4A
Authority: CN
Inventors: 胡方扬; 魏彦兆; 李宝宝; 唐海波
Original assignee: Xiaozhou Technology Co ltd
Current assignee: Xiaozhou Technology Co ltd
Priority date: 2024-01-17
Filing date: 2024-01-17
Publication date: 2024-02-20
Anticipated expiration: 2044-01-17
Also published as: CN117562542B

Abstract

The application discloses an emotion recognition method, computer equipment and storage medium based on an electroencephalogram signal, wherein the method comprises the steps of obtaining the electroencephalogram signal generated by a user under a preset stimulus source comprising a positive emotion stimulus source, a negative emotion stimulus source and a neutral emotion stimulus source; performing characteristic processing on the electroencephalogram signals to obtain first characteristic data which comprises frequency domain data and time domain data and corresponds to a first preset frequency range of the electroencephalogram signals; confirming a target frequency range in a first preset frequency range according to the frequency domain data, amplifying time domain data corresponding to the target frequency range, and obtaining second characteristic data; acquiring target feature data for feature fusion of the first feature data and the second feature data; and inputting the target characteristic data into a pre-trained emotion recognition model, generating an emotion recognition result, and completing emotion recognition of the electroencephalogram signals. According to the method, the key emotion related features are effectively enhanced, so that the expression and recognition capability of the key emotion features can be remarkably improved.

Description

Emotion recognition method based on electroencephalogram signals, computer equipment and storage medium

Technical Field

The invention belongs to the technical field of brain-computer interfaces, and particularly relates to an emotion recognition method based on brain-computer signals, computer equipment and a storage medium.

Background

Emotion computing technology is widely required in the fields of man-machine interaction, education and medical treatment and the like. Identifying the emotional state of the individual is significant in improving the intelligence and humanization of human-computer interaction. The brain electrical signal is used as a physiological signal directly reflecting brain activities, and has unique advantages in emotion calculation. However, since the electroencephalogram signals are susceptible to various noises and exhibit time-varying and nonlinear characteristics, the direct application to emotion recognition still faces a plurality of difficulties.

The current emotion recognition research based on brain electricity mainly adopts the following technology that 1) a frequency domain analysis method is adopted, and the correlation between different frequency bands such as alpha, beta, gamma and the like and specific emotion is analyzed through fast Fourier transformation, and the frequency spectrum is used as a characteristic. The method ignores time domain information and cannot locate emotion characteristics of key time. 2) The time-frequency analysis method applies wavelet transformation and the like to obtain a time-frequency diagram which represents the time evolution characteristics of the brain electrical signals. However, this method gives uniform weights to each time and each frequency band, and cannot focus on the critical area. 3) The deep learning method uses a convolutional neural network to directly perform end-to-end learning on the original brain electrical signals. This approach is sensitive to noise and it is difficult to obtain a stable representation of the features.

In summary, the existing method has the defects in the aspects of noise suppression, key feature expression, time-frequency information fusion and adaptation to the nonstationary characteristics of the electroencephalogram signals. The lack of effective pretreatment and feature extraction of signals, and the direct deep learning-based method also faces the problems of unstable training process, poor model interpretation and the like. Therefore, how to propose an electroencephalogram emotion recognition method which can effectively represent key emotion characteristics and can remarkably improve recognition performance is a technical problem which needs to be solved currently.

Disclosure of Invention

The embodiment of the application provides an emotion recognition method, computer equipment and storage medium based on an electroencephalogram signal, which can solve the technical problem of how to stably and accurately decode complex dynamic intention of a user.

In a first aspect, an embodiment of the present application provides an emotion recognition method based on an electroencephalogram signal, including:

acquiring an electroencephalogram signal generated by a user under a preset stimulus source; the preset stimulus source comprises a positive emotion stimulus source, a negative emotion stimulus source and a neutral emotion stimulus source;

performing characteristic processing on the electroencephalogram signals to obtain first characteristic data of a first preset frequency range corresponding to the electroencephalogram signals; wherein the first characteristic data includes frequency domain data and time domain data;

Confirming a target frequency range in the first preset frequency range according to the frequency domain data, amplifying time domain data corresponding to the target frequency range, and acquiring second characteristic data according to the frequency domain data corresponding to the target frequency range and the amplified time domain data;

performing feature fusion on the first feature data and the second feature data to obtain target feature data;

inputting the target characteristic data into a pre-trained emotion recognition model, and generating an emotion recognition result by the emotion recognition model according to the target characteristic data to complete emotion recognition of the electroencephalogram signal; wherein the emotion recognition result at least comprises positive emotion, negative emotion and neutral emotion.

In some implementations of the first aspect, the first preset frequency range includes at least an alpha frequency range, a beta frequency range, and a gamma frequency range; the identifying a target frequency range in the first preset frequency range according to the frequency domain data includes:

respectively acquiring a first energy enhancement value of frequency domain data corresponding to the alpha wave frequency range, a second energy enhancement value of frequency domain data corresponding to the beta wave frequency range and a third energy enhancement value of frequency domain data corresponding to the gamma wave frequency range;

And determining the frequency range corresponding to the maximum value in the first energy enhancement value, the second energy enhancement value and the third energy enhancement value as the target frequency range.

In some implementations of the first aspect, the feature fusing the first feature data and the second feature data to obtain target feature data includes:

acquiring third characteristic data according to frequency domain data and time domain data corresponding to a frequency range except the target frequency range in the alpha wave frequency range, the beta wave frequency range and the gamma wave frequency range;

and carrying out feature fusion on the first feature data, the second feature data and the third feature data to obtain target feature data.

inputting the first characteristic data and the second characteristic data into a pre-trained characteristic fusion model, wherein the characteristic fusion model stores a weight mapping relation, and the weight mapping relation comprises a preset weight value and a frequency range corresponding to the preset weight value;

and acquiring the target feature data generated by feature fusion of the first feature data and the second feature data by the feature fusion model according to the weight mapping relation.

In some implementations of the first aspect, before the identifying a target frequency range in the first preset frequency range according to the frequency domain data, the method further includes:

performing first-order differential calculation on the first characteristic data to obtain a first-order differential value corresponding to the first characteristic data;

performing second-order differential calculation on the first characteristic data to obtain an acceleration value corresponding to the first characteristic data;

respectively performing time domain differentiation and frequency domain differentiation on the first characteristic data to obtain a first partial derivative value and a second partial derivative value;

and acquiring a fluctuation feature matrix according to the first-order differential value, the acceleration value, the first partial derivative value and the second partial derivative value, and updating the first feature data according to the fluctuation feature matrix.

performing first-order difference in the target frequency range according to the fluctuation feature matrix to obtain target fluctuation data;

and carrying out feature fusion on the updated first feature data and the target fluctuation data to obtain target feature data.

In some implementations of the first aspect, the target feature data includes a target feature matrix; before the inputting of the target feature data into the pre-trained emotion recognition model, the method further comprises:

performing feature segmentation on the target feature matrix at preset time intervals to obtain a plurality of sub-target feature matrices;

obtaining feature vectors in each sub-target feature matrix, and calculating a correlation coefficient matrix and a divergence matrix corresponding to the sub-target feature matrix according to the feature vectors;

obtaining a matching interference degree matrix corresponding to the sub-target feature matrix according to the correlation coefficient matrix and the divergence matrix;

and inputting the matched interference degree matrix into a pre-trained attention fusion model, generating an attention weight matrix by the attention fusion model according to the matched interference degree matrix, and fusing the attention weight matrix and the target feature matrix based on a self-attention mechanism to update the target feature matrix.

In some implementations of the first aspect, the emotion recognition model includes any one of a convolutional neural network or a recurrent neural network; before the inputting of the target feature data into the pre-trained emotion recognition model, the method further comprises:

Acquiring emotion information corresponding to the electroencephalogram signals;

inputting the target characteristic data into an emotion recognition model to be trained, and obtaining a predicted emotion recognition result generated by the emotion recognition model to be trained;

and training the sensory recognition model according to the predicted emotion recognition result and the emotion information.

In a second aspect, the present application further provides a computer device, including a processor and a memory, where the memory is configured to store a computer program, where the computer program is executed by the processor to implement the emotion recognition method based on electroencephalogram signals according to the first aspect.

In a third aspect, the present application further provides a computer readable storage medium storing a computer program, which when executed by a processor implements the electroencephalogram signal-based emotion recognition method according to the first aspect.

Compared with the prior art, the application has the following beneficial effects:

acquiring an electroencephalogram signal generated by a user under a preset stimulus source comprising a positive emotion stimulus source, a negative emotion stimulus source and a neutral emotion stimulus source, and acquiring electroencephalogram signals representing various emotions of the user, so that the comprehensiveness of a training sample is improved; performing characteristic processing on the electroencephalogram signals to obtain first characteristic data of a first preset frequency range corresponding to the electroencephalogram signals, and selecting a specific first preset frequency range to screen out the electroencephalogram signals capable of expressing preset characteristics; confirming a target frequency range in a first preset frequency range according to the frequency domain data, amplifying time domain data corresponding to the target frequency range, and acquiring second characteristic data according to the frequency domain data corresponding to the target frequency range and the amplified time domain data, so that key emotion related characteristics can be effectively enhanced; and carrying out feature fusion on the first feature data and the second feature data to obtain target feature data, carrying out cross-band feature fusion by using complementary emotion information contained in different electroencephalogram frequency intervals, learning internal association among different frequency components, and dynamically adjusting weight distribution of different time-space points. Compared with the traditional flat attention structure, the hierarchical design can more comprehensively express complex emotion characteristics through cross-level collaborative learning.

In addition, the application also fuses various characteristic expression means such as time-frequency analysis, pearson correlation, JS divergence measurement and the like. The multi-model fusion improves the robustness of the features and enhances the diversity of emotion representation. Through layer-by-layer feature deepening, the finally formed feature expression matrix integrates multiple aspects of information of the original electroencephalogram data, and particularly, key emotion related features are effectively enhanced.

In conclusion, the expression and recognition capability of key emotion characteristics can be remarkably improved. Compared with a single model, the integrated multiple feature extraction and fusion modules work cooperatively to form an information-rich and organically fused feature representation space.

Drawings

Fig. 1 is a schematic flow chart of an emotion recognition method based on an electroencephalogram signal according to an embodiment of the present application;

fig. 2 is a schematic structural diagram of an emotion recognition system based on an electroencephalogram signal according to an embodiment of the present application;

fig. 3 is a schematic structural diagram of a computer device according to an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.

Referring to fig. 1, fig. 1 is a schematic flow chart of an emotion recognition method based on an electroencephalogram according to an embodiment of the present application. The electroencephalogram signal-based emotion recognition method can be applied to computer equipment, wherein the computer equipment comprises, but is not limited to, smart phones, notebook computers, tablet computers, desktop computers, physical servers, cloud servers and the like. As shown in fig. 1, the emotion recognition method based on an electroencephalogram signal of the present embodiment includes steps S101 to S105, and is described in detail as follows:

step S101, acquiring an electroencephalogram signal generated by a user under a preset stimulus source; the preset stimulus sources comprise a positive emotion stimulus source, a negative emotion stimulus source and a neutral emotion stimulus source.

In this step, a flexible, adjustable electrode array is used as an electroencephalogram signal acquisition device, which can be easily attached to the scalp surface to prevent electrode movement and signal interruption, ensure stability of electrode position, so as to maintain continuity of signals during long-time acquisition and provide a stable signal acquisition environment. Each electrode is connected to a signal amplifier and transmits a signal through a guide wire into a receiver. The provided method requires pre-screening of a subject with a predisposed emotion to obtain an obvious emotional response electroencephalogram signal. These people may have a high neuro-cytoplasmic character and may develop significant brain electrical responses in the face of emotional stimuli. Simultaneously preparing three types of external stimulus materials, namely positive, negative and neutral, and carrying out different types of emotion excitation on the tested materials.

Alternatively, a method is provided for selecting a 32-lead electrode cap that meets international standards, and determining electrode placement with reference to a 10-20 system. The 10-20 system is a standard system for equally distributing sampling points on the whole scalp area according to the anatomical position of the scalp of a person, and comprises key parts such as forehead, top, rear part, temporal parts on two sides and the like. For example, fpz denotes the forehead median, cz is the top median, pz is the occipital median, T3 denotes the left temporal, T4 is the right temporal, etc., a total of 32 sampling lead points are determined. By adopting the electrode layout distributed at intervals, the brain electrical activity signals of different parts of the scalp can be obtained.

Alternatively, the positive stimulus material may include beautiful natural scenes, cheerful music, humorous video clips, etc., the negative stimulus material may include offensive horror images, low-mood sad music, thrill nervous movie clips, etc., and the neutral stimulus material may include pictures of ordinary buildings, fluent classical music, general recorded video, etc. The three types of stimulation materials are combined into a stimulation sequence according to a certain sequence, and the stimulation sequence is sequentially presented to a tested person for watching or listening. So that the acquired electroencephalogram signals can effectively characterize positive, negative and neutral emotions.

Optionally, the brain electrical activity data of the subject is acquired simultaneously during the subject's viewing of the listening stimulation sequence. And the testee scores the current emotional state according to subjective feeling, for example, 1-9 points are used for representing the emotional state, 1 points represent the very negative state, 9 points represent the very positive state and 5 points are used for representing the neutral state. And taking the subjective scoring results as emotion labels, and recording the corresponding relation between the stimulus type and the emotion labels. Meanwhile, the gesture action of the tested person needs to be controlled, and the interference of excessive irrelevant physiological activities on signals is avoided. During the acquisition phase, care is taken to electromagnetically shield the device to reduce the effect of external noise on the signal quality. The acquired original brain electrical signals are digitized by an analog-to-digital converter and then stored, and generally, the sampling frequency of 250Hz or 500Hz is selected for discretization.

Optionally, in addition to the electroencephalogram signal, multimodal fusion is performed in combination with other various biological signals, such as eye movements, facial expressions, heart rate, etc. Through integrating and decoding the signals acquired by different sensors, the intention and the internal state of the user can be more comprehensively understood, and the accuracy and the adaptability of brain-computer interaction are improved.

Step S102, performing feature processing on the electroencephalogram signals to obtain first feature data of a first preset frequency range corresponding to the electroencephalogram signals; wherein the first characteristic data includes frequency domain data and time domain data.

In this embodiment, the electroencephalogram signal needs to be expressed from two dimensions of the time domain and the frequency domain to obtain richer first feature data containing more emotion.

Alternatively, the main purpose of the frequency domain analysis of the electroencephalogram signal is to analyze the situation where the signal energy is distributed in different frequency intervals. The frequency domain analysis can be based on a fast fourier transform (Fast Fourier Transform, FFT) method and the like to obtain the spectral representation of the brain electrical signal and display the magnitudes of different frequency components.

When the fast fourier transform is adopted for frequency domain analysis, the calculation formula is as follows:

X(k) = ∑_{n=0}^{N-1} x(n) e^{-j2πkn/N} , k = 0,1,...,N-1；

wherein X (N) represents a time domain signal with a length of N, j represents an imaginary unit, k is a frequency index, and X (k) represents an FFT conversion coefficient at a kth frequency point and represents a complex amplitude of the frequency component.

Optionally, a frequency domain feature extraction algorithm used for extracting frequency domain data of the electroencephalogram signal is used for extracting relevant features from spectrum distribution of the signal, and the frequency domain feature extraction algorithm includes, but is not limited to:

Mean frequency (MeanFrequency): mf=Σ (fi×pi)/Σ (Pi), where fi represents the i-th frequency point and Pi represents the power spectral density of the corresponding frequency point.

Energy band ratio (BandPowerRatio): bp=Σ (Pi)/Σ (Pj), where Pi and Pj represent power spectral densities in different frequency band ranges, the energy distributions of the different frequency bands can be compared.

Variance frequency (VarianceFrequency): vf=Σ ((fi-mf) Pi)/Σ (Pi), where fi represents the i-th frequency point, pi represents the power spectral density of the corresponding frequency point, and mf represents the mean frequency.

Peak of spectrum (peakamplite): pk=max (ASD), where ASD represents the amplitude spectral density, taking the maximum value of the spectrum.

Frequency characteristic percentage (frequency featurescore): ffp = Σ (Pi)/Σ (ASD), where Pi represents the power spectral density of the corresponding frequency bin and ASD represents the amplitude spectral density.

Optionally, the time domain analysis of the electroencephalogram signal mainly includes that a time-frequency diagram can be obtained through short-time fourier transformation, and the frequency components and the intensities of the frequency components are corresponding to different moments of the signal on a time axis. The short-time Fourier transform calculation formula is as follows:

S(t, f) = sum_{n=-inf}^{inf} x(n) w(n-t) e^{-j2πfn}；

where x (n) is the original signal, n is the time sample index, w (n) and w (n-t) are sliding window functions, t is the time index, f is the frequency bin, e-j 2 pi fn is the exponential kernel, containing the imaginary unit j. The short-time Fourier transform is performed by sliding windows on the signals to obtain local signals at different moments t and performing Fourier transform to obtain frequency spectrums at corresponding moments. Finally, an original time-frequency matrix S (t, f) is obtained, which represents the time-frequency distribution of the brain electrical signals.

Optionally, the time domain feature extraction algorithm for extracting time domain data of the electroencephalogram signal is mainly applied to the time domain signal and used for describing temporal features of the signal. Time domain feature extraction algorithms include, but are not limited to:

mean (Mean) = (x1+x2+ … +xn)/n, where xi represents the i-th sample point and n represents the number of sample points.

Variance (Variance): σ= ((x 1- μ) x+ (x 2- μ) x+ … + (xn- μ) j,)/n, where, mu represents the average value, xi represents the ith sample point, and n represents the number of sample points.

Standard deviation (standard device): σ= v (σ (j)), where σ represents variance.

Maximum value (Maximum): max=max (x 1, x2, …, xn), where xi represents the i-th sample point.

Minimum (Minimum): min=min (x 1, x2, …, xn), where xi represents the i-th sample point.

Slope (Slope): slope= (xn-x 1)/(n-1), where xi represents the i-th sample point and n represents the number of sample points.

In some embodiments, the frequency range corresponding to the electroencephalogram signal is a second preset frequency range, and the second preset frequency range is greater than the first preset frequency range; before the feature processing is performed on the electroencephalogram signal, and the step S102 is performed to obtain first feature data of a first preset frequency range corresponding to the electroencephalogram signal, the method further includes: inputting the electroencephalogram signals into a preset filter to obtain electroencephalogram signals in the first preset frequency range; wherein the predetermined filter comprises at least one or more of a band reject filter, a finite impulse response filter, or an infinite impulse response filter.

Because various physiological noises and external interference can influence the original brain electrical signals, the collected signals need to be preprocessed so as to improve the reliability of subsequent analysis. According to the noise frequency distribution characteristics and the energy size characteristics recorded in the early-stage test, a digital filter can be designed to denoise the signals.

Physiological noise mainly includes two major classes, electrooculogram and myoelectricity. The eye electricity is generally concentrated in the low frequency band, mainly distributed at 0-1Hz, and the secondary energy is concentrated in the higher frequency range of 15-30 Hz. Myoelectric noise is mainly distributed in the higher 90-200Hz frequency range. Therefore, a band-stop filter can be designed, a proper passband frequency range is selected, and the effective components of the electroencephalogram signals are reserved while noise is filtered. For example, by designing a band-stop filter with a passband in the range of 1-70Hz, the electro-oculogram power lower than 1Hz and the myoelectric noise higher than 70Hz can be effectively eliminated, and the main alpha, beta, gamma and other rhythms of the electroencephalogram signals are reserved.

Alternatively, the digital filter may select an FIR (finite impulse response) or IIR (infinite impulse response) structure. And determining the type of the filter and related design parameters according to the characteristics of the noise power spectrum. For example, for electro-oculogram noise, IIR butterworth filters may be selected, while for some random short-time random disturbances, FIR filters work better. The order of the filter needs to be set during design, and the ripple index of the passband is controlled to obtain good smooth frequency response. And meanwhile, the attenuation index of the stop band is adjusted, so that the noise is fully restrained. Besides designing a filter according to typical noise spectrum statistical parameters, after obtaining the tested brain electrical signals, frequency domain analysis can be carried out on the signals, spectrum distribution of the signals is observed, so that the actual effective frequency band range is determined, and the optimal filter cut-off frequency or passband parameters are selected to realize self-adaptive filtering. The electroencephalogram signals after digital filtering can greatly reduce the influence of various physiological noises and external random interference, and improve the signal quality for subsequent feature extraction and model training. For example, one test original brain electrical signal has 0-5Hz low frequency eye electrical noise and 120-200Hz high frequency myoelectrical noise. For this case, a set of low pass filters of 1-30Hz and a set of high pass filters below 100Hz may be designed for cascaded filtering. The processed clean electroencephalogram signals can better retain key rhythm components such as alpha waves, beta waves, gamma waves and the like, and the analyzability of emotion related features is improved.

Step S103, confirming a target frequency range in a first preset frequency range according to the frequency domain data, amplifying time domain data corresponding to the target frequency range, and acquiring second characteristic data according to the frequency domain data corresponding to the target frequency range and the amplified time domain data.

By analyzing the first characteristic data corresponding to the electroencephalogram signals, which target frequency ranges have key effects on the emotion recognition task can be determined, and then the time domain data corresponding to the target frequency ranges are amplified, so that a basis can be provided for the subsequent construction of frequency domain characteristics.

In some embodiments, the first preset frequency range includes at least an alpha frequency range, a beta frequency range, and a gamma frequency range; the step S103 includes steps S1031 to S1032, which are specifically as follows:

step S1031, respectively obtaining a first energy enhancement value of the frequency domain data corresponding to the α -wave frequency range, a second energy enhancement value of the frequency domain data corresponding to the β -wave frequency range, and a third energy enhancement value of the frequency domain data corresponding to the γ -wave frequency range.

Through FFT analysis, the energy of the brain electrical signal is mainly concentrated in different frequency rhythm corresponding frequency bands such as delta wave (0-4 Hz), theta wave (4-8 Hz), alpha wave (8-13 Hz), beta wave (13-30 Hz), gamma wave (30-50 Hz) and the like. In the methods provided herein, the above-described brain electrical rhythms are believed to be highly correlated with the mood of humans, with reference to existing reports. For example, the alpha wave is associated with an individual's relaxed state, the beta wave is associated with a positive emotion, and the gamma wave is associated with a negative emotion of anxiety, tension, etc. Accordingly, the brain electrical signal FFT power spectrum P (k) of the α wave, the β wave, and the γ wave is analyzed as a first energy boost value, a second energy boost value, and a third energy boost value, respectively. The energy enhancement value is the power spectrum of the signal, namely the square of the amplitude at each frequency point, and the calculation formula is as follows:

P(k) = |X(k)|^2；

Wherein P (k) is an energy boost value; k is the frequency index; x (k) is the FFT transform coefficient at the kth frequency bin and P (k) is used to reflect the signal energy distribution at each frequency k.

Step S1032, determining the frequency range corresponding to the maximum value of the first energy enhancement value, the second energy enhancement value and the third energy enhancement value as the target frequency range.

By determining the first energy enhancement value, the second energy enhancement value, and the third energy enhancement value of the α wave, the β wave, and the γ wave, it is possible to determine a frequency range at a maximum of the first energy enhancement value, the second energy enhancement value, and the third energy enhancement value as a target frequency range [ fmin, fmax ], which has a key effect on the emotion recognition task. These bands are typically associated with the frequency bands of the alpha, beta, gamma, etc. rhythms. The method and the device provide basis for the subsequent construction of frequency domain features by determining the key high-frequency ranges.

Illustratively, the provided method first performs a 2048-point FFT to obtain a complete power spectrum P (k) of 0-100 Hz. Then analyzing the relative energy change of the three frequency bands of the alpha wave (8-13 Hz), the beta wave (13-30 Hz) and the gamma wave (30-50 Hz) under different emotion states. If the relative energy enhancement percentage of the gamma wave band is the maximum under the stimulation of the negative emotion, determining that the key frequency band range is [30Hz, 50Hz ], and enhancing the weight of the frequency range in the subsequent feature extraction so as to improve the identification effect of the negative emotion.

Optionally, in the present application, amplifying the time domain data corresponding to the target frequency range, by obtaining an index coefficient of the time domain data corresponding to the target frequency range in a preset index transformation function, and increasing the index coefficient, amplification of the time domain data corresponding to the target frequency range is completed; the preset exponential transformation function is as follows:

the scaling ratio of the time domain data is the time domain data, the scaling ratio of the time domain data is the first preset frequency, the exponential coefficient is the exponential function, and the amplitude corresponding to the time domain data is the exponential function.

In order to strengthen the emotion related characteristics of the key high frequency range in the time-frequency matrix, the authority adopts exponential mapping as a nonlinear transformation function, for example, gamma wave band (30-50 Hz) is selected as the key high frequency range, and y is an output result after exponential function conversion. By adjusting the positive and negative values of the parameter b, nonlinear mapping for amplifying or reducing the frequency range corresponding to the input x can be realized.

In some embodiments, such as where the gamma band (30-50 Hz) is determined by analysis to be more sensitive to the current emotion recognition task, it is desirable to enhance the characterization of that band in a time-frequency matrix. The method and the device extract the time-frequency component amplitude x corresponding to the gamma wave band in the time-frequency matrix to serve as the input of the index mapping. Setting a parameter a=1, keeping the overall proportion unchanged, and setting a positive value of 0.5 to a parameter b to construct an exponential gain mapping function. Then, the input amplitude x of the gamma wave band is substituted into an exponential function y=exp (0.5 x) to perform nonlinear transformation, so as to obtain a new output y. Here, along with the increase of x, the exponential function can realize the amplification mapping of the output y to the input x, so as to achieve the purpose of enhancing the gamma-band time-frequency characteristic. And combining the mapped gamma-band time-frequency components y back to corresponding time-frequency positions in the original time-frequency matrix to replace the original gamma-band components, thereby obtaining a new time-frequency matrix S_gamma (t, f) with enhanced key gamma-band information.

In order to verify the effectiveness of parameter setting, the time-frequency diagram of the original time-frequency matrix and the matrix with enhanced index mapping in the gamma wave band is visually compared. At the moment corresponding to negative emotion stimulation of an experimental brain electrical signal, a weak short-time power rising part exists in an original gamma wave band time-frequency diagram, and the weak short-time power rising part represents potential negative emotion fluctuation characteristics. After exponential transformation, in the gamma-band time-frequency diagram at the same moment, the pulse characteristics are obviously amplified, the fluctuation mode is clearer and more discernable than the original diagram, and the recognition degree of the key emotion characteristics is effectively improved.

Compared with linear mapping, the customized exponential transformation function can pertinently amplify key frequency dimensions in the time-frequency matrix, so that a frequency band more sensitive to emotion recognition tasks is enhanced. The technical means can improve the expression capability and separability of key features and provide input containing more obvious emotion information for subsequent construction of an identification model.

In some embodiments, before said identifying a target frequency range in said first preset frequency range from said frequency domain data, said method further comprises: performing first-order differential calculation on the first characteristic data to obtain a first-order differential value corresponding to the first characteristic data; performing second-order differential calculation on the first characteristic data to obtain an acceleration value corresponding to the first characteristic data; respectively performing time domain differentiation and frequency domain differentiation on the first characteristic data to obtain a first partial derivative value and a second partial derivative value; and acquiring a fluctuation feature matrix according to the first-order differential value, the acceleration value, the first partial derivative value and the second partial derivative value, and updating the first feature data according to the fluctuation feature matrix.

The electroencephalogram signals often exhibit sudden wave characteristics due to emotional correlations. In order to enhance the key characteristic, the time-frequency expression corresponding to the embodiment of the application can be subjected to differential operation, and fluctuation components in the time-frequency expression can be extracted.

For example, when the γ -band signal is the target band, the trend of the time-frequency matrix corresponding to the first characteristic data at the adjacent time may be represented by first-order difference. Let the time-frequency matrix value at the frequency f be S (t, f) at the time t, then perform the first order difference calculation:

ΔS(t,f) = S(t,f) - S(t-1,f)；

Δs (t, f) represents the first-order difference result of S, which reflects the amount of change in the frequency component at that time with respect to the previous time. The emotion-related sudden brain wave motion can cause the time-frequency matrix to locally generate a large-amplitude first-order differential value.

Further performing second-order difference, and obtaining the acceleration representation of the feature matrix of the first feature data:

δS(t,f) = ΔS(t,f) - ΔS(t-1,f)

δs (t, f) represents a second-order difference result of Δs, and the acceleration of the change of the time-frequency matrix is represented by calculating a difference value between the current moment and the previous two moments.

The burst characteristic representing key emotion fluctuation in the time-frequency matrix can be highlighted through continuous first-order second-order differential operation.

And the partial derivative calculation is performed by expanding the difference technology to a bivariate function. For example, time-domain differentiation of the time-frequency matrix ∂ S/∂ t=s (t, f) -S (t-1, f) to represent time-varying fluctuations of the specific frequency components.

Similarly, frequency domain differentiation may be performed, ∂ S/∂ f=s (t, f) -S (t, f-1) reflecting the relative change of each frequency component at a fixed time. The dynamic fluctuation effect of the feature matrix can be further enhanced by combining time domain and frequency domain bivariate differentiation.

And then, superposing the differential and partial derivative results to obtain a new feature matrix P (t, f) containing comprehensive dynamic fluctuation information. And taking the time-frequency matrix after the difference operation as a new characteristic layer to represent the fluctuation component of the brain electrical data with the steady-state background removed. Compared with the original linear time-frequency characteristic, the method remarkably enhances key emotion-related burst characteristics and provides characteristic expression with higher recognition degree.

And finally, performing first-order difference on the gamma frequency band of the fluctuation feature matrix P (t, f) to obtain Pgamma (t, f), and obtaining the burst dynamic change process of the frequency component. Specific: taking the corresponding element of gamma frequency band (30-50 Hz) in the P (t, f) matrix, namely the Pgamma (t, f) submatrix. First order difference calculation is performed on the pγ (t, f) submatrix, Δpγ (t, f) =pγ (t, f) -pγ (t-1, f).

Further, by calculating Δpγ (t, f), the difference between adjacent time points can be obtained when the γ band signal is the target band. Thereby effectively expressing key mood swings. The first order difference is only performed because the gamma band itself is already the critical emotion related band determined by prior analysis and only the sudden change thereof needs to be emphasized, without the second order difference being needed to further amplify the features. Compared with a source time-frequency matrix, the newly extracted fluctuation features can effectively highlight key emotion related components, remove steady-state background interference and improve emotion recognition performance.

It should be noted that, the feature fusion of the first feature data and the second feature data to obtain target feature data includes: performing first-order difference in the target frequency range according to the fluctuation feature matrix to obtain target fluctuation data; and carrying out feature fusion on the updated first feature data and the target fluctuation data to obtain target feature data.

Because F (t, F) in the analysis contains the whole time-frequency information, the basic time-frequency distribution characteristic of the signal is reflected. However, F (t, F) alone cannot clearly express a slight change in the gamma band inside the region most sensitive to emotion recognition. Δpγ (t, f) extracts dynamic fluctuation characteristics inside the γ band, which emphasizes sudden changes of key points in the γ band by difference. F (t, F) and ΔPγ (t, F) each express two different aspects of the time-frequency signal, namely the overall time-frequency characteristic and the γ -band internal dynamic characteristic. The use of both separately may lose or confuse information specific to the other party. Therefore, it needs to be fused with it to form a new feature expression S' (t, f) that is more informative.

Specifically, a dual-input feature fusion module can be established, wherein the first input channel is a new comprehensive time-frequency expression matrix F (t, F) reflecting the time-frequency distribution feature, and the second input channel is a first-order differential result delta Pgamma (t, F) of the gamma wave band of the time-frequency matrix obtained in the last step, and the first-order differential result delta Pgamma (t, F) represents the sudden change component of emotion fluctuation in the matrix and is extracted through differential operation.

The feature fusion module can dynamically learn the weight coefficients of two feature inputs at different moments and frequency bands by adopting an attention mechanism, and performs weighted fusion. Firstly, respectively inputting a time-frequency expression feature F (t, F) and a fluctuation feature delta Pgamma (t, F) into an attention network, wherein the network comprises two layers of convolution layers to extract local features of a feature map, and then outputting attention weight matrixes A (t, F) and B (t, F) by a full-connection layer, wherein A (t, F) represents weight coefficients of the time-frequency expression feature F at each time-space point (t, F), and B (t, F) represents weight coefficients of the fluctuation feature delta Pgamma at each time-space point. Finally, weighting fusion is carried out:

S'(t,f) = A(t,f) * F(t,f) + B(t,f) * ΔPγ(t,f)；

wherein S' (t, f) represents the new feature expression obtained by fusion. The attention network will give the spatiotemporal points representing key emotion fluctuations a higher weight B through training learning, suppressing the weights of the uncorrelated spatiotemporal points, thus enhancing these key emotion fluctuation features in the fusion features S' (t, f).

For example, assume that a segment of EEG signals is collected that is being tested when subjected to positive emotional stimuli. The method comprises the steps of performing feature extraction and fusion to obtain a time-frequency matrix F (t, F), observing gamma frequency band dynamic features delta Pgamma (t, F), finding that an obvious front fluctuation peak appears in 2-3 seconds, inputting F (t, F) and delta Pgamma (t, F) into an attention network for fusion, wherein after network training, the user can understand that delta Pgamma (t, F) is given a high weight in a gamma frequency band region of 2-3 seconds; according to the weight, S '(t, F) is formed after weighted fusion, and due to the enhancement of the contribution of 2-3 seconds gamma frequency band, the positive features of S' (t, F) are more obvious than those of F (t, F) in the area, so that the key positive emotion fluctuation features originally hidden in delta Pgamma (t, F) are fused to obtain clearer expression. Compared with the single use of the original time-frequency matrix or the fluctuation feature, the fusion feature can provide richer and obvious feature input for the subsequent recognition task.

According to the embodiment of the application, on the basis of time-frequency expression, the fluctuation characteristics are enhanced by adopting differential operation, and key characteristics representing emotion fluctuation are extracted. And then, the attention network is applied to realize dynamic weighted fusion of the time-frequency expression characteristics and the fluctuation characteristics, and important characteristic intervals are highlighted. And further learning the feature correlation in a local time window, generating a matching interference degree matrix, guiding a high-level attention module to weight the frequency bands at all moments, and obtaining the enhanced new feature expression.

The method realizes layered attention design and can deeply dig the diversity of emotion characteristics. The bottom layer attention promotes the fusion effect of time-frequency expression and fluctuation characteristics. The higher level of attention can capture a larger range of feature associations and dynamically adjust the weight distribution of different spatio-temporal points. Compared with the traditional flat attention structure, the hierarchical design can more comprehensively represent complex emotion characteristics through cross-level collaborative learning.

Step S104, feature fusion is carried out on the first feature data and the second feature data, and target feature data are obtained.

In addition to enhancing critical bands, different electroencephalogram frequency intervals may also contain complementary affective information. Therefore, cross-band feature fusion is needed, and internal correlations among different frequency components are learned to obtain comprehensive target feature data.

In some embodiments, the step S104 includes steps S1041 to S1042, which are specifically as follows:

step S1041, inputting the first feature data and the second feature data to a pre-trained feature fusion model, where the feature fusion model stores a weight mapping relationship, and the weight mapping relationship includes a preset weight value and a frequency range corresponding to the preset weight value.

The network structure of the feature fusion model can adopt a multi-layer feedforward full-connection structure, for example, the network structure comprises 2 hidden layers, and the number of hidden layer nodes is 256 and 128 respectively by constructing a multi-input neural network as the feature fusion model. The training process of the network is based on a large number of time-frequency expression samples marked with emotion type labels. And training network parameters through a back propagation algorithm, learning the inherent correlation among different input frequency band characteristics, and automatically obtaining different weights contributing to emotion recognition tasks. The weight of the input features corresponding to the alpha band may be smaller, while the weight of the input features corresponding to the beta and gamma bands is larger, indicating that the two bands are more sensitive to emotion recognition. Finally, the output layer of the network integrates the weighted features from different frequency bands to form a new comprehensive time-frequency expression F (t, F), and compared with the expression of each frequency band of the directly stacked time-frequency matrix, the fusion mode can highlight the time-frequency components more sensitive to emotion recognition, obtain a comprehensive new feature expression and provide richer information input for subsequently improving recognition accuracy.

Step S1042, obtaining the target feature data generated by feature fusion of the feature fusion model on the first feature data and the second feature data according to the weight mapping relationship.

The input of the feature fusion model comprises an original electroencephalogram time-frequency expression matrix S (t, f), an enhanced gamma frequency band matrix S_gamma (t, f) and other frequency band time-frequency subgraphs such as S_alpha (t, f) of an alpha wave band and S_beta (t, f) of a beta wave band. And further, cross-band feature fusion is realized, and the accuracy of emotion recognition is improved.

Illustratively, in some embodiments, the step S104 includes: acquiring third characteristic data according to frequency domain data and time domain data corresponding to a frequency range except the target frequency range in the alpha wave frequency range, the beta wave frequency range and the gamma wave frequency range; and carrying out feature fusion on the first feature data, the second feature data and the third feature data to obtain target feature data.

The first feature data, the second feature data, and the third feature data are fused for the following reasons:

(1) The time-frequency expression matrix S (t, f) of the original brain electrical signal contains complete frequency band information such as delta, theta, alpha, beta, gamma and the like, which is the most original and complete characteristic expression and reflects the time-frequency distribution condition of the full frequency band.

(2) The enhanced gamma frequency band characteristic matrix S_gamma (t, f) further strengthens emotion related characteristics in the gamma frequency band through nonlinear mapping. The foregoing analysis of the present application shows that the gamma frequency band is highly correlated with negative emotions. So S _ y (t, f) has a strong negative emotion detection capability.

(3) Other frequency band time-frequency sub-graphs of the time-frequency matrix, such as s_α (t, f) and s_β (t, f), represent time-frequency information of other frequency bands, such as α, β, etc., except the γ frequency band. Studies have shown that the alpha band is associated with an individual's relaxed state and the beta band is associated with positive mood. They contain other complementary affective information.

By fusing the three, complementary emotion information in different frequency bands can be used more fully. For example, the gamma band emphasizes negative emotions, the alpha band emphasizes positive relaxed emotions, and the combination of both can more accurately distinguish emotional states. Compared with the single frequency band characteristic, the fusion characteristic can improve the accuracy of emotion recognition.

Step S105, inputting target feature data into a pre-trained emotion recognition model, and generating an emotion recognition result by the emotion recognition model according to the target feature data to complete emotion recognition of the electroencephalogram signals; wherein the emotion recognition result at least comprises positive emotion, negative emotion and neutral emotion.

And constructing a convolutional neural network or a cyclic neural network as an emotion recognition model. The network structure of the emotion recognition model comprises a convolution layer and a pooling layer. Specifically, a plurality of groups of convolution kernels are arranged to realize multi-scale receptive fields, and mode relations of different scales are learned. The convolution layer is followed by pooling layer downsampling to further extract influencing features. And a plurality of convolution pooling layer stacks are arranged, so that the level improvement of the abstract degree is realized. The full connection layer carries out weighted summation on the output of the last convolution pooling layer, decomposes the output into coefficients of different categories and outputs probabilities of each category. And the model evaluation uses a test set, compares the predicted category with the real category, calculates indexes such as classification accuracy, sensitivity, recall rate and the like, and carries out quantitative evaluation. Finally, the new feature matrix S '' (t, f) is input into the trained emotion prediction model, and predictions of different emotion types, such as positive, neutral and negative emotion states, are output.

In some embodiments, prior to step S105, the method further comprises: acquiring emotion information corresponding to the electroencephalogram signals; inputting the target characteristic data into an emotion recognition model to be trained, and obtaining a predicted emotion recognition result generated by the emotion recognition model to be trained; and training the sensory recognition model according to the predicted emotion recognition result and the emotion information.

The method provided by the application selects cross entropy as the loss function, for example, using an adam optimizer for optimization. Setting a learning rate decrementing and early stopping strategy during training prevents overfitting. During model training, the data is randomly partitioned into training and validation sets using a cross-validation method. And (3) calculating the accuracy of the training set and the verification set after each round of iteration, and observing the error of the training set and the verification set to track the fitting degree. And ending the training process when the optimal point is reached.

It should be noted that, acquiring emotion information corresponding to the electroencephalogram signal may synchronously acquire electroencephalogram activity data during the process of watching and listening to a stimulation sequence by a subject. And the testee scores the current emotional state according to subjective feeling, for example, 1-9 points are used for representing the emotional state, 1 points represent the very negative state, 9 points represent the very positive state and 5 points are used for representing the neutral state. And taking the subjective scoring results as emotion labels, and recording the corresponding relation between the stimulus type and the emotion labels. Meanwhile, the gesture action of the tested person needs to be controlled, and the interference of excessive irrelevant physiological activities on signals is avoided. During the acquisition phase, care is taken to electromagnetically shield the device to reduce the effect of external noise on the signal quality. The acquired original brain electrical signals are digitized by an analog-to-digital converter and then stored, and generally, the sampling frequency of 250Hz or 500Hz is selected for discretization.

In some embodiments, the target feature data comprises a target feature matrix, and prior to step S105, the method further comprises: performing feature segmentation on the target feature matrix at preset time intervals to obtain a plurality of sub-target feature matrices; obtaining feature vectors in each sub-target feature matrix, and calculating a correlation coefficient matrix and a divergence matrix corresponding to the sub-target feature matrix according to the feature vectors; obtaining a matching interference degree matrix corresponding to the sub-target feature matrix according to the correlation coefficient matrix and the divergence matrix; and inputting the matched interference degree matrix into a pre-trained attention fusion model, generating an attention weight matrix by the attention fusion model according to the matched interference degree matrix, and fusing the attention weight matrix and the target feature matrix based on a self-attention mechanism to update the target feature matrix.

Illustratively, after obtaining the fused new feature matrix S' (t, f), it is necessary to learn further the feature expression in its local area to provide the key local features. The feature matrix S' is segmented in a sliding window mode, the window length is set to be L, and the sliding step length is set to be tau.

The sliding window divides the whole feature matrix S' into a plurality of local intervals of length L according to the time dimension, each window representing the expression of the original feature in a local continuous time period. The window length L needs to be set according to the time correlation of the original signal, for example, preferably 2-3 seconds. The sliding step τ controls the size of the overlap between windows, and is typically equal to or less than L, such as half L, to ensure continuity of information between windows.

Within each truncated local sliding window S' _w (t, f) of length L, a pearson correlation coefficient matrix between feature vectors within the window can be calculated to evaluate the interaction of the local samples on the linear correlation. The pearson correlation coefficient reflects the linear dependence between the two variables. Assuming that any two feature vectors in the window are X and Y respectively, the pearson correlation coefficient is calculated as:

r = cov(X,Y) / (σ_X * σ_Y)；

where cov denotes the covariance of X and Y, σ_X and σ_Y are the standard deviations of X and Y, respectively. The correlation coefficient matrix reflects the overall distribution condition of the linear correlation of the feature vectors in the current window by comparing the correlation relationship between each feature vector and all other vectors.

It should be noted that, the correlation coefficient calculation may also use Spearman correlation coefficient, coherence method, granger causality, etc. the method for calculating the correlation coefficient in the embodiment of the present application is not limited.

In addition, a JS divergence matrix of the feature vector in each sliding window is calculated to evaluate the consistency of the feature distribution in the area. The JS divergence is defined as the relative entropy of two probability distributions:

JS(P||Q) = (KL(P||M) + KL(Q||M))/2；

wherein m= (p+q)/2, kl is the relative entropy. The JS divergence ranges from 0 to 1, with a smaller value indicating that the two distributions are more consistent. And calculating the JS divergence of the vectors in the window to obtain a matrix reflecting the consistency of the distribution.

The correlation coefficient and the JS divergence matrix can be combined to form a matching interference degree matrix Z, which reflects the statistical correlation and the distribution consistency among the feature vectors in the current local sliding window area and can guide the follow-up attention module to identify and focus key features. Specific: the pearson correlation coefficients between the feature vectors in each sliding window are calculated to form a correlation coefficient matrix. And then, calculating JS divergence values among the feature vectors in the window to form a JS divergence matrix. Each element in the two matrices is normalized so that their values are within the same range. Then, an averaging operation is performed for each element position corresponding to the normalized two matrices. Through the polymerization, a comprehensive matrix is obtained, and the matrix gives the overall matching degree of each feature and other features by using two indexes of a correlation coefficient and JS divergence. Such a composite matrix can be calculated for all sliding window windows by repeating the above procedure. These composite matrices are defined as the matched interference matrix Z.

Illustratively, after the matching interference degree matrix Z of the feature matrix in the local area is obtained by the sliding window method, the matching interference degree matrix Z reflects the statistical correlation and the distribution consistency between the feature vectors in the current window. Based on the matching matrix Z, a dynamic attention module can be constructed to focus on the enhanced key emotion features.

The two inputs to the attention module are the match disturbance factor matrix Z and the new feature expression matrix S' (t, f), respectively. The matching matrix Z can guide the attention module to evaluate the importance of different moments and frequency bands in the feature matrix S' (t, f) to emotion recognition tasks. Specifically, the attention module takes as input the matching matrix Z, and learns to generate the attention weight matrix a through a multi-layer fully connected network. The network comprises an input layer, two hidden layers and an output layer, wherein the hidden layer nodes are 128 and 64 respectively, and the activation function adopts a ReLU. The output layer generates an attention weight matrix A through Softmax operation, and the dimension is the same as the input matching matrix Z and the original feature S'. Then, weighted fusion is carried out to generate a new feature matrix S '' (t, f):

S''(t,f) = ΣΣ A(i,j) * S'(t+i,f+j)；

wherein A (i, j) represents the weight value at the corresponding time t and frequency point f in the attention matrix, and S '' (t, f) is a new feature matrix after fusion weighting. The attention network may learn to identify which spatiotemporal regions of the matching matrix Z have more relevance and consistency in their features and give these critical regions higher weight in the attention matrix a. In the fusion feature s″ (t, f), these more weighted moments and bands are correspondingly emphasized.

For example, given that the new feature expression matrix S' (t, f) and its local matching matrix Z the attention network finds that Z indicates more significant feature consistency in the region corresponding to time t1 and frequency f1, the weight a (t 1, f 1) corresponding to time t1 and frequency f1 in the attention matrix a will output a larger value. This spatiotemporal region (t 1, f 1) in the original feature matrix S '(t, f) corresponding thereto is emphasized by the greater attention weight in the fused feature S' (t, f).

The parameters of the attention module can be trained and updated through a back propagation algorithm, so that the learned attention matrix A can more accurately strengthen a space-time region with obvious emotion recognition effect in the feature matrix S' (t, f). After dynamic focusing of attention, the local features representing key emotion fluctuations in the new feature matrix s″ (t, f) will be expressed in an enhanced manner, providing a more robust and reliable feature input for the method of the present application on the classification task of emotion recognition.

The method integrates various characteristic expression means such as time-frequency analysis, pearson correlation, JS divergence measurement and the like. The multi-model fusion improves the robustness of the features and enhances the diversity of emotion representation. Through layer-by-layer feature deepening, the finally formed feature expression matrix integrates multiple aspects of information of the original electroencephalogram data, and particularly, key emotion related features are effectively enhanced.

In the embodiment, the expression and recognition capability of key emotion features can be remarkably improved by designing an emotion recognition scheme of hierarchical dynamic attention multi-mode fusion. Compared with a single model, the integrated multiple feature extraction and fusion modules work cooperatively to form an information-rich and organically fused feature representation space, so that complex emotion features can be more comprehensively represented.

It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic of each process, and should not limit the implementation process of the embodiment of the present application in any way.

In order to execute the emotion recognition method based on the electroencephalogram signals corresponding to the embodiment of the method, corresponding functions and technical effects are achieved. Referring to fig. 2, fig. 2 shows a block diagram of a emotion recognition system based on an electroencephalogram signal according to an embodiment of the present application. For convenience of explanation, only the portions related to this embodiment are shown, and the emotion recognition system based on an electroencephalogram signal provided in this embodiment of the present application includes:

the stimulation module 201 is configured to obtain an electroencephalogram signal generated by a user under a preset stimulation source; the preset stimulus source comprises a positive emotion stimulus source, a negative emotion stimulus source and a neutral emotion stimulus source;

The processing module 202 is configured to perform feature processing on the electroencephalogram signal, and obtain first feature data of a first preset frequency range corresponding to the electroencephalogram signal; wherein the first characteristic data includes frequency domain data and time domain data;

a confirmation module 203, configured to confirm a target frequency range in the first preset frequency range according to the frequency domain data, amplify time domain data corresponding to the target frequency range, and obtain second feature data according to the frequency domain data corresponding to the target frequency range and the amplified time domain data;

the fusion module 204 is configured to perform feature fusion on the first feature data and the second feature data, so as to obtain target feature data;

the recognition module 205 is configured to input the target feature data into a pre-trained emotion recognition model, where the emotion recognition model generates an emotion recognition result according to the target feature data, and completes emotion recognition of the electroencephalogram signal; wherein the emotion recognition result at least comprises positive emotion, negative emotion and neutral emotion.

In some embodiments, the first preset frequency range includes at least an alpha frequency range, a beta frequency range, and a gamma frequency range; the determining module 203 is specifically configured to determine a target frequency range in the first preset frequency range according to the frequency domain data:

Illustratively, the fusing module 204 is specifically configured to:

In some embodiments, the fusion module 204 is specifically configured to:

In some embodiments, the emotion recognition system based on brain electrical signals further includes a difference module 206, where before the identifying a target frequency range in the first preset frequency range according to the frequency domain data, the difference module 206 is configured to:

Illustratively, the fusing module 204 is specifically configured to:

In some embodiments, the electroencephalogram-based emotion recognition system further comprises a segmentation module 207, the segmentation module 207 for, prior to the inputting of the target feature data into the pre-trained emotion recognition model:

In some embodiments, the emotion recognition model includes any of a convolutional neural network or a recurrent neural network; the emotion recognition system based on electroencephalogram signals further includes a training module 208, and before the target feature data is input into the pre-trained emotion recognition model, the training module 208 is configured to:

The emotion recognition system based on the electroencephalogram signals can implement the emotion recognition method based on the electroencephalogram signals in the embodiment of the method. The options in the method embodiments described above are also applicable to this embodiment and will not be described in detail here. The rest of the embodiments of the present application may refer to the content of the method embodiments described above, and in this embodiment, no further description is given.

Fig. 3 is a schematic structural diagram of a computer device according to an embodiment of the present application. As shown in fig. 3, the computer device 3 of this embodiment includes: at least one processor 30 (only one is shown in fig. 3), a memory 31 and a computer program 32 stored in the memory 31 and executable on the at least one processor 30, the processor 30 implementing the steps in any of the method embodiments described above when executing the computer program 32.

The computer device 3 may be a smart phone, a tablet computer, a desktop computer, a cloud server, or other computing devices. The computer device may include, but is not limited to, a processor 30, a memory 31. It will be appreciated by those skilled in the art that fig. 3 is merely an example of the computer device 3 and is not meant to be limiting as the computer device 3, and may include more or fewer components than shown, or may combine certain components, or different components, such as may also include input-output devices, network access devices, etc.

The processor 30 may be a central processing unit (Central Processing Unit, CPU), the processor 30 may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 31 may in some embodiments be an internal storage unit of the computer device 3, such as a hard disk or a memory of the computer device 3. The memory 31 may in other embodiments also be an external storage device of the computer device 3, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card) or the like, which are provided on the computer device 3. Further, the memory 31 may also include both an internal storage unit and an external storage device of the computer device 3. The memory 31 is used for storing an operating system, application programs, boot loader (BootLoader), data, other programs etc., such as program codes of the computer program etc. The memory 31 may also be used for temporarily storing data that has been output or is to be output.

In addition, the embodiment of the present application further provides a computer readable storage medium, where a computer program is stored, where the computer program is executed by a processor to implement the steps in any of the above-mentioned method embodiments.

The present embodiments provide a computer program product which, when run on a computer device, causes the computer device to perform the steps of the method embodiments described above.

In several embodiments provided herein, it will be understood that each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.

The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device to perform all or part of the steps of the method described in the various embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

The foregoing embodiments have been provided for the purpose of illustrating the objects, technical solutions and advantages of the present application in further detail, and it should be understood that the foregoing embodiments are merely examples of the present application and are not intended to limit the scope of the present application. It should be noted that any modifications, equivalent substitutions, improvements, etc. made by those skilled in the art, which are within the spirit and principles of the present application, are intended to be included within the scope of the present application.

Claims

1. The emotion recognition method based on the electroencephalogram signal is characterized by comprising the following steps of:

2. The method of claim 1, wherein the first predetermined frequency range includes at least an alpha frequency range, a beta frequency range, and a gamma frequency range; the identifying a target frequency range in the first preset frequency range according to the frequency domain data includes:

3. The method of claim 2, wherein feature fusing the first feature data and the second feature data to obtain target feature data comprises:

4. The method of claim 1, wherein feature fusing the first feature data and the second feature data to obtain target feature data comprises:

5. The method of claim 1, wherein prior to said identifying a target frequency range in said first preset frequency range from said frequency domain data, said method further comprises:

6. The method of claim 5, wherein feature fusing the first feature data and the second feature data to obtain target feature data comprises:

7. The method of claim 1, wherein the target feature data comprises a target feature matrix; before the inputting of the target feature data into the pre-trained emotion recognition model, the method further comprises:

8. The method of claim 1, wherein the emotion recognition model comprises any one of a convolutional neural network or a recurrent neural network; before the inputting of the target feature data into the pre-trained emotion recognition model, the method further comprises:

9. A computer device comprising a processor and a memory for storing a computer program which when executed by the processor implements the steps of the emotion recognition method of any of claims 1 to 8.

10. A computer-readable storage medium, characterized in that it stores a computer program which, when executed by a processor, implements the steps of the emotion recognition method of any one of claims 1 to 8.