CN114970641A

CN114970641A - Emotion category identification method and device, processor and electronic equipment

Info

Publication number: CN114970641A
Application number: CN202210727036.4A
Authority: CN
Inventors: 饶宇熹; 冯智斌; 黎明欣; 黄淋
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2022-06-24
Filing date: 2022-06-24
Publication date: 2022-08-30

Abstract

The application discloses a method and a device for recognizing emotion types, a processor and electronic equipment. Relates to the field of artificial intelligence, and the method comprises the following steps: acquiring a fusion electrophysiological signal of an object to be identified, wherein the fusion electrophysiological signal is determined according to an electroencephalogram physiological signal, a blood flow physiological signal and an electrocardio physiological signal; use mixed neural network model to carry out the analysis to fusing electrophysiological signal, confirm the emotion classification that fuses electrophysiological signal and correspond, wherein, mixed neural network model is for using multiunit training data to train out through machine learning, and every group training data in the multiunit training data all includes: and fusing the electrophysiological signals and the emotion categories calibrated by each fused electrophysiological signal. By the method and the device, the problem that emotion recognition cannot be carried out according to electrophysiological signals in the related art is solved.

Description

Emotion category identification method and device, processor and electronic equipment

Technical Field

The application relates to the field of artificial intelligence, in particular to a method, a device, a processor and electronic equipment for recognizing emotion categories.

Background

In the field of emotion recognition, three different dimensions are commonly used for recognition, including emotion recognition of three categories of human face, voice and physiology.

Face emotion recognition

The emotion of the human is identified through feature analysis of facial expression, muscle tendency and the like, and the emotion of happiness, sadness, anger and the like can be distinguished. Generally, a deep neural network is adopted for emotion recognition, firstly, preprocessing is carried out on a picture, including face detection, face alignment, data enhancement and face normalization, and the steps play a key role in the accuracy of subsequent emotion recognition; deep feature learning, typically based on CNN network models, some feature extraction using Deep Belief Networks (DBNs), and finally some LSTM based on sequence modeling.

Second, speech emotion recognition

The emotion of the human is recognized through the characteristics of the speech speed, the tone, the pitch and the like when the human speaks and a deep learning method. After the original voiceprint signal is converted into a spectrogram, the CNN firstly uses two different convolution kernels to respectively extract time domain characteristics and frequency domain characteristics, and then combines the two characteristics to train by using the deep CNN.

Third, physiological emotion recognition

Human emotion is recognized by analyzing physiological signals such as posture action performance, respiration, heart rate, body temperature and the like in behaviors. In the existing technical research, feature extraction is performed on physiological signals, and then the physiological signals are classified by using methods such as the traditional machine learning method (PCA, KNN).

In the face emotion recognition technology, the features commonly used in machine recognition are mainly divided into: three types of gray scale features, motion features and frequency features are adopted, but certain bottleneck exists when the classification performance is improved by taking the three types of gray scale features, motion features and frequency features as classified gold standard features, and the facial expression can cause failure of an identification task through disguise; in the speech emotion recognition technology, characteristics and distribution rules such as time sequence characteristics, amplitude, fundamental frequency structure and formant structure in a tone signal are often used for feature extraction, a study also has the effect of converting a voiceprint signal into a spectrogram for feature extraction, and finally emotion classification is carried out, so that certain difference exists in precision relative to face recognition. In the first two technologies, expressions and voices are controlled by subjective consciousness of individuals to a greater or lesser extent, real emotional feedback of a tested body cannot be objectively fed back under certain special backgrounds, and physiological signals are only regulated and controlled by a human nervous system and a secretory system and can be completely independent of subjective ideas of people.

Aiming at the problem that emotion recognition can not be carried out according to electrophysiological signals in the related art, an effective solution is not provided at present.

Disclosure of Invention

The present application mainly aims to provide a method, an apparatus, a processor and an electronic device for emotion classification identification, so as to solve the problem in the related art that emotion identification cannot be performed according to electrophysiological signals.

In order to achieve the above object, according to one aspect of the present application, a method for recognizing emotion categories is provided. The method comprises the following steps: acquiring a fusion electrophysiological signal of an object to be identified, wherein the fusion electrophysiological signal is determined according to an electroencephalogram physiological signal, a blood flow physiological signal and an electrocardio physiological signal; analyzing the fused electrophysiological signal by using a hybrid neural network model, and determining an emotion category corresponding to the fused electrophysiological signal, wherein the hybrid neural network model is trained by machine learning using a plurality of sets of training data, and each set of training data in the plurality of sets of training data includes: and fusing the electrophysiological signals and the emotion categories calibrated by each fused electrophysiological signal.

Optionally, the acquiring of the fused electrophysiological signal of the object to be identified comprises: acquiring an electroencephalogram physiological signal of the object to be identified in a preset time period; acquiring a blood flow physiological signal of the object to be identified in the preset time period; acquiring the electrocardio physiological signals of the object to be identified in the preset time period; and determining the fused electrophysiological signal according to the electroencephalogram physiological signal, the blood flow physiological signal and the electrocardio physiological signal of the object to be identified in the same preset time period.

Optionally, determining the fused electrophysiological signal according to the electroencephalogram physiological signal, the blood flow physiological signal, and the electrocardiograph physiological signal in the same preset time period includes: determining the electroencephalogram physiological signals acquired through the first channel as a first-class signal matrix; determining the blood flow physiological signals acquired through the second channel as a second type signal matrix; determining the electrocardio-physiological signals acquired through the third channel as a third signal matrix; and splicing the first type signal matrix, the second type signal matrix and the third type signal matrix in the column dimension of the matrix to generate the fused electrophysiological signal.

Optionally, before analyzing the fused electrophysiological signal by using a hybrid neural network model to determine an emotion category corresponding to the fused electrophysiological signal, the method further includes: obtaining a sample signal of at least one sample object for a preset category of emotions, wherein the sample signal at least comprises: the electroencephalogram physiological signal, the blood flow physiological signal and the electrocardio physiological signal in the same sample time interval; preprocessing the sample signal to obtain a training signal; and calibrating the training signals according to the emotion of the preset category to generate the training data.

Optionally, the preprocessing the sample signal to obtain a training signal includes: performing baseline wandering removal processing on the sample signal to obtain a first processing signal; carrying out notch processing on the first processed signal to obtain a second processed signal; and performing preset noise removal processing on the electroencephalogram physiological signal in the second processed signal to obtain the training signal.

Optionally, before analyzing the fused electrophysiological signal by using a hybrid neural network model to determine an emotion category corresponding to the fused electrophysiological signal, the method further includes: extracting the spatial features of the fused electrophysiological signals in each group of training data by using a first convolution kernel to obtain a spatial feature map; extracting the time characteristics of the fused electrophysiological signals in each group of training data by using a second convolution kernel to obtain a time characteristic diagram; combining the spatial characteristic diagram and the temporal characteristic diagram to obtain a physiological signal characteristic diagram; and performing pooling processing on the physiological signal characteristic diagram to generate a characteristic sequence vector.

Optionally, after pooling the physiological signal feature map to generate a feature sequence vector, the method further includes: processing the characteristic sequence vector by using a preset long-short term memory artificial neural network to generate a classification characteristic diagram; outputting a classification result of the emotion classification according to the classification feature map by using a preset classifier; and adjusting parameters by using a preset optimizer according to the classification result of the emotion classification to obtain the hybrid neural network model.

In order to achieve the above object, according to another aspect of the present application, there is provided an emotion classification recognition apparatus. The device includes: the system comprises an acquisition unit, a recognition unit and a processing unit, wherein the acquisition unit is used for acquiring a fusion electrophysiological signal of an object to be recognized, and the fusion electrophysiological signal is determined according to an electroencephalogram physiological signal, a blood flow physiological signal and an electrocardio physiological signal; an analysis unit, configured to analyze the fused electrophysiological signal using a hybrid neural network model, and determine an emotion category corresponding to the fused electrophysiological signal, where the hybrid neural network model is trained through machine learning using multiple sets of training data, and each set of training data in the multiple sets of training data includes: and fusing the electrophysiological signals and the emotion categories calibrated by each fused electrophysiological signal.

To achieve the above object, according to another aspect of the present application, there is provided a processor. The processor is used for running a program, wherein the program executes the emotion classification identification method during running.

To achieve the above object, according to another aspect of the present application, there is provided an electronic device. The electronic device comprises one or more processors and memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method for emotion classification identification described above.

Through the application, the following steps are adopted: acquiring a fusion electrophysiological signal of an object to be identified, wherein the fusion electrophysiological signal is determined according to an electroencephalogram physiological signal, a blood flow physiological signal and an electrocardio physiological signal; use mixed neural network model to carry out the analysis to fusing electrophysiological signal, confirm the emotion classification that fuses electrophysiological signal and correspond, wherein, mixed neural network model is for using multiunit training data to train out through machine learning, and every group training data in the multiunit training data all includes: the electrophysiological signals and the emotion types calibrated by each electrophysiological signal are fused, so that the problem that emotion recognition cannot be performed according to the electrophysiological signals in the related technology is solved. Thereby achieving the technical effect of emotion recognition according to the electrophysiological signals.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the application and, together with the description, serve to explain the application and are not intended to limit the application. In the drawings:

FIG. 1 is a flow chart of a method for recognizing emotion categories provided according to an embodiment of the present application;

fig. 2 is a schematic diagram of a signal processing flow provided according to an embodiment of the present application;

FIG. 3 is a schematic diagram of a hybrid neural network model provided in accordance with an embodiment of the present application;

FIG. 4 is a schematic diagram of a ten-fold cross-validation provided in accordance with an embodiment of the present application;

FIG. 5 is a schematic diagram of an emotion classification recognition apparatus according to an embodiment of the present application;

fig. 6 is a schematic diagram of an electronic device according to an embodiment of the application.

Detailed Description

It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.

In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the accompanying drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It should be understood that the data so used may be interchanged under appropriate circumstances such that embodiments of the application described herein may be used. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

For convenience of description, some terms or expressions referred to in the embodiments of the present application are explained below:

EEG: brain waves, a method of recording brain activity using electrophysiological indicators.

EOG: an electro-oculogram is an electrophysiological examination that records the functional status of the outer retina and retinal pigment epithelium under both dark and light adaptation conditions, and detects changes in the resting potential of the eye.

ECG: electrocardiography is a technique for recording a pattern of change in electrical activity generated every cardiac cycle of the heart from the body surface by an electrocardiograph.

CNN: the convolutional neural network is a feedforward neural network which comprises convolutional calculation and has a deep structure, has characterization (feature) learning capacity and can carry out translation invariant classification on input information according to a hierarchical structure of the convolutional neural network.

LSTM: the long-term memory network is specially designed for solving the long-term dependence problem of a general RNN (recurrent neural network), and all RNNs have a chain form of a repeated neural network module.

A convolution pooling layer: the convolutional neural network is a general term for a convolutional layer and a max-pooling layer.

It should be noted that relevant information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for presentation, analyzed data, etc.) referred to in the present disclosure are information and data that are authorized by the user or sufficiently authorized by various parties. For example, an interface is provided between the system and the relevant user or organization, before obtaining the relevant information, an obtaining request needs to be sent to the user or organization through the interface, and after receiving the consent information fed back by the user or organization, the relevant information is obtained.

The present invention is described below with reference to preferred implementation steps, and fig. 1 is a flowchart of a method for identifying emotion categories according to an embodiment of the present application, as shown in fig. 1, the method includes the following steps:

s102, acquiring a fusion electrophysiological signal of an object to be identified, wherein the fusion electrophysiological signal is determined according to the electroencephalogram physiological signal, the blood flow physiological signal and the electrocardio physiological signal;

step S104, analyzing the fused electrophysiological signals by using a hybrid neural network model, and determining emotion categories corresponding to the fused electrophysiological signals, wherein the hybrid neural network model is trained by using multiple groups of training data through machine learning, and each group of training data in the multiple groups of training data comprises: and fusing the electrophysiological signals and the emotion categories calibrated by each fused electrophysiological signal.

In the above step S102, the electroencephalogram physiological signal is determined from the electroencephalogram EEG, and the electrocardiograph physiological signal is determined from the electrocardiogram ECG.

In step S104, the hybrid neural network model may be a hybrid network CNN-LSTM model, and the collected multi-modal physiological signals (i.e., the fused electrophysiological signals) are subjected to emotion recognition and classification by combining the feature extraction advantages of the convolutional neural network CNN model and the feature fusion and timing prediction advantages of the long-term and short-term memory artificial neural network LSTM model.

It should be noted that the Convolutional Neural Network (CNN) can extract spatial features in the previously fused electrophysiological signals, and the long-term memory network (LSTM) performs feature fusion on the features output by the CNN and extracts time series information.

In step S104, the emotion categories at least include: fear, anger, sadness, happiness, disgust.

Optionally, in the emotion classification recognition method provided in the embodiment of the present application, the acquiring a fused electrophysiological signal of an object to be recognized includes: acquiring an electroencephalogram physiological signal of an object to be identified in a preset time period; acquiring a blood flow physiological signal of an object to be identified in a preset time period; acquiring an electrocardio physiological signal of an object to be identified in a preset time period; and determining a fused electrophysiological signal according to the electroencephalogram physiological signal, the blood flow physiological signal and the electrocardio physiological signal of the object to be identified in the same preset time period.

According to the embodiment of the invention, the fused electrophysiological signal is generated according to the electroencephalogram physiological signal, the blood flow physiological signal and the electrocardio physiological signal in the same preset time period, and the emotion category corresponding to the fused electrophysiological signal is determined, so that the emotion category of the object to be identified can be determined based on a plurality of electrophysiological signals of the object to be identified, and the technical effect of accurately identifying the emotion category of the object to be identified is realized.

Optionally, before analyzing the fused electrophysiological signal by using the hybrid neural network model and determining the emotion category corresponding to the fused electrophysiological signal, the method further includes: training a hybrid neural network model through machine learning using a plurality of sets of training data, wherein the plurality of sets of training data used to train the hybrid neural network model include: fused electrophysiological signals and corresponding emotion classifications for a plurality of sample objects.

Optionally, before analyzing the fused electrophysiological signal by using the hybrid neural network model and determining the emotion category corresponding to the fused electrophysiological signal, the method further includes: acquiring an electroencephalogram physiological signal of a sample object in a preset time period; acquiring a blood flow physiological signal of a sample object in a preset time period; acquiring an electrocardio physiological signal of a sample object in a preset time period; determining a fused electrophysiological signal of the sample object according to the electroencephalogram physiological signal, the blood flow physiological signal and the electrocardio physiological signal of the sample object in the same preset time period; and calibrating the emotion types corresponding to the fused electrophysiological signals of the sample object.

It should be noted that, in the case of identifying the emotion type, the fused electrophysiological signal may be a fusion of an electroencephalogram physiological signal, a blood stream physiological signal, and an electrocardiograph physiological signal of the object to be identified; under the condition of training a mixed neural network model for recognizing emotion classes, the fused electrophysiological signals can be the fusion of electroencephalogram physiological signals, blood flow physiological signals and electrocardio physiological signals of the sample object.

Optionally, the electroencephalogram physiological signal, the blood flow physiological signal and the electrocardiograph physiological signal can be collected through different channels, and then the electrophysiological signals collected by the channels are fused.

Optionally, in the emotion classification identification method provided in the embodiment of the present application, determining, according to the electroencephalogram physiological signal, the blood stream physiological signal, and the electrocardiograph physiological signal in the same preset time period, that the fused electrophysiology signal includes: determining the electroencephalogram physiological signals acquired through the first channel as a first-class signal matrix; determining the blood flow physiological signals acquired through the second channel as a second type signal matrix; determining the electrocardio-physiological signals acquired through the third channel as a third signal matrix; and splicing the first type signal matrix, the second type signal matrix and the third type signal matrix in the column dimension of the matrix to generate the fused electrophysiological signal.

Optionally, the number of the first channels may be multiple, and the number of the rows of the first type signal matrix is determined according to the number of the first channels; the number of the second channels can be multiple, and the number of the rows of the second type signal matrix is determined according to the number of the second channels; the number of the third channels can be multiple, and the number of the rows of the third-type signal matrix is determined according to the number of the third channels.

According to the embodiment of the invention, the electroencephalogram physiological signals are collected through the first channel to obtain the first-class signal matrix, the blood flow physiological signals are collected through the second channel to obtain the second-class signal matrix, the electrocardio physiological signals are collected through the third channel to obtain the third-class signal matrix, and the first-class signal matrix, the second-class signal matrix and the third-class signal matrix are spliced in the column dimension of the matrix to obtain the fused electrophysiology signals, so that the fusion of the electroencephalogram physiological signals, the blood flow physiological signals and the electrocardio physiological signals can be realized.

Optionally, the electroencephalogram physiological signal, the blood flow physiological signal and the electrocardiograph physiological signal may be acquired according to a preset sampling frequency in a preset time period, and a signal matrix is determined according to the electrophysiological signal, where the signal matrix includes: the signal matrix comprises a first type signal matrix, a second type signal matrix and a third type signal matrix; detecting whether the size of the signal matrix is higher than a preset size, and combining the electrophysiological signals of a plurality of acquisition points with adjacent acquisition time under the condition that the size of the signal matrix is higher than the preset size, wherein each acquisition point represents a time point for acquiring the electrophysiological signals.

Alternatively, the electrophysiological signals for a preset number of acquisition points may be combined.

Optionally, combining the electrophysiological signals of the plurality of collection points comprises: an average of the electrophysiological signals for the plurality of acquisition points is determined.

As an alternative example, if the sampling frequency at the time of acquisition is determined to be 500Hz (i.e. 500 points are acquired in 1 second), then data for a period of 5 minutes is taken as one sample, and the final signal matrix size is (150000, 18). If this sample is too large, it is rendered unmanageable by the server and can be down-sampled.

Optionally, the down-sampling refers to: an average is taken of every four adjacent points so that the sampling frequency is reduced to 125Hz, and also to 2 minutes in time, thereby reducing the single sample matrix to (18000, 18).

Optionally, in the emotion classification recognition method provided in the embodiment of the present application, before analyzing the fused electrophysiological signal using the hybrid neural network model and determining the emotion classification corresponding to the fused electrophysiological signal, the method further includes: acquiring a sample signal of at least one sample object for a preset category of emotions, wherein the sample signal at least comprises: the electroencephalogram physiological signal, the blood flow physiological signal and the electrocardio physiological signal in the same sample time interval; preprocessing a sample signal to obtain a training signal; and calibrating the training signals according to the emotion of the preset category to generate training data.

According to the embodiment of the invention, the electrophysiological signals of a plurality of sample objects based on different emotion types are collected, and the collected electrophysiological signals are calibrated to obtain a plurality of groups of training data.

Alternatively, the sample signal may be a fused electrophysiological signal determined from the electroencephalogram physiological signal, the blood flow physiological signal, and the electrocardiograph physiological signal of the sample object.

Optionally, each set of training data comprises: the same sample object aims at multiple electrophysiological signals of the same emotion class.

For example, if the emotion of the sample object a at the time a is an open heart, acquiring the electroencephalogram physiological signal, the blood flow physiological signal and the electrocardiograph physiological signal acquired at the time a is an open heart electrophysiological signal.

The invention needs to collect the electrophysiological signals of five emotions of a plurality of subjects (namely sample objects) in 2-3 hours in a resting state environment, wherein the electrophysiological signals comprise electroencephalogram physiological signals, blood flow physiological signals, electrocardio physiological signals and the like, which can be collected in a non-invasive way, and 5 emotions comprise fear, anger, sadness, happiness and disgust.

Optionally, a single subject (i.e., a sample object), when directed to produce a corresponding emotion, breaks the recording of the electrophysiological signal for later segmentation. For example, a piece of happy video is played, and after the expression of the subject (i.e., sample object) is observed as a happy state, a label is directly marked on the record or the time occurring at that time is recorded.

Optionally, the emotional state of the subject (i.e. sample subject) in different video states is guided repeatedly, so that 5 different emotional states exist in one subject (i.e. sample subject).

Optionally, 20 pieces of data of different emotions of the tested object can be acquired, and then the acquired data of electrophysiological signals in each 2-minute time period are labeled correspondingly with emotion labels by referring to the training of the images on deep learning. Setting the frequency of the acquisition instrument to be 500Hz, the electroencephalogram physiological signals to be 24 channels, the blood flow physiological signals to be 4 channels and the electrocardio physiological signals to be 2 channels, and then (60000x30) matrix data represents data with the duration of 2 minutes and corresponds to an emotion label.

Alternatively, after acquiring the electroencephalogram physiological signal, the blood flow physiological signal, and the electrocardiograph physiological signal, in order to ensure the accuracy of the acquired electrophysiology signal and the accuracy of the trained hybrid neural network model, it is necessary to preprocess the electrophysiology signal of the target object to remove signal noise before training the hybrid neural network model using the electrophysiology signal of the sample object.

Optionally, in the method for identifying an emotion category provided in the embodiment of the present application, the preprocessing the sample signal to obtain a training signal includes: carrying out baseline wandering removal processing on the sample signal to obtain a first processing signal; carrying out notch processing on the first processing signal to obtain a second processing signal; and performing preset noise removal processing on the electroencephalogram physiological signal in the second processed signal to obtain a training signal.

Fig. 2 is a schematic diagram of a signal processing flow according to an embodiment of the present application, and as shown in fig. 2, since a lot of noise exists in electrical physiological signals such as an electroencephalogram physiological signal, a blood flow physiological signal, and an electrocardiograph physiological signal, before a hybrid application network model is trained based on the electrical physiological signals, artifact processing such as baseline drift, notch processing (power frequency interference), and eye electrical removal of an EEG signal needs to be sequentially removed according to a data processing flow; then, the electrophysiological signal data is normalized, so that the training speed of a model can be increased later.

Optionally, in the emotion classification identification method provided in the embodiment of the present application, the performing baseline wander removal processing on the sample signal to obtain a first processed signal includes: identifying missing values for each of the sample signals; determining a missing time period corresponding to the missing value; deleting the plurality of signals of the missing period; and carrying out data smoothing processing on the signals connected in the missing time interval to obtain a first processed signal.

In the above embodiment of the present invention, in the processing process of removing the baseline wander, it is required to first check whether there is a missing value in the electrophysiological signals collected by all channels, then delete all the electroencephalogram physiological signals, the blood flow physiological signals, and the electrocardiograph physiological signals in the time period corresponding to the missing value, and keep the smoothness of the connection at the deleted data position as much as possible, or use linear interpolation at the deleted position to ensure the smoothness of the data, thereby implementing the baseline wander removal processing on the electrophysiological signals.

It should be noted that, although the whole data acquisition is in a rest environment, it is difficult to avoid power frequency interference (an interference caused by a power system) during data acquisition, and in order to solve this problem, a butterworth band-pass filter with a stop band lower limit cut-off frequency of 49Hz and an upper limit cut-off frequency of 51Hz is used for carrying out notch processing in an experiment, so as to eliminate power frequency interference of 50Hz and realize notch processing on electrophysiological signals.

It should be noted that, in order to ensure that another experimental signal or interference signal is not mixed in the single-source signal, the experiment removes noise components such as electrooculogram, myoelectricity, and head movement signals from the EEG signal.

Alternatively, the EEG signal was artifact removed, mainly using blind source signal separation technique, Independent Component Analysis (ICA) method, with noise rejection by the eeglab v14.1.1 toolbox in MATLAB 2018 b. The whole steps are as follows:

a) and introducing a data signal (namely, an electroencephalogram physiological signal of the sample object) into the eeglab, and introducing each channel data into the working area according to the sampling frequency.

b) And loading channel position information according to the channel position configuration files, visualizing the electroencephalogram physiological signals of the whole brain scalp, and drawing a 2-dimensional image.

c) And processing the channel data by using an ICA algorithm to obtain a plurality of components of the channel, and sequentially checking whether each component contains a noise component.

Optionally, the blink component: the components are distributed at the front end of a scalp topographic map, small squares are arranged in an ERP image, low-frequency energy is high in a power spectrogram, and the components are ranked more front;

optionally, the eye movement component: the red and blue are distributed on two sides of the front end of the scalp topographic map, the red and the blue are opposite, in the ERP image, the red and the blue are in a strip shape, the low-frequency energy in a power spectrogram is high, the components are ranked in the front, but generally behind blinking.

Optionally, the head movement component: the ERP images appear long and have very significant (very long) drift within a single trial.

After ICA analysis and noise signal elimination, the preprocessed electrophysiological signals are trained by using a neural network (or a hybrid neural network model) to have high precision and recall rate.

As an alternative embodiment, the hybrid neural network model comprises: CNN networks (i.e., convolutional neural networks) with feature extraction advantages, training a hybrid neural network model using multiple sets of training data, including: and extracting the features of the fused electrophysiological signals by using the CNN network to generate a feature sequence vector.

Optionally, in the emotion classification recognition method provided in the embodiment of the present application, before analyzing the fused electrophysiological signal using the hybrid neural network model and determining the emotion classification corresponding to the fused electrophysiological signal, the method further includes: extracting the spatial features of the fused electrophysiological signals in each group of training data by using a first convolution kernel to obtain a spatial feature map; extracting the time characteristics of the fused electrophysiological signals in each group of training data by using a second convolution kernel to obtain a time characteristic diagram; combining the spatial characteristic diagram and the temporal characteristic diagram to obtain a physiological signal characteristic diagram; and performing pooling processing on the physiological signal characteristic diagram to generate a characteristic sequence vector.

As an alternative embodiment, the hybrid neural network model comprises: the LSTM network (namely the preset long-short term memory artificial neural network) with the advantages of feature fusion and time sequence prediction uses a plurality of groups of training data to train a mixed neural network model, and comprises the following steps: and processing the characteristic sequence vector by using an LSTM network, and realizing the classification of the emotion classes based on the processing result.

Optionally, after pooling the physiological signal feature map to generate the feature sequence vector, the method further includes: processing the characteristic sequence vector by using a preset long-short term memory artificial neural network to generate a classification characteristic diagram; outputting a classification result of the emotion classification according to the classification feature map by using a preset classifier; and adjusting parameters by using a preset optimizer according to the classification result of the emotion classification to obtain a hybrid neural network model.

Fig. 3 is a schematic diagram of a hybrid neural network model provided according to an embodiment of the present application, and as shown in fig. 3, in order to fully mine features of electrophysiological signals in the spatio-temporal dimension, in an initial CNN network, two convolution kernels (3x1) and (1x3) are used to perform feature extraction in the temporal dimension and the spatial dimension (i.e., a first convolution kernel is used to extract spatial features, a second convolution kernel is used to extract temporal features), and the two convolution kernels are fused and placed in a deep convolution neural network with four convolution pooling layers later to perform feature extraction, and then an LSTM network is used to perform feature fusion and time series feature emotion prediction on the previously extracted features, and finally a fully-connected and preset classifier softmax is used to perform classification.

Optionally, the input of the whole system is a multi-modal electrophysiological signal, which is a fused electrophysiological signal determined by an electroencephalogram physiological signal, a blood flow physiological signal, and an electrocardiograph physiological signal, wherein the size of a single sample data (i.e., a single fused electrophysiological signal) is (60000x30), before entering the whole mixed neural network model, two convolution kernels with different sizes are used to extract spatial features and temporal features respectively, so as to obtain feature maps (i.e., a spatial feature map and a temporal feature map) with sizes of (60000x10) and (20000x30), which can also accelerate the speed of the subsequent network training. And then combining the two obtained feature maps to re-form a physiological signal feature map (namely combining the spatial feature map and the temporal feature map to obtain the physiological signal feature map).

Optionally, the physiological signal feature map is sent to the next four convolution pooling layers, one convolution pooling layer comprising one convolution kernel of size (3x 3); then, carrying out batch standardization (increasing the training speed and improving the robustness); then a maximum pooling layer with a convolution kernel size of (2x2) and step size of 2 is accessed.

Optionally, the (7500x2) signature sequence vector obtained by the four convolution pooling layers is flattened into a 15000x1 vector and transmitted into the LSTM network. In the LSTM network, each LSTM unit consists of four main components, an input gate, a neuron with self-loop connections, a forgetting gate, and an output gate.

Optionally, the LSTM network of the present invention includes: 3 hidden layers and a fully connected layer, wherein each hidden layer is a basic cyclic network composed of a plurality of LSTMs and defined as LSTM basic units. A single LSTM base unit contains 128 neurons, using a default tanh activation function. Dropout is then used between each neuron to reduce overfitting, and it is noted that this dropout is used between individual LSTM neurons, and therefore is not used as much as CNN, and is no longer used at 4 gates per neuron.

Optionally, finally, a full connection layer is used to re-pass the obtained local features through the weight matrix to form a formal feature map (i.e., a classification feature map). And finally, mapping the output of the full connection layer into a probability distribution for classification through a preset classifier softmax. The preset optimizer Adam with an initial learning rate of 0.01 is used for training in the training process, and meanwhile, the model improves the robustness of the model on test data by using exponential weighted smooth average.

As an alternative embodiment, training the hybrid neural network model using the plurality of sets of training data includes: dividing the fused electrophysiological signals of the plurality of sample objects into training data and testing data according to a first preset proportion, wherein the training data is used for training the hybrid neural network model, and the testing data is used for testing the hybrid neural network model; training a hybrid neural network model using training data includes: and dividing the plurality of groups of training data into training samples and verification samples according to a second preset proportion, training the mixed neural network model by using the training samples, and then verifying the trained mixed neural network model by using the verification samples.

Optionally, a sample data set of the sample object may be obtained according to a plurality of electrophysiological signals of a plurality of target objects, wherein the sample data set comprises: sample signals of a plurality of target objects; that is, the sample data set includes: fused electrophysiological signals of a plurality of sample objects.

Optionally, after the sample data set is obtained, the sample data set may be divided into training sets according to a ratio of 7: 3: test set and train the CNN-LSTM model (i.e., the hybrid neural network model) using cross-validation of ten folds.

Fig. 4 is a schematic diagram of ten-fold cross validation provided according to an embodiment of the present application, and as shown in fig. 4, data in the entire training set is randomly divided into 10 parts, where 9 parts are used for training and 1 part is used for validation. And after the whole experiment is subjected to ten-fold cross validation, the model is one epoch, 30 epoch trainings are set in total, finally, a model with a higher average value is obtained and used as a final trained model, then, the model is used for a test set, and the final result is the accuracy of the model.

According to the technical scheme provided by the invention, from the beginning of data (namely, a plurality of electrophysiological signals) acquisition, in order to ensure the performance of a model, artifact removal is carried out on original data (namely, the acquired electrophysiological signals), then the processed data (namely, fused electrophysiological signals) are sent to a constructed CNN-LSTM network (namely, a mixed neural network model) for training, and the maximum soft classification is carried out; compared with the traditional human face image and voiceprint data, the electrophysiological signals are only regulated and controlled by a human body nervous system and a secretion system, can be completely independent of the subjective idea of a human, and have higher reliability; the multi-modal electrophysiological signals are fused into fused electrophysiological signals, so that robustness can be provided for emotion model identification, and the accuracy of model classification prediction is improved; feature extraction is carried out in the space dimension and the time dimension respectively through convolution, and a basic feature template is provided for the subsequent CNN-LSTM (namely a hybrid neural network model) to accelerate model training.

Optionally, the hybrid neural network model provided by the invention adopts a convolutional neural network with 4 convolutional pooling layer depths, so that the physiological signal characteristics can be better extracted, and then an LSTM network is used for carrying out characteristic fusion and emotion prediction; thus, the hybrid neural network model of the present application combines the two advantages of CNN networks and LSTM networks.

According to the emotion category identification method provided by the embodiment of the application, fusion electrophysiological signals of an object to be identified are obtained, wherein the fusion electrophysiological signals are determined according to electroencephalogram physiological signals, blood flow physiological signals and electrocardio physiological signals; use mixed neural network model to carry out the analysis to fusing electrophysiological signal, confirm the emotion classification that fuses electrophysiological signal and correspond, wherein, mixed neural network model is for using multiunit training data to train out through machine learning, and every group training data in the multiunit training data all includes: the electrophysiological signals and the emotion types calibrated by each electrophysiological signal are fused, so that the problem that emotion recognition cannot be performed according to the electrophysiological signals in the related technology is solved. And further achieves the technical effect of emotion recognition according to the electrophysiological signals.

It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer-executable instructions and that, although a logical order is illustrated in the flowcharts, in some cases, the steps illustrated or described may be performed in an order different than presented herein.

The embodiment of the present application further provides an emotion classification recognition device, and it should be noted that the emotion classification recognition device in the embodiment of the present application may be used to execute the emotion classification recognition method provided in the embodiment of the present application. The following describes an emotion classification recognition apparatus provided in an embodiment of the present application.

FIG. 5 is a schematic diagram of an emotion classification recognition apparatus according to an embodiment of the present application. As shown in fig. 5, the apparatus includes: an obtaining unit 52, configured to obtain a fused electrophysiological signal of the object to be identified, where the fused electrophysiological signal is determined according to the electroencephalogram physiological signal, the blood flow physiological signal, and the electrocardiograph physiological signal; an analyzing unit 54, configured to analyze the fused electrophysiological signal by using a hybrid neural network model, and determine an emotion category corresponding to the fused electrophysiological signal, where the hybrid neural network model is trained by machine learning using multiple sets of training data, and each set of training data in the multiple sets of training data includes: and fusing the electrophysiological signals and the emotion categories calibrated by each fused electrophysiological signal.

It should be noted that the obtaining unit 52 in this embodiment may be configured to execute step S102 in this embodiment, and the analyzing unit 54 in this embodiment may be configured to execute step S104 in this embodiment. The above units are the same as the examples and application scenarios realized by the corresponding steps, but are not limited to the disclosure of the above embodiments.

Optionally, in the apparatus for identifying an emotion category provided in an embodiment of the present application, the obtaining unit includes: the first acquisition module is used for acquiring the electroencephalogram physiological signals of the object to be identified in a preset time period; the second acquisition module is used for acquiring a blood flow physiological signal of the object to be identified in the preset time period; the third acquisition module is used for acquiring the electrocardio physiological signals of the object to be identified in the preset time period; the first determining module is used for determining the fusion electrophysiological signal according to the electroencephalogram physiological signal, the blood flow physiological signal and the electrocardio physiological signal of the object to be identified in the same preset time period.

Optionally, in the apparatus for identifying an emotion category provided in the embodiment of the present application, the first determining module includes: the first determining submodule is used for determining the electroencephalogram physiological signals acquired through the first channel as a first-class signal matrix; the second determining submodule is used for determining the blood flow physiological signals acquired through the second channel into a second signal matrix; the third determining submodule is used for determining the electrocardio-physiological signals acquired through the third channel as a third signal matrix; and the fourth determining submodule is used for splicing the first type signal matrix, the second type signal matrix and the third type signal matrix in the column dimension of the matrix to generate a fused electrophysiological signal.

Optionally, in the apparatus for identifying an emotion category provided in the embodiment of the present application, the apparatus further includes: a fourth obtaining module, configured to obtain a sample signal of an emotion of at least one sample object with respect to a preset category before analyzing the fused electrophysiological signal using a hybrid neural network model and determining an emotion category corresponding to the fused electrophysiological signal, where the sample signal at least includes: the electroencephalogram physiological signal, the blood flow physiological signal and the electrocardio physiological signal in the same sample time interval; the preprocessing module is used for preprocessing the sample signal to obtain a training signal; and the calibration module is used for calibrating the training signals according to the emotion of the preset category to generate the training data.

Optionally, in the apparatus for identifying an emotion category provided in the embodiment of the present application, the preprocessing module includes: the first processing module is used for carrying out baseline wandering removal processing on the sample signal to obtain a first processing signal; the second processing module is used for carrying out notch processing on the first processing signal to obtain a second processing signal; and the third processing module is used for carrying out preset noise removal processing on the electroencephalogram physiological signal in the second processing signal to obtain the training signal.

Optionally, in the apparatus for identifying an emotion category provided in the embodiment of the present application, the first processing module includes: an identification module for identifying missing values of each of the sample signals; the second determining module is used for determining a missing time interval corresponding to the missing value; a deleting module for deleting a plurality of signals of the missing period; and the fourth processing module is used for carrying out data smoothing processing on the signals connected in the missing time period to obtain the first processed signal.

Optionally, in the apparatus for identifying an emotion category provided in the embodiment of the present application, the apparatus further includes: the first extraction module is used for extracting the spatial features of the fused electrophysiological signals in each group of training data by using a first convolution kernel to obtain a spatial feature map before analyzing the fused electrophysiological signals by using a hybrid neural network model and determining the emotion types corresponding to the fused electrophysiological signals; the second extraction module is used for extracting the time characteristics of the fused electrophysiological signals in each group of training data by using a second convolution kernel to obtain a time characteristic diagram; the combination processing module is used for combining the spatial characteristic diagram and the time characteristic diagram to obtain a physiological signal characteristic diagram; and the fifth processing module is used for performing pooling processing on the physiological signal characteristic diagram to generate a characteristic sequence vector.

Optionally, in the apparatus for identifying an emotion category provided in the embodiment of the present application, the apparatus further includes: the preset long-short term memory artificial neural network module is used for processing the characteristic sequence vector to generate a classification characteristic diagram; the preset classifier is used for outputting the classification result of the emotion classification according to the classification feature map; and the preset optimizer is used for adjusting parameters according to the classification result of the emotion classification to obtain the hybrid neural network model.

The emotion category identification device provided by the embodiment of the application acquires the fused electrophysiological signal of the object to be identified, wherein the fused electrophysiological signal is determined according to the electroencephalogram physiological signal, the blood flow physiological signal and the electrocardio physiological signal; use mixed neural network model to carry out the analysis to fusing electrophysiological signal, confirm the emotion classification that fuses electrophysiological signal and correspond, wherein, mixed neural network model is for using multiunit training data to train out through machine learning, and every group training data in the multiunit training data all includes: the electrophysiological signals and the emotion categories calibrated by each electrophysiological signal are fused, so that the problem that emotion recognition cannot be performed according to the electrophysiological signals in the related technology is solved, and the technical effect of performing emotion recognition according to the electrophysiological signals is further achieved.

The emotion classification recognition device comprises a processor and a memory, wherein the units and the like are stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions.

The processor comprises a kernel, and the kernel calls the corresponding program unit from the memory. One or more than one kernel can be set, and the technical effect of emotion recognition according to electrophysiological signals is achieved by adjusting kernel parameters.

The memory may include volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM), including at least one memory chip.

An embodiment of the present invention provides a computer-readable storage medium, on which a program is stored, which, when executed by a processor, implements the method for recognizing emotion categories.

The embodiment of the invention provides a processor, which is used for running a program, wherein the method for identifying the emotion category is executed when the program runs.

Fig. 6 is a schematic diagram of an electronic device according to an embodiment of the present application, and as shown in fig. 6, an embodiment of the present invention provides an electronic device 60, which includes a processor 62, a memory 64, and a program stored in the memory and executable on the processor, and the processor executes the program to implement the following steps: acquiring a fusion electrophysiological signal of an object to be identified, wherein the fusion electrophysiological signal is determined according to the electroencephalogram physiological signal, the blood flow physiological signal and the electrocardio physiological signal; use mixed neural network model to carry out the analysis to fusing electrophysiological signal, confirm the emotion classification that fuses electrophysiological signal and correspond, wherein, mixed neural network model is for using multiunit training data to train out through machine learning, and every group training data in the multiunit training data all includes: and fusing the electrophysiological signals and the emotion categories calibrated by each fused electrophysiological signal.

Optionally, the processor executes the program to implement the following steps: acquiring an electroencephalogram physiological signal of an object to be identified in a preset time period; acquiring a blood flow physiological signal of an object to be identified in a preset time period; acquiring an electrocardio physiological signal of an object to be identified in a preset time period; and determining a fused electrophysiological signal according to the electroencephalogram physiological signal, the blood flow physiological signal and the electrocardio physiological signal of the object to be identified in the same preset time period.

Optionally, the processor executes the program to implement the following steps: determining the electroencephalogram physiological signals acquired through the first channel as a first-class signal matrix; determining the blood flow physiological signals acquired through the second channel as a second type signal matrix; determining the electrocardio-physiological signals acquired through the third channel as a third signal matrix; and splicing the first type signal matrix, the second type signal matrix and the third type signal matrix in the column dimension of the matrix to generate the fused electrophysiological signal.

Optionally, the processor executes the program to implement the following steps: before analyzing the fused electrophysiological signal by using the hybrid neural network model and determining the emotion category corresponding to the fused electrophysiological signal, obtaining sample signals of at least one sample object for the emotion of a preset category, wherein the sample signals at least comprise: the electroencephalogram physiological signal, the blood flow physiological signal and the electrocardio physiological signal in the same sample time interval; preprocessing a sample signal to obtain a training signal; and calibrating the training signals according to the emotion of the preset category to generate training data.

Optionally, the processor executes the program to implement the following steps: carrying out baseline wandering removal processing on the sample signal to obtain a first processing signal; carrying out notch processing on the first processing signal to obtain a second processing signal; and performing preset noise removal processing on the electroencephalogram physiological signal in the second processed signal to obtain a training signal.

Optionally, the processor executes the program to implement the following steps: identifying missing values for each of the sample signals; determining a missing time period corresponding to the missing value; deleting the plurality of signals of the missing period; and carrying out data smoothing processing on the signals connected in the missing time interval to obtain a first processed signal.

Optionally, the processor executes the program to implement the following steps: before analyzing the fused electrophysiological signals by using a hybrid neural network model and determining emotion categories corresponding to the fused electrophysiological signals, extracting spatial features of the fused electrophysiological signals in each group of training data by using a first convolution kernel to obtain a spatial feature map; extracting the time characteristics of the fused electrophysiological signals in each group of training data by using a second convolution kernel to obtain a time characteristic diagram; combining the spatial characteristic diagram and the temporal characteristic diagram to obtain a physiological signal characteristic diagram; and performing pooling processing on the physiological signal characteristic diagram to generate a characteristic sequence vector.

Optionally, the processor executes the program to implement the following steps: processing the characteristic sequence vector by using a preset long-short term memory artificial neural network to generate a classification characteristic diagram; outputting a classification result of the emotion classification according to the classification feature map by using a preset classifier; and adjusting parameters by using a preset optimizer according to the classification result of the emotion classification to obtain the hybrid neural network model.

The device herein may be a server, a PC, a PAD, a mobile phone, etc.

The present application further provides a computer program product adapted to perform a program for initializing the following method steps when executed on a data processing device: acquiring a fusion electrophysiological signal of an object to be identified, wherein the fusion electrophysiological signal is determined according to an electroencephalogram physiological signal, a blood flow physiological signal and an electrocardio physiological signal; use mixed neural network model to carry out the analysis to fusing electrophysiological signal, confirm the emotion classification that fuses electrophysiological signal and correspond, wherein, mixed neural network model is for using multiunit training data to train out through machine learning, and every group training data in the multiunit training data all includes: and fusing the electrophysiological signals and the emotion categories calibrated by each fused electrophysiological signal.

Alternatively, when executed on a data processing device, is adapted to perform a procedure for initializing the following method steps: acquiring an electroencephalogram physiological signal of an object to be identified in a preset time period; acquiring a blood flow physiological signal of an object to be identified in a preset time period; acquiring an electrocardio physiological signal of an object to be identified in a preset time period; and determining a fused electrophysiological signal according to the electroencephalogram physiological signal, the blood flow physiological signal and the electrocardio physiological signal of the object to be identified in the same preset time period.

Alternatively, it is adapted to perform a procedure when executed on a data processing device, which initializes the following method steps: determining the electroencephalogram physiological signals acquired through the first channel as a first-class signal matrix; determining the blood flow physiological signals acquired through the second channel as a second type signal matrix; determining the electrocardio-physiological signals acquired through the third channel as a third signal matrix; and splicing the first type signal matrix, the second type signal matrix and the third type signal matrix in the column dimension of the matrix to generate the fused electrophysiological signal.

Alternatively, it is adapted to perform a procedure when executed on a data processing device, which initializes the following method steps: before analyzing the fused electrophysiological signal by using the hybrid neural network model and determining the emotion category corresponding to the fused electrophysiological signal, obtaining sample signals of at least one sample object for the emotion of a preset category, wherein the sample signals at least comprise: the electroencephalogram physiological signal, the blood flow physiological signal and the electrocardio physiological signal in the same sample time interval; preprocessing a sample signal to obtain a training signal; and calibrating the training signals according to the emotion of the preset category to generate training data.

Alternatively, it is adapted to perform a procedure when executed on a data processing device, which initializes the following method steps: carrying out baseline wandering removal processing on the sample signal to obtain a first processing signal; carrying out notch processing on the first processing signal to obtain a second processing signal; and performing preset noise removal processing on the electroencephalogram physiological signal in the second processed signal to obtain a training signal.

Alternatively, it is adapted to perform a procedure when executed on a data processing device, which initializes the following method steps: identifying missing values of each signal in the sample signal; determining a missing time period corresponding to the missing value; deleting a plurality of signals of the missing period; and carrying out data smoothing processing on the signals connected in the missing time interval to obtain a first processed signal.

Alternatively, it is adapted to perform a procedure when executed on a data processing device, which initializes the following method steps: before analyzing the fused electrophysiological signals by using a hybrid neural network model and determining emotion categories corresponding to the fused electrophysiological signals, extracting spatial features of the fused electrophysiological signals in each group of training data by using a first convolution kernel to obtain a spatial feature map; extracting the time characteristics of the fused electrophysiological signals in each group of training data by using a second convolution kernel to obtain a time characteristic diagram; combining the spatial characteristic diagram and the temporal characteristic diagram to obtain a physiological signal characteristic diagram; and performing pooling processing on the physiological signal characteristic diagram to generate a characteristic sequence vector.

Alternatively, it is adapted to perform a procedure when executed on a data processing device, which initializes the following method steps: processing the characteristic sequence vector by using a preset long-short term memory artificial neural network to generate a classification characteristic diagram; outputting a classification result of the emotion classification according to the classification feature map by using a preset classifier; and adjusting parameters by using a preset optimizer according to the classification result of the emotion classification to obtain the hybrid neural network model.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.

The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for recognizing emotion categories is characterized by comprising the following steps:

acquiring a fusion electrophysiological signal of an object to be identified, wherein the fusion electrophysiological signal is determined according to an electroencephalogram physiological signal, a blood flow physiological signal and an electrocardio physiological signal;

analyzing the fused electrophysiological signal by using a hybrid neural network model, and determining an emotion category corresponding to the fused electrophysiological signal, wherein the hybrid neural network model is trained by machine learning using a plurality of sets of training data, and each set of training data in the plurality of sets of training data includes: and fusing the electrophysiological signals and the emotion categories calibrated by each fused electrophysiological signal.

2. The method of claim 1, wherein acquiring the fused electrophysiological signal of the object to be identified comprises:

acquiring an electroencephalogram physiological signal of the object to be identified in a preset time period;

acquiring a blood flow physiological signal of the object to be identified in the preset time period;

acquiring the electrocardio physiological signals of the object to be identified in the preset time period;

and determining the fused electrophysiological signal according to the electroencephalogram physiological signal, the blood flow physiological signal and the electrocardio physiological signal of the object to be identified in the same preset time period.

3. The method of claim 2, wherein determining the fused electrophysiological signal from the EEG physiological signal, the blood flow physiological signal, and the ECG physiological signal for the same predetermined time period comprises:

determining the electroencephalogram physiological signals acquired through the first channel as a first-class signal matrix;

determining the blood flow physiological signals acquired through the second channel as a second type signal matrix;

determining the electrocardio-physiological signals acquired through the third channel as a third signal matrix;

and splicing the first type signal matrix, the second type signal matrix and the third type signal matrix in the column dimension of the matrix to generate the fused electrophysiological signal.

4. The method of claim 1, wherein prior to analyzing the fused electrophysiological signal using a hybrid neural network model to determine an emotion classification to which the fused electrophysiological signal corresponds, the method further comprises:

obtaining a sample signal of at least one sample object for a preset category of emotions, wherein the sample signal at least comprises: the electroencephalogram physiological signal, the blood flow physiological signal and the electrocardio physiological signal in the same sample time interval;

preprocessing the sample signal to obtain a training signal;

and calibrating the training signals according to the emotion of the preset category to generate the training data.

5. The method of claim 4, wherein preprocessing the sample signal to obtain a training signal comprises:

performing baseline wandering removal processing on the sample signal to obtain a first processing signal;

carrying out notch processing on the first processed signal to obtain a second processed signal;

and performing preset noise removal processing on the electroencephalogram physiological signal in the second processed signal to obtain the training signal.

6. The method of claim 1, wherein prior to analyzing the fused electrophysiological signal using a hybrid neural network model to determine an emotion classification to which the fused electrophysiological signal corresponds, the method further comprises:

extracting the spatial features of the fused electrophysiological signals in each group of training data by using a first convolution kernel to obtain a spatial feature map;

extracting the time characteristics of the fused electrophysiological signals in each group of training data by using a second convolution kernel to obtain a time characteristic diagram;

combining the spatial characteristic diagram and the temporal characteristic diagram to obtain a physiological signal characteristic diagram;

and performing pooling processing on the physiological signal characteristic diagram to generate a characteristic sequence vector.

7. The method of claim 6, wherein after pooling the physiological signal feature map to generate a feature sequence vector, the method further comprises:

processing the characteristic sequence vector by using a preset long-short term memory artificial neural network to generate a classification characteristic diagram;

outputting a classification result of the emotion classification according to the classification feature map by using a preset classifier;

and adjusting parameters by using a preset optimizer according to the classification result of the emotion classification to obtain the hybrid neural network model.

8. An apparatus for emotion classification, comprising:

the system comprises an acquisition unit, a recognition unit and a processing unit, wherein the acquisition unit is used for acquiring a fusion electrophysiological signal of an object to be recognized, and the fusion electrophysiological signal is determined according to an electroencephalogram physiological signal, a blood flow physiological signal and an electrocardio physiological signal;

an analysis unit, configured to analyze the fused electrophysiological signal using a hybrid neural network model, and determine an emotion category corresponding to the fused electrophysiological signal, where the hybrid neural network model is trained through machine learning using multiple sets of training data, and each set of training data in the multiple sets of training data includes: and fusing the electrophysiological signals and the emotion categories calibrated by each fused electrophysiological signal.

9. A processor, characterized in that the processor is configured to run a program, wherein the program is configured to execute the method for identifying an emotion classification according to any one of claims 1 to 7 when the program is run.

10. An electronic device comprising one or more processors and memory storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of sentiment category identification of any one of claims 1 to 7.