CN114494969A - Emotion recognition method based on multimode voice information complementary AND gate control - Google Patents
Emotion recognition method based on multimode voice information complementary AND gate control Download PDFInfo
- Publication number
- CN114494969A CN114494969A CN202210106236.8A CN202210106236A CN114494969A CN 114494969 A CN114494969 A CN 114494969A CN 202210106236 A CN202210106236 A CN 202210106236A CN 114494969 A CN114494969 A CN 114494969A
- Authority
- CN
- China
- Prior art keywords
- features
- fusion
- representation
- mode
- text
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 28
- 230000008909 emotion recognition Effects 0.000 title claims abstract description 26
- 230000000295 complement effect Effects 0.000 title 1
- 230000004927 fusion Effects 0.000 claims abstract description 45
- 230000007246 mechanism Effects 0.000 claims abstract description 24
- 230000008451 emotion Effects 0.000 claims abstract description 9
- 230000002457 bidirectional effect Effects 0.000 claims abstract description 7
- 238000010606 normalization Methods 0.000 claims description 12
- 230000002708 enhancing effect Effects 0.000 claims description 6
- 238000012549 training Methods 0.000 claims description 5
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 3
- 238000012512 characterization method Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 230000010354 integration Effects 0.000 description 4
- 239000013598 vector Substances 0.000 description 4
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 238000002679 ablation Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000001149 cognitive effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002996 emotional effect Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000004880 explosion Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 230000001755 vocal effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/047—Probabilistic or stochastic networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Probability & Statistics with Applications (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Image Analysis (AREA)
Abstract
The invention provides an emotion recognition method based on multimode voice information complementation and gate control, which belongs to the technical field of multimode emotion recognition and comprises the following steps: s1, extracting audio features and text features in the target video; s2, performing feature bidirectional fusion on the audio features and the text features; s3, the proportion of fusion representation in the result of bidirectional fusion in S2 is adjusted through a learnable door control mechanism and output; and S4, splicing the output of the learnable door control mechanism in S3, and finally obtaining emotion category output. According to the invention, a gating mechanism is applied to the cross attention module to determine whether to retain source modal information or cover target modal information, and the proportion of the source modal information and the target modal information is adjusted, so that the identification accuracy and the model parameters are balanced.
Description
Technical Field
The invention relates to the technical field of multi-modal emotion recognition, in particular to an emotion recognition method based on multi-modal voice information complementation and gate control.
Background
Emotion plays a key role in interpersonal communication, and not only linguistic information, but also acoustic information conveys an individual's emotional state. In many areas, such as human-computer interaction, healthcare, and cognitive sciences, there is considerable interest in developing tools to identify emotions in human vocal expressions. The recent explosion of deep learning has also promoted the development of emotion recognition, and in addition, the demand of applications has promoted the development of high-performance lightweight models.
There are many existing works to improve the performance of speech emotion recognition based on audio-only features. The LLDs-based characterization is extracted by deep learning networks, such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), etc. Some variant modular structures, such as CNN-LSTM, are also used in this area to extract feature sequences and capture time dependencies.
However, the language information and the sound information are equally important for emotion recognition. Therefore, both text and audio modalities should be taken into account in order to accomplish the task of multi-modal emotion recognition. For audio modalities, the process of feature extraction is similar to that of single-modality speech emotion recognition. For text mode, a word embedding model like GloVe is typically used. What makes multimodal emotion recognition more challenging than single modality emotion recognition is the process of modality fusion. Some early work incorporated different features as inputs to deep neural networks, and in order to fuse modes at a deeper level, the Transformer architecture was widely used such that the learned modal fusion characterization was enhanced.
Despite the improvements made by previous work, few considerations are given to the scale and balance of modal fusion characterizations.
Disclosure of Invention
The invention aims to provide an emotion recognition method based on multi-mode voice information complementation and gate control, which can adjust the proportion of modal fusion characterization and realize the balance of emotion recognition accuracy and model parameter quantity.
In order to achieve the purpose, the invention adopts the technical scheme that:
the emotion recognition method based on multimode voice information complementation and gate control comprises the following steps: s1, extracting audio features and text features in the target video; s2, performing feature bidirectional fusion on the audio features and the text features; s3, the proportion of fusion representation in the result of bidirectional fusion in S2 is adjusted through a learnable door control mechanism and output; and S4, splicing the output of the learnable door control mechanism in S3, and finally obtaining emotion category output.
S2 includes: the text features are taken as a source modality, the audio features are taken as a target modality, the text features are taken as first original modality representations, and the text features and the audio features are fused through a Transformer cross attention mechanism to obtain first fusion representations; and taking the audio features as a source modality, taking the text features as a target modality, taking the audio features as a second original modality representation, and fusing the audio features and the text features through a Transformer cross attention mechanism to obtain a second fused representation.
S2 includes: taking the text characteristic as a source mode, taking the audio characteristic as a target mode, and then taking the text characteristic as a first original mode representation; fusing the text features and the audio features through a Transformer cross attention mechanism; performing cross-layer connection and normalization through a residual error module to obtain a first intermediate fusion representation; enhancing the first intermediate fusion representation through the full-connection layer and normalization to obtain a first fusion representation; the audio features are characterized by a second original mode by taking the audio features as a source mode and the text features as a target mode; fusing the audio features and the text features through a Transformer cross attention mechanism; performing cross-layer connection and normalization through a residual error module to obtain a second intermediate fusion representation; and enhancing the second intermediate fusion representation through the full-connection layer and normalization to obtain a second fusion representation.
S3 is: and respectively fusing the first fusion characteristics and the first original mode characteristics according to a proportion through a learnable door control mechanism to obtain first intermediate output, and fusing the second fusion characteristics and the second original mode characteristics according to a proportion to obtain second intermediate output.
S4 is: and splicing the first intermediate output and the second intermediate output to finally obtain emotion category output.
The method is deployed on a public data set CMU-MOSEI, and optimized by using an Adam optimizer in the training process.
Drawings
The invention and its features, aspects and advantages will become more apparent from reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings. Like reference symbols in the various drawings indicate like elements. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention.
FIG. 1 is an architecture diagram of an emotion recognition method based on multi-mode voice information complementation and gate control provided by the present invention;
fig. 2 is a comparison graph of parameters and F1 values of the CMU-MOSEI in different models provided by the present invention.
Detailed Description
The invention will be further described with reference to the following drawings and specific examples, which are not intended to limit the invention thereto.
In the prior art, most speech emotion recognition models only consider information of a speech modality but do not consider texts, namely semantic information of the texts, and lack balance fusion of the semantic information and audio information; most of current networks are influenced by large-scale pre-training models, parameters are huge, and the networks are difficult to fall on the ground under scenes with high requirements on instantaneity and light weight.
According to the emotion recognition method based on multimode speech information complementation and gate control, as shown in fig. 1, firstly, audio features and text features in a target video are extracted, for a text mode, pre-trained GloVe word embedding is used for processing, and the embedding is a 300-dimensional vector. For audio modalities, covapr is used to extract low-level 74-dimensional vectors including 12 mel-frequency cepstral coefficients (MFCCs), pitch tracking and acoustic/non-acoustic segmentation features, peak slope parameters and pitch maxima.
Respectively extracting audio features and text features by using CNN-BilSTM and BilSTM, and performing extraction on text sequencesIt is encoded using BilSTM, which can be expressed as
Ht=BiLSTM(Xt)
For audio sequences, the audio sequence is represented asUsing one-dimensional convolution operations, i.e.
Then, BilSTM encodes it again as input,
After extracting the features, the cross attention module of the Transformer reinforces the features of one mode with the features of the other mode, wherein a door mechanism is used as a flow control unit to balance the proportion of the two modes.
Define a source modality asThe target modality is defined asWherein { S, T }. belongs to { T, a }, namely, the source mode is text, the target mode is audio fusion, the source mode is audio and the target mode is text bidirectional fusion. In particular, let dS=dTD, i.e. the source modality is consistent with the target modality dimension.
In this embodiment, the text feature is used as the source modality HtTargeting audio features to modality HaIf the text feature is the first original mode representation; text features and audio features are fused through a transform Cross Attention mechanism, namely Cross Attention, and the formula is as follows:
Q=WQ×Ha
K=WK×Ht
V=WV×Ht
whereinIs characterized by the linear transformation of the original features. Then, performing cross-layer connection and normalization through a residual error module:
ht→a=LN(H′+Ha)
whereinA first intermediate fusion characterization is represented that fuses the source modality to the target modality.
And finally, enhancing the first intermediate fusion representation again by using the full-connection layer and normalization to obtain:
Ht→a′=LN((ht→a+FFN(ht→a))
whereinRepresents the source modality HtTowards target modality HaThe first fused representation of the fusion, FFN, represents the fully-connected layer.
Audio feature-based source modality HaTargeting text features as a target modality HtIf so, the audio feature is a second original mode representation; and fusing the audio features and the text features through a Transformer cross attention mechanism, wherein the formula is as follows:
Q=WQ×Ht
K=WK×Ha
V=WV×Ha
whereinIs characterized by the linear transformation of the original features. Then, performing cross-layer connection and normalization through a residual error module:
ha→t=LN(H′+Ht)
whereinA second intermediate fusion characterization representing a fusion of the source modality to the target modality.
And finally, enhancing the second intermediate fusion representation again by using the full-connection layer and the normalization to obtain:
Ha→t′=LN((ht→a+FFN(ht→a))
whereinRepresents the source modality HaTowards target modality HtThe second fused representation of the fusion, FFN, represents the fully-connected layer.
And then proportionally fusing the first fusion characteristic and the first original modal characteristic through a learnable gate control mechanism to obtain a first intermediate output:
Ht→a=Ht→a′×Gi+Ha×Gr
whereinA representation integration gate for adjusting the information weight after the integration,indicating a reservation gate for adjusting the weight of the original information.
And proportionally fusing the second fusion characteristic and the second original modal characteristic through a learnable door control mechanism to obtain a second intermediate output:
Ha→t=Ha→t′×Gi+Ht×Gr
whereinA representation integration gate for adjusting the information weight after the integration,indicating a reservation gate for adjusting the weight of the original information.
And finally, splicing the first intermediate output and the second intermediate output to finally obtain emotion category output, wherein the formula is as follows:
wherein,a vector representation representing the fusion of a source modality to a target modality,a vector representation representing the fusion of the target modality to the source modality, [ ·,·]representing a splicing operation, the Transformer representing a Transformer encoder,representing predicted emotion categories.
Finally, the invention applies the method to the public data set CMU-MOSEI, using an Adam optimizer in the training process, using a learning rate attenuation technique. The experimental result shows that the model based on the method achieves the best performance with the minimum parameter number (only 0.432M), the method balances the accuracy and the parameter number, has considerable accuracy under the condition of ensuring practical application, and pays attention to light-weight application under the condition of ensuring the accuracy. The specific alignment is shown in fig. 2.
In summary, the invention fuses various modal information based on the Transformer cross complementation module, and the learnable gating mechanism controls information flow and stabilizes the training process. In the experiments, models based on the present method were compared to baseline models and further ablation studies were performed. The results show the effectiveness of each component, which is the basis for the performance and lightweight of the inventive model. The algorithm balances accuracy and parameter quantity, compared with the existing method, the model provided by the invention can achieve the best effect with the minimum parameter quantity, and an experimental result shows that the method is more beneficial to the application of an actual scene.
The present invention applies a gating control mechanism to the cross attention module to decide whether to retain the source modality information or to override the target modality information. In addition, most of the existing models rely on a large number of learnable parameters, and potential applications in some fields which have prospects but need real-time and lightweight models, such as human-computer interaction, are ignored. Therefore, a lightweight model is necessary to improve the feasibility and utility of speech emotion recognition applications.
The above description is of the preferred embodiment of the invention; it is to be understood that the invention is not limited to the particular embodiments described above, in that devices and structures not described in detail are understood to be implemented in a manner common in the art; any person skilled in the art can make many possible variations and modifications, or modify equivalent embodiments, without departing from the technical solution of the invention, without affecting the essence of the invention; therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention are still within the scope of the protection of the technical solution of the present invention, unless the contents of the technical solution of the present invention are departed.
Claims (6)
1. The emotion recognition method based on multimode voice information complementation and gate control is characterized by comprising the following steps of:
s1, extracting audio features and text features in the target video;
s2, performing feature bidirectional fusion on the audio features and the text features;
s3, adjusting the proportion of fusion representation in the result of the bidirectional fusion in the S2 through a learnable door control mechanism, and outputting the proportion;
and S4, splicing the outputs of the learnable door control mechanism in the S3, and finally obtaining emotion category outputs.
2. The method for emotion recognition based on multi-mode speech information complementation and gate control as claimed in claim 1, wherein said S2 comprises:
the text features are taken as a source modality, the audio features are taken as a target modality, the text features are taken as first original modality representations, and the text features and the audio features are fused through a Transformer cross attention mechanism to obtain first fusion representations;
and taking the audio features as a source modality, taking the text features as a target modality, taking the audio features as a second original modality representation, and fusing the audio features and the text features through a Transformer cross attention mechanism to obtain a second fused representation.
3. The method for emotion recognition based on multi-mode speech information complementation and gate control as claimed in claim 1, wherein said S2 comprises:
the text characteristic is taken as a source mode, the audio characteristic is taken as a target mode, and the text characteristic is taken as a first original mode representation;
fusing text features and audio features through a Transformer cross attention mechanism; performing cross-layer connection and normalization through a residual error module to obtain a first intermediate fusion representation;
enhancing the first intermediate fusion representation through the full-connection layer and normalization to obtain a first fusion representation;
the audio features are characterized by a second original mode by taking the audio features as a source mode and the text features as a target mode;
fusing the audio features and the text features through a Transformer cross attention mechanism; performing cross-layer connection and normalization through a residual error module to obtain a second intermediate fusion representation;
and enhancing the second intermediate fusion representation through the full-connection layer and normalization to obtain a second fusion representation.
4. The method for emotion recognition based on multi-mode speech information complementation and gate control as claimed in claim 2 or 3, wherein said S3 is:
and respectively fusing the first fusion characteristics and the first original mode characteristics according to a proportion through a learnable door control mechanism to obtain first intermediate output, and fusing the second fusion characteristics and the second original mode characteristics according to a proportion to obtain second intermediate output.
5. The method for emotion recognition based on multi-mode speech information complementation and gate control as claimed in claim 4, wherein said S4 is:
and splicing the first intermediate output and the second intermediate output to finally obtain emotion category output.
6. The method for emotion recognition based on multimode speech information complementation and gate control as claimed in claim 1, wherein the method is deployed on a public data set CMU-MOSEI, optimized using Adam optimizer during training.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210106236.8A CN114494969A (en) | 2022-01-28 | 2022-01-28 | Emotion recognition method based on multimode voice information complementary AND gate control |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210106236.8A CN114494969A (en) | 2022-01-28 | 2022-01-28 | Emotion recognition method based on multimode voice information complementary AND gate control |
Publications (1)
Publication Number | Publication Date |
---|---|
CN114494969A true CN114494969A (en) | 2022-05-13 |
Family
ID=81477008
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202210106236.8A Pending CN114494969A (en) | 2022-01-28 | 2022-01-28 | Emotion recognition method based on multimode voice information complementary AND gate control |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN114494969A (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115238749A (en) * | 2022-08-04 | 2022-10-25 | 中国人民解放军军事科学院***工程研究院 | Feature fusion modulation identification method based on Transformer |
CN117423168A (en) * | 2023-12-19 | 2024-01-19 | 湖南三湘银行股份有限公司 | User emotion recognition method and system based on multi-modal feature fusion |
-
2022
- 2022-01-28 CN CN202210106236.8A patent/CN114494969A/en active Pending
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115238749A (en) * | 2022-08-04 | 2022-10-25 | 中国人民解放军军事科学院***工程研究院 | Feature fusion modulation identification method based on Transformer |
CN115238749B (en) * | 2022-08-04 | 2024-04-23 | 中国人民解放军军事科学院***工程研究院 | Modulation recognition method based on feature fusion of transducer |
CN117423168A (en) * | 2023-12-19 | 2024-01-19 | 湖南三湘银行股份有限公司 | User emotion recognition method and system based on multi-modal feature fusion |
CN117423168B (en) * | 2023-12-19 | 2024-04-02 | 湖南三湘银行股份有限公司 | User emotion recognition method and system based on multi-modal feature fusion |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Huang et al. | Attention assisted discovery of sub-utterance structure in speech emotion recognition. | |
Gu et al. | Speech intention classification with multimodal deep learning | |
JP2023509031A (en) | Translation method, device, device and computer program based on multimodal machine learning | |
CN114694076A (en) | Multi-modal emotion analysis method based on multi-task learning and stacked cross-modal fusion | |
Shashidhar et al. | Combining audio and visual speech recognition using LSTM and deep convolutional neural network | |
Seo et al. | Wav2kws: Transfer learning from speech representations for keyword spotting | |
CN115329779B (en) | Multi-person dialogue emotion recognition method | |
CN114494969A (en) | Emotion recognition method based on multimode voice information complementary AND gate control | |
Zhang et al. | Multi-head attention fusion networks for multi-modal speech emotion recognition | |
CN111382257A (en) | Method and system for generating dialog context | |
CN112597841B (en) | Emotion analysis method based on door mechanism multi-mode fusion | |
CN110569869A (en) | feature level fusion method for multi-modal emotion detection | |
CN111274412A (en) | Information extraction method, information extraction model training device and storage medium | |
Qu et al. | Lipsound2: Self-supervised pre-training for lip-to-speech reconstruction and lip reading | |
CN114882862A (en) | Voice processing method and related equipment | |
Xu et al. | A comprehensive survey of automated audio captioning | |
CN117892237A (en) | Multi-modal dialogue emotion recognition method and system based on hypergraph neural network | |
CN116955579B (en) | Chat reply generation method and device based on keyword knowledge retrieval | |
Singh et al. | A lightweight 2D CNN based approach for speaker-independent emotion recognition from speech with new Indian Emotional Speech Corpora | |
CN116860943A (en) | Multi-round dialogue method and system for dialogue style perception and theme guidance | |
CN114373443A (en) | Speech synthesis method and apparatus, computing device, storage medium, and program product | |
CN116052291A (en) | Multi-mode emotion recognition method based on non-aligned sequence | |
Liu et al. | Keyword retrieving in continuous speech using connectionist temporal classification | |
Shin et al. | Performance Analysis of a Chunk-Based Speech Emotion Recognition Model Using RNN. | |
Song et al. | Towards realizing sign language to emotional speech conversion by deep learning |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |