CN116189064A - Barrage emotion analysis method and system based on joint model - Google Patents

Barrage emotion analysis method and system based on joint model Download PDF

Info

Publication number
CN116189064A
CN116189064A CN202310458854.3A CN202310458854A CN116189064A CN 116189064 A CN116189064 A CN 116189064A CN 202310458854 A CN202310458854 A CN 202310458854A CN 116189064 A CN116189064 A CN 116189064A
Authority
CN
China
Prior art keywords
barrage
surrounding
emotion
comment
feature
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202310458854.3A
Other languages
Chinese (zh)
Other versions
CN116189064B (en
Inventor
宋彦
陈伟东
罗常凡
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
University of Science and Technology of China USTC
Original Assignee
University of Science and Technology of China USTC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by University of Science and Technology of China USTC filed Critical University of Science and Technology of China USTC
Priority to CN202310458854.3A priority Critical patent/CN116189064B/en
Publication of CN116189064A publication Critical patent/CN116189064A/en
Application granted granted Critical
Publication of CN116189064B publication Critical patent/CN116189064B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/16—Matrix or vector computation, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Multimedia (AREA)
  • Medical Informatics (AREA)
  • Pure & Applied Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Computational Mathematics (AREA)
  • Mathematical Analysis (AREA)
  • Mathematical Optimization (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computational Linguistics (AREA)
  • Algebra (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

The invention discloses a barrage emotion analysis method and system based on a joint model, wherein barrage comments are input into the trained joint model to output emotion tendencies corresponding to the barrage comments, the joint model comprises a coding module and a decoding module, the coding module comprises a video coding module, a text coding module, a gating fusion module and a multi-mode fusion module, the decoding module comprises a barrage reconstruction module and an emotion analysis module, and the decoding module takes the output of the coding module as input to output emotion analysis tendencies corresponding to the barrage comments; the barrage emotion analysis method and system utilize a gating fusion screening mechanism to take surrounding barrage comments as context information of target barrage comments, and utilize a multi-mode fusion mode to take video information into consideration, and fully utilize useful information to strengthen characteristic representation of video barrages so as to accurately identify emotion tendencies of the target barrage comments.

Description

Barrage emotion analysis method and system based on joint model
Technical Field
The invention relates to the technical field of barrage emotion analysis, in particular to a barrage emotion analysis method and system based on a joint model.
Background
The emotion analysis of the video barrage refers to the emotion polarity of real-time comments of the video.
The existing video barrage emotion analysis method is prone to extracting sentence-level features for emotion analysis and classification, and is based on grammar and semantics of rules, but the barrage is characterized in that: the method is short, has serious aphasia, uses special characters to represent specific meanings, has extremely irregular grammar and the like, so the traditional emotion analysis method cannot accurately segment the bullet screen properly, analyze grammar and the like, and further cannot accurately analyze emotion.
In addition, the existing barrage comment is short, insufficient context information is not available, grammar is very irregular, the existing barrage comment is related to a video theme at the time, interactivity is strong, instantaneity is strong, and the like, so that the existing method cannot effectively and accurately analyze emotion of the existing barrage comment in a short time.
Disclosure of Invention
Based on the technical problems in the background technology, the invention provides a barrage emotion analysis method and system based on a joint model, which can accurately identify emotion tendencies of target barrage comments.
According to the barrage emotion analysis method based on the joint model, barrage comments are input into the trained joint model to output emotion tendencies corresponding to the barrage comments;
the training process of the joint model is as follows:
s1: constructing a training sample set, the training sample set comprising moments
Figure SMS_2
Bullet comment>
Figure SMS_5
Time->
Figure SMS_7
To the point of
Figure SMS_3
Inward bullet comment->
Figure SMS_4
Surrounding video->
Figure SMS_6
And comment on bullet screen->
Figure SMS_8
Surrounding barrage comment +.>
Figure SMS_1
；
S2: for the video
Figure SMS_9
Coding and concatenating to obtain coded video feature ∈ ->
Figure SMS_10
Comment on the barrage->
Figure SMS_11
And the surrounding barrage comment->
Figure SMS_12
Coding to obtain the coded target barrage characteristic +.>
Figure SMS_13
And surrounding barrage features
Figure SMS_14
；
S3: based on the target barrage feature
Figure SMS_15
For the surrounding barrage feature->
Figure SMS_16
After screening and filtering, connecting in series to obtain all surrounding barrage comments +.>
Figure SMS_17
；
S4: video characterization through self-attention layers and cross-attention layers
Figure SMS_18
Target barrage feature->
Figure SMS_19
Comment on surrounding barrage->
Figure SMS_20
Enhancement processing to obtain enhanced video features>
Figure SMS_21
Enhanced target barrage feature->
Figure SMS_22
And reinforcing the surrounding barrage->
Figure SMS_23
；
S5, performing S5; enhancement of video features based on multi-layered multi-headed pairs of attention layers
Figure SMS_24
Enhanced target barrage feature->
Figure SMS_25
Reinforcing surrounding barrage->
Figure SMS_26
Reconstructing to obtain reconstructed barrage comments, and constructing a barrage reconstructed loss function by using the reconstructed barrage comments and the real barrage comments by using cross entropy>
Figure SMS_27
；
S6: for enhanced video features
Figure SMS_28
Enhanced target barrage feature->
Figure SMS_29
Reinforcing surrounding barrage->
Figure SMS_30
Sequentially carrying out regularization and normalization operations, and outputting the barrage comment +.>
Figure SMS_31
Corresponding to predicted barrage emotion +.>
Figure SMS_32
；
S7: predicted barrage emotion using cross entropy
Figure SMS_33
And true barrage emotion->
Figure SMS_34
Construction of a loss function for emotion prediction>
Figure SMS_35
Loss function based on barrage reconstruction>
Figure SMS_36
And loss function of emotion prediction->
Figure SMS_37
Calculating the overall loss function->
Figure SMS_38
Updating parameters of the joint model based on the total loss function and the back propagation algorithm until the performance of the joint model reaches a set expected value;
the surrounding barrage comments
Figure SMS_39
The calculation formula is as follows:
Figure SMS_40
Figure SMS_41
Figure SMS_42
wherein ,
Figure SMS_44
for post-selection->
Figure SMS_47
Comment on the surrounding bullet screen->
Figure SMS_50
Is->
Figure SMS_45
Comment on the surrounding bullet screen->
Figure SMS_48
Is (are) peripheral features of->
Figure SMS_51
，/>
Figure SMS_53
Is a learnable gate matrix +.>
Figure SMS_43
Is a learnable gate offset vector, +.>
Figure SMS_46
For ReLU function>
Figure SMS_49
Representing series connection,/->
Figure SMS_52
Representing the product.
Further, the video features
Figure SMS_54
Is calculated by the formula of (2)The following are provided:
Figure SMS_55
the target barrage feature
Figure SMS_56
The calculation formula of (2) is as follows:
Figure SMS_57
the surrounding barrage feature
Figure SMS_58
The calculation formula of (2) is as follows:
Figure SMS_59
wherein ,
Figure SMS_60
，/>
Figure SMS_61
，/>
Figure SMS_62
representing series connection,/->
Figure SMS_63
Representing a video encoder>
Figure SMS_64
Representing a long and short term memory network.
Further, in step S4: video characterization through self-attention layers and cross-attention layers
Figure SMS_65
Target barrage feature->
Figure SMS_66
Comment on surrounding barrage->
Figure SMS_67
Enhancement processing to obtain enhanced video features>
Figure SMS_68
Enhanced target barrage feature->
Figure SMS_69
And reinforcing the surrounding barrage->
Figure SMS_70
Specifically, the method comprises the following steps:
characterizing video
Figure SMS_71
Target barrage feature->
Figure SMS_72
Comment on surrounding barrage->
Figure SMS_73
Inputting as a first layer of the self-attention layer and the cross-attention layer and performing an L-layer iteration, wherein the L-layer is the total layer number of the self-attention layer and the cross-attention layer;
in the first place
Figure SMS_74
Layer input video feature->
Figure SMS_75
Obtaining the input video feature of the next layer +.>
Figure SMS_76
The following are provided:
Figure SMS_77
/>
in the first place
Figure SMS_78
Layer input target barrage feature->
Figure SMS_79
Obtaining the input target barrage feature of the next layer>
Figure SMS_80
：
Figure SMS_81
In the first place
Figure SMS_82
Layer input surrounding barrage comment->
Figure SMS_83
Obtaining the comment +.>
Figure SMS_84
：
Figure SMS_85
Where SA represents the self-attention layer and CA represents the cross-attention layer.
Further, in step S5, the barrage reconstructed loss function
Figure SMS_86
The construction formula is as follows:
Figure SMS_87
Figure SMS_88
wherein ,
Figure SMS_89
indicating batch processing, +.>
Figure SMS_90
Representing cross entropy loss, < >>
Figure SMS_91
Representing a reconstruction module->
Figure SMS_92
Comment of bullet generated by the reconstruction module is represented, < ->
Figure SMS_93
Indicating time->
Figure SMS_94
Is a true bullet comment;
specifically, the bullet comments generated by the reconstruction module are specifically expressed in the following form:
Figure SMS_95
wherein
Figure SMS_96
Representing a multi-layer perceptron, LN representing regularization operation, MHA representing cross-multi-headed attention.
Further, in step S6, predicted barrage emotion
Figure SMS_97
The calculation formula is as follows:
Figure SMS_98
Figure SMS_99
wherein ,
Figure SMS_100
is a Softmax function, LN represents a layer regularization operation, +.>
Figure SMS_101
Representing a multi-layer sensor->
Figure SMS_102
For a learnable emotion prediction matrix, +.>
Figure SMS_103
Is a learnable video emotion matrix, +.>
Figure SMS_104
As a learnable ambient barrage emotion matrix,
Figure SMS_105
representing a learnable target barrage emotion matrix, < ->
Figure SMS_106
Representing a tandem operation, representing a product.
Further, in step S7, the loss function of emotion prediction
Figure SMS_107
The construction formula is as follows:
Figure SMS_108
the overall loss function
Figure SMS_109
The calculation process of (2) is as follows:
Figure SMS_110
wherein ,
Figure SMS_111
for predicted barrage emotion, +.>
Figure SMS_112
Representing cross entropy loss, < >>
Figure SMS_113
Is true barrage emotion +.>
Figure SMS_114
Representing loss balance parameters, +.>
Figure SMS_115
Indicating batch processing.
A barrage emotion analysis system based on a joint model inputs barrage comments into the trained joint model to output emotion tendencies corresponding to the barrage comments;
the analysis system comprises a construction module, a video coding module, a text coding module, a door control fusion module, a multi-mode fusion module, a barrage reconstruction module, a barrage emotion prediction module and a loss calculation module;
the construction module is used for constructing a training sample set, and the training sample set comprises moments
Figure SMS_117
Bullet comment>
Figure SMS_119
Time->
Figure SMS_121
To->
Figure SMS_118
Inward bullet comment->
Figure SMS_120
Surrounding video->
Figure SMS_122
And comment on bullet screen->
Figure SMS_123
Surrounding barrage comment +.>
Figure SMS_116
；
The video coding module is used for coding the video
Figure SMS_124
Coding and concatenating to obtain coded video features
Figure SMS_125
；
The text coding module is used for commenting on the barrage
Figure SMS_126
And the surrounding barrage comment->
Figure SMS_127
Coding to obtain the coded target barrage characteristic +.>
Figure SMS_128
And surrounding barrage feature->
Figure SMS_129
；
The gating fusion module is used for processing the video
Figure SMS_131
Coding and concatenating to obtain coded video features
Figure SMS_135
Comment on the barrage->
Figure SMS_137
And the surrounding barrage comment->
Figure SMS_132
Coding to obtain the characteristics of the coded target barrage
Figure SMS_133
And surrounding barrage feature->
Figure SMS_136
Based on the target barrage feature +.>
Figure SMS_138
For the surrounding barrage feature
Figure SMS_130
After screening and filtering, connecting in series to obtain all surrounding barrage comments +.>
Figure SMS_134
；
The multi-mode fusion module is used for video features through the self-attention layer and the cross-attention layer
Figure SMS_139
Target barrage feature->
Figure SMS_140
Comment on surrounding barrage->
Figure SMS_141
Processing to obtain enhanced video features->
Figure SMS_142
Enhanced target barrage feature->
Figure SMS_143
And reinforcing the surrounding barrage->
Figure SMS_144
；
The bullet screen reconstruction module is used for enhancing video features based on multi-layer multi-head attention layer pairs
Figure SMS_145
Enhanced target barrage feature->
Figure SMS_146
Reinforcing surrounding barrage->
Figure SMS_147
Reconstructing to obtain reconstructed barrage comments, and constructing a barrage reconstructed loss function by using the reconstructed barrage comments and the real barrage comments by using cross entropy>
Figure SMS_148
；
The barrage emotion prediction module is used for enhancing video features
Figure SMS_149
Enhanced target barrage feature->
Figure SMS_150
Reinforcing surrounding barrage->
Figure SMS_151
Sequentially carrying out regularization and normalization operations, and outputting the barrage comment +.>
Figure SMS_152
Corresponding to predicted barrage emotion +.>
Figure SMS_153
；
The loss calculation module is used for predicting bullet screen emotion by using cross entropy
Figure SMS_154
And true barrage emotion->
Figure SMS_155
Construction of a loss function for emotion prediction>
Figure SMS_156
Loss function based on barrage reconstruction>
Figure SMS_157
And loss function for emotion prediction
Figure SMS_158
Calculating the overall loss function->
Figure SMS_159
Updating parameters of the joint model based on the total loss function and the back propagation algorithm until the performance of the joint model reaches a set expectation;
the surrounding barrage comments
Figure SMS_160
The calculation formula is as follows:
Figure SMS_161
Figure SMS_162
Figure SMS_163
/>
wherein ,
Figure SMS_165
for post-selection->
Figure SMS_168
Comment on the surrounding bullet screen->
Figure SMS_170
Is->
Figure SMS_166
Comment on the surrounding bullet screen->
Figure SMS_169
Is (are) peripheral features of->
Figure SMS_172
，/>
Figure SMS_174
Is a learnable gate matrix +.>
Figure SMS_164
Is a learnable gate offset vector, +.>
Figure SMS_167
For ReLU function>
Figure SMS_171
Representing series connection,/->
Figure SMS_173
Representing the product.
The barrage emotion analysis method and system based on the joint model provided by the invention have the advantages that: according to the barrage emotion analysis method and system based on the joint model, the video information is included through the multi-mode fusion module, the relation between the video theme and the barrage is fully considered, the enhanced characteristic representation is obtained, and the emotion analysis performance of the joint model on the target barrage comment is improved; the video information is included through the multi-mode fusion module, the relation between the video theme and the barrage is fully considered, the enhanced characteristic representation is obtained, and the performance of the combined model for carrying out emotion analysis on the target barrage comment is improved; and the bullet screen reconstruction module is utilized to promote the overall learning effect of each module and improve the performance of the emotion analysis module.
Drawings
FIG. 1 is a schematic diagram of the structure of the present invention;
fig. 2 is a schematic view of a module frame according to the present invention.
Detailed Description
In the following detailed description of the present invention, numerous specific details are set forth in order to provide a thorough understanding of the present invention. The invention may be embodied in many other forms than described herein and similarly modified by those skilled in the art without departing from the spirit or scope of the invention, which is therefore not limited to the specific embodiments disclosed below.
As shown in fig. 1 and 2, according to the barrage emotion analysis method based on the combined model, barrage comments are input into the trained combined model so as to output emotion tendencies corresponding to the barrage comments; the combined model uses a coding-decoding architecture, and comprises a coding module and a decoding module, wherein the coding module comprises a video coding module, a text coding module, a door control fusion module and a multi-mode fusion module, the decoding module comprises a barrage reconstruction module and an emotion analysis module, the emotion analysis module comprises a barrage emotion prediction module and a loss calculation module, and the decoding module takes the output of the coding module as input so as to output emotion analysis trends corresponding to barrage evaluation.
The method mainly comprises the steps of taking surrounding comments as context information of a target barrage by using a gating screening mechanism in a joint model, taking video information into consideration by using a multi-mode fusion mode, fully utilizing useful information to strengthen characteristic representation of the video barrage, constructing the joint model based on a residual convolution neural network, a long-short-period memory network, a gating fusion self-attention layer, a cross-attention layer and the like, training and learning parameters in the joint model, and optimizing the learning parameters to realize the effect of accurately identifying emotion tendencies of the target barrage comments by the joint model, and is specifically as follows.
The training process of the joint model is as follows:
s1: constructing a training sample set, the training sample set comprising moments
Figure SMS_177
Bullet comment>
Figure SMS_178
Time->
Figure SMS_180
To the point of
Figure SMS_175
Inward bullet comment->
Figure SMS_179
Surrounding video->
Figure SMS_181
And comment on bullet screen->
Figure SMS_182
Surrounding barrage comment +.>
Figure SMS_176
；
Video frequency
Figure SMS_184
There is->
Figure SMS_187
Frame video->
Figure SMS_189
Surrounding barrage comment->
Figure SMS_185
There is->
Figure SMS_186
Comment on bullet screen
Figure SMS_188
Surrounding barrage comment->
Figure SMS_190
Is at the comment->
Figure SMS_183
Surrounding comments.
For example, the bullet comment y "for itself, in the example shown in FIG. 2, insists on-! "as input, ambient barrage comment
Figure SMS_191
"beautiful and" good stature "as the context of y, and video corresponding to the comment y of the bullet screen is given +.>
Figure SMS_192
Together as an input.
S2: for the video
Figure SMS_193
Coding and concatenating to obtain coded video feature ∈ ->
Figure SMS_194
Comment on the barrage->
Figure SMS_195
And said surroundingsBullet comment->
Figure SMS_196
Coding to obtain the coded target barrage characteristic +.>
Figure SMS_197
And surrounding barrage features
Figure SMS_198
；
Encoding within a video encoding module using a residual convolutional neural network
Figure SMS_199
Frame video->
Figure SMS_200
And concatenating the obtained encoded vectors to obtain the encoded frame-level video feature +.>
Figure SMS_201
：
Figure SMS_202
wherein ,
Figure SMS_203
representing a video encoder>
Figure SMS_204
Representing a tandem operation;
in the text coding module, a long-term and short-term memory network is used
Figure SMS_205
) Comment on the barrage respectively->
Figure SMS_206
And its surroundings
Figure SMS_207
Comment of bullet screen->
Figure SMS_208
Coding to obtain the coded target barrage characteristic +.>
Figure SMS_209
And surrounding barrage feature->
Figure SMS_210
：
Figure SMS_211
I.e.
Figure SMS_212
wherein ,
Figure SMS_213
；/>
Figure SMS_214
=/>
Figure SMS_215
；
it should be understood that the first
Figure SMS_216
Comment on the surrounding bullet screen->
Figure SMS_217
Is characterized by->
Figure SMS_218
。
S3: based on the target barrage feature
Figure SMS_219
For the surrounding barrage feature->
Figure SMS_220
After screening and filtering, connecting in series to obtain all surrounding barrage comments +.>
Figure SMS_221
；
Based on the characteristics of the video barrage, some surrounding useful surrounding barrage comments with the same emotion can be used as the context information of the target barrage comments to provide assistance, so that the video barrage comments can be utilized by the gating fusion module
Figure SMS_222
To pair(s)
Figure SMS_223
Screening and filtering operation is carried out to obtain the +.>
Figure SMS_224
Comment on the surrounding bullet screen->
Figure SMS_225
：
Figure SMS_226
Figure SMS_227
wherein ,
Figure SMS_228
for post-selection->
Figure SMS_233
Comment on the surrounding bullet screen->
Figure SMS_236
Is->
Figure SMS_229
Comment on the surrounding bullet screen->
Figure SMS_232
Is (are) peripheral features of->
Figure SMS_235
，/>
Figure SMS_238
Is a learnable gate matrix +.>
Figure SMS_231
For a learnable gate offset vector, the function +.>
Figure SMS_237
For ReLU function>
Figure SMS_239
Representing series connection,/->
Figure SMS_240
Representing the product>
Figure SMS_230
and />
Figure SMS_234
Are all learnable parameters, and parameter optimization is carried out in the combined model training process so as to achieve the expected effect by using the input model;
1 st to 1 st
Figure SMS_241
Comment on the surrounding bullet screen->
Figure SMS_242
All surrounding barrage comments are obtained by connecting in series>
Figure SMS_243
：/>
Figure SMS_244
wherein ,
Figure SMS_245
representing a series operation.
S4: video characterization through self-attention layers and cross-attention layers
Figure SMS_246
Target barrage feature->
Figure SMS_247
Comment on surrounding barrage->
Figure SMS_248
Enhancement processing to obtain enhanced video features>
Figure SMS_249
Enhanced target barrage feature->
Figure SMS_250
And reinforcing the surrounding barrage->
Figure SMS_251
；
The multi-mode fusion module consists of an L-layer self-attention layer and a cross-attention layer, and is used for characterizing video
Figure SMS_252
Target barrage feature->
Figure SMS_253
Comment on surrounding barrage->
Figure SMS_254
As input of the first layer of the multi-mode fusion module, after multi-layer iteration (i.e. after processing of the L layers of self-attention layers and cross-attention layers), corresponding enhanced video features fused with other modes are obtained in the last layer>
Figure SMS_255
Enhanced target barrage feature->
Figure SMS_256
And reinforcing the surrounding barrage->
Figure SMS_257
。
In the first place
Figure SMS_258
Layer input video feature->
Figure SMS_259
Obtaining the input video feature of the next layer +.>
Figure SMS_260
The following are provided:
Figure SMS_261
in the first place
Figure SMS_262
Layer input target barrage feature->
Figure SMS_263
Obtaining the input target barrage feature of the next layer>
Figure SMS_264
：
Figure SMS_265
In the first place
Figure SMS_266
Layer input surrounding barrage comment->
Figure SMS_267
Obtaining the comment +.>
Figure SMS_268
：
Figure SMS_269
Where SA represents the self-attention layer and CA represents the cross-attention layer.
S5, performing S5; based on multiple layersMulti-headed attention layer pair enhanced video features
Figure SMS_270
Enhanced target barrage feature->
Figure SMS_271
Reinforcing surrounding barrage->
Figure SMS_272
Reconstructing to obtain reconstructed barrage comments, and constructing a barrage reconstructed loss function by using the reconstructed barrage comments and the real barrage comments by using cross entropy>
Figure SMS_273
；
The decoding module consists of a barrage reconstruction module and an emotion analysis module, and the decoding module encodes the enhanced video features obtained in the module
Figure SMS_274
Enhanced target barrage feature->
Figure SMS_275
Reinforcing surrounding barrage->
Figure SMS_276
As input;
and in the barrage reconstruction module, the reconstruction loss is analyzed and calculated by the barrage reconstruction module and added into closed-loop training to promote the learning effect of the multi-mode fusion module and promote the effect of the emotion analysis module.
The barrage reconstruction module consists of a plurality of multi-head attention layers, and a loss function of barrage reconstruction
Figure SMS_277
The method comprises the following steps:
Figure SMS_278
Figure SMS_279
wherein ,
Figure SMS_280
representing batch processing, CE representing cross entropy loss, < ->
Figure SMS_281
Representing a reconstruction module->
Figure SMS_282
Comment of bullet generated by the reconstruction module is represented, < ->
Figure SMS_283
Indicating time->
Figure SMS_284
Is a true bullet comment; />
Specifically, the bullet comments generated by the reconstruction module are specifically expressed in the following form:
Figure SMS_285
wherein
Figure SMS_286
Representing a multi-layer perceptron, LN representing regularization operation, MHA representing cross-multi-headed attention.
S6: for enhanced video features
Figure SMS_287
Enhanced target barrage feature->
Figure SMS_288
Reinforcing surrounding barrage->
Figure SMS_289
Sequentially carrying out regularization and normalization operations, and outputting the barrage comment +.>
Figure SMS_290
Corresponding to predictedBullet screen emotion->
Figure SMS_291
；
The emotion analysis module comprises a barrage emotion prediction module and a loss calculation module; wherein the bullet screen emotion predicted in the bullet screen emotion prediction module
Figure SMS_292
The calculation formula is as follows:
Figure SMS_293
Figure SMS_294
wherein ,
Figure SMS_296
is a Softmax function, LN represents a layer regularization operation, +.>
Figure SMS_299
Representing a multi-layer sensor->
Figure SMS_302
For a learnable emotion prediction matrix, +.>
Figure SMS_297
Is a learnable video emotion matrix, +.>
Figure SMS_298
As a learnable ambient barrage emotion matrix,
Figure SMS_301
representing a learnable target barrage emotion matrix, < ->
Figure SMS_303
Represents a series operation, & represents a product, & lt + & gt>
Figure SMS_295
、/>
Figure SMS_300
、/>
Figure SMS_304
All are learnable parameters, and parameter optimization is performed in the combined model training process so as to achieve the expected effect by using the input model.
S7: predicted barrage emotion using cross entropy
Figure SMS_305
And true barrage emotion->
Figure SMS_306
Construction of a loss function for emotion prediction>
Figure SMS_307
Loss function based on barrage reconstruction>
Figure SMS_308
And loss function of emotion prediction->
Figure SMS_309
Calculating the overall loss function->
Figure SMS_310
Updating parameters of the joint model based on the total loss function and the back propagation algorithm until the performance of the joint model reaches a set expected value;
loss function for emotion prediction in loss calculation module
Figure SMS_311
The construction formula is as follows:
Figure SMS_312
wherein ,
Figure SMS_313
indicating batch processing, +.>
Figure SMS_314
For the predicted bullet screen emotion, the predicted bullet screen emotion is bullet screen comment +.>
Figure SMS_315
Predicted barrage emotion output through the joint model, < ->
Figure SMS_316
The true barrage emotion is barrage comment +.>
Figure SMS_317
Corresponding actual emotion;
overall loss function
Figure SMS_318
The calculation process of (2) is as follows:
Figure SMS_319
wherein ,
Figure SMS_320
and representing the loss balance parameters, and updating the learnable parameters of the joint model based on the loss and the back propagation algorithm until the model performance achieves the expected effect.
First: step S3 provides a gating fusion mechanism, and utilizes the target barrage comments to carry out screening and filtering operation on surrounding barrage comments, so that some surrounding useful barrage comments with the same emotion can be used as context information of the target barrage comments to provide assistance, the problems that the barrage comments are short, insufficient context information exists and the like are solved, and the quality of the target barrage is improved.
Second,: step S4 provides a multi-mode fusion enhancement mechanism, video information is included through the multi-mode fusion module, the relation between a video theme and a barrage is fully considered, enhanced feature representation is obtained, and the performance of the joint model for carrying out emotion analysis on target barrage comments is improved.
Third, steps S5 to S7 provide a barrage reconstruction and emotion analysis mechanism, and the barrage reconstruction module is utilized to promote the overall learning effect of each module and the performance of the emotion analysis module.
The embodiment is mainly applied to emotion analysis of video real-time comments, for example, a comment is sent by a user at a certain moment, and emotion tendency of the comment is judged.
The foregoing is only a preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art, who is within the scope of the present invention, should make equivalent substitutions or modifications according to the technical scheme of the present invention and the inventive concept thereof, and should be covered by the scope of the present invention.

Claims (7)

1. The barrage emotion analysis method based on the joint model is characterized in that barrage comments are input into the trained joint model to output emotion tendencies corresponding to the barrage comments;
the training process of the joint model is as follows:
s1: constructing a training sample set, the training sample set comprising moments
Figure QLYQS_1
Bullet comment>
Figure QLYQS_4
Time->
Figure QLYQS_5
To->
Figure QLYQS_3
Inward bullet comment->
Figure QLYQS_6
Surrounding video->
Figure QLYQS_7
And bullet screenComment->
Figure QLYQS_8
Surrounding barrage comment +.>
Figure QLYQS_2
；
S2: for the video
Figure QLYQS_9
Coding and concatenating to obtain coded video feature ∈ ->
Figure QLYQS_10
Comment on the barrage->
Figure QLYQS_11
And the surrounding barrage comment->
Figure QLYQS_12
Coding to obtain the coded target barrage characteristic +.>
Figure QLYQS_13
And surrounding barrage feature->
Figure QLYQS_14
；
S3: based on the target barrage feature
Figure QLYQS_15
For the surrounding barrage feature->
Figure QLYQS_16
After screening and filtering, connecting in series to obtain all surrounding barrage comments +.>
Figure QLYQS_17
；
S4: video characterization through self-attention layers and cross-attention layers
Figure QLYQS_18
Target barrage feature->
Figure QLYQS_19
Comment on surrounding barrage->
Figure QLYQS_20
Enhancement processing to obtain enhanced video features>
Figure QLYQS_21
Enhanced target barrage feature->
Figure QLYQS_22
And reinforcing the surrounding barrage->
Figure QLYQS_23
；
S5, performing S5; enhancement of video features based on multi-layered multi-headed pairs of attention layers
Figure QLYQS_24
Enhanced target barrage feature->
Figure QLYQS_25
Reinforcing surrounding barrage->
Figure QLYQS_26
Reconstructing to obtain reconstructed barrage comments, and constructing a barrage reconstructed loss function by using the reconstructed barrage comments and the real barrage comments by using cross entropy>
Figure QLYQS_27
；
S6: for enhanced video features
Figure QLYQS_28
Enhanced target barrage feature->
Figure QLYQS_29
Reinforcing surrounding barrage->
Figure QLYQS_30
Sequentially carrying out regularization and normalization operations, and outputting the barrage comment +.>
Figure QLYQS_31
Corresponding to predicted barrage emotion +.>
Figure QLYQS_32
；
S7: predicted barrage emotion using cross entropy
Figure QLYQS_33
And true barrage emotion->
Figure QLYQS_34
Constructing a loss function for emotion prediction
Figure QLYQS_35
Loss function based on barrage reconstruction>
Figure QLYQS_36
And loss function of emotion prediction->
Figure QLYQS_37
Calculated overall loss function
Figure QLYQS_38
Based on the overall loss function->
Figure QLYQS_39
And updating parameters of the joint model by a back propagation algorithm until the performance of the joint model reaches a set expected value;
the surrounding barrage comments
Figure QLYQS_40
The calculation formula is as follows:
Figure QLYQS_41
Figure QLYQS_42
Figure QLYQS_43
wherein ,
Figure QLYQS_45
for post-selection->
Figure QLYQS_48
Comment on the surrounding bullet screen->
Figure QLYQS_52
Is->
Figure QLYQS_46
Comment on the surrounding bullet screen->
Figure QLYQS_50
Is (are) peripheral features of->
Figure QLYQS_53
，/>
Figure QLYQS_54
Is a learnable gate matrix +.>
Figure QLYQS_44
Is a learnable gate offset vector, +.>
Figure QLYQS_47
For ReLU function>
Figure QLYQS_49
Representing series connection,/->
Figure QLYQS_51
Representing the product.
2. The method of collaborative model-based barrage emotion analysis of claim 1, wherein the video features
Figure QLYQS_55
The calculation formula of (2) is as follows:
Figure QLYQS_56
/>
the target barrage feature
Figure QLYQS_57
The calculation formula of (2) is as follows:
Figure QLYQS_58
the surrounding barrage feature
Figure QLYQS_59
The calculation formula of (2) is as follows:
Figure QLYQS_60
wherein ,
Figure QLYQS_61
，/>
Figure QLYQS_62
，/>
Figure QLYQS_63
representing series connection,/->
Figure QLYQS_64
Representing a video encoder>
Figure QLYQS_65
Representing a long and short term memory network.
3. The barrage emotion analysis method based on joint model as set forth in claim 1, characterized in that in step S4: video characterization through self-attention layers and cross-attention layers
Figure QLYQS_66
Target barrage feature->
Figure QLYQS_67
Comment on surrounding bullet screen
Figure QLYQS_68
Enhancement processing to obtain enhanced video features>
Figure QLYQS_69
Enhanced target barrage feature->
Figure QLYQS_70
And reinforcing the surrounding barrage
Figure QLYQS_71
Specifically, the method comprises the following steps:
characterizing video
Figure QLYQS_72
Target barrage feature->
Figure QLYQS_73
Comment on surrounding barrage->
Figure QLYQS_74
Inputting as a first layer of the self-attention layer and the cross-attention layer and performing an L-layer iteration, wherein the L-layer is the total layer number of the self-attention layer and the cross-attention layer;
in the first place
Figure QLYQS_75
Layer input video feature->
Figure QLYQS_76
Obtaining the input video feature of the next layer +.>
Figure QLYQS_77
The following are provided:
Figure QLYQS_78
in the first place
Figure QLYQS_79
Layer input target barrage feature->
Figure QLYQS_80
Obtaining the input target barrage feature of the next layer>
Figure QLYQS_81
：
Figure QLYQS_82
In the first place
Figure QLYQS_83
Layer input surrounding barrage comment->
Figure QLYQS_84
Obtaining the comment +.>
Figure QLYQS_85
：
Figure QLYQS_86
Where SA represents the self-attention layer and CA represents the cross-attention layer.
4. A combined model-based barrage emotion analysis method as claimed in claim 3, characterized in that in step S5, the barrage reconstructed loss function
Figure QLYQS_87
The construction formula is as follows:
Figure QLYQS_88
Figure QLYQS_89
wherein ,
Figure QLYQS_90
indicating batch processing, +.>
Figure QLYQS_91
Representing cross entropy loss, < >>
Figure QLYQS_92
Representing a reconstruction module->
Figure QLYQS_93
Comment of bullet generated by the reconstruction module is represented, < ->
Figure QLYQS_94
Indicating time->
Figure QLYQS_95
Is a true bullet comment;
specifically, the bullet comments generated by the reconstruction module are specifically expressed in the following form:
Figure QLYQS_96
wherein ,
Figure QLYQS_97
representing a multi-layer perceptron, LN representing regularization operation, MHA representing cross-multi-headed attention. />
5. The method of claim 4, wherein in step S6, predicted barrage emotion is predicted
Figure QLYQS_98
The calculation formula is as follows:
Figure QLYQS_99
Figure QLYQS_100
wherein ,
Figure QLYQS_101
is a Softmax function, LN represents a layer regularization operation, +.>
Figure QLYQS_102
Representing a multi-layer sensor->
Figure QLYQS_103
For a learnable emotion prediction matrix, +.>
Figure QLYQS_104
Is a learnable video emotion matrix, +.>
Figure QLYQS_105
Is a surrounding barrage emotion matrix which can be learned, < + >>
Figure QLYQS_106
Representing a learnable target barrage emotion matrix, < ->
Figure QLYQS_107
Representing a tandem operation, representing a product.
6. The method of collaborative model-based barrage emotion analysis according to claim 5, wherein in step S7, the emotion predicted penalty function
Figure QLYQS_108
The construction formula is as follows:
Figure QLYQS_109
the overall loss function
Figure QLYQS_110
The calculation process of (2) is as follows:
Figure QLYQS_111
wherein ,
Figure QLYQS_112
for predicted barrage emotion, +.>
Figure QLYQS_113
Is true barrage emotion +.>
Figure QLYQS_114
Representing cross entropy loss, < >>
Figure QLYQS_115
Representing loss balance parameters, +.>
Figure QLYQS_116
Indicating batch processing.
7. The barrage emotion analysis system based on the joint model is characterized in that barrage comments are input into the trained joint model to output emotion tendencies corresponding to the barrage comments;
the analysis system comprises a construction module, a video coding module, a text coding module, a door control fusion module, a multi-mode fusion module, a barrage reconstruction module, a barrage emotion prediction module and a loss calculation module;
the construction module is used for constructing a training sample set, and the training sample set comprises moments
Figure QLYQS_117
Bullet comment>
Figure QLYQS_120
Time of day
Figure QLYQS_121
To->
Figure QLYQS_118
Inward bullet comment->
Figure QLYQS_122
Surrounding video->
Figure QLYQS_123
And comment on bullet screen->
Figure QLYQS_124
Video in the same frameSurrounding barrage comments in->
Figure QLYQS_119
；
The video coding module is used for coding the video
Figure QLYQS_125
Coding and concatenating to obtain coded video feature ∈ ->
Figure QLYQS_126
；
The text coding module is used for commenting on the barrage
Figure QLYQS_127
And the surrounding barrage comment->
Figure QLYQS_128
Coding to obtain the coded target barrage characteristic +.>
Figure QLYQS_129
And surrounding barrage feature->
Figure QLYQS_130
；
The gating fusion module is used for processing the video
Figure QLYQS_131
Coding and concatenating to obtain coded video feature ∈ ->
Figure QLYQS_134
Comment on the barrage->
Figure QLYQS_136
And the surrounding barrage comment->
Figure QLYQS_132
Coding to obtain coded target bulletCurtain characteristics->
Figure QLYQS_135
And surrounding barrage feature->
Figure QLYQS_137
Based on the target barrage feature +.>
Figure QLYQS_139
For the surrounding barrage feature->
Figure QLYQS_133
After screening and filtering, connecting in series to obtain all surrounding barrage comments +.>
Figure QLYQS_138
；
The multi-mode fusion module is used for video features through the self-attention layer and the cross-attention layer
Figure QLYQS_140
Target barrage feature->
Figure QLYQS_141
Comment on surrounding barrage->
Figure QLYQS_142
Processing to obtain enhanced video features->
Figure QLYQS_143
Enhanced target barrage feature->
Figure QLYQS_144
And reinforcing the surrounding barrage->
Figure QLYQS_145
；
The bullet screen reconstruction module is used for enhancing video features based on multi-layer multi-head attention layer pairs
Figure QLYQS_146
Enhanced target barrage feature->
Figure QLYQS_147
Reinforcing surrounding barrage->
Figure QLYQS_148
Reconstructing to obtain reconstructed barrage comments, and constructing a barrage reconstructed loss function by using the reconstructed barrage comments and the real barrage comments by using cross entropy>
Figure QLYQS_149
；
The barrage emotion prediction module is used for enhancing video features
Figure QLYQS_150
Enhanced target barrage feature->
Figure QLYQS_151
Reinforcing surrounding barrage->
Figure QLYQS_152
Sequentially carrying out regularization and normalization operations, and outputting the barrage comment +.>
Figure QLYQS_153
Corresponding to predicted barrage emotion +.>
Figure QLYQS_154
；
The loss calculation module is used for predicting bullet screen emotion by using cross entropy
Figure QLYQS_155
And true barrage emotion->
Figure QLYQS_156
Construction of emotionPredicted loss function->
Figure QLYQS_157
Loss function based on barrage reconstruction>
Figure QLYQS_158
And loss function of emotion prediction->
Figure QLYQS_159
Calculating the overall loss function->
Figure QLYQS_160
Updating parameters of the joint model based on the overall loss function and the back propagation algorithm until the performance of the joint model reaches a set expectation;
the surrounding barrage comments
Figure QLYQS_161
The calculation formula is as follows:
Figure QLYQS_162
Figure QLYQS_163
Figure QLYQS_164
wherein ,
Figure QLYQS_165
for post-selection->
Figure QLYQS_170
Comment on the surrounding bullet screen->
Figure QLYQS_172
Is->
Figure QLYQS_166
Comment on the surrounding bullet screen->
Figure QLYQS_168
Is (are) peripheral features of->
Figure QLYQS_171
，/>
Figure QLYQS_174
Is a learnable gate matrix +.>
Figure QLYQS_167
Is a learnable gate offset vector, +.>
Figure QLYQS_169
For ReLU function>
Figure QLYQS_173
Representing series connection,/->
Figure QLYQS_175
Representing the product. />
CN202310458854.3A 2023-04-26 2023-04-26 Barrage emotion analysis method and system based on joint model Active CN116189064B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310458854.3A CN116189064B (en) 2023-04-26 2023-04-26 Barrage emotion analysis method and system based on joint model

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310458854.3A CN116189064B (en) 2023-04-26 2023-04-26 Barrage emotion analysis method and system based on joint model

Publications (2)

Publication Number Publication Date
CN116189064A true CN116189064A (en) 2023-05-30
CN116189064B CN116189064B (en) 2023-08-29

Family

ID=86446571

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310458854.3A Active CN116189064B (en) 2023-04-26 2023-04-26 Barrage emotion analysis method and system based on joint model

Country Status (1)

Country Link
CN (1) CN116189064B (en)

Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101782898A (en) * 2010-03-25 2010-07-21 中国科学院计算技术研究所 Method for analyzing tendentiousness of affective words
US20190287142A1 (en) * 2018-02-12 2019-09-19 Baidu Online Network Technology (Beijing) Co., Ltd. Method, apparatus for evaluating review, device and storage medium
US20200050705A1 (en) * 2018-08-10 2020-02-13 International Business Machines Corporation Relative weighting for social collaboration comments
US20200104369A1 (en) * 2018-09-27 2020-04-02 Apple Inc. Sentiment prediction from textual data
CN111683294A (en) * 2020-05-08 2020-09-18 华东师范大学 Bullet screen comment recommendation method for information extraction
CN112036187A (en) * 2020-07-09 2020-12-04 上海极链网络科技有限公司 Context-based video barrage text auditing method and system
CN114119136A (en) * 2021-10-29 2022-03-01 中国工商银行股份有限公司 Product recommendation method and device, electronic equipment and medium
US11302360B1 (en) * 2020-10-08 2022-04-12 Adobe Inc. Enhancing review videos
CN114817536A (en) * 2022-04-20 2022-07-29 上海电力大学 Network short text sentiment analysis method based on fusion characteristics
CN114896386A (en) * 2021-09-24 2022-08-12 武汉工程大学 Film comment semantic emotion analysis method and system based on BilSTM
WO2022183138A2 (en) * 2021-01-29 2022-09-01 Elaboration, Inc. Automated classification of emotio-cogniton
CN115361595A (en) * 2022-07-28 2022-11-18 华中科技大学 Video bullet screen generation method

Patent Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101782898A (en) * 2010-03-25 2010-07-21 中国科学院计算技术研究所 Method for analyzing tendentiousness of affective words
US20190287142A1 (en) * 2018-02-12 2019-09-19 Baidu Online Network Technology (Beijing) Co., Ltd. Method, apparatus for evaluating review, device and storage medium
US20200050705A1 (en) * 2018-08-10 2020-02-13 International Business Machines Corporation Relative weighting for social collaboration comments
US20200104369A1 (en) * 2018-09-27 2020-04-02 Apple Inc. Sentiment prediction from textual data
CN111683294A (en) * 2020-05-08 2020-09-18 华东师范大学 Bullet screen comment recommendation method for information extraction
CN112036187A (en) * 2020-07-09 2020-12-04 上海极链网络科技有限公司 Context-based video barrage text auditing method and system
US11302360B1 (en) * 2020-10-08 2022-04-12 Adobe Inc. Enhancing review videos
WO2022183138A2 (en) * 2021-01-29 2022-09-01 Elaboration, Inc. Automated classification of emotio-cogniton
CN114896386A (en) * 2021-09-24 2022-08-12 武汉工程大学 Film comment semantic emotion analysis method and system based on BilSTM
CN114119136A (en) * 2021-10-29 2022-03-01 中国工商银行股份有限公司 Product recommendation method and device, electronic equipment and medium
CN114817536A (en) * 2022-04-20 2022-07-29 上海电力大学 Network short text sentiment analysis method based on fusion characteristics
CN115361595A (en) * 2022-07-28 2022-11-18 华中科技大学 Video bullet screen generation method

Non-Patent Citations (5)

* Cited by examiner, † Cited by third party
Title
QIAN SU: "Research on the Emotional Polarity Classification of Barrage Texts", 《2021 4TH INTERNATIONAL CONFERENCE ON COMPUTER INFORMATION SCIENCE AND APPLICATION TECHNOLOGY (CISAT 2021) 》 *
付兵: "基于双模态的弹幕情感分析", 《中国优秀硕士学位论文全文数据库》 *
庄须强;刘方爱;: "基于AT-LSTM的弹幕评论情感分析", 数字技术与应用, no. 02 *
辛明远: "基于元学习与字幕特征的知识类视频弹幕分类研究", 《中国优秀硕士学位论文全文数据库》 *
金马，宋彦，戴礼荣: "基于卷积神经网络的语种识别***", <数据采集与处理>, vol. 34, no. 2 *

Also Published As

Publication number Publication date
CN116189064B (en) 2023-08-29

Similar Documents

Publication Publication Date Title
CN108681610B (en) generating type multi-turn chatting dialogue method, system and computer readable storage medium
ALIAS PARTH GOYAL et al. Z-forcing: Training stochastic recurrent networks
WO2022095682A1 (en) Text classification model training method, text classification method and apparatus, device, storage medium, and computer program product
CN111860785A (en) Time sequence prediction method and system based on attention mechanism cyclic neural network
CN107484017A (en) Supervision video abstraction generating method is had based on attention model
CN111274398A (en) Method and system for analyzing comment emotion of aspect-level user product
CN110083702B (en) Aspect level text emotion conversion method based on multi-task learning
CN111626764A (en) Commodity sales volume prediction method and device based on Transformer + LSTM neural network model
CN111931549B (en) Human skeleton motion prediction method based on multi-task non-autoregressive decoding
CN113128527B (en) Image scene classification method based on converter model and convolutional neural network
CN113177666A (en) Prediction method based on non-invasive attention preprocessing process and BilSTM model
CN115841119B (en) Emotion cause extraction method based on graph structure
CN112529071B (en) Text classification method, system, computer equipment and storage medium
CN111144130A (en) Context-aware-based fine-grained emotion classification method for hybrid neural network
CN117475038B (en) Image generation method, device, equipment and computer readable storage medium
Skatchkovsky et al. Learning to time-decode in spiking neural networks through the information bottleneck
CN114444561A (en) PM2.5 prediction method based on CNNs-GRU fusion deep learning model
CN110175338B (en) Data processing method and device
Koryakovskiy et al. One-shot model for mixed-precision quantization
CN116189064B (en) Barrage emotion analysis method and system based on joint model
CN113989933A (en) Online behavior recognition model training and detecting method and system
US20240119713A1 (en) Channel Fusion for Vision-Language Representation Learning
CN116863920B (en) Voice recognition method, device, equipment and medium based on double-flow self-supervision network
US20240104352A1 (en) Contrastive Learning and Masked Modeling for End-To-End Self-Supervised Pre-Training
CN113469399A (en) Service prediction method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant