CN116680363A

CN116680363A - Emotion analysis method based on multi-mode comment data

Info

Publication number: CN116680363A
Application number: CN202310711727.XA
Authority: CN
Inventors: 陈碧云; 周国泉
Original assignee: Yancheng Teachers University
Current assignee: Yancheng Teachers University
Priority date: 2023-06-15
Filing date: 2023-06-15
Publication date: 2023-09-01

Abstract

The invention discloses an emotion analysis method based on multi-mode comment data, which relates to the technical field of multi-mode emotion analysis, and comprises the following steps: extracting different modal characteristics appearing in comments and analyzing intrinsic emotion information through multi-modal emotion analysis recommendation by using a multi-modal fusion technology, reclassifying the emotion information, introducing a time sequence prediction method to comprehensively analyze future emotion trends of commodities, and finally extracting commodity keywords worth recommending and commodity keywords with reduced future emotion so as to provide assistance for improving commodities and recommending vocabulary entries for sellers; according to the invention, through quantifying the emotion information contained in the multi-mode comments of the user, the difference between the original score and the real preference is reduced, and the recommendation quality is effectively improved and the commodity characteristics are accurately output through predicting the updated score information.

Description

Emotion analysis method based on multi-mode comment data

Technical Field

The invention belongs to the technical field of multi-mode emotion analysis, and particularly relates to an emotion analysis method based on multi-mode comment data.

Background

Nowadays, online comments on websites and social networks are considered as rich sources of implicit information, people are in a complex environment of multi-domain interaction, more and more people express views through social media, and generated multi-mode data are rich in complex emotion information such as information of pictures, texts, scores and the like. How to mine the potential relation among the multi-mode information, and to perform fusion analysis on the emotion of people and apply the emotion is already a current hot topic.

At present, the recommended algorithm is endless. Deng et al solve the data sparseness problem in the traditional collaborative filtering recommendation algorithm by using a K-medoids clustering method, but do not consider the potential characteristics between users and commodities; huang contacts the context of the user and the merchandise, but the way of analyzing the problem is still single; and the predictive recommendation model has recently received a great deal of attention in the fields of electronic commerce, social networks, online advertisements and the like. The method is mainly used for predicting the scoring or clicking rate of the user on the object, so that the mining of the user preference is realized, and personalized recommendation service is provided for the user. The deep learning, time sequence model, knowledge graph and semantic technology are widely applied to the prediction recommendation model. Zhang builds a recommendation model based on deep learning, but does not incorporate potential features of timing sequence; xu adds a time sequence, so that the reliability of prediction is enhanced; zhang utilizes knowledge graph and semantic technology, and uses semantic network of online comments to construct a recommendation model, so that the accuracy of recommendation is greatly improved.

The multi-modal technique also provides a new idea for recommending algorithms. Liu provides a mixed depth recommendation system, so that accuracy of cross-field commodity recommendation is improved; farahnakain et al propose a multi-modal framework, fusing multiple sensor information for marine vessel detection; li Wenan and the like explore a multi-mode recommendation algorithm, and compared with a general algorithm, the algorithm has higher accuracy and more effective recommendation; aiming at user comment information, korean Teng Yue provides a multi-mode sequence recommendation algorithm based on a comparison learning technology, extracts commodity characteristics, analyzes emotion of user comments, makes the recommendation algorithm more superior, but does not consider picture information possibly existing in the user comment information, and searches for potential interests and preference information of a user by mining the comment information of frequent users to obtain the most true emotion information of the commodity, so that the method becomes a hot point of current research.

Disclosure of Invention

Aiming at the problems of how to search potential interests and preference information of a user and obtain the most true emotion information of a commodity by mining the comment information of the user with frequent and frequent items in the prior art, the invention provides an emotion analysis method based on multi-modal comment data.

An emotion analysis method based on multi-mode comment data comprises the following steps:

obtaining comment data of the commodity; the comment data comprises picture mode data and text mode data;

inputting the image mode data into a Swin TensorFlow model to extract an Embedding characteristic matrix of the image mode;

using TextBlob to primarily classify emotion of the text modal data to obtain a text matrix;

inputting the text matrix into a Bert model to extract an Embedding feature matrix of the text mode;

inputting the image modal Embedding feature matrix and the text modal Embedding feature matrix into a transform model for emotion analysis to obtain emotion indexes of comment data;

clustering analysis is carried out on the emotion indexes based on the K-Shape, so that emotion characteristics of commodity comment data are obtained;

and carrying out time sequence prediction on the emotion characteristics of the commodity comment data by using the Prophet model, and analyzing the future emotion trend of the commodity according to the prediction result.

Further, the extracting step of the image modal data comprises the following steps of:

extracting features in the picture modal data;

inputting the extracted picture features into a SwinTransformer model for calculation;

And obtaining the feature vector of the picture according to the calculation result.

Further, the step of inputting the extracted picture features into a Swin Transformer model for calculation comprises the following steps:

performing Patch encoding on an input picture, dividing the picture into a plurality of small blocks, and converting each small block into a vector;

inputting each small block into an Encoder of a Transformer, calculating a weight for each small block, and weighting a feature vector of each small block;

after the obtained weighted vector is transformed and feature extracted by using the fully connected neural network, the weighted vector is mapped back to the original dimension by using matrix multiplication, and the feature vector of each small block is converted into a new vector to obtain the picture feature vector.

Further, the extracting step of the text modal data comprises the following steps of:

carrying out emotion analysis on comment data;

performing preliminary emotion classification on text modal data in the comment data;

constructing a new Text feature matrix T by using a Text Blob;

normalizing the new text feature matrix T;

inputting the processed new text feature matrix T into a Bert model for feature extraction to obtain an Embedding feature matrix.

Further, the method of deep learning is adopted for performing preliminary emotion classification on the text modal data in the comment data, and the method of deep learning is used for reducing redundancy of text data features.

Further, the analysis process of emotion analysis is carried out on the evaluation data by adopting a pretrained characterization model Bert, and comprises the following steps:

carrying out emotion marking on the text modal data by using a dictionary-based analysis method;

according to the part of speech, the negative word, the degree adverb, the punctuation mark and the emotion mark marked in the text modal data, the emotion value of the text modal data is calculated, and the calculation formula is as follows:

S _{i_ad} ＝MAX(-1,MIN(S _i *S _ad ,1))

wherein the polarity score represents the emotion value of the comment text, K represents the emotion word number in the comment text, and S _i A value representing a current emotion word; s is S _{i_ad} An emotion value representing a susceptible word with a degree adverb; n represents the number of negatives associated with the inflicted word; s is S _punc A emotion value representing punctuation; s is S _em The emotion value representing the emotion symbol, MAX representing the maintenance maximum value, and MIN representing the maintenance minimum value.

Further, the step of obtaining the emotion index of the comment data includes:

respectively transmitting the matrix X to a plurality of different Self-attribute, and calculating to obtain a plurality of output matrixes;

Splicing the obtained multiple output matrixes by using a Multi-Head attribute, and then transmitting the multiple output matrixes into a Linear layer to obtain a final output result of the Multi-Head attribute; wherein the Multi-Head Attention comprises a plurality of Self-Attention layers;

after the output results are connected through the residual, the emotion value is output by using a Softmax function.

Further, the calculation process of the output matrix includes the following steps:

inputting an Embedding feature matrix X of a picture mode and a text mode into Self-Attention to obtain moment matrices Q, K and V;

according to the matrix Q, K, V, calculating the output of Self-attribute, wherein the calculation formula is as follows:

wherein d_k Is the number of columns of the Q, K matrix, i.e. the vector dimension.

Further, the K-Shape based clustering analysis is performed on the comment data, and the method comprises the following steps:

acquiring a plurality of pieces of data appearing in the same time point of different data;

determining the similarity of a plurality of sequences with equal length of data through normalized cross correlation NCC to obtain translation conditions of all the sequences

According to the translation of all sequencesCalculating to obtain cross-correlation coefficient->Further, calculating a distance metric SBD in the KShape algorithm, wherein the calculation formula is as follows:

wherein ,representation using R ₀ To calculate the similarity of x and y at each step, calculate dot product at corresponding position, R ₀ Is the sum of the dot products of the active area;

calculating the optimal offset of all time sequences in the class;

taking the cluster center as a reference and aligning all sequences with the sequence of the reference;

generating clusters of the time series according to the SBD distance measurement and the shape extraction method;

and (3) obtaining comprehensive emotion data of the comment set of the same day by analyzing the data clustering condition in the same day.

Further, the Prophet model is expressed as:

y(t)＝g(t)+s(t)+h(t)+ε(t)

wherein g (t) represents the trend of the time series in non-periodic aspects, s (t) represents a periodic term or a seasonal term, typically used in cycles in units of weeks or years, h (t) represents a holiday term, ε (t) is an error term, represents the unpredictable fluctuations of the model, and ε (t) follows a Gaussian distribution; predicting future emotion tendencies through piecewise linear functions, wherein the expression is as follows:

g(t)＝(k+α(t) ^T δ·t+(m+α(t) ^T γ))

where k represents the growth rate, and δ represents the amount of change in the growth rate, in the range of δ _j -Laplace (0,0.05); thus, when gamma approaches zero, delta _j Also trending towards zero, the growth function at this time will become a full-segment logistic regression function or a linear function; m represents an offset, and when the growth rate k is adjusted, the offset m corresponding to each changepoint is appropriately adjusted, so as to connect the last time point of each segment, and the expression is as follows:

Obtaining the corresponding change of the increment rate as delta _j Laplace (0, gamma), and fitting the emotion trend of future commodities.

The invention provides an emotion analysis method based on multi-mode comment data, which has the following beneficial effects:

according to the method, through a multi-mode fusion technology, the image modal data and text modal data features appearing in comments are extracted, the intrinsic emotion information is analyzed, after the emotion information is reclassified, a time sequence prediction method is introduced to comprehensively analyze future emotion trends of commodities, finally commodity keywords which are worth recommending and commodity keywords with reduced future emotion values are extracted, and assistance of improving commodities and recommending vocabulary entries is provided for sellers; according to the invention, through quantifying emotion information contained in the multi-mode comments of the user, the difference between the original score and the real preference is reduced, and the recommendation quality is effectively improved and commodity characteristics are accurately output through predicting the updated score information; meanwhile, preference information in the user comments is extracted, and commodity manufacturers are helped to aim at interests and preferences of users so as to improve commodities and modify advertisement words to realize revenues.

Drawings

FIG. 1 is a flow chart of multi-modal emotion analysis in an embodiment of the present invention;

FIG. 2 is a schematic diagram of a text feature extraction process according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of a process for extracting image features according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of a multi-modal fusion process in accordance with an embodiment of the present invention;

FIG. 5 is a schematic diagram of a Self-Attention structure in an embodiment of the present invention;

FIG. 6 is a schematic diagram of a linear transformation process in accordance with an embodiment of the present invention;

FIG. 7 is a schematic diagram of a Multi-head attention structure in an embodiment of the present invention;

FIG. 8 is a graph showing the distribution of the number of supporters in an embodiment of the present invention;

FIG. 9 is a timing clustering prediction emotion classification diagram in an embodiment of the present invention;

FIG. 10 is a graph showing the results of a time-series cluster analysis in an embodiment of the invention;

FIG. 11 is a graph showing the emotion distribution of a "1780674880" book in an embodiment of the present invention;

FIG. 12 is a graph showing the predicted outcome of "1780674880" timing emotion in an embodiment of the present invention;

FIG. 13 is a graph of "1780674880" emotion tendencies in an embodiment of the present invention;

FIG. 14 is a graph showing the emotion prediction for "1635615372" merchandise in an embodiment of the present invention;

FIG. 15 is a graph of "1635615372" emotion tendencies in an embodiment of the present invention;

FIG. 16 is a graph showing the emotional tendency of "B017WJ5PR4" and "B001C4VLZQ" according to the embodiment of the present invention;

FIG. 17 is a bar graph of word frequency of "B017WJ5PR4" in an embodiment of the present invention;

FIG. 18 is a graph showing a word cloud distribution of "B017WJ5PR4" in an embodiment of the present invention;

FIG. 19 is a bar graph of word frequency for "B001C4VLZQ" in accordance with an embodiment of the present invention;

FIG. 20 is a graph showing the cloud pattern of the word "B001C4VLZQ" according to an embodiment of the present invention.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments.

With the rapid development of the Internet, people are in an age of continuous expansion of information, and the information quantity required to be processed by users is far greater than the information receiving capability of the users, so that the final decision is influenced, a recommendation system is generated, and the problem of information overload caused by complicated data is effectively relieved. The personalized recommendation system is taken as an important branch of artificial intelligence, and can conveniently and rapidly search out more favorable commodity from complicated information to recommend in the field of electronic commerce. Zahra integrates the geographic position and demographics of the user, and proposes a GHRS algorithm, so that the problem of cold start is effectively solved; the consistent emotion perception algorithm of Polignano makes the accuracy of the recommendation a new step by analyzing the emotion of the music itself.

And the predictive recommendation model has recently received a great deal of attention in the fields of electronic commerce, social networks, online advertisements and the like. The method is mainly used for predicting the scoring or clicking rate of the user on the object, so that the mining of the user preference is realized, and personalized recommendation service is provided for the user. The deep learning, time sequence model, knowledge graph and semantic technology are widely applied to the prediction recommendation model. Zhang builds a recommendation model based on deep learning, but does not incorporate potential features of timing sequence; xu adds a time sequence, so that the reliability of prediction is enhanced; zhang utilizes knowledge graph and semantic technology, and uses semantic network of online comments to construct a recommendation model, so that the accuracy of recommendation is greatly improved.

Currently, multimodal fusion techniques are used in many fields, such as computer vision, NLP, and speech recognition, and have achieved significant results in these fields. Visual information and text characteristics are combined, emotion recognition is performed, and emotion or attitude information is captured in text analysis. The understanding and perception capabilities of the computer to the real world can also be improved through a multi-modal approach, thereby designing novel virtual/augmented reality applications. The multi-modal technique also provides a new idea for recommending algorithms.

However, the multi-modality based recommendation algorithm still faces the following challenges:

1. user comments are numerous in false, so that the reliability of the model is low;

2. the accuracy of the recommendation algorithm is improved by adopting a fusion mode;

3. how to mine the correlation between different data in the fusion analysis;

4. the heterogeneity matching problem of the data in the fusion analysis;

5. how the potential features present in the massive comment information are handled.

According to the emotion analysis method based on the multi-modal comment data, which is shown in the figure 1, the multi-modal emotion analysis recommendation utilizes a multi-modal fusion technology to extract different modal characteristics appearing in comments and analyze intrinsic emotion information; after the emotion information is reclassified, introducing a time sequence prediction method to comprehensively analyze future emotion trends of the commodity; and finally, extracting commodity keywords which are worth recommending and commodity keywords with reduced emotion in the future, and providing assistance for improving commodities and recommending vocabulary entries for sellers. The method comprises the following steps:

(1) Extracting picture characteristics; and inputting a Swin TensorFlow model to extract an Embedding feature matrix for the uploaded picture mode data. As shown in fig. 3.

Since the book evaluation picture has complex characteristics and background and has strong similarity to postcards, cards and the like, the picture processing adopts a sliding window-based mechanism and has a Swin transform with hierarchical design (downsampling layer), and the model has the following advantages: have the ability to globally model; feature information of each scale is effectively extracted; effectively processing pictures of greater resolution. Swin transducer has a layered structure like a convolutional neural network, and can extract multi-scale features, so that the Swin transducer is easy to use in downstream tasks. The Swin transform has the advantage that hierarchy, locality, translation invariance (translational invariance) and the like are introduced into the transform network structure design a priori on the basis of ViT so as to obtain better performance in visual tasks. And whether the comment pictures are books or not can be judged, so that the picture characteristics can be effectively analyzed.

The user comment information comprises a plurality of pictures, the pictures reflect the authenticity of the user comment, and the implicit emotion characteristics can reflect the comprehensive evaluation of the user on the commodity.

Firstly, the characteristics in a plurality of pictures are required to be extracted, and the Swin transducer comprising a sliding window level design is used, so that the picture information contained in the comments can be effectively extracted.

The Swin transducer model is a transducer-based image classification model that uses a self-attention mechanism to process image features. In this model, image features are treated as input sequences and the structure of the transducer is used to extract features. When inputting an image into a Swin transform model, firstly performing Patch Embedding on the image; this process partitions the image into a number of patches and converts each patch into a vector; this vector represents the information of the patch, which can be used as input for subsequent processing; for each tile, it is input into the transform's Encoder (feed-forward layer) for processing.

The first Layer of the Encoder is Self-Attention Layer. The main function of this layer is to calculate a weight for each patch to capture the relationship between patches. Specifically, for each patch, we calculate its similarity to other patches and normalize the similarity to a probability value. These probability values are used to weight the feature vectors of the patches to obtain a vector representing the relationship with other patches.

These weight vectors are passed to the feed forward layer for processing. A fully connected neural network is used to transform and extract features from vectors. Again using matrix multiplication to map it back to the original dimension. The process converts the feature vector of each small block into a new vector, which contains the relation and different features with other small blocks, and finally obtains the picture feature vector.

(2) Processing text characteristics; for text modal data, firstly, performing preliminary classification on emotion by using a TextBlob (the TextBlob is a python middle library, wherein the emotion analysis function can finish marking the text emotion), constructing new features, and inputting Bert to extract an Embedding feature matrix for fusion classification; as shown in fig. 2.

Text data appearing in comment data needs emotion analysis to construct true emotion expressed by comment information, and accuracy of preference of commodities is maintained. Generally, there are three main approaches to text classification: compared with a traditional text classification model, the deep learning method can obtain higher-level and more abstract semantic representation through multi-level semantic operation, and feature extraction is integrated into a model construction process so as to reduce incompleteness and redundancy of manually designed features.

The method has the advantages that the data volume is huge, the characteristics are more, the redundancy of the characteristics is reduced by adopting a deep learning method, the comment data is required to be subjected to emotion analysis before emotion is quantized, the comment data is subjected to semantic analysis through a TextBlob text processing library, and the comment data is subjected to preliminary classification after emotion values of each word are given. And performing deep learning after classification to further extract the characteristics.

By using the pretraining characterization model Bert, deep features of text comments can be mined, and deep two-way language characterization can be performed by fusing left and right context information, so that comment data are more objective, and emotion analysis is more accurate.

Because the comment information has no emotion tendency of comment users, a training set of the text information needs to be initially marked, but the Wajdi Aljedaani presents that manual marking is time-consuming and too subjective, and a dictionary-based analysis method is used for marking the emotion of the text mode.

TextBlob comprehensively considers parts of speech, negatives, degree adverbs, punctuation and emotion symbols in calculating emotion values. In each text message, the attribute 'sender' (meaning the emotion polarity value obtained by emotion analysis of the text) returns a named tuple in the form of "sender (polarity)". The polarity score represents the emotional value of the comment text and reflects the positive or negative degree of the comment text.

S _{i_ad} ＝MAX(-1，MIN(S _i *S _ad ，1)) (2)

Wherein K represents the number of emotion words in comment text, S _i A value representing a current emotion word; s is S _{i_ad} An emotion value representing a susceptible word with a degree adverb; n represents the number of negatives associated with the inflicted word; s is S _punc A emotion value representing punctuation; s is S _em The emotion value representing the emotion symbol. MAX represents the hold maximum value and MIN represents the hold minimum value.

The emotion value varies within the range of [ -1.0,1.0], where-1.0 is very negative and 1.0 is very positive. Subjectivity varied within the range [0.01.0], with 0.0 being very objective and 1.0 being very subjective. Emotion analysis is a subjective analysis, so only inputs other than 0 are taken as training set features.

The text features are required to be extracted before feature fusion is carried out, the feature extraction is carried out by adopting WordPiece in Berts, and the WordPiece algorithm is mainly implemented in a mode called BPE (Byte-Pair Encoding) double-Byte Encoding. The BPE process splits a word again, so that the meaning and tense of the word can be separated, and the number of word lists is effectively reduced. The data is then input as a representation of each token, which is made up of three parts, the corresponding token, segmentation and location empeddings, respectively.

In order to separate the different sentence token, a segmentation token ([ SEP ]) is inserted in the sequence token. Then, to distinguish the sentences to which each token belongs, a learnable segmentation is added to each token.

The text is considered as a sequence in which each word is an element. The text sequence is first embedded in a high-dimensional space and the embedded vector is sent to a multi-layer transducer encoder. An attention mechanism is used to calculate the relative weights between each word and the other words and the corresponding weight vectors are accumulated to calculate the output of the layer. In the feed-forward sub-layer, we use a fully connected neural network to process the output of this layer to further extract the features of the text. The BERT takes each word vector as input and calculates the attention distribution between words to represent the context between the words, thereby extracting the features of the text and finally obtaining the text feature vector.

(3) Multimodal fusion; inputting the Embedding feature matrix of the text mode and the picture mode in the step 1 and the step 2 into a transducer for emotion analysis, and outputting emotion indexes; as shown in fig. 4.

As comment data often has two forms of pictures and texts, the invention adopts a multi-mode fusion technology to analyze the emotion information of user comments, thereby enhancing the accuracy. Because the data set is larger, the characteristic values are more, and the emotion analysis is carried out by adopting a fusion-first mode and using a transducer. In 2014, bengio team proposed a transducer model that utilized the attention mechanism to increase the training speed of the deep learning model in various fields and was widely used in recent years. Because Swin transformers and Bert are deep learning frameworks based on the transformers, the emotion analysis results can be obtained only by inputting the Embedding values of pictures and texts extracted by the Swin transformers and Bert into the transformers. And pre-training a depth bidirectional representation model from unlabeled features through an Encoder-Decoder, and finally outputting emotion indexes of the user through softMax.

Before emotion analysis, data consistency needs to be ensured, heterogeneity differences of different modes are made up, and accuracy of emotion recognition is improved. The comment data are multi-mode data composed of pictures and texts, the picture mode extraction features are required to be independently encoded, and the text mode is required to be pre-classified in emotion. Deep learning analysis is used to comment on emotion through a multi-modal fusion technology. Through the multi-modal architecture of fig. 4, multi-modal features of pictures and text can be fused to enhance the accuracy of emotion analysis.

Inputting the obtained Embedding vectors of the text mode and the picture mode into Self-Attention (Self-Attention mechanism); FIG. 5 is a Self-Attention structure, which requires the use of the matrix Q (query), K (key value), V (value) in the computation. In practice, self-Attention receives the input (matrix X of word representation vectors X) or the output of the last Encoder block. Q, K, V are obtained by linear transformation through the input of Self-Attention.

The Self-Attention input is represented by matrix X, and then Q, K, V can be calculated using linear variable matrix matrices WQ, WK, WV. Calculation as shown in fig. 6, note that each row of X, Q, K, V represents a set of features.

After the matrix Q, K and V are obtained, the output of Self-Attention can be calculated, and the calculation formula is as follows:

It can be seen from fig. 7 that the Multi-Head Attention contains a plurality of Self-Attention layers, and first, the input X is transferred to h different Self-Attention layers respectively, and h output matrices Z are calculated. The Multi-Head stations splice them together (Concat) and then pass into a Linear layer to get the final output Z of the Multi-Head stations. After connecting by residual, the emotion value was output using Softmax.

(4) Based on time sequence cluster analysis; and (3) carrying out time sequence clustering on the emotion data processed in the step (3), and integrating a plurality of pieces of comment data appearing in one day.

And through the integrated emotion information obtained after fusion, the hidden potential value information in the comment data can be accurately analyzed by using the time sequence clustering. Time-series clustering is divided into three methods: full time sequence clustering, sub-sequence clustering, and time point clustering.

The invention uses a clustering method of full time sequence to cluster a plurality of sequence data in a group of time sequences and analyze the potential emotion characteristics. K-Shape is used as a time sequence clustering method which is independent of the field, high in precision and high in efficiency, and compared with a traditional clustering method, the time sequence clustering effect is better. According to the method, the comment emotion data is subjected to cluster analysis based on K-Shape, and potential information among commodity comments and emotion fluctuation of the commodity comments along with time are searched. After the potential emotion features are found through clustering, overall evaluation of the commodity in the day is analyzed through multiple comment emotion appearing in the day.

For a plurality of pieces of comment data appearing in one day, a plurality of indexes appear after emotion analysis, and at the moment, time sequence cluster analysis is needed for comprehensive emotion indexes in the day, so that potential rules are found. For multiple pieces of data that occur within the same point in time of different data, multiple time sequences need to be clustered,

The KShape principle is similar to Kmeans except that it improves the distance calculation method and optimizes the centroid calculation method. On one hand, amplitude scaling and translation invariance are supported, on the other hand, the calculation efficiency is higher, parameters are not manually set, and the method is convenient to expand to more fields. The algorithm has a distance measure SBD of invariance to scaling and movement.

Similarity of equal length sequences was determined by Normalized cross-correlation NCC (Normalized cross-correlation). Knowing all sequence translationsRear part of the shoeTo calculate the cross-correlation coefficient->With the cross-correlation coefficients, the definition of SBD can be given:

wherein ,representation using R ₀ To calculate the similarity of x and y at each step, to calculate the dot product at the corresponding (both x and y present) position, R ₀ Is the sum of the dot products of the active area (the sum of the products of the patches on each pair), R ₀ The larger the two sequences the more similar.

Then, an optimal offset is calculated for all time series in the class, each time series is first assigned to the nearest centroid cluster to update cluster membership, and then each cluster centroid is updated to reflect the change in cluster membership in the previous step as the centroid is updated. These two steps will be repeated until the cluster membership has not changed or the maximum number of iterations allowed is reached; taking the cluster center obtained by previous calculation as a reference and aligning all sequences with the sequence of the reference, the clusters of the time series are efficiently generated depending on the SBD distance measurement and the shape extraction method. And then, the comprehensive emotion data of the comment set of the same day is obtained by analyzing the data clustering condition in the same day.

(5) A time series prediction recommendation; predicting the processed time sequence in the step 4, analyzing the future emotion trend of the commodity, and recommending more superior commodity; and provides comment features to provide improved recommended vocabulary entry and merchandise functionality for the seller.

And carrying out time sequence prediction according to the emotion data processed by the K-Shape time clustering, and obtaining emotion trend of future commodities, thereby recommending high-quality commodities more accurately. Because the time sequence of the comment data set is not continuous, but is clustered in a time sequence of one section to another section, the comment emotion data can have a large number of periodic characteristics, the situation of partial missing values can be well processed by using Prophet for time sequence prediction, and the expected prediction result can be obtained in a short time. Prophet can get the trend without feature engineering, but cannot use more information, and emotion data is analyzed through a multi-mode fusion technology, so that the Prophet can concentrate on prediction to achieve a good fitting result.

In the same day, a large number of comments exist, the data after K-Shape clustering analysis are integrated, and the situation that comment data time sequence is discontinuous still exists.

Propset is the process of predicting time series data based on nonlinear trends and additional models of annual, weekly and daily seasonal, and holiday effects. Prophet's design and implementation focused on circumventing several drawbacks that exist in traditional time series models (such as ARIMA), such as inadequate processing of outlier data, difficulty in modeling trends and seasonal changes, and the like.

Prophet is a time sequence addition model, and the Prophet algorithm obtains a time sequence predicted value through fitting and accumulation.

y(t)＝g(t)+s(t)+h(t)+ε(t) (5)

In this model, g (t) represents the trend of the time series in terms of non-period; and s (t) represents a periodic term or a seasonal term, typically used in cycles of weeks or years; h (t) represents holiday terms, which represent the effect of a potential, non-stationary period holiday on the predicted value; epsilon (t), the error term or residual term, represents the unpredictable fluctuations of the model, epsilon (t) obeys a gaussian distribution; future emotional trends can be predicted by piecewise linear functions.

g(t)＝(k+α(t) ^T δ·t+(m+α(t) ^T γ)) (6)

Where k represents the growth rate, the trend trace changes in a specific period or a potential periodic curve, and the model defines a point corresponding to the change of the growth rate k, which is called changepoints. Delta represents the change amount of the growth rate, and the growth rate of the change point is satisfied Laplace distribution in the range of delta _j -Laplace (0,0.05); thus, when gamma approaches zero, delta _j Also trending towards zero, the growth function at this time will become a full-segment logistic regression function or a linear function. m represents an offset, and when the growth rate k is adjusted, the offset m corresponding to each changepoint is appropriately adjusted, so as to connect the last time point of each segment, and the expression is as follows:

from the data of length T historically, s change points can be selected, which correspond to a change in growth rate of delta _j Laplace (0, gamma). Thereby fitting the emotion trend of the future commodity and recommending.

(6) Experiments prove that.

1. Experimental environment: the hardware and software platform is a PC using Inter core I7-980 x 3.33GHz, and the GPU uses GTX 3090 and 32G memory. The experimental environment is generally shown in table 1.

Table 1 experimental hardware and software environment

2. Evaluation index: the invention selects the traditional evaluation index: precision, recall, accuracy, and scoring. F1 P N T F (positive) and P N T F (negative) indicate the determination results of the model, true and False indicate whether the determination results of the model are correct or not. The calculation results are as follows:

where TP is the number of samples correctly classified as positive, FP is the number of samples incorrectly assigned to the initiative, and FN is the number of samples originally belonging to the negative class but assigned to the other class.

To further describe the accuracy of the model, we also select Mean Square Error (MSE) and Mean Absolute Error (MAE). In MSE and MAE, when the predicted value is exactly identical to the real value, it is equal to 0, the perfect model. The larger the error, the larger its value. Where the predicted value for the sample is the true value for the ith sample.

3. Introduction of data set: classical data set Amazon Review Dataset as a recommendation system records user ratings for amazon website merchandise. The data sets are divided into sub-data sets such as Books, electronics, movies and TV, CDs and vinyl according to commodity categories, and the sub-data sets comprise two types of information, wherein one type is commodity information, and the other type is user scoring record information.

The data used in the present invention is Books data, which contains 4w pictures and 2.7kw comment data, contains fields, and covers all scoring information from 1990 to 2018. Table 2 lists the relevant attributes in the dataset.

Table 2 amazon dataset tags

4. Data preprocessing: since the data downloaded from Amazon Review Data is JSON data, JSON files need to be read and converted into CSV files. The pictures are analyzed and marked in the form of URL, and the crawler technology is used for reading commodity picture information. The duplicate values are removed therefrom and the converted missing values are replaced with NAN values. When key information such as pictures, text, time, etc. is deleted directly if the missing values in a row are too many, to improve the robustness of the algorithm and the accuracy of the prediction. For noise data, the value is unreasonable, removed as an outlier.

Because comment text data on the original features is composed of one or thousands of words, the data features are redundant, and the low-dimensional data of the mapping data are used for constructing preliminary classification, so that the calculation complexity is reduced.

Since false comments may be included in the data, by observing the distribution of the number of support people for 1w comments before the comments, as shown in fig. 8, it can be determined that when the number of support people is greater than 5, the comment information can be regarded as effective information, and since the comment dataset has 2.7kw data, training is performed by adopting 2w data containing pictures and texts in a random sampling manner.

5. Experimental results and analysis.

Text feature extraction experiment: and carrying out preliminary emotion classification on the Text modes in the comment data set, constructing a new feature T by using the Text Blob, and carrying out normalization processing on the new feature. The range of T is controlled within [0,1], and when T is closer to 0, emotion is more negative and when T is closer to 1, emotion is more positive. And then inputting the T text matrix into the Bert for feature extraction to obtain an Embedding feature matrix.

And (3) carrying out picture feature extraction experiments: and extracting the picture features by using the SwinTransformer, wherein one comment in the comment data set possibly corresponds to a plurality of pictures, so that an Embedding feature matrix of the plurality of pictures needs to be constructed, and the weight proportion of the features between the picture mode and the text mode is adjusted to align the data features.

Multimode fusion characteristic experiment: inputting the obtained text and picture coding feature matrix into a converter for encoding and decoding, and finally outputting a final emotion score e-score through softMax, wherein the e-score is between [0,1], and the more negative is, the more positive is the more close to 1.

Time sequence clustering of emotion: the final emotion scores e-score obtained in the experiment were clustered over time, so that the emotion scores were analyzed to be classified into 6 categories, and emotion was classified into 6 major categories according to the interval shown in fig. 9 below.

And then performing dimension reduction processing by using a distance function SBD so as to determine the comprehensive emotion index of the current day. As can be seen from fig. 10, when score is greater than 0, emotion is positive, and when score is less than 0, emotion is negative, and if comment data on day 21 is most negative, it belongs to class 6.

Emotion timing prediction recommendation, comprising:

determination of a timing prediction algorithm: through previous experiments, we can obtain daily emotion score, and select a book as an example, here we select the number "1780674880" to perform experiments, and we can obtain emotion score time series from 1998 to 2018 of the book, as shown in fig. 11. For better time series prediction, we used the modeling analysis of the four different algorithms Prophet, ARIMA, SARIMA and LSTM, and the prediction analysis was performed on the number 1780674880, and the results are shown in table 3 below.

TABLE 3 timing prediction accuracy

From table 3, the highest propset prediction accuracy was found, and the accuracy of both MAE and MSE was optimal, so we selected propset timing prediction model.

Different book timing analysis: and predicting and analyzing two different books with numbers of 1780674880 and 1635615372 by adopting a Prophet algorithm to obtain emotion data and emotion tendencies of the two books in the next year. From fig. 12 and 13, emotion data and emotion variation trend in the next year are analyzed, and it is easy to see that the buying emotion of book class "1780674880" is in an increasing trend, which further reflects that the book is more and more popular with the masses.

Meanwhile, the commodity fitting of the book of 1635615372 is predicted, as shown in fig. 14, the buying emotion is predicted to be gradually decreased in one year, and the buyer emotion is found to show a descending trend through analyzing the trend, as shown in fig. 15, so that the book is possibly disliked by the public, and the store needs to change the sales strategy.

Timing prediction recommendation: randomly selecting 5 different books, carrying out a recommendation experiment, and predicting the preference degree of consumers to the books; by analyzing the emotional trends of five different books, as shown in fig. 16, the books of "B017WJ5PR4" are not difficult to find to show an increasing trend, and the books are favored by the public in the future; the emotion index is higher when the 'B001C 4 VLZQ' book just appears than when other four books just appear, but the emotion presents abrupt drop along with the time, and the emotion index cannot be accepted in the future.

In order to further analyze consumer preference, the invention selects the number 'B017 WJ5PR 4' with the largest ascending trend and the number 'B001C 4 VLZQ' with the most obvious descending trend for research, and the specific reasons of the ascending trend and the descending trend are respectively selected. 17, 18, 19 and 20, by counting the comment word frequency of two books, extracting words in comment data, and analyzing to obtain that 'B017 WJ5PR 4' is a Haribot series, the books are oriented to students in middle and primary schools and colleges, and the degree of promotion is high; the "B001C4VLZQ" books are books related to talking habits, mainly facing business people, and the business can update related advertisement words to attract more clients.

Ablation test: the accuracy of the multi-mode emotion analysis is up to 94%, and in order to ensure the effectiveness of the multi-mode emotion analysis, an ablation test is performed. The test results are shown in Table 4 below:

table 4 ablation test results

	Precision	Accuracy	Recall	F1	MAE	MSE
							Multi-modality	0.9411	0.9406	0.9396	0.9398	0.05944	0.5944
FastText	0.6044	0.6065	0.597	0.5685	0.3935	0.3935
							Bert	0.25	0.4999	0.5	0.3316	0.5001	0.5001
Convnext	0.5392	0.944	0.5117	0.5249	0.05031	0.03025
							Restnet50	0.6054	0.9182	0.5657	0.5837	0.0753	0.0753

The invention carries out independent emotion analysis on the text mode, and experiments show that the precision and the accuracy of FastText are only 60 percent. The Bert deep learning emotion classification effect is worse, and the precision and the accuracy are only 25% and 49% respectively.

The picture data is singly identified, the accuracy of using Convnext is only 94% of 53% accuracy, and the identification is not accurate enough due to more features, so that the accuracy is far lower than that of a multi-mode fusion algorithm; the picture is identified using the RestNet50 with 60% and 91% accuracy, respectively, which is improved compared to Convnext but the remaining indicators are not as good as Convnext.

Ablation experiments show that the multi-modal fusion technology is more accurate in identifying the emotion of comments, and the accuracy of the post-prediction recommendation model is ensured.

The invention carries out independent emotion analysis on the text mode, and experiments show that the precision and the accuracy of FastText are only 60 percent. The Bert deep learning emotion classification effect is worse, and the precision and the accuracy are only 25% and 49% respectively. The picture data is singly identified, the accuracy of using Convnext is only 94% of 53% accuracy, and the identification is not accurate enough due to more features, so that the accuracy is far lower than that of a multi-mode fusion algorithm; the picture is identified using the RestNet50 with 60% and 91% accuracy, respectively, which is improved compared to Convnext but the remaining indicators are not as good as Convnext. Ablation experiments show that the multi-modal fusion technology is more accurate in identifying the emotion of comments, and the accuracy of the post-prediction recommendation model is ensured.

The foregoing is only a preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art, who is within the scope of the present invention, should make equivalent substitutions or modifications according to the technical scheme of the present invention and the inventive concept thereof, and should be covered by the scope of the present invention.

Claims

1. An emotion analysis method based on multi-mode comment data is characterized by comprising the following steps:

2. The emotion analysis method based on multi-mode comment data according to claim 1, wherein the extracting step of the image modal data comprises:

Extracting features in the picture modal data;

3. The emotion analysis method based on multi-modal comment data according to claim 2, wherein the calculating process includes:

4. The emotion analysis method based on multi-modal comment data according to claim 1, wherein the extracting step of the text modal data comprises:

carrying out emotion analysis on comment data;

constructing a new Text feature matrix T by using a Text Blob;

normalizing the new text feature matrix T;

5. The emotion analysis method based on multi-modal comment data of claim 4 wherein said preliminary emotion classification of text modal data in comment data employs a deep learning method for reducing redundancy of text data features.

6. The emotion analysis method based on multi-modal comment data according to claim 4, wherein emotion analysis is performed on the comment data by using a pre-training characterization model Bert, and the analysis process includes:

S _{i_ad} ＝MAX(-1,MIN(S _i *S _ad ,1))

7. The emotion analysis method based on multi-modal comment data according to claim 1, wherein the step of obtaining an emotion index of comment data includes:

8. The emotion analysis method based on multi-modal comment data according to claim 7, wherein the calculation process of the output matrix includes the steps of:

inputting an Embedding feature matrix X of a picture mode and a text mode into Self-Attention to obtain matrices Q, K and V;

9. The emotion analysis method based on multi-modal comment data according to claim 8, wherein the step of performing cluster analysis on comment data based on K-Shape includes:

calculating the optimal offset of all time sequences in the class;

10. The emotion analysis method based on multi-modal comment data according to claim 1, wherein the Prophet model is expressed as:

y(t)＝g(t)+s(t)+h(t)+ε(t)

g(t)＝(k+α(t) ^T δ·t+(m+α(t) ^T γ))

where k represents the growth rate, and δ represents the amount of change in the growth rate, in the range of δ _j -Laplace (0,0.05); thus, when gamma approaches zero, delta _j Also trending towards zero, the growth function at this time will become a full-segment logistic regression function or a linear function; m represents the offset, and after the growth rate k is adjusted, the offset m corresponding to each changepoint point is properly adjusted, therebyThe last time point of each segment is followed by the expression: