CN110287320A - A kind of deep learning of combination attention mechanism is classified sentiment analysis model more - Google Patents

A kind of deep learning of combination attention mechanism is classified sentiment analysis model more Download PDF

Info

Publication number
CN110287320A
CN110287320A CN201910553755.7A CN201910553755A CN110287320A CN 110287320 A CN110287320 A CN 110287320A CN 201910553755 A CN201910553755 A CN 201910553755A CN 110287320 A CN110287320 A CN 110287320A
Authority
CN
China
Prior art keywords
word
cnn
text
model
network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201910553755.7A
Other languages
Chinese (zh)
Other versions
CN110287320B (en
Inventor
刘磊
孙应红
陈浩
李静
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing University of Technology
Original Assignee
Beijing University of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing University of Technology filed Critical Beijing University of Technology
Priority to CN201910553755.7A priority Critical patent/CN110287320B/en
Publication of CN110287320A publication Critical patent/CN110287320A/en
Application granted granted Critical
Publication of CN110287320B publication Critical patent/CN110287320B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/35—Clustering; Classification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/30—Semantic analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Evolutionary Computation (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Databases & Information Systems (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Machine Translation (AREA)

Abstract

The present invention relates to a kind of deep learning of combination attention mechanism mostly classification sentiment analysis models, belong to natural language processing technique field, the present invention analyzes the weakness of existing CNN network and LSTM network in terms of text emotion analysis, proposes a kind of deep learning mostly classification sentiment analysis model of combination attention mechanism.The model use attention mechanism blends the word order feature of local feature and LSTM model extraction that CNN network extracts, and the thought of integrated model is used in classification layer, the affective characteristics that CNN network and LSTM network extract are spliced respectively, the affective characteristics finally extracted as model.By comparative experiments, it is found that the accuracy rate of the model has significant raising.

Description

A kind of deep learning of combination attention mechanism is classified sentiment analysis model more
Technical field
The invention belongs to text information processing field, emotion of classifying the deep learning for being related to a kind of combination attention mechanism more Analysis model.
Background technique
With the continuous rise of the social networks such as microblogging, Twitter, internet has been not only that people obtain daily information Source, while also become people express the indispensable platform of oneself viewpoint.People Web Community comment on focus incident, express The viewpoint that writes a motion picture review and description Product Experience etc. can all generate a large amount of text for having emotional color (such as: happiness, anger, grief and joy) Information, and effective sentiment analysis is carried out to these text informations, the interest tendency and concern journey of user can be best understood from Degree.But there are the text with emotional color of magnanimity in the increase with people to network information attention rate, Web Community daily It generates, if only being much unable to complete this task by handmarking, this, which allows for text emotion analysis, becomes certainly One research hotspot in right Language Processing field.
Successful application with deep learning method in computer vision direction, more and more depth learning technologies also by Applied to natural language processing direction.The advantage of deep learning is, can not only automatically extract the feature of text, but also to big Data have stronger ability to express.The text emotion analysis method based on deep learning of mainstream mainly includes convolutional Neural at present Network (Convolutional Neural Network, CNN) and Recognition with Recurrent Neural Network (Recurrent Neural Network, RNN) two kinds, the accuracy rate of the sentiment analysis model based on both methods is all lower, mainly there is following side The reason of face:
Firstly, convolutional neural networks are effectively captured by expanding convolution kernel size during the sentiment analysis of text To the emotion information of different location, and then get the local affective characteristics of text.But during convolution, text is often had ignored Context in this between word order.But in text emotion analytic process, the precedence relationship of word order is again particularly significant, without word order Characteristic information result must be caused to have certain deviation.
Secondly, Recognition with Recurrent Neural Network network utilizes front and back dependence, the sequencing of text data is effectively simulated, it can The word order relationship and semantic information of text are extracted, therefore good effect can be reached in the sentiment analysis of text.But work as sample When notebook data is longer or language contexts are more complex, the interval of useful emotion information varies, different in size, length memory Therefore the performance of network (Long Short-Term Memory, LSTM) is also restricted.
The present invention takes full advantage of attention mechanism, CNN network, LSTM network, proposes and realize a kind of combination to pay attention to The deep learning of power mechanism mostly classification sentiment analysis model.This model can effectively improve the accuracy rate of text emotion analysis.
Summary of the invention
The invention proposes a kind of deep learning based on attention mechanism mostly classification sentiment analysis models.The models coupling CNN network and LSTM network carry out affective characteristics fusion.Text to be analyzed is extracted first with the multiple dimensioned convolution kernel of CNN network Then this local feature utilizes attention mechanism, the local feature that CNN network extracts is dissolved into LSTM network.Finally Using the thought of integrated model, the feature extraction result of the pond layer result of CNN network and LSTM network is spliced, as Final model output.Experiment shows that in text emotion analysis, the accuracy rate of the model has significant raising.
To achieve the above object, the present invention adopts the following technical scheme that:
A kind of sentiment analysis method 1. deep learning of combination attention mechanism is classified more, it is characterised in that including following step It is rapid:
Step (1) data prediction
If affection data set representations are as follows: G=[(segtxt1,y1),(segtxt2,y2),......,(segtxtN,yN)], Wherein, segtxtiIndicate i-th of sample, yiIt is then corresponding emotional category label, N indicates number of samples in data set G, to G Middle sample carries out data prediction,
Data set G after pretreatment, is expressed as G '=[(seg1,y1),(seg2,y2),...,(segM,yM)], in which: segiIt is expressed as i-th of sample, y in data set G 'iIt is then corresponding emotional category label, M indicates the middle sample of data set G ' Number；
The input of step (2) building model
Sample data (seg, y) to be analyzed for any one in data set G ', by it, further refinement is indicated are as follows:
Seg=[w1,w2,w3,...,wd]T (1)
Y=[0,0,1 ..., 0] (2)
Wherein: wi∈RεRefer to and encoded according to one-hot of the vocabulary wordList to the i-th word in text to be analyzed, ε is The size of vocabulary wordList, d indicate that the sentence of the text is long.y∈RpIt is the one-hot coding according to emotional category, p indicates mould Type class number to be divided.Then the term vector embeded matrix of the sample may be expressed as:
X=seg*ET (3)
Wherein: X ∈ Rd×m, X=[x1,x2,...,xd]TIt is indicated for the term vector matrix of text to be analyzed, m is term vector Dimension, xi∈RmIt is indicated for the term vector of i-th of vocabulary in the text, E is the expression of term vector embeding layer；
Step (3) constructs deep learning mostly classification sentiment analysis model
Deep learning sentiment analysis model of more classifying includes local shape factor stage based on CNN network and based on LSTM The word order relationship characteristic of network extracts the stage, by the pond layer result C in the local shape factor stage based on CNN networkCnnAnd base The result C' in stage is extracted in the word order relationship characteristic of LSTM networkRnnSplicing, i.e. vector [CCnn；C'Rnn] finally mentioned as model The feature vector taken.Then by feature vector [CCnn；C'Rnn] obtain final model output vector by full articulamentumWherein p indicates model class number to be divided.
The local shape factor stage based on CNN network, including the following contents:
The input of local shape factor stage is that the term vector matrix of the text to be analyzed of formula 3 indicates X；
The local shape factor stage is based on CNN network, altogether includes two layers, i.e. one layer of convolutional layer, one layer of pond layer, In:
Convolutional layer carries out convolution, and same scale convolution kernel to text to be analyzed using the convolution kernel of n kind different scale Filter, that is, neuron each k；
The resulting vector of convolution is done down-sampling using the method for maximum pond layer by pond layer, selects local optimum feature, Therefore each filter becomes a scalar by maximum pond layer, and it is special which represents emotion optimal in the filter Sign；
The output of local shape factor module is CCnn=[c1,c2,...,cnk], i.e., it will be various sizes of more in the layer of pond The optimal characteristics that a filter is chosen are spliced together CCnn=[c1,c2,...,cnk] output as this module, wherein CCnn ∈Rnk, nk is the number of all filters in convolutional layer；
The word order relationship characteristic based on LSTM network extracts stage, including the following contents:
Multiple dimensioned CNN network local shape factor: convolutional layer in the local shape factor stage based on CNN network is same The convolution results of k filter of convolution scale are spliced, and set Z is obtainedCnn, then will set ZCnnIn each vector ZiInput Into GLU mechanism, i.e., gate convolutional network, obtained result are denoted as { π1,π2,...,πn, complete multiple dimensioned CNN network part The extraction of feature.
Wherein, ZCnn={ Z1,Z2,...,Zn, ZiFor the splicing for multiple filter convolution results that scale is i；
Wherein,ZiRepresent k filter convolution results of a certain scale Splicing, W1, W2∈Rλ×qFor weight matrix, λ indicates the dimension of respective weights matrix, b1, b2∈RqFor amount of bias, σ is indicated Sigmoid function, πi∈Rq, q is the output dimension of LSTM network；
Then, using attention mechanism, by multiple dimensioned CNN network local shape factor result { π1,π2,...,πnIncorporate Into LSTM network, the output result C' that the word order relationship characteristic based on LSTM network extracts the stage is obtainedRnn, i.e.,
Wherein,Indicate the output of LSTM module corresponding to the last one word in text to be analyzed,It indicates wait divide The output of LSTM module corresponding to first word in text is analysed, the present invention uses two-way LSTM model, i.e. BiLSTM model,
Using forward-propagating, specific calculating process is as follows:
D is the length of text to be analyzed, the corresponding LSTM module of each word order in the text,
During forward-propagating, the output of the t-1 LSTM module isThe then output of t-th of LSTM moduleIt calculates Formula is as follows:
Wherein:It is the dot product of two vectors, also referred to as scoring functions, is for calculating previous word The output of LSTMWith the similarity of current local feature vectors,
Wherein: αt,i∈ R represents feature πiWeight,
Wherein: st-1∈RqIt is the weighted results of multiple convolution features, utilizes st-1Instead ofIn conjunction with current term word to Measure xtAcquire the output of current LSTM moduleFormula is as follows:
Using backpropagation, specific calculating process is as forward-propagating, and details are not described herein again；
Step (4) model training: inputting sentiment analysis models of classifying for training data more, using cross entropy loss function, In conjunction with backpropagation BP algorithm adjusting parameter, is returned using softmax as sorting algorithm, complete training；
Step (5) model analysis: it is analysed to the model that text input training is completed, after final output analyzes the text Emotional semantic classification result.
The preprocessing process the following steps are included:
1) it segments, removal deactivates, English capitalization turns small letter, traditional font turns simplified.
2) word that frequency in data set G is more than or equal to σ is chosen, vocabulary wordList={ word is constructed1, word2,...wordε, wherein wordiIndicate that i-th of word in vocabulary wordlist, ε indicate that word frequency is more than in data set G The word sum of σ.
3) sample is deleted if length is greater than d to each sample in data set G, if length is less than d, uses symbol </>polishing.
The convolutional layer calculation formula of the local shape factor module based on CNN network is as follows:
Z=f (∑ WT*xi:i+s-1+b) (8)
Wherein: z indicates a neuron to the resulting feature vector of the convolution of text to be analyzed, and f () indicates activation letter Number, W ∈ Rs×mIndicate that the weight matrix of neuron, the same neuron parameter sharing, s × m indicate the size of convolution kernel size, b Indicate threshold value, xi:i+s-1Indicate the term vector by i-th of word in text sentence to i+s-1 word.
The training data is the data after pretreatment.
The convolutional layer in the local shape factor stage based on CNN network uses the convolution kernel of 4 kinds of different scales.Institute The training termination condition stated is that accuracy rate no longer changes or reach setting the number of iterations.
Detailed description of the invention
Fig. 1 flow chart of the method for the present invention；
The deep learning of Fig. 2 combination attention mechanism mostly classification sentiment analysis model structure schematic diagram.
Specific embodiment
Below with reference to chart and embodiment, a specific embodiment of the invention is described in further detail.Following reality Example is applied for illustrating the present invention, but is not intended to limit the scope of the invention.
Method proposed by the present invention is successively realized according to the following steps:
Step (1) data prediction
Emotion language dataset representation are as follows: G=[(segtxt1,y1),(segtxt2,y2),......,(segtxtN,yN)], Wherein, segtxtiIndicate i-th of sample, yiIt is then corresponding emotional category label.N indicates number of samples in data set G, emotion Label takes " happiness ", " indignation ", " detest ", " low " four major class, and N takes 80000, wherein four each 20000 of class emotion samples. Data prediction is carried out to sample in G including the following steps:
1) it segments, removal deactivates, English capitalization turns small letter, traditional font turns simplified.
2) word that frequency in data set G is more than or equal to σ is chosen, vocabulary wordList={ word is constructed1, word2,...wordε, wherein wordiIndicate i-th of word in data set G, word frequency is more than the word of σ in ε expression data set G Sum.σ takes 2, and in finally obtained data set G, word frequency is more than or equal to word totally 41763 of 2, i.e. ε is 41763.
3) by after above-mentioned processing, the sample is deleted, if length if length is greater than d to each sample in data set G Less than d, then symbol is used</>polishing.D takes 64.
Data set G after pretreatment, is expressed as G '=[(seg1,y1),(seg2,y2),...,(segM,yM)].Wherein: segiIt is expressed as i-th of sample, y in data set G 'iIt is then corresponding emotional category label, M indicates the middle sample of data set G ' Number.The number of the middle sample of final data collection G ' is 73150, and the sample strip number of each emotional category is as shown in table 1:
Sample size of all categories after table 1 pre-processes
The input of step (2) model
Sample data (seg, y) to be analyzed for any one in data set G ', by it, further refinement is indicated are as follows:
Seg=[w1,w2,w3,...,wd]T (1)
Y=[0,0,1 ..., 0] (2)
Wherein: wi∈RεRefer to and encoded according to one-hot of the vocabulary wordList to the i-th word in text to be analyzed, ε is The size of vocabulary wordList, the long d of the sentence of text are 64.y∈RpIt is the one-hot coding according to emotional category, p indicates model Class number to be divided, p 4.Then the term vector embeded matrix of the sample may be expressed as:
X=seg*ET (3)
Wherein: X ∈ Rd×m, X=[x1,x2,...,xd]TIt is indicated for the term vector matrix of text to be analyzed, term vector dimension m Take 256.xi∈RmIt indicates that term vector embeding layer indicates E for the term vector of i-th of vocabulary in the text, is increased income using wikipedia Word2vec term vector, next using X as the input of network model.
Step (3) constructs deep learning mostly classification sentiment analysis model
Deep learning sentiment analysis model of more classifying includes local shape factor stage based on CNN network and based on LSTM The word order relationship characteristic of network extracts the stage, by the pond layer result C in the local shape factor stage based on CNN networkCnnAnd base The result C' in stage is extracted in the word order relationship characteristic of LSTM networkRnnSplicing, i.e. vector [CCnn；C'Rnn] finally mentioned as model The feature vector taken.Then by feature vector [CCnn；C'Rnn] obtain final model output vector by full articulamentumWherein p indicates model class number to be divided.
The local shape factor stage based on CNN network, including the following contents:
The input of local shape factor stage is that the term vector matrix of the text to be analyzed of formula 3 indicates X；
The local shape factor stage is based on CNN network, altogether includes two layers, i.e. one layer of convolutional layer, one layer of pond layer, In:
Convolutional layer carries out convolution, and same scale convolution kernel to text to be analyzed using the convolution kernel of n kind different scale Filter, that is, neuron each k, n of the present invention takes 4, k to take 128.
The resulting vector of convolution is done down-sampling using the method for maximum pond layer by pond layer, selects local optimum feature, Therefore each filter becomes a scalar by maximum pond layer, and it is special which represents emotion optimal in the filter Sign；
The output of local shape factor module is CCnn=[c1,c2,...,cnk], i.e., it will be various sizes of more in the layer of pond The optimal characteristics that a filter is chosen are spliced together CCnn=[c1,c2,...,cnk] output as this module, wherein CCnn ∈Rnk, nk be convolutional layer in all filters number, totally 512；
The word order relationship characteristic based on LSTM network extracts stage, including the following contents:
Multiple dimensioned CNN network local shape factor: convolutional layer in the local shape factor stage based on CNN network is same The convolution results of k filter of convolution scale are spliced, and set Z is obtainedCnn, then will set ZCnnIn each vector ZiInput Into GLU mechanism, i.e., gate convolutional network, obtained result are denoted as { π1,π2,...,πn, complete multiple dimensioned CNN network part The extraction of feature.
Wherein, ZCnn={ Z1,Z2,...,Zn, ZiFor the splicing for multiple filter convolution results that scale is i；
Wherein,ZiRepresent k filter convolution results of a certain scale Splicing, W1, W2∈Rλ×qFor weight matrix, λ indicates the dimension of respective weights matrix, b1, b2∈RqFor amount of bias, σ is indicated Sigmoid function, πi∈Rq, q is the output dimension of LSTM network, and q takes 256；
Then, using attention mechanism, by multiple dimensioned CNN network local shape factor result { π1,π2,...,πnIncorporate Into LSTM network, the output result C' that the word order relationship characteristic based on LSTM network extracts the stage is obtainedRnn, i.e.,
Wherein,Indicate the output of LSTM module corresponding to the last one word in text to be analyzed,It indicates wait divide The output of LSTM module corresponding to first word in text is analysed, the present invention uses two-way LSTM model, i.e. BiLSTM model,
Using forward-propagating, specific calculating process is as follows:
D is the length of text to be analyzed, the corresponding LSTM module of each word order in the text,
During forward-propagating, the output of the t-1 LSTM module isThe then output of t-th of LSTM moduleIt calculates Formula is as follows:
Wherein:It is the dot product of two vectors, also referred to as scoring functions, is for calculating previous word The output of LSTMWith the similarity of current local feature vectors,
Wherein: αt,i∈ R represents feature πiWeight,
Wherein: st-1∈RqIt is the weighted results of multiple convolution features, utilizes st-1Instead ofIn conjunction with current term word to Measure xtAcquire the output of current LSTM moduleFormula is as follows:
Using backpropagation, specific calculating process is as forward-propagating, and details are not described herein again；
Step (4) model training: inputting sentiment analysis models of classifying for training data more, using cross entropy loss function, In conjunction with backpropagation BP algorithm adjusting parameter, is returned using softmax as sorting algorithm, complete training.
Step (5) model analysis: it is analysed to the model that text input training is completed, after final output analyzes the text Emotional semantic classification result.
The convolutional layer calculation formula of the local shape factor module based on CNN network is as follows:
Z=f (∑ WT*xi:i+s-1+b) (8)
Wherein: z indicates a neuron to the resulting feature vector of the convolution of text to be analyzed, and f () indicates activation letter Number, W ∈ Rs×mIndicate that the weight matrix of neuron, the same neuron parameter sharing, s × m indicate the size of convolution kernel size, b Indicate threshold value, xi:i+s-1It indicates to take [2,3,4,5] four by the term vector of i-th of word in text sentence to i+s-1 word, s The different convolution size of kind, f () use RELU activation primitive.
The training data is the data after pretreatment.
The convolutional layer in the local shape factor stage based on CNN network uses the convolution kernel of 4 kinds of different scales.Institute The training termination condition stated is that accuracy rate no longer changes or reach setting the number of iterations.
1. experimental analysis
Test phase chooses happy, angry, detest, low emotion corpus of all categories respectively accounts for 2000.Use accuracy rate Acc (Accuracy) is used as evaluation index, and the parameter of test phase model remains unchanged, and the results are shown in Table 2 for test set:
2 sentiment analysis Comparative result of table
The test result comparison of several models is given in table 2, wherein experiment 1 be general convolution kernel having a size of 3 list Scale CNN network model, experiment 2 is general LSTM network, and experiment 3 is then the text proposed in this paper based on attention mechanism Sentiment analysis model.
It is proposed in this paper based on note by the comparative analysis of experiment as it can be seen that comparing common CNN network and LSTM network The accuracy rate of the sentiment analysis model for power mechanism of anticipating all significantly improves, and illustrates that method proposed by the present invention can be extracted effectively The local feature information of CNN network and the word order characteristic information of LSTM network, illustrate the validity of this method.

Claims (6)

  1. A kind of sentiment analysis method 1. deep learning of combination attention mechanism is classified more, it is characterised in that the following steps are included:
    Step (1) data prediction
    If affection data set representations are as follows: G=[(segtxt1,y1),(segtxt2,y2),...,(segtxtN,yN)], wherein segtxtiIndicate i-th of sample, yiIt is then corresponding emotional category label, N indicates number of samples in data set G, to sample in G Data prediction is carried out,
    Data set G after pretreatment, is expressed as G '=[(seg1,y1),(seg2,y2),...,(segM,yM)], in which: segiTable It is shown as i-th of sample, y in data set G 'iIt is then corresponding emotional category label, M indicates the middle number of samples of data set G '；
    The input of step (2) building model
    Sample data (seg, y) to be analyzed for any one in data set G ', by it, further refinement is indicated are as follows:
    Seg=[w1,w2,...,wi,...,wd]T (1)
    Y=[0,0,1 ..., 0] (2)
    Wherein: wi∈RεRefer to and encoded according to one-hot of the vocabulary wordList to the i-th word in text to be analyzed, ε is vocabulary The size of wordList, d indicate that the sentence of the text is long, y ∈ RpIt is the one-hot coding according to emotional category, p indicates that model waits for The class number divided, then the term vector embeded matrix of the sample may be expressed as:
    X=seg*ET (3)
    Wherein: X ∈ Rd×m, X=[x1,x2,...,xd]TIt is indicated for the term vector matrix of text to be analyzed, m is the dimension of term vector Degree, xi∈RmIt is indicated for the term vector of i-th of vocabulary in the text, E is the expression of term vector embeding layer；
    Step (3) constructs deep learning mostly classification sentiment analysis model
    Deep learning sentiment analysis model of more classifying includes local shape factor stage based on CNN network and based on LSTM network Word order relationship characteristic extract the stage, by the pond layer result C in the local shape factor stage based on CNN networkCnnBe based on The word order relationship characteristic of LSTM network extracts the result C' in stageRnnSplicing, i.e. vector [CCnn；C'Rnn] finally extracted as model Feature vector, then by feature vector [CCnn；C'Rnn] obtain final model output vector by full articulamentumWherein p indicates model class number to be divided,
    The local shape factor stage based on CNN network, including the following contents:
    The input of local shape factor stage is that the term vector matrix of the text to be analyzed of formula 3 indicates X；
    The local shape factor stage is based on CNN network, altogether includes two layers, i.e. one layer of convolutional layer, one layer of pond layer, in which:
    Convolutional layer carries out convolution, and the filtering of same scale convolution kernel to text to be analyzed using the convolution kernel of n kind different scale Device, that is, neuron each k；
    The resulting vector of convolution is done down-sampling using the method for maximum pond layer by pond layer, selects local optimum feature, therefore Each filter becomes a scalar by maximum pond layer, which represents affective characteristics optimal in the filter；
    The output of local shape factor module is CCnn=[c1,c2,...,cnk], i.e., by multiple filtering various sizes of in the layer of pond The optimal characteristics that device is chosen are spliced together CCnn=[c1,c2,...,cnk] output as this module, wherein CCnn∈Rnk, nk For the number of filters all in convolutional layer；
    The word order relationship characteristic based on LSTM network extracts stage, including the following contents:
    Multiple dimensioned CNN network local shape factor: by the same convolution of convolutional layer in the local shape factor stage based on CNN network The convolution results of k filter of scale are spliced, and set Z is obtainedCnn, then will set ZCnnIn each vector ZiIt is input to In GLU mechanism, i.e., gate convolutional network, obtained result are denoted as { π1,π2,...,πn, it is special to complete multiple dimensioned CNN network part The extraction of sign,
    Wherein, ZCnn={ Z1,Z2,...,Zn, ZiFor the splicing for multiple filter convolution results that scale is i；
    Wherein,ZiRepresent the spelling of k filter convolution results of a certain scale It connects, W1, W2∈Rλ×qFor weight matrix, λ indicates the dimension of respective weights matrix, b1, b2∈RqFor amount of bias, σ indicates sigmoid Function, πi∈Rq, q is the output dimension of LSTM network；
    Then, using attention mechanism, by multiple dimensioned CNN network local shape factor result { π1,π2,...,πnBe dissolved into In LSTM network, the output result C' that the word order relationship characteristic based on LSTM network extracts the stage is obtainedRnn, i.e.,
    Wherein,Indicate the output of LSTM module corresponding to the last one word in text to be analyzed,Indicate text to be analyzed The output of LSTM module corresponding to first word in this, the present invention use two-way LSTM model, i.e. BiLSTM model,
    Using forward-propagating, specific calculating process is as follows:
    D is the length of text to be analyzed, the corresponding LSTM module of each word order in the text,
    During forward-propagating, the output of the t-1 LSTM module isThe then output of t-th of LSTM moduleCalculation formula It is as follows:
    Wherein:It is the dot product of two vectors, also referred to as scoring functions, is the LSTM for calculating previous word OutputWith the similarity of current local feature vectors,
    Wherein: αt,i∈ R represents feature πiWeight,
    Wherein: st-1∈RqIt is the weighted results of multiple convolution features, utilizes st-1Instead ofIn conjunction with the term vector x of current termt Acquire the output of current LSTM moduleFormula is as follows:
    Using backpropagation, specific calculating process is as forward-propagating, and details are not described herein again；
    Step (4) model training: inputting sentiment analysis models of classifying for training data more, using cross entropy loss function, in conjunction with Backpropagation BP algorithm adjusting parameter is returned as sorting algorithm using softmax, completes training；
    Step (5) model analysis: it is analysed to the model that text input training is completed, final output is to the feelings after text analysis Feel classification results.
  2. The sentiment analysis method 2. a kind of deep learning of combination attention mechanism according to claim 1 is classified more, it is special Sign is, the preprocessing process the following steps are included:
    1) segment, remove deactivate, English capitalization turn small letter, traditional font turn it is simplified,
    2) word that frequency in data set G is more than or equal to σ is chosen, vocabulary wordList={ word is constructed1,word2, ...wordε, wherein wordiIndicate i-th of word in vocabulary wordlist, word frequency is more than the word of σ in ε expression data set G Language sum,
    3) sample is deleted if length is greater than d to each sample in data set G, if length is less than d, uses symbol</> Polishing.
  3. The sentiment analysis method 3. a kind of deep learning of combination attention mechanism according to claim 1 is classified more, it is special Sign is that the convolutional layer calculation formula of the local shape factor module based on CNN network is as follows:
    Z=f (∑ WT*xi:i+s-1+b) (8)
    Wherein: z indicates a neuron to the resulting feature vector of the convolution of text to be analyzed, and f () indicates activation primitive, W ∈Rs×mIndicate that the weight matrix of neuron, the same neuron parameter sharing, s × m indicate the size of convolution kernel size, b is indicated Threshold value, xi:i+s-1Indicate the term vector by i-th of word in text sentence to i+s-1 word.
  4. The sentiment analysis method 4. a kind of deep learning of combination attention mechanism according to claim 1 is classified more, it is special Sign is that the training data is the data after pretreatment.
  5. The sentiment analysis method 5. a kind of deep learning of combination attention mechanism according to claim 1 is classified more, it is special Sign is that the convolutional layer in the local shape factor stage based on CNN network uses the convolution kernel of 4 kinds of different scales.
  6. The sentiment analysis method 6. a kind of deep learning of combination attention mechanism according to claim 1 is classified more, it is special Sign is that the training termination condition is that accuracy rate no longer changes or reach setting the number of iterations.
CN201910553755.7A 2019-06-25 2019-06-25 Deep learning multi-classification emotion analysis model combining attention mechanism Active CN110287320B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910553755.7A CN110287320B (en) 2019-06-25 2019-06-25 Deep learning multi-classification emotion analysis model combining attention mechanism

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910553755.7A CN110287320B (en) 2019-06-25 2019-06-25 Deep learning multi-classification emotion analysis model combining attention mechanism

Publications (2)

Publication Number Publication Date
CN110287320A true CN110287320A (en) 2019-09-27
CN110287320B CN110287320B (en) 2021-03-16

Family

ID=68005491

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910553755.7A Active CN110287320B (en) 2019-06-25 2019-06-25 Deep learning multi-classification emotion analysis model combining attention mechanism

Country Status (1)

Country Link
CN (1) CN110287320B (en)

Cited By (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110855474A (en) * 2019-10-21 2020-02-28 广州杰赛科技股份有限公司 Network feature extraction method, device, equipment and storage medium of KQI data
CN110866113A (en) * 2019-09-30 2020-03-06 浙江大学 Text classification method based on sparse self-attention mechanism fine-tuning Bert model
CN111079985A (en) * 2019-11-26 2020-04-28 昆明理工大学 Criminal case criminal period prediction method based on BERT and fused with distinguishable attribute features
CN111079547A (en) * 2019-11-22 2020-04-28 武汉大学 Pedestrian moving direction identification method based on mobile phone inertial sensor
CN111291832A (en) * 2020-03-11 2020-06-16 重庆大学 Sensor data classification method based on Stack integrated neural network
CN111339768A (en) * 2020-02-27 2020-06-26 携程旅游网络技术（上海）有限公司 Sensitive text detection method, system, electronic device and medium
CN111402953A (en) * 2020-04-02 2020-07-10 四川大学 Protein sequence classification method based on hierarchical attention network
CN111582397A (en) * 2020-05-14 2020-08-25 杭州电子科技大学 CNN-RNN image emotion analysis method based on attention mechanism
CN111881262A (en) * 2020-08-06 2020-11-03 重庆邮电大学 Text emotion analysis method based on multi-channel neural network
CN111914084A (en) * 2020-01-09 2020-11-10 北京航空航天大学 Deep learning-based emotion label text generation and evaluation system
CN112597279A (en) * 2020-12-25 2021-04-02 北京知因智慧科技有限公司 Text emotion analysis model optimization method and device
CN112598065A (en) * 2020-12-25 2021-04-02 天津工业大学 Memory-based gated convolutional neural network semantic processing system and method
CN112818123A (en) * 2021-02-08 2021-05-18 河北工程大学 Emotion classification method for text
CN113177111A (en) * 2021-05-28 2021-07-27 中国人民解放军国防科技大学 Chinese text emotion analysis method and device, computer equipment and storage medium
CN113239199A (en) * 2021-05-18 2021-08-10 重庆邮电大学 Credit classification method based on multi-party data set
CN113268592A (en) * 2021-05-06 2021-08-17 天津科技大学 Short text object emotion classification method based on multi-level interactive attention mechanism
CN113377901A (en) * 2021-05-17 2021-09-10 内蒙古工业大学 Mongolian text emotion analysis method based on multi-size CNN and LSTM models
CN113379818A (en) * 2021-05-24 2021-09-10 四川大学 Phase analysis method based on multi-scale attention mechanism network
WO2021174922A1 (en) * 2020-03-02 2021-09-10 平安科技（深圳）有限公司 Statement sentiment classification method and related device
CN114298025A (en) * 2021-12-01 2022-04-08 国家电网有限公司华东分部 Emotion analysis method based on artificial intelligence
CN114662547A (en) * 2022-04-07 2022-06-24 天津大学 MSCRNN emotion recognition method and device based on electroencephalogram signals
CN115116448A (en) * 2022-08-29 2022-09-27 四川启睿克科技有限公司 Voice extraction method, neural network model training method, device and storage medium
US20230160942A1 (en) * 2020-04-22 2023-05-25 Qingdao Topscomm Communication Co., Ltd Fault arc signal detection method using convolutional neural network

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108460089A (en) * 2018-01-23 2018-08-28 哈尔滨理工大学 Diverse characteristics based on Attention neural networks merge Chinese Text Categorization
CN109670169A (en) * 2018-11-16 2019-04-23 中山大学 A kind of deep learning sensibility classification method based on feature extraction
CN109710761A (en) * 2018-12-21 2019-05-03 中国标准化研究院 The sentiment analysis method of two-way LSTM model based on attention enhancing

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108460089A (en) * 2018-01-23 2018-08-28 哈尔滨理工大学 Diverse characteristics based on Attention neural networks merge Chinese Text Categorization
CN109670169A (en) * 2018-11-16 2019-04-23 中山大学 A kind of deep learning sensibility classification method based on feature extraction
CN109710761A (en) * 2018-12-21 2019-05-03 中国标准化研究院 The sentiment analysis method of two-way LSTM model based on attention enhancing

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
MING-HSIANG SU.ETL: "LSTM-based Text Emotion Recognition Using Semantic and Emotional Word Vectors", 《 2018 FIRST ASIAN CONFERENCE ON AFFECTIVE COMPUTING AND INTELLIGENT INTERACTION》 *
THITITORN SENEEWONG NA AYUTTHAYA.ETL: "Thai Sentiment Analysis via Bidirectional LSTM-CNN Model with Embedding Vectors and Sentic Features", 《2018 INTERNATIONAL JOINT SYMPOSIUM ON ARTIFICIAL INTELLIGENCE AND NATURAL LANGUAGE PROCESSING》 *
关鹏飞等: "注意力增强的双向LSTM情感分析", 《中文信息学报》 *

Cited By (34)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110866113A (en) * 2019-09-30 2020-03-06 浙江大学 Text classification method based on sparse self-attention mechanism fine-tuning Bert model
CN110866113B (en) * 2019-09-30 2022-07-26 浙江大学 Text classification method based on sparse self-attention mechanism fine-tuning burt model
CN110855474B (en) * 2019-10-21 2022-06-17 广州杰赛科技股份有限公司 Network feature extraction method, device, equipment and storage medium of KQI data
CN110855474A (en) * 2019-10-21 2020-02-28 广州杰赛科技股份有限公司 Network feature extraction method, device, equipment and storage medium of KQI data
CN111079547A (en) * 2019-11-22 2020-04-28 武汉大学 Pedestrian moving direction identification method based on mobile phone inertial sensor
CN111079985A (en) * 2019-11-26 2020-04-28 昆明理工大学 Criminal case criminal period prediction method based on BERT and fused with distinguishable attribute features
CN111914084A (en) * 2020-01-09 2020-11-10 北京航空航天大学 Deep learning-based emotion label text generation and evaluation system
CN111339768B (en) * 2020-02-27 2024-03-05 携程旅游网络技术（上海）有限公司 Sensitive text detection method, system, electronic equipment and medium
CN111339768A (en) * 2020-02-27 2020-06-26 携程旅游网络技术（上海）有限公司 Sensitive text detection method, system, electronic device and medium
WO2021174922A1 (en) * 2020-03-02 2021-09-10 平安科技（深圳）有限公司 Statement sentiment classification method and related device
CN111291832A (en) * 2020-03-11 2020-06-16 重庆大学 Sensor data classification method based on Stack integrated neural network
CN111402953A (en) * 2020-04-02 2020-07-10 四川大学 Protein sequence classification method based on hierarchical attention network
CN111402953B (en) * 2020-04-02 2022-05-03 四川大学 Protein sequence classification method based on hierarchical attention network
US11860216B2 (en) * 2020-04-22 2024-01-02 Qingdao Topscomm Communication Co., Ltd Fault arc signal detection method using convolutional neural network
US20230160942A1 (en) * 2020-04-22 2023-05-25 Qingdao Topscomm Communication Co., Ltd Fault arc signal detection method using convolutional neural network
CN111582397B (en) * 2020-05-14 2023-04-07 杭州电子科技大学 CNN-RNN image emotion analysis method based on attention mechanism
CN111582397A (en) * 2020-05-14 2020-08-25 杭州电子科技大学 CNN-RNN image emotion analysis method based on attention mechanism
CN111881262A (en) * 2020-08-06 2020-11-03 重庆邮电大学 Text emotion analysis method based on multi-channel neural network
CN111881262B (en) * 2020-08-06 2022-05-20 重庆邮电大学 Text emotion analysis method based on multi-channel neural network
CN112598065A (en) * 2020-12-25 2021-04-02 天津工业大学 Memory-based gated convolutional neural network semantic processing system and method
CN112597279A (en) * 2020-12-25 2021-04-02 北京知因智慧科技有限公司 Text emotion analysis model optimization method and device
CN112598065B (en) * 2020-12-25 2023-05-30 天津工业大学 Memory-based gating convolutional neural network semantic processing system and method
CN112818123A (en) * 2021-02-08 2021-05-18 河北工程大学 Emotion classification method for text
CN113268592A (en) * 2021-05-06 2021-08-17 天津科技大学 Short text object emotion classification method based on multi-level interactive attention mechanism
CN113377901A (en) * 2021-05-17 2021-09-10 内蒙古工业大学 Mongolian text emotion analysis method based on multi-size CNN and LSTM models
CN113377901B (en) * 2021-05-17 2022-08-19 内蒙古工业大学 Mongolian text emotion analysis method based on multi-size CNN and LSTM models
CN113239199A (en) * 2021-05-18 2021-08-10 重庆邮电大学 Credit classification method based on multi-party data set
CN113379818B (en) * 2021-05-24 2022-06-07 四川大学 Phase analysis method based on multi-scale attention mechanism network
CN113379818A (en) * 2021-05-24 2021-09-10 四川大学 Phase analysis method based on multi-scale attention mechanism network
CN113177111A (en) * 2021-05-28 2021-07-27 中国人民解放军国防科技大学 Chinese text emotion analysis method and device, computer equipment and storage medium
CN114298025A (en) * 2021-12-01 2022-04-08 国家电网有限公司华东分部 Emotion analysis method based on artificial intelligence
CN114662547A (en) * 2022-04-07 2022-06-24 天津大学 MSCRNN emotion recognition method and device based on electroencephalogram signals
CN115116448A (en) * 2022-08-29 2022-09-27 四川启睿克科技有限公司 Voice extraction method, neural network model training method, device and storage medium
CN115116448B (en) * 2022-08-29 2022-11-15 四川启睿克科技有限公司 Voice extraction method, neural network model training method, device and storage medium

Also Published As

Publication number Publication date
CN110287320B (en) 2021-03-16

Similar Documents

Publication Publication Date Title
CN110287320A (en) A kind of deep learning of combination attention mechanism is classified sentiment analysis model more
CN111767741B (en) Text emotion analysis method based on deep learning and TFIDF algorithm
CN109740148B (en) Text emotion analysis method combining BiLSTM with Attention mechanism
CN113254599B (en) Multi-label microblog text classification method based on semi-supervised learning
CN109241255B (en) Intention identification method based on deep learning
CN107608956B (en) Reader emotion distribution prediction algorithm based on CNN-GRNN
Alwehaibi et al. Comparison of pre-trained word vectors for arabic text classification using deep learning approach
CN110287323B (en) Target-oriented emotion classification method
CN107818084B (en) Emotion analysis method fused with comment matching diagram
CN107862087A (en) Sentiment analysis method, apparatus and storage medium based on big data and deep learning
CN110765769B (en) Clause feature-based entity attribute dependency emotion analysis method
CN112818861A (en) Emotion classification method and system based on multi-mode context semantic features
CN110851601A (en) Cross-domain emotion classification system and method based on layered attention mechanism
CN109271636B (en) Training method and device for word embedding model
CN112860896A (en) Corpus generalization method and man-machine conversation emotion analysis method for industrial field
CN112287106A (en) Online comment emotion classification method based on dual-channel hybrid neural network
CN107818173B (en) Vector space model-based Chinese false comment filtering method
CN114462385A (en) Text segmentation method and device
CN113204624B (en) Multi-feature fusion text emotion analysis model and device
CN113159831A (en) Comment text sentiment analysis method based on improved capsule network
Siddique et al. Bilingual word embeddings for cross-lingual personality recognition using convolutional neural nets
CN111414755A (en) Network emotion analysis method based on fine-grained emotion dictionary
CN109635289B (en) Entry classification method and audit information extraction method
CN111694960A (en) E-commerce comment emotion analysis model based on combination of part-of-speech characteristics and viewpoint characteristics and convolutional neural network
Vayadande et al. Mood detection and emoji classification using tokenization and convolutional neural network

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant