CN110750645A - Cross-domain false comment identification method based on countermeasure training - Google Patents
Cross-domain false comment identification method based on countermeasure training Download PDFInfo
- Publication number
- CN110750645A CN110750645A CN201910976702.6A CN201910976702A CN110750645A CN 110750645 A CN110750645 A CN 110750645A CN 201910976702 A CN201910976702 A CN 201910976702A CN 110750645 A CN110750645 A CN 110750645A
- Authority
- CN
- China
- Prior art keywords
- training
- model
- word
- countermeasure
- cross
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012549 training Methods 0.000 title claims abstract description 128
- 238000000034 method Methods 0.000 title claims abstract description 49
- 239000013598 vector Substances 0.000 claims abstract description 66
- 238000012360 testing method Methods 0.000 claims abstract description 17
- 238000007781 pre-processing Methods 0.000 claims abstract description 5
- 230000008485 antagonism Effects 0.000 claims description 33
- 230000006870 function Effects 0.000 claims description 17
- 230000003042 antagnostic effect Effects 0.000 claims description 11
- 238000003062 neural network model Methods 0.000 claims description 11
- 238000013528 artificial neural network Methods 0.000 claims description 7
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 5
- 238000005457 optimization Methods 0.000 claims description 5
- 238000012795 verification Methods 0.000 claims description 5
- 230000004913 activation Effects 0.000 claims description 4
- 238000011156 evaluation Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 claims description 4
- 238000013145 classification model Methods 0.000 claims description 3
- 238000009826 distribution Methods 0.000 claims description 3
- 238000002372 labelling Methods 0.000 claims description 3
- 239000011159 matrix material Substances 0.000 claims description 3
- 230000003247 decreasing effect Effects 0.000 claims description 2
- 238000003058 natural language processing Methods 0.000 abstract description 2
- 239000010410 layer Substances 0.000 description 22
- 238000002474 experimental method Methods 0.000 description 10
- 238000011160 research Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 4
- 238000010200 validation analysis Methods 0.000 description 4
- 238000013179 statistical model Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000012552 review Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000007477 logistic regression Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000007721 medicinal effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000002356 single layer Substances 0.000 description 1
- 238000012706 support-vector machine Methods 0.000 description 1
- 230000001225 therapeutic effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/35—Clustering; Classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/02—Marketing; Price estimation or determination; Fundraising
- G06Q30/0201—Market modelling; Market analysis; Collecting market data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/02—Marketing; Price estimation or determination; Fundraising
- G06Q30/0201—Market modelling; Market analysis; Collecting market data
- G06Q30/0203—Market surveys; Market polls
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Finance (AREA)
- Strategic Management (AREA)
- Theoretical Computer Science (AREA)
- Accounting & Taxation (AREA)
- Development Economics (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Entrepreneurship & Innovation (AREA)
- Economics (AREA)
- General Business, Economics & Management (AREA)
- Game Theory and Decision Science (AREA)
- General Engineering & Computer Science (AREA)
- Marketing (AREA)
- Biomedical Technology (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Databases & Information Systems (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Machine Translation (AREA)
Abstract
The invention relates to the technical field of natural language processing, in particular to a cross-domain false comment identification method based on countermeasure training, which comprises the following steps: the method comprises the following steps: preprocessing the data sets of the three fields; step two: pre-training; step three: constructing a confrontation training model; step four: training a model; step five: testing the model; step six: evaluating, analyzing and summarizing the judgment result of the step four, and improving the performance of the model by using feedback; firstly, input word vectors generate disturbance, then the disturbance is input into an LSTM model for confrontation training, and finally a final prediction is output; the results show that the proposed model outperforms previous models in the results of mixed datasets of three domains and has better generalization capability on cross-domain false comment tasks.
Description
[ technical field ] A method for producing a semiconductor device
The invention relates to the technical field of natural language processing, in particular to a cross-domain false comment identification method based on countermeasure training.
[ background of the invention ]
In the web2.0 era, more and more users can make purchasing decisions by referring to online comments, positive comments can bring huge economic benefits and reputation to business organizations and individuals, negative comments can also influence sales performance of competitors, and thus false comments are greatly multiplied.
False comment identification studies have been continuing for the last decade since the concept of false comments by Jindal and Liu (2008) and a number of research methods have been proposed. There are two main categories of these research methods: traditional statistical models and neural network models. The traditional statistical models used for the false comment recognition comprise a support vector machine, naive Bayes, logistic regression and the like, the traditional statistical models are often combined with various features to classify texts, common text features comprise bag-of-words features, part-of-speech features, LIWC features and the like, and neural network models used for the false comment recognition comprise CNN/RNN/LSTM and the like.
For the false comment identification task, the sample size of the data set is relatively small, and the collection of the labeled data set is time-consuming and labor-consuming, so that the related work of cross-domain false comment identification is yet to be further researched. In the existing cross-domain false comment identification research, the prediction effect of a similar domain is good, but the prediction effect performance of a domain with large difference is poor.
[ summary of the invention ]
The invention aims to provide a cross-domain false comment identification method based on countermeasure training, aiming at the defects and shortcomings of the prior art.
The cross-domain false comment identification method based on the countermeasure training adopts the following steps:
the method comprises the following steps: preprocessing a data set in three fields, namely a hotel field, a retaurant field and a doctor field;
step two: pre-training, namely initializing word vectors and LSTM weights by using a pre-training recursive language model trained by unlabeled and labeled samples;
step three: constructing a confrontation training model;
step four: model training, namely inputting an embedded layer formed by word vectors of a training set into a confrontation training model, and inputting the obtained data into a classifier formed by a softmax function through a full connection layer to carry out confrontation training and obtain training parameters;
step five: the model test is carried out, wherein an embedded layer formed by word vectors of a test set is input into a trained model, obtained data firstly pass through a full connection layer and then are input into a classifier formed by a softmax function, and the classifier outputs text categories;
step six: and evaluating, analyzing and summarizing the judgment result of the step four, and improving the performance of the model by using feedback.
Further, in the step one, the data sets of the three fields are preprocessed, and the following steps are adopted:
step I, word frequency statistics: collecting text corpora of the three fields, and carrying out word frequency statistics to generate a unitary bow characteristic;
step II, data set division: dividing a mixed data set of the three fields into a training set, a verification set and a test set;
step III, labeling: and according to the category number of the corpora, assigning one-hot type labels to different categories of the corpora.
Further, the confrontational training model described in step three is based on the LSTM neural network model.
Further, in the confrontation training model described in step three, { w (t)1, K, T represents a set of T word sequences, and the corresponding target output is y; to represent a discrete word input as a continuous vector, a word vector matrix V ∈ R is defined(K +1)×DWhere K represents the number of words in the vocabulary, and each row h corresponds to a word vector for the ith word. Note that the K +1 word vectors represent the eos symbol V at the end of the sequenceeos. At step t, the input is a discrete word w(t)Corresponding word vector is v(t)。
Further, in the confrontation training model in the third step, the word vector is disturbed; perturbations are standard bounded, but the model can make them insignificant by learning word vectors with very large norms; to prevent this, when applying the antagonism and virtual antagonism training to the model defined above, we apply the word vector vkReplacement by normalized word vectorsIt is defined as:
wherein: f. ofiThe calculated frequency of the ith word in all training samples.
Further, the countermeasure training of step three is a new classifier regularization method for improving robustness to small, near worst case disturbances; x represents the input, θ represents the parameters of the classifier; when training the classifier, the loss function formula is as follows:
wherein: r represents the perturbation to the input and,is a constant representing the parameters of the classifier; at each step in the training process, the current model is determinedIs radvAnd training the robustness of the model to such disturbances by minimizing the formula; for an accurate minimization solution of r, Goodfellow proposes through linearizationApproximately solve the value by linear approximation and L in the formula2The norm constraint of (2) and the anti-disturbance formula are as follows:
calculating the disturbance by utilizing back propagation in the neural network; the virtual antagonism training is a regularization method closely related to the antagonism training; the additional losses due to virtual antagonistic training are as follows:
wherein: KL [ p | | q |)]Representing the KL divergence between distributions p and q, the classifier can be trained to be smooth by minimizing the formula, which can be seen as having the classifier on the current modelThe most sensitive direction resists disturbances;
the virtual antagonism loss formula only needs to input x without an actual label y, and the label y is needed for antagonism loss, so that the virtual antagonism training can be applied to semi-supervised learning; for calculating virtual countermeasure loss, a method of calculating an approximation equation using back propagation in a neural network;
to define the countering perturbations in the word vector, s denotes a normalized word vector sequencep (y | s; θ) represents the y conditional probability model given s; wherein: θ represents a parameter of the model; then the antagonistic perturbation on s radvIs defined as:
in order to make the model robust to the antagonistic perturbations defined in the formula, the antagonistic loss is defined by the following formula:
wherein: n is the number of marked samples; the antagonism training is to apply a negative log-likelihood plus a stochastic gradient decreasing LadvMinimization;
in virtual antagonism training of a text classification model, in each training step, the following approximate virtual antagonism perturbation is computed:
wherein: d is a small random vector in the TD dimension; corresponding to a single iteration of the second order taylor expansion and power function method described above for the formula using the method proposed by Miyato; the virtual confrontation loss is defined as:
wherein: n' is the number of marked and unmarked instances.
Further: step four, model training, namely adding a hidden layer between the softmax layer and the last LSTM layer of the output target y, wherein the dimensionality is 30, and the activation function of the hidden layer is ReLU; for optimization, using Adam optimizer, initial learning rate was set to 0.0005, exponentially decaying learning rate was set to 0.9998, batch _ size was set to 64; gradient clipping is applied and a gradient clipping threshold of 1.0 is set for all parameters except the word vector.
Further, the model training described in the step four optimizes two scalar hyperparameters by using a validation set, wherein one scalar hyperparameter is the dropout rate of the word vector, and the other scalar hyperparameter is the specification constraint epsilon of the countermeasure training and the virtual countermeasure training.
Further, the evaluation, analysis and summarization in step six comprises: evaluating the accuracy, recall rate and F value of the predicted data; the summary summarizes the main types and causes of error occurrences.
The invention has the following advantages and beneficial effects:
1. the conventional countermeasure training is mostly used for image classification tasks, and the countermeasure training is used in the field of false comment identification, so that cross-field false comment identification research is carried out;
2. based on the data sets in the three mentioned fields, the past work only carries out cross-field research from hotel to restaurant and from hotel to doctor, and the invention comprehensively carries out cross-field experiments, including three groups of experiments from hotel to restaurant and doctor, from restaurant to hotel and doctor, from doctor to hotel and restaurant;
3. the method firstly uses a large number of label-free samples for pre-training to obtain characteristics, then trains the model, and uses a countermeasure training method on the basis of a one-way LSTM model. Experimental results show that the results of the mixed data sets in the three fields exceed the previous models, and the method has better generalization capability on cross-field false comment tasks.
[ description of the drawings ]
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, and are not to be considered limiting of the invention, in which:
FIG. 1 is an overall framework topology of the proposed model in the present invention;
FIG. 2 is a neural network model topology of the LSTM in the present invention;
FIG. 3 is a topological diagram of a neural network model in the present invention after a perturbation is applied to a word vector.
[ detailed description ] embodiments
The present invention will now be described in detail with reference to the drawings and specific embodiments, wherein the exemplary embodiments and descriptions are provided only for the purpose of illustrating the present invention and are not to be construed as limiting the present invention.
As shown in fig. 1 to fig. 3, the cross-domain false comment identification method based on countermeasure training according to the present embodiment includes the following steps:
the method comprises the following steps: preprocessing a data set in three fields, namely a hotel field, a retaurant field and a doctor field;
step two: pre-training, namely initializing word vectors and LSTM weights by using a pre-training recursive language model (Bengio) trained by unlabeled and labeled samples;
step three: constructing a confrontation training model;
step four: model training, namely inputting an embedded layer formed by word vectors of a training set into a confrontation training model, and inputting the obtained data into a classifier formed by a softmax function through a full connection layer to carry out confrontation training and obtain training parameters;
step five: the model test is carried out, wherein an embedded layer formed by word vectors of a test set is input into a trained model, obtained data firstly pass through a full connection layer and then are input into a classifier formed by a softmax function, and the classifier outputs text categories;
step six: and evaluating, analyzing and summarizing the judgment result of the step four, and improving the performance of the model by using feedback.
Further, in the step one, the data sets of the three fields are preprocessed, and the following steps are adopted:
step I, word frequency statistics: collecting text corpora of the three fields, and carrying out word frequency statistics to generate a unitary bow characteristic;
step II, data set division: dividing a mixed data set of the three fields into a training set, a verification set and a test set;
step III, labeling: and according to the category number of the corpora, assigning one-hot type labels to different categories of the corpora.
Further, the confrontational training model described in step three is based on the LSTM neural network model.
Further, in the confrontation training model described in step three, { w (t)1, K, T represents a set of T word sequences, and the corresponding target output is y; to represent a discrete word input as a continuous vector, a word vector matrix V ∈ R is defined(K +1)×DWhere K represents the number of words in the vocabulary, and each row h corresponds to a word vector for the ith word. Note that the K +1 word vectors represent the eos symbol V at the end of the sequenceeos. At step t, the input is a discrete word w(t)Corresponding word vector is v(t)。
Further, in the confrontation training model in the third step, the word vector is disturbed; the perturbations are standard bounded, but the model can be modeled by learning word vectors with very large normsThe disturbance becomes insignificant; to prevent this, when applying the antagonism and virtual antagonism training to the model defined above, we apply the word vector vkReplacement by normalized word vectorsIt is defined as:
wherein: f. ofiThe calculated frequency of the ith word in all training samples.
Further, the countermeasure training of step three is a new classifier regularization method for improving robustness to small, near worst case disturbances; x represents the input, θ represents the parameters of the classifier; when training the classifier, the loss function formula is as follows:
wherein: r represents the perturbation to the input and,is a constant representing the parameters of the classifier; at each step in the training process, the current model is determinedIs radvAnd training the robustness of the model to such disturbances by minimizing the formula; however, we cannot compute this value accurately in the general case, because for many interesting models (such as neural networks) it is difficult to minimize the accuracy of r; goodfellow et al propose by linearizationApproximately solve the value by linear approximation and L in the formula2Norm constraints of, against disturbance equations such asThe following:
this perturbation can be easily calculated using back propagation in the neural network; the virtual antagonism training is a regularization method closely related to the antagonism training; the additional losses due to virtual antagonistic training are as follows:
wherein: KL [ p | | q |)]Representing the KL divergence between distributions p and q, the classifier can be trained to be smooth by minimizing the formula, which can be seen as having the classifier on the current modelThe most sensitive direction resists disturbances;
the virtual antagonism loss formula only needs to input x without an actual label y, but needs the label y in the antagonism loss, so that the application of the virtual antagonism training to semi-supervised learning becomes possible; in general, we cannot accurately calculate the virtual confrontation loss, and Miyato et al propose a method of efficiently calculating an approximation equation using back propagation.
To define the countering perturbations in the word vector, s denotes a normalized word vector sequencep (y | s; theta) represents the y conditional probability model given s, where theta represents a parameter of the model; then we perturb the antagonism at s radvIs defined as:
to make the model robust to the antagonistic perturbations defined in the formula, we define the antagonistic loss with the following formula:
wherein: n is the number of marked samples; in the experiment, the antagonism training is to reduce the negative log-likelihood plus the random gradientadvMinimization;
in virtual antagonism training of our text classification model, in each training step, we calculate the following approximate virtual antagonism perturbation:
wherein: d is a small random vector in the TD dimension; this approximation corresponds to a single iteration of the second order taylor expansion and power function method described above for the formula using the method proposed by Miyato; the virtual confrontation loss is defined as:
wherein: n' is the number of marked and unmarked instances.
Further: step four, model training, namely adding a hidden layer between the softmax layer and the last LSTM layer of the output target y, wherein the dimensionality is 30, and the activation function of the hidden layer is ReLU;
for optimization, using Adam optimizer, initial learning rate was set to 0.0005, exponentially decaying learning rate was set to 0.9998, batch _ size was set to 64; gradient clipping is applied and a gradient clipping threshold of 1.0 is set for all parameters except the word vector.
Further, the model training described in step four optimizes two scalar hyperparameters by using the validation set, wherein one is the dropout rate of the word vector, and the other is the specification constraint of the countermeasure training and the virtual countermeasure training.
Further, the evaluation, analysis and summarization in step six comprises: evaluating the accuracy, recall rate and F value of the predicted data; the summary summarizes the main types and causes of error occurrences.
The general framework of the invention is shown in fig. 1; firstly, disturbing a word vector to generate a confrontation sample, then carrying out confrontation training on the basis of a baseline model LSTM, and enabling the confrontation sample to participate in training to generate confrontation loss; finally, a final representation of the predicted false comment is generated. In order to verify the validity of the proposed model, the invention performed multiple sets of experiments.
Firstly, the performance of different network models (SVM, LSTM and the proposed model) on a false comment identification task is experimentally discussed on a mixed data set; then, on data sets of different fields, cross-field false comment identification experiments are carried out by using the LSTM and the proposed model respectively, and the generalization ability of the countertraining on the cross-field false comment identification model is contrastively analyzed.
The neural network model of LSTM is shown in fig. 2, and the neural network model after perturbation is applied to the word vectors is shown in fig. 3.
The corpus adopted by the invention contains real and false comments in three fields of hotels, restaurants and doctors (hotel field, restaurant field and doc field). In each domain, real reviews are sourced from customers, false reviews are sourced from Turkers and professional employees. Firstly, text corpora of three fields are collected, and word frequency statistics is carried out to generate a unitary bow characteristic. In order to compare the effectiveness of various models, the invention uses the mixed data sets of the three fields to carry out a group of experiments; the mixed data set was randomly divided into a training set, a test set, and a validation set at a ratio of 80: 10.
The present invention initializes word vectors and LSTM weights using a pre-trained recursive language model trained from unlabeled and labeled samples. The present invention uses a unidirectional single layer LSTM with 256 hidden units. The word vector dimension D is set to 256 and the sample softmax penalty from 256 candidate samples is used for training. For optimization, the parameter is learned using Adam optimizer, during training, batch _ size is set to 64, initial learning rate is set to 0.001, and exponentially decaying learning rate is set to 0.9999, for a total of 10000 steps. A gradient clipping threshold of 1.0 is set for all parameters except the word vector, and a dropout rate of 0.5 is set at the word vector level.
After pre-training, the classifier is trained using virtual confrontation training. And a hidden layer is added between the softmax layer and the last LSTM layer of the output target y, the dimension is 30, and the activation function of the hidden layer is ReLU. For optimization, we again used the Adam optimizer with the initial learning rate set to 0.0005, the exponentially decaying learning rate set to 0.9998, and the batch _ size set to 64. Gradient clipping is again applied, setting the gradient clipping threshold to 1.0 for all parameters except the word vector.
For each data set, dividing an original training set into a training set and a verification set, roughly optimizing some hyper-parameters by using all methods, such as adjusting a model architecture, batch size, training steps and the like, and verifying the performance of a basic model by using a word vector dropout. For each method, we optimize two scalar superparameters using the validation set, one is the dropout rate of the word vector and the other is the canonical constraint e for the countermeasure training and the virtual countermeasure training. Note that for antagonism training and virtual antagonism training, we find that the model works best when perturbation is generated after the word vector dropout is applied.
The data output by the fusion layer further passes through the full connection layer and finally enters the classifier. The classifier selects the softmax function, which normalizes the input. When initializing, the class number of the data set needs to be preset, the class number of the used corpus is 2 (real or false), softmax outputs the probability that the sample belongs to the 2 classes, and the class label corresponding to the maximum value is used as the judgment of the class of the test sample. And comparing the test result output by the classifier with the label to know whether the test result is correct or not, and then counting the test accuracy as an evaluation index for evaluating the model performance obtained through the training process.
Constructing other comparison models, and performing comparison test, wherein the experimental results on the mixed data set are shown in Table 1
Table 1: hybrid dataset results
Method | Accuracy(%) | Marco-F1(%) |
SVM | 68.8 | 69.0 |
LSTM | 75.7 | 74.2 |
our model | 84.3 | 83.9 |
Ren(2016) | 84.1 | 83.9 |
The results of the experiments on the mixed data set of the three fields for the different models are given in table 1. The traditional discrete model SVM, combined with a variety of features, gives only 68.8% accuracy. For the neural network model, the LSTM achieved 75.7% accuracy, which indicates the effectiveness of the neural network model over the discrete model. Based on the LSTM model, the results using the challenge training were better than the discrete model and the LSTM model, achieving an accuracy of 84.3%. In addition, we have compared our method with Ren et al's gated recurrent neural network with attention mechanism, and experimental data show that our model has more results on mixed data sets than their proposed model, i.e. we achieve higher accuracy with similar values of Marco-F1.
The experimental results of the cross-domain dataset for the LSTM model are shown in table 2:
table 2: cross-domain dataset results for LSTM models
Table 2 gives the experimental results of the LSTM model based on the data set of each domain, respectively. Wherein, the model trained by hotel field data has an accuracy rate of 70.3% when predicting the restaurant field and an accuracy rate of only 50.3% when predicting the doctor field. This is because hotel domain data and restaurant domain data have many similar attributes, such as description of environment, location or service, while data content in the doctor domain mostly relates to medicine, medical or therapeutic effects, etc., and the prediction result is much lower than that in the restaurant domain due to the large word vector difference. Similarly, restaurant domain models also show differences in predicting hotel and doctor domains, i.e., the hotel domain predicts higher results than the doctor domain. The model trained by the doctor field data has basically similar results when predicting the restaurant field data, and the accuracy rate only differs by 0.2 percent, because the prediction results are relatively similar due to the similarity of the test sets caused by the similar attributes of the hotel field data and the restaurant field data.
The experimental results of the cross-domain data set of the proposed model are shown in table 3:
table 3: cross-domain dataset results
Table 3 gives the experimental results of our model based on the data set of each domain, respectively.
Firstly, compared with the data in the table 2, the results of all experiments of the model disclosed by the invention, no matter the accuracy or the Macro-F1 value, are improved to different degrees compared with the baseline model LSTM, and the effectiveness of the model disclosed by the invention on cross-domain false comment identification tasks is demonstrated.
Secondly, through observation and discovery, the improvement is obvious results of the model in the hotel or restaurant field in the doctor field, the prediction accuracy rate from the hotel field to the doctor field is improved by about 16 percentage points, and the prediction accuracy rate from the restaurant field to the doctor field is also improved by nearly 11 percentage points, which shows that the capability of the model for adapting to new data is improved.
Table 4: ren et al Cross-Domain dataset results
As shown in table 4, compared with the results of Ren and the like (Ren and the like only predicts from the hotel field to the restaurant and doctor field), the classifier trained by the hotel model shows that the results of the invention are worse than Ren and the like in the prediction of the restaurant field, but the performance of the invention is improved by nearly 9 percent in the accuracy rate and the Marco-F1 value is also improved in the doctor field compared with Ren and the like, which shows that the model of the invention has stronger adaptability to strange data and can cope with the difference of word vectors caused by different fields.
That is to say, the model of the invention has stronger generalization capability on the cross-domain false comment identification task, and can weaken the difference of different domains, so that the effect reaches a balance.
In addition, not only the experiment in the hotel field, but also the classifier trained by data in a certain field can be used according to other two groups of experiment results, so that the effect difference is obviously reduced compared with Ren and the like when the data in the other two fields are predicted. For example, when the hotel field is used for predicting the restaurant and doctor fields, the accuracy difference of Ren and the like is about 26 percent, while the model accuracy difference of the invention is 14 percent; when the restaurant field is used for predicting the hotel field and the doctor field, the difference of the accuracy rate is 3.5 percent; when the doctor field is used for predicting the fields of hotels and restaurants, the accuracy rate only differs by 0.2 percent.
Therefore, the difference of the prediction results in different fields is obviously reduced by the model, and the generalization capability of the model on the false comment identification task is further proved to be stronger than that of Ren and the like.
The method is based on mixed data sets of three fields (hotel, restaurant and sector), different machine learning models are adopted to model the comment text, and the effectiveness of the proposed models is verified; secondly, based on the data set of each field, cross-field false recognition research is carried out, namely, a data training model of one field is used for verifying the classification performance of the other two fields. Specifically, the perturbation is generated on the input word vector, then the perturbation is input into the LSTM model for countertraining, and finally a final prediction is output. The results show that the proposed model outperforms previous models in the results of mixed datasets of three domains and has better generalization capability on cross-domain false comment tasks.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention, and all equivalent changes and modifications made based on the features and principles described in the claims of the present invention are included in the scope of the present invention.
Claims (9)
1. The cross-domain false comment identification method based on the countermeasure training is characterized by comprising the following steps: the method comprises the following steps: preprocessing a data set in three fields, namely a hotel field, a retaurant field and a doctor field;
step two: pre-training, namely initializing word vectors and LSTM weights by using a pre-training recursive language model trained by unlabeled and labeled samples;
step three: constructing a confrontation training model;
step four: model training, namely inputting an embedded layer formed by word vectors of a training set into a confrontation training model, and inputting the obtained data into a classifier formed by a softmax function through a full connection layer to carry out confrontation training and obtain training parameters;
step five: the model test is carried out, wherein an embedded layer formed by word vectors of a test set is input into a trained model, obtained data firstly pass through a full connection layer and then are input into a classifier formed by a softmax function, and the classifier outputs text categories;
step six: and evaluating, analyzing and summarizing the judgment result of the step four, and improving the performance of the model by using feedback.
2. The cross-domain false comment recognition method based on countermeasure training of claim 1, wherein: preprocessing the data sets in the three fields in the first step, and adopting the following steps:
step I, word frequency statistics: collecting text corpora of the three fields, and carrying out word frequency statistics to generate a unitary bow characteristic;
step II, data set division: dividing a mixed data set of the three fields into a training set, a verification set and a test set;
step III, labeling: and according to the category number of the corpora, assigning one-hot type labels to different categories of the corpora.
3. The cross-domain false comment recognition method based on countermeasure training of claim 1, wherein: the confrontational training model described in step three is based on the LSTM neural network model.
4. The cross-domain false comment recognition method based on countermeasure training of claim 1, wherein: in the confrontation training model described in step three, { w(t)1, K, T represents a set of T word sequences, and the corresponding target output is y; to represent a discrete word input as a continuous vector, a word vector matrix V ∈ R is defined(K+1)×DWhere K represents the number of words in the vocabulary, and each row h corresponds to a word vector for the ith word. Note that the K +1 word vectors represent the eos symbol V at the end of the sequenceeos. At step t, the input is a discrete word w(t)Corresponding word vector is v(t)。
5. The cross-domain false comment recognition method based on countermeasure training of claim 1, wherein: in the confrontation training model in the third step, the word vector is disturbed; perturbations are standard bounded, but the model can make them insignificant by learning word vectors with very large norms; to prevent this, when applying the antagonism and virtual antagonism training to the model defined above, we apply the word vector vkReplacement by normalized word vectorsIt is defined as:
wherein: f. ofiThe calculated frequency of the ith word in all training samples.
6. The cross-domain false comment recognition method based on countermeasure training of claim 1, wherein: the countermeasure training in the third step is a new classifier regularization method, which is used for improving the robustness to small and approximate worst case disturbance; x represents the input, θ represents the parameters of the classifier; when training the classifier, the loss function formula is as follows:
wherein: r represents the perturbation to the input and,is a constant representing the parameters of the classifier; at each step in the training process, the current model is determinedIs radvAnd training the robustness of the model to such disturbances by minimizing the formula; for an accurate minimization solution of r, Goodfellow proposes through linearizationApproximately solve the value by linear approximation and L in the formula2The norm constraint of (2) and the anti-disturbance formula are as follows:
calculating the disturbance by utilizing back propagation in the neural network; the virtual antagonism training is a regularization method closely related to the antagonism training; the additional losses due to virtual antagonistic training are as follows:
wherein: KL [ p | | q |)]Representing the KL divergence between distributions p and q, the classifier can be trained to be smooth by minimizing the formula, which can be seen as having the classifier on the current modelThe most sensitive direction resists disturbances;
the virtual antagonism loss formula only needs to input x without an actual label y, and the label y is needed for antagonism loss, so that the virtual antagonism training can be applied to semi-supervised learning; for calculating virtual countermeasure loss, a method of calculating an approximation equation using back propagation in a neural network;
to define the countering perturbations in the word vector, s denotes a normalized word vector sequencep (y | s; θ) represents the y conditional probability model given s; wherein: θ represents a parameter of the model; then the antagonistic perturbation on s radvIs defined as:
in order to make the model robust to the antagonistic perturbations defined in the formula, the antagonistic loss is defined by the following formula:
wherein: n is the number of marked samples; the antagonism training is to apply a negative log-likelihood plus a stochastic gradient decreasing LadvMinimization;
in virtual antagonism training of a text classification model, in each training step, the following approximate virtual antagonism perturbation is computed:
wherein: d is a small random vector in the TD dimension; corresponding to a single iteration of the second order taylor expansion and power function method described above for the formula using the method proposed by Miyato; the virtual confrontation loss is defined as:
wherein: n' is the number of marked and unmarked instances.
7. The cross-domain false comment recognition method based on countermeasure training of claim 1, wherein: step four, model training, namely adding a hidden layer between the softmax layer and the last LSTM layer of the output target y, wherein the dimensionality is 30, and the activation function of the hidden layer is ReLU; for optimization, using Adam optimizer, initial learning rate was set to 0.0005, exponentially decaying learning rate was set to 0.9998, batch _ size was set to 64; gradient clipping is applied and a gradient clipping threshold of 1.0 is set for all parameters except the word vector.
8. The cross-domain false comment recognition method based on countermeasure training of claim 1, wherein: and fourthly, model training is carried out, two scalar hyperparameters are optimized by using a verification set, one scalar hyperparameter is the dropout rate of the word vector, and the other scalar hyperparameter is the specification constraint E of the countermeasure training and the virtual countermeasure training.
9. The cross-domain false comment recognition method based on countermeasure training of claim 1, wherein: the evaluation, analysis and summarization in step six comprises: evaluating the accuracy, recall rate and F value of the predicted data; the summary summarizes the main types and causes of error occurrences.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910976702.6A CN110750645B (en) | 2019-10-15 | 2019-10-15 | Cross-domain false comment identification method based on countermeasure training |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910976702.6A CN110750645B (en) | 2019-10-15 | 2019-10-15 | Cross-domain false comment identification method based on countermeasure training |
Publications (2)
Publication Number | Publication Date |
---|---|
CN110750645A true CN110750645A (en) | 2020-02-04 |
CN110750645B CN110750645B (en) | 2023-11-07 |
Family
ID=69278310
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201910976702.6A Active CN110750645B (en) | 2019-10-15 | 2019-10-15 | Cross-domain false comment identification method based on countermeasure training |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN110750645B (en) |
Cited By (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111475649A (en) * | 2020-04-02 | 2020-07-31 | 中国人民解放军国防科技大学 | False news prediction method, system, device and medium based on deep learning |
CN111681091A (en) * | 2020-08-12 | 2020-09-18 | 腾讯科技(深圳)有限公司 | Financial risk prediction method and device based on time domain information and storage medium |
CN111984762A (en) * | 2020-08-05 | 2020-11-24 | 中国科学院重庆绿色智能技术研究院 | Text classification method sensitive to attack resistance |
CN112434471A (en) * | 2020-12-11 | 2021-03-02 | 恩亿科(北京)数据科技有限公司 | Method, system, electronic device and storage medium for improving model generalization capability |
CN112765319A (en) * | 2021-01-20 | 2021-05-07 | 中国电子信息产业集团有限公司第六研究所 | Text processing method and device, electronic equipment and storage medium |
CN113128199A (en) * | 2021-05-11 | 2021-07-16 | 济南大学 | Word vector generation method based on pre-training language model and multiple word information embedding |
CN113239926A (en) * | 2021-06-17 | 2021-08-10 | 北京邮电大学 | Multi-modal false information detection model based on countermeasures |
CN113269228A (en) * | 2021-04-20 | 2021-08-17 | 重庆邮电大学 | Method, device and system for training graph network classification model and electronic equipment |
CN117573814A (en) * | 2024-01-17 | 2024-02-20 | 中电科大数据研究院有限公司 | Public opinion situation assessment method, device and system and storage medium |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107025284A (en) * | 2017-04-06 | 2017-08-08 | 中南大学 | The recognition methods of network comment text emotion tendency and convolutional neural networks model |
-
2019
- 2019-10-15 CN CN201910976702.6A patent/CN110750645B/en active Active
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107025284A (en) * | 2017-04-06 | 2017-08-08 | 中南大学 | The recognition methods of network comment text emotion tendency and convolutional neural networks model |
Non-Patent Citations (1)
Title |
---|
张晓辉;于双元;王全新;徐保民;: "基于对抗训练的文本表示和分类算法" * |
Cited By (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111475649A (en) * | 2020-04-02 | 2020-07-31 | 中国人民解放军国防科技大学 | False news prediction method, system, device and medium based on deep learning |
CN111475649B (en) * | 2020-04-02 | 2023-04-07 | 中国人民解放军国防科技大学 | False news prediction method, system, device and medium based on deep learning |
CN111984762B (en) * | 2020-08-05 | 2022-12-13 | 中国科学院重庆绿色智能技术研究院 | Text classification method sensitive to attack resistance |
CN111984762A (en) * | 2020-08-05 | 2020-11-24 | 中国科学院重庆绿色智能技术研究院 | Text classification method sensitive to attack resistance |
CN111681091A (en) * | 2020-08-12 | 2020-09-18 | 腾讯科技(深圳)有限公司 | Financial risk prediction method and device based on time domain information and storage medium |
CN112434471A (en) * | 2020-12-11 | 2021-03-02 | 恩亿科(北京)数据科技有限公司 | Method, system, electronic device and storage medium for improving model generalization capability |
CN112765319A (en) * | 2021-01-20 | 2021-05-07 | 中国电子信息产业集团有限公司第六研究所 | Text processing method and device, electronic equipment and storage medium |
CN112765319B (en) * | 2021-01-20 | 2021-09-03 | 中国电子信息产业集团有限公司第六研究所 | Text processing method and device, electronic equipment and storage medium |
CN113269228A (en) * | 2021-04-20 | 2021-08-17 | 重庆邮电大学 | Method, device and system for training graph network classification model and electronic equipment |
CN113269228B (en) * | 2021-04-20 | 2022-06-10 | 重庆邮电大学 | Method, device and system for training graph network classification model and electronic equipment |
CN113128199B (en) * | 2021-05-11 | 2022-06-21 | 济南大学 | Word vector generation method based on pre-training language model and multiple word information embedding |
CN113128199A (en) * | 2021-05-11 | 2021-07-16 | 济南大学 | Word vector generation method based on pre-training language model and multiple word information embedding |
CN113239926B (en) * | 2021-06-17 | 2022-10-25 | 北京邮电大学 | Multi-modal false information detection model system based on countermeasure |
CN113239926A (en) * | 2021-06-17 | 2021-08-10 | 北京邮电大学 | Multi-modal false information detection model based on countermeasures |
CN117573814A (en) * | 2024-01-17 | 2024-02-20 | 中电科大数据研究院有限公司 | Public opinion situation assessment method, device and system and storage medium |
CN117573814B (en) * | 2024-01-17 | 2024-05-10 | 中电科大数据研究院有限公司 | Public opinion situation assessment method, device and system and storage medium |
Also Published As
Publication number | Publication date |
---|---|
CN110750645B (en) | 2023-11-07 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN110750645A (en) | Cross-domain false comment identification method based on countermeasure training | |
CN108363804B (en) | Local model weighted fusion Top-N movie recommendation method based on user clustering | |
Qu et al. | Question answering over freebase via attentive RNN with similarity matrix based CNN | |
CN107861951A (en) | Session subject identifying method in intelligent customer service | |
Gui et al. | Embedding learning with events in heterogeneous information networks | |
Wang et al. | Mulde: Multi-teacher knowledge distillation for low-dimensional knowledge graph embeddings | |
Tang et al. | Multi-label patent categorization with non-local attention-based graph convolutional network | |
CN108875809A (en) | The biomedical entity relationship classification method of joint attention mechanism and neural network | |
CN110807086A (en) | Text data labeling method and device, storage medium and electronic equipment | |
Gao et al. | CE-HEAT: an aspect-level sentiment classification approach with collaborative extraction hierarchical attention network | |
Mu et al. | Keyphrase extraction with span-based feature representations | |
Li et al. | Semi-supervised learning for text classification by layer partitioning | |
Durga et al. | Deep-Sentiment: An Effective Deep Sentiment Analysis Using a Decision-Based Recurrent Neural Network (D-RNN) | |
Chen et al. | Utilizing generative adversarial networks for recommendation based on ratings and reviews | |
Eshmawi et al. | Design of Automated Opinion Mining Model Using Optimized Fuzzy Neural Network. | |
CN108694176A (en) | Method, apparatus, electronic equipment and the readable storage medium storing program for executing of document sentiment analysis | |
Xu et al. | Neural topic modeling with deep mutual information estimation | |
Patidar et al. | Automatic conversational helpdesk solution using seq2seq and slot-filling models | |
Guohao et al. | Competency analysis in human resources using text classification based on deep neural network | |
Yeo et al. | Sentiment analysis on time-series data using weight priority method on deep learning | |
Cheng et al. | Knowledge-aware method for confusing charge prediction | |
CN116049349A (en) | Small sample intention recognition method based on multi-level attention and hierarchical category characteristics | |
Jaiswal et al. | Genetic approach based bug triage for sequencing the instance and features | |
Chan et al. | A self-weighting module to improve sentiment analysis | |
Zhang et al. | Hierarchical attention networks for grid text classification |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |