CN111813938A - Record question-answer classification method based on ERNIE and DPCNN - Google Patents
Record question-answer classification method based on ERNIE and DPCNN Download PDFInfo
- Publication number
- CN111813938A CN111813938A CN202010654746.XA CN202010654746A CN111813938A CN 111813938 A CN111813938 A CN 111813938A CN 202010654746 A CN202010654746 A CN 202010654746A CN 111813938 A CN111813938 A CN 111813938A
- Authority
- CN
- China
- Prior art keywords
- model
- ernie
- word
- dpcnn
- sentence
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/35—Clustering; Classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/332—Query formulation
- G06F16/3329—Natural language query formulation or dialogue systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/3331—Query processing
- G06F16/334—Query execution
- G06F16/3346—Query execution using probabilistic model
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Artificial Intelligence (AREA)
- Mathematical Physics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Probability & Statistics with Applications (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention relates to a record question-answer classification method based on ERNIE and DPCNN, which mainly comprises the steps of preprocessing a record text data set, inputting the processed data into an ERNIE model to train to obtain a word vector sequence, and then inputting the obtained word vector sequence into a DPCNN model to train.
Description
Technical Field
The invention belongs to the technical field of natural language processing, and discloses a method for classifying question-answer pairs in a record based on ERNIE and DPCNN.
Background
As time goes on, more and more records texts are accumulated, however, the records texts contain a large amount of case key information, how to efficiently obtain case information with important value from the data becomes a research hotspot, questions and answers, search, information extraction, crime analysis and the like in the records are all application fields of natural language processing, however, classification of the questions and answers in the records is the basis of the technologies, and thus, the requirement on the classification accuracy of the questions and answers in the records is very high.
With the rapid development of the deep learning field, the accuracy of the text classification technology is continuously improved, for example, garbage information classification and intention identification have good effects, but with the development of the internet, criminals have more and more diversified crime modes by using a network technology means, and correspondingly, higher accuracy is urgently needed for the classification technology of case information.
Most of the traditional text classification techniques adopt RNN or CNN models, but the models have great defects, such as that the RNN model is a recurrent neural network, namely, the result of the next layer depends on the output of the previous layer, and in common, the output needs to be performed word by word, which is obviously not a friendly parallel processing mode, meanwhile, in recent years, some researchers adopt the ultra-deep CNN to classify texts in a large-scale training data environment, which undoubtedly increases the training complexity, the BERT model published in 2018, although good results are obtained in the natural language field, the precision of the BERT model for Chinese classification is not friendly, and the priori knowledge in Chinese sentences is not considered only through context prediction.
Disclosure of Invention
The purpose of the invention is as follows: the accuracy of the classification of the question and answer of the record is further improved, so that the record is conveniently subjected to data processing such as information extraction and retrieval in a subsequent process.
In order to achieve the above purpose, the invention provides a record question-answer classification method based on ERNIE and DPCNN, which mainly comprises the following steps:
step 1, dividing labels of question and answer pairs in a record text, wherein the types of the division of different case records are different;
step 3.1, firstly, processing the data set into a data format required by the ERNIE model, and utilizing a split function to tjDividing the content into two parts, namely content and category;
step 3.2, utilizing tokenize in the model to carry out word segmentation processing on the content to obtain Where W denotes each sample data, i.e. content, W in the data setkRepresenting the kth word in a sentence;
step 3.3, splicing [ CLS ] at the beginning of the sample W to obtain W', and setting the fixed length of a sentence as seg _ size;
step 3.4, converting W ' into a vector W ' by using a model, if the length of W ' is less than seg _ size, filling 0, otherwise, truncating;
step 3.5, the processed data are stored, so that the processed data only need to be loaded when the model is debugged each time, and the efficiency of the model for processing the data is improved;
step 3.6, establishing an iterator, setting the number of samples in each group as batch, and then transmitting the processed data into an ERNIE model to obtain a word vector sequence;
step 4, carrying out operations such as convolution, pooling and the like on the sequence subjected to vector quantization by using a DPCNN model;
step 4.1, performing convolution operation twice on a word vector sequence obtained by training an ERNIE model by using a DPCNN model, activating by using a nonlinear activation function Relu after each convolution, and performing padding operation, namely boundary filling, in order to prevent boundary information from missing;
step 4.2, after the operation of 4.1, performing maximum pooling operation on the obtained data to obtain a feature vector X;
4.3, setting a judgment condition length, if the actual length of the sentence is greater than the length, performing convolution operation on the feature vector X for two times, performing Relu and padding operation on the convolved data each time to obtain a feature vector F, and performing feature fusion on the feature vectors X and F to obtain a feature vector X';
and 4.4, performing maximum pooling operation on the new feature vector X ' to obtain a feature vector X ', reducing the dimension of the X ' through a full connection layer to obtain a feature vector X ', wherein X ' is { X ', and dimension reduction is performed on the feature vector X ', and1,x2,x3,…,xNn represents the number of text categories;
step 4.5, inputting the feature vector X' into the softmax layer to obtain a predicted question-answer category probability value P, wherein P is { P ═ P { (P)1,p2,p3,…,pm,…,pN},pmThe probability value representing the mth category, Max (P), is the final classification of the question and answer;
step 5, training the ERNIE-DPCNN model by utilizing a training set and a verification set;
optimizing parameters in the model by adopting a gradient descent method, defining cross entropy as a loss function of the model, setting an imporv _ num parameter, and finishing the training of the model if the model effect is not improved when the model training exceeds the parameter value;
step 6, probability classification of the text of the notes, questions and answers;
inputting data to be predicted into a trained ERNIE-DPCNN model for processing, inputting a feature vector obtained through model training into a softmax layer, and using the maximum probability as the category of the predicted data as in the step 4.5.
Drawings
FIG. 1 is a schematic flow chart of a record question-answer classification method based on ERNIE and DPCNN of the present invention;
FIG. 2 is an internal structure diagram of the ERNIE-DPCNN model of the present invention.
Detailed Description
The following describes a specific implementation process of the present invention with reference to fig. 1 and 2, and the following contents are as follows:
(1) notes question-answer classification
As shown in fig. 1, firstly, the question and answer pairs in the record text are classified into categories, such as "personal situation", "crime passing", "physical condition", and the like, and the categories of the classification of different case records are different, such as the categories of "drug source", "drug sucking mode", and the like, recorded by the drug-sucking case record, and the categories of "stealing means", "stealing place", and the like, recorded by the stealing case record;
(2) data pre-processing
Since the original entry information may be a question or a question answer both above and below, the original entry text T is preprocessed, where T ═ T {1,T2,T3,…,Ti,…,Tlen(T)}, len (T) denotes the number of text entries, TiRepresenting the ith entry text, and grouping the questions and answers in the entry to obtain a data set T', T ═ T1,t2,t3,…,tj,…,tlen(T′)},tjRepresenting the jth stroke-recording question-answer pair, and len (T') representing the sample number of the question-answer pair;
(3) vectorization of data set T' based on ERNIE model
(3.1) processing the data set into the data format required by the ERNIE model, namely seg _ len, input _ ids, input _ mask and cable, wherein seg _ len represents the actual length of a sentence, input _ ids represents that a word in the sentence is represented by id in a word list in the ERNIE model, input _ mask represents that a word in the sentence is represented by 1, and label represents the id number of a label of the sentence in a corresponding classified word list. Then using split function to pair tjDividing the content into two parts, namely content and category;
(3.2) carrying out word segmentation processing by using token in the model to obtainWhere W denotes content, W per datasetkRepresenting the kth word in a sentence;
(3.3) splicing [ CLS ] at the beginning of a sample W to obtain W ', setting a fixed length seg _ size of a sentence, converting W' into a vector W 'by using an ERNIE model, namely input _ ids, filling 0 if the length of W' is less than seg _ size, and otherwise, truncating, wherein input _ mask indicates that a word in the sentence is represented by 1, and the rest positions are filled by 0;
(3.4) establishing an iterator, namely the number of each group of samples is batch, and inputting the processed data into an ERNIE model to train to obtain a word vector sequence;
(4) performing operations such as convolution, pooling and the like on word vector sequence based on DPCNN model
(4.1) performing convolution operation twice on the word vector sequence obtained by training the ERNIE model by using the DPCNN model, as shown in FIG. 2, wherein the size of a convolution kernel is 3, the number of the convolution kernels is 250, and the size of the word vector sequence after convolution is noutThe formula is as follows:
wherein n isinRepresenting the size of an original word vector sequence, p representing the size of padding, f representing the size of a convolution kernel, s being a step length, activating by using a nonlinear activation function Relu after each convolution, and in addition, in order to prevent boundary information from missing, padding operation, namely boundary filling, is needed;
(4.2) after the operation, performing maximum pooling operation, namely down-sampling operation on the obtained data, performing maximum pooling by using a pooling layer with the size of 3 and the step size of 2, namely generating a new internal representation of the document by taking the maximum value of 3 continuous internal vectors by the pool layer, and obtaining a feature vector X after the operation;
(4.3) setting length as a judgment condition, if the actual length of a sentence is greater than the length, performing convolution operation on the feature vector X twice, as shown in FIG. 2, performing Relu and padding operation on data convolved each time to obtain a feature vector F, and performing feature fusion on the feature vector X and the feature vector F to obtain a feature vector X';
(4.4) performing maximum pooling operation on the new feature vector X ' to obtain a feature vector X ', performing dimensionality reduction on the X ' through a full connection layer to obtain a dimensionality-reduced feature vector X ', wherein X ' (X)1,x2,x3,…,xNN stands for text category number, and the feature vector X' ″ is input to the softmax layer, which is formulated as follows:
obtaining a predicted question-answer class probability value P, P ═ P1,p2,p3,…,pm,…,pN},pmThe probability value representing the mth category, Max (P), is the final classification of the question and answer;
(5) ERNIE-DPCNN model training
Parameters in the model are optimized by adopting a gradient descent method, cross entropy is defined as a loss function loss of the model, and a multi-classification cross entropy formula is as follows:
H(D,Y)=-∑D(x)log Y(x) (3)
and D is a true value, Y is a predicted value, an inprov _ num parameter is set, and when the model training exceeds the parameter value, if the model effect is not improved, the model training is ended.
Claims (4)
1. A record question-answer classification method based on ERNIE and DPCNN is characterized by comprising the following steps:
step 1, classifying the notes, questions and answers;
step 2, preprocessing data;
step 3, vectorizing the data set T' based on the ERNIE model;
step 4, carrying out operations such as convolution, pooling and the like on the word vector sequence based on the DPCNN model;
and 5, training an ERNIE-DPCNN model.
2. The ERNIE and DPCNN-based bibliographic question-answer classification method according to claim 1, characterized by the steps of3, processing the data set into data formats required by the ERNIE model, namely seg _ len, input _ ids, input _ mask and cable, wherein seg _ len represents the actual length of a sentence, input _ ids represents that a word in the sentence is represented by id in a word list in the ERNIE model, input _ mask represents that a word in the sentence is represented by 1, and label represents the id number of a label of the sentence in a corresponding classified word list. Then using split function to pair tjDividing the content into two parts, namely content and category, wherein the specific content of the step is as follows:
performing word segmentation processing by using token in model to obtainWhere W denotes each sample data, i.e. content, W in the data setkRepresenting the kth word in a sentence;
splicing [ CLS ] at the beginning of a sample W to obtain W ', setting a fixed length seg _ size of a sentence, converting W' into a vector W 'by using a model, namely input _ ids, filling 0 if the length of W' is less than the seg _ size, otherwise, truncating, wherein input _ mask indicates that a word in the sentence is represented by 1, and filling the rest positions by 0;
and establishing an iterator, namely the number of samples in each group is batch, and then transmitting the processed data into an ERNIE model to train to obtain a word vector sequence.
3. The method for classifying the scripts and the questions and the answers based on the ERNIE and the DPCNN as claimed in claim 1, wherein in step 4, the DPCNN model is used to perform operations such as convolution and pooling on the word vector sequence obtained by training the ERNIE model, and the specific contents are as follows:
firstly, performing convolution operation twice on a word vector sequence, wherein the size of convolution kernels is 3, the number of the convolution kernels is 250, and the size of the word vector sequence after convolution is noutAfter each convolution, a nonlinear activation function Relu is used for activation, and in addition, in order to prevent boundary information from missing, padding operation, namely boundary filling, is required;
after the operation, performing maximum pooling operation, namely downsampling operation on the obtained data, performing maximum pooling by using a pooling layer with the size of 3 and the step length of 2, namely generating a new internal representation of the document by taking the maximum value of 3 continuous internal vectors by the pooling layer, and obtaining a feature vector X after the operation;
setting length as a judgment condition, if the actual length of a sentence is greater than length, performing convolution operation on a feature vector X twice, similarly performing Relu and padding operation on data after each convolution to obtain a feature vector F, performing feature fusion on the feature vectors X and F to obtain a feature vector X ', performing maximum pooling operation on a new feature vector X ' to obtain a feature vector X ', and performing dimension reduction on the X ' through a full connection layer to obtain a dimension-reduced feature vector X ', wherein X ' is { X ', and X ', where X ' is1,x2,x3,...,xNN represents the number of text categories, and the feature vector X' ″ is input into the softmax layer to obtain a predicted probability value of question-answer categories P, { P ═ P { (P)1,p2,p3,...,pm,...,pN},pmThe probability value representing the mth category, max (p), is the final classification of the question and answer.
4. The method as claimed in claim 1, wherein in step 5, parameters in the model are optimized by using a gradient descent method in the training of the ERNIE-DPCNN model, cross entropy is defined as a loss function loss of the model, an inprov _ num parameter is set, and when the number of times of model training exceeds the parameter value, if the model effect is not improved, the model training is ended.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010654746.XA CN111813938A (en) | 2020-07-09 | 2020-07-09 | Record question-answer classification method based on ERNIE and DPCNN |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010654746.XA CN111813938A (en) | 2020-07-09 | 2020-07-09 | Record question-answer classification method based on ERNIE and DPCNN |
Publications (1)
Publication Number | Publication Date |
---|---|
CN111813938A true CN111813938A (en) | 2020-10-23 |
Family
ID=72843254
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202010654746.XA Pending CN111813938A (en) | 2020-07-09 | 2020-07-09 | Record question-answer classification method based on ERNIE and DPCNN |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN111813938A (en) |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180240012A1 (en) * | 2017-02-17 | 2018-08-23 | Wipro Limited | Method and system for determining classification of text |
CN111160017A (en) * | 2019-12-12 | 2020-05-15 | 北京文思海辉金信软件有限公司 | Keyword extraction method, phonetics scoring method and phonetics recommendation method |
-
2020
- 2020-07-09 CN CN202010654746.XA patent/CN111813938A/en active Pending
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180240012A1 (en) * | 2017-02-17 | 2018-08-23 | Wipro Limited | Method and system for determining classification of text |
CN111160017A (en) * | 2019-12-12 | 2020-05-15 | 北京文思海辉金信软件有限公司 | Keyword extraction method, phonetics scoring method and phonetics recommendation method |
Non-Patent Citations (2)
Title |
---|
RIE JOHNSON等: "Deep Pyramid Convolutional Neural Networks for Text Categorization", 《PROCEEDINGS OF THE 55TH ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINDUISTICS》 * |
李舟军等: "面向自然语言处理的预训练技术研究综述", 《计算机科学》 * |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN113254599B (en) | Multi-label microblog text classification method based on semi-supervised learning | |
CN112115238B (en) | Question-answering method and system based on BERT and knowledge base | |
CN109766277B (en) | Software fault diagnosis method based on transfer learning and DNN | |
CN109766544B (en) | Document keyword extraction method and device based on LDA and word vector | |
CN109684626A (en) | Method for recognizing semantics, model, storage medium and device | |
CN111488739A (en) | Implicit discourse relation identification method based on multi-granularity generated image enhancement representation | |
CN110647612A (en) | Visual conversation generation method based on double-visual attention network | |
CN110750635B (en) | French recommendation method based on joint deep learning model | |
CN111966812B (en) | Automatic question answering method based on dynamic word vector and storage medium | |
CN110851594A (en) | Text classification method and device based on multi-channel deep learning model | |
CN112434164B (en) | Network public opinion analysis method and system taking topic discovery and emotion analysis into consideration | |
KR20200105057A (en) | Apparatus and method for extracting inquiry features for alalysis of inquery sentence | |
CN112036705A (en) | Quality inspection result data acquisition method, device and equipment | |
CN112100212A (en) | Case scenario extraction method based on machine learning and rule matching | |
CN113051887A (en) | Method, system and device for extracting announcement information elements | |
CN117236338B (en) | Named entity recognition model of dense entity text and training method thereof | |
CN111159405B (en) | Irony detection method based on background knowledge | |
CN113988079A (en) | Low-data-oriented dynamic enhanced multi-hop text reading recognition processing method | |
CN114742047A (en) | Text emotion recognition method based on maximum probability filling and multi-head attention mechanism | |
CN112632272A (en) | Microblog emotion classification method and system based on syntactic analysis | |
CN116821297A (en) | Stylized legal consultation question-answering method, system, storage medium and equipment | |
Nouhaila et al. | Arabic sentiment analysis based on 1-D convolutional neural network | |
Zhao et al. | Commented content classification with deep neural network based on attention mechanism | |
Hutagalung et al. | Hiragana Handwriting Recognition Using Deep Neural Network Search. | |
CN111813938A (en) | Record question-answer classification method based on ERNIE and DPCNN |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
WD01 | Invention patent application deemed withdrawn after publication | ||
WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20201023 |