CN114780677A - Chinese event extraction method based on feature fusion - Google Patents
Chinese event extraction method based on feature fusion Download PDFInfo
- Publication number
- CN114780677A CN114780677A CN202210354653.4A CN202210354653A CN114780677A CN 114780677 A CN114780677 A CN 114780677A CN 202210354653 A CN202210354653 A CN 202210354653A CN 114780677 A CN114780677 A CN 114780677A
- Authority
- CN
- China
- Prior art keywords
- network
- event
- input
- word
- layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000605 extraction Methods 0.000 title claims abstract description 116
- 230000004927 fusion Effects 0.000 title claims abstract description 34
- 238000012549 training Methods 0.000 claims abstract description 57
- 238000000034 method Methods 0.000 claims abstract description 37
- 238000012360 testing method Methods 0.000 claims abstract description 16
- 230000009193 crawling Effects 0.000 claims abstract description 5
- 238000013526 transfer learning Methods 0.000 claims abstract description 4
- 239000013598 vector Substances 0.000 claims description 58
- 238000013507 mapping Methods 0.000 claims description 20
- 239000011159 matrix material Substances 0.000 claims description 20
- 238000001514 detection method Methods 0.000 claims description 19
- 230000006870 function Effects 0.000 claims description 10
- 238000002372 labelling Methods 0.000 claims description 8
- 230000004913 activation Effects 0.000 claims description 6
- 230000007246 mechanism Effects 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000010606 normalization Methods 0.000 claims description 4
- 230000000737 periodic effect Effects 0.000 claims description 4
- 230000011218 segmentation Effects 0.000 claims description 4
- 230000003044 adaptive effect Effects 0.000 claims description 2
- 238000004422 calculation algorithm Methods 0.000 claims description 2
- 238000007500 overflow downdraw method Methods 0.000 abstract description 7
- 238000004458 analytical method Methods 0.000 abstract description 5
- 230000010365 information processing Effects 0.000 abstract description 3
- 238000012502 risk assessment Methods 0.000 abstract description 2
- 238000011160 research Methods 0.000 description 6
- 238000004880 explosion Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000010276 construction Methods 0.000 description 3
- 238000004088 simulation Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- BQCADISMDOOEFD-UHFFFAOYSA-N Silver Chemical compound [Ag] BQCADISMDOOEFD-UHFFFAOYSA-N 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 229910052709 silver Inorganic materials 0.000 description 1
- 239000004332 silver Substances 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/3331—Query processing
- G06F16/334—Query execution
- G06F16/3344—Query execution using natural language analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/35—Clustering; Classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/30—Semantic analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Databases & Information Systems (AREA)
- Machine Translation (AREA)
Abstract
The invention discloses a Chinese event extraction method based on feature fusion, which comprises the following steps: 1) constructing a Chinese event extraction network BERT-FF; 2) constructing a training data set; 3) downloading a pre-training parameter file and optimizing by using a contrast learning method; 4) loading the optimized pre-training parameter file in a word level feature extraction network by using a transfer learning method; 5) training by using a training data set to obtain a trained Chinese event extraction network BERT-FF; 6) and crawling a text describing the event from the open network, inputting the text serving as a test data set into a trained Chinese event extraction network BERT-FF for event extraction, and outputting structured event information, namely an event extraction result. The method enhances the semantic expression capability of the model through a feature fusion method, improves the performance of extracting Chinese events, and can be used in the fields of news public opinion analysis, information processing, financial risk assessment and the like.
Description
Technical Field
The invention belongs to the technical field of artificial intelligence, relates to natural language processing, and particularly relates to a Chinese event extraction method based on feature fusion, which can be used for public opinion analysis and information processing.
Background
The main objective of the event extraction task is to extract structured event information from unstructured text, so as to reduce the difficulty of acquiring and processing information for users. This typically includes determining the type of event (event type) contained in the text, and identifying an event argument (argument) and determining the role (role) it plays in the event. The technology can be applied to the fields of news public opinion analysis, information processing, financial risk assessment and the like. The research of the event extraction system has been developed over a decade, and many experts and scholars have obtained a great deal of effective research results, but most of the research is directed at the english corpus, the research in the aspect of chinese event extraction is relatively immature, not only is a high-quality data set lacking, but also the model performance is poorer than that of english event extraction.
The traditional event extraction method converts event detection and event argument extraction into classification problems by acquiring characteristic information of event trigger words and vocabulary, syntax, semantics and the like of event arguments, and the core of the traditional event extraction method is the extraction of characteristics and the construction of a classifier. Although various characteristics such as vocabulary, syntax, semantics and the like can be used as input of the classifier, the construction of higher-level linguistic characteristics such as part-of-speech tagging, syntactic dependency analysis and the like requires professional knowledge of linguistics and related fields, which limits the adaptability and universality of the classifier.
With the development of neural networks, deep learning has made remarkable achievements in both event detection performance and event argument extraction performance. In a neural network, input layers can adopt simple representation of original data as input, and each layer can convert input of a shallow layer into more abstract and complex characteristics through learning and then input the more abstract and complex characteristics into a deeper layer until output characteristics of a deepest layer are used for classification. Compared with the traditional method, the deep learning can greatly reduce the difficulty of the characteristic engineering, and the detection and classification precision is superior to that of the traditional method. In 2018, in 10 months, *** AI research institute issued a pre-trained language model BERT based on a Transformer encoder structure, and a research on applying the pre-trained language model to an event extraction task was also activated. Before that, the mainstream event detection method is to detect an event trigger word from a text and judge the event type according to the event trigger word; with the development of an event extraction model based on a pre-training network, due to the strong semantic feature representation capability of the event extraction model, an event detection method based on the full text becomes the mainstream.
Disclosure of Invention
In order to overcome the above drawbacks of the prior art, the present invention provides a method for extracting chinese events based on feature fusion, so as to improve the accuracy and recall rate of chinese event extraction.
In order to achieve the purpose, the invention adopts the technical scheme that:
the Chinese event extraction method based on feature fusion comprises the following steps:
step 1, constructing a Chinese event extraction network BERT-FF
The Chinese event extraction network BERT-FF comprises a word level feature extraction network, a feature fusion network and a back-end classification network;
the word level feature extraction network is based on a BERT pre-training language model and is used for extracting word level features of an input text; the word level feature extraction network is used for extracting word level features of an input text; the feature fusion network fuses the extracted word-level features and the word-level features through an attention mechanism so as to enhance the semantic representation capability of the model and obtain fusion feature vectors; the back-end classification network is used for respectively inputting the fusion feature vectors into the event detection back-end network and the event argument extraction back-end network to obtain a final event extraction result;
step 2, constructing a training data set
The training data set consists of texts which are crawled from an open network and describe events and annotation files which are in one-to-one correspondence with the texts;
step 3, training the Chinese event extraction network BERT-FF
And 4, crawling a text describing the event from the open network, inputting the text as a test data set into a trained Chinese event extraction network BERT-FF for event extraction, outputting structured event information, obtaining an event extraction result, and calculating the event extraction accuracy and recall rate.
Compared with the prior art, the invention has the beneficial effects that:
firstly, the invention uses a transfer learning method to load the pre-training parameter file of the BERT into the word level feature extraction network constructed in the step (1a), so that the word level feature extraction network can rapidly learn the word level semantic features of the text, the integral convergence speed of the model is accelerated, and the time complexity of event extraction is reduced.
Secondly, the invention optimizes the pre-training parameters of the BERT by using a contrast learning method, relieves the anisotropy problem of the BERT semantic feature vector space, and then uses the optimized model to replace the original model as a feature extraction network, thereby improving the event detection performance of the model.
Thirdly, the invention uses a feature fusion method based on an attention mechanism to fuse the extracted word-level feature information with the word-level feature information aiming at the characteristics of Chinese text data, thereby enhancing the context semantic representation capability of the model and improving the accuracy rate and the recall rate of the model in event detection and event argument extraction.
Drawings
FIG. 1 is a schematic structural diagram of a Chinese event extraction network BERT-FF.
FIG. 2 is a schematic diagram of the structure of the BERT pre-training model.
FIG. 3 is a schematic diagram of a multi-head attention mechanism.
Fig. 4 is an exemplary diagram of event extraction.
Detailed Description
The embodiments of the present invention will be described in detail below with reference to the drawings and examples.
The invention relates to a Chinese event extraction method based on feature fusion, which comprises the following implementation steps.
Step 1, referring to fig. 1, a Chinese event extraction network BERT-FF is constructed.
The Chinese event extraction network BERT-FF comprises: the system comprises a word level feature extraction network, a feature fusion network and a back-end classification network.
The word level feature extraction network is based on a BERT pre-training language model and is used for extracting word level features of an input text; the word level feature extraction network is used for extracting word level features of the input text; the feature fusion network fuses the extracted word-level features and the word-level features through an attention mechanism so as to enhance the semantic representation capability of the model and obtain fusion feature vectors; the back-end classification network is used for inputting the fusion feature vectors into the event detection back-end network and the event argument extraction back-end network respectively to obtain a final event extraction result.
The key steps are as follows:
1.1) building a character level feature extraction network of a Chinese event extraction network BERT-FF:
referring to fig. 2, the structural relationship is: input layer → word embedding layer → position coding → N concatenated semantic coders → output layer.
The specific parameters and implementation modes of the modules are as follows:
the input of the input layer is a token sequence obtained after text word segmentation, in order to avoid the problem of OOM (out of memory) of display memory, the maximum length of the token sequence during training is set to be 128, and if the maximum length exceeds 128, truncation is carried out. In order to Batch the input text data, the token sequence length in each Batch (Batch) must be kept equal, and if not, the token sequence length Padding is the longest in the Batch. Let token sequence length be sequence length1。
The embedding dimension embeded size of the word embedding layer is 768, which meets the input dimension requirement of BERT, that is, the word vector of each token is a column vector of 768 dimensions.
The position coding method adopts Sinusoid position coding, and is shown as formula (1) and formula (2):
wherein pos refers to the position of the current token in the sequence, and the value range is [0, sequence length ]1) I refers to the dimension number of the word vector, and the value range is [0, embedded size/2) ], i.e., the dimension of the position code is consistent with the dimension of the word vector, and d refers to the dimension of the word vector. The formula (1) and formula (2)) is a set of sine and cosine formulas, which are respectively calculation formulas of position coding when the dimension serial number i of the word vector is even number and odd number, thereby generating different periodic changes. As can be seen from the above equation, as i increases, the frequency of the periodic variation becomes lower and lower, eventually yielding a unique texture containing location information at each different location. The method is used as a position code to be added into a word vector, so that a model can learn the dependency relationship between positions and the time sequence characteristic of a natural language.
The main body of the word-level feature extraction network is N cascaded semantic encoders, wherein N is 12. Each semantic encoder consists of two parts: a Multi-Head Self-Attention (Multi-Head Self-Attention) module containing a residual network and a Forward propagation (Feed Forward) module containing a residual network.
The multi-head self-attention module comprising the residual error network is formed by splicing a multi-head attention module and a residual error module, and the multi-head attention module refers to fig. 3. Input of multi-head attention module1And input2The three inputs of each Attention Head are respectively a Query (Query) vector sequence Q, a Key (Key) vector sequence K and a Value (Value) vector sequence V, and the number h of the Attention Head is 12. Q is input1Obtained through the full connection layer, K and V are from input2Obtained by full connection layer. In practical use, the dimension d of the input word vector is usually a large number, and since multiple groups of attention weights need to be calculated, in order to avoid the excessive number of network parameters, the mapping matrix is generally selectedDimension reduction is performed, that is, in each extension head, the original d-dimensional word vector is projected to d/h dimension, so that the dimension of each fully-connected layer mapping matrix is 768 × 64. Each Attention Head generates a set of Q, K and V, each set of Q, K and V is input to a scaled dot product Attention module to compute a context feature vector. Finally, splicing the obtained multiple groups of context feature vectors, inputting the multiple groups of context feature vectors into a full connection layer to obtain the output of the multi-head attention module, wherein the dimensionality of a full connection layer mapping matrix is 768 multiplied by 768. The residual module adds the input and output of its pre-module and performs layer normalization.
The forward propagation module containing the residual error network is formed by splicing two full connection layers, a GeLU activation function and a residual error module, wherein the GeLU activation function is positioned between the two full connection layers. The first fully-connected layer mapping matrix has dimensions of 768 × 2048, and the second fully-connected layer mapping matrix has dimensions of 2048 × 768. The residual module adds the input and output of its pre-module and performs layer normalization. The GeLU activation function is expressed as:
where x represents the output of the pre-module and erf (×) represents the gaussian error computation function.
1.2) building a word level feature extraction network of a Chinese event extraction network BERT-FF:
the structural relationship is as follows: input layer → word embedding layer → position coding → fully connected layer → output layer.
The specific parameters and implementation of each module are as follows:
the input of the input layer is a token sequence obtained after the word segmentation of the text,in order to avoid the video on-board OOM problem, the maximum length of the token sequence during training is set to be 128, and if the maximum length exceeds 128, truncation is carried out. In order to Batch the input text data, the token sequence length in each Batch (Batch) must be kept equal, and if not, the token sequence length Padding is the longest in the Batch. Let token sequence length be sequence length2。
The embedding dimension embeddedsize of the word embedding layer is 128, i.e. the word vector of each token is a column vector of 128 dimensions.
The position coding method adopts Sinussoid position coding.
The dimensionality of the fully-connected layer mapping matrix is 128 multiplied by 768, so that the dimensionality of the word-level feature vector is equal to the dimensionality of the word-level feature vector, and subsequent calculation is facilitated.
1.3) constructing a feature fusion network of a Chinese event extraction network BERT-FF:
the structural relationship is as follows: input layer → multi-headed attention module → output layer.
The specific parameters and implementation of each module are as follows:
the input of the input layer is composed of two parts, namely a word-level feature vector input1And word-level feature vector input2,input1Has a dimension of sequence length1×768,input2Has a dimension of sequence length2×768。
The three inputs of each Attention Head of the multi-Head Attention module are respectively a query vector sequence Q, a key vector sequence K and a value vector sequence V, and through experimental parameter adjustment, the model performance is optimal when the number of the Attention heads is set to be 24. Q is input1Obtained through the full connection layer, K and V are input2And obtaining the mapping matrix through the full connection layer, wherein the dimensionality of each full connection layer mapping matrix is 768 multiplied by 32. Each Attention Head generates a set of Q, K and V, each set of Q, K and V is input to a scaled dot product Attention module to compute a context feature vector. Finally, splicing the obtained multiple groups of context feature vectors, inputting the multiple groups of context feature vectors into a full connection layer to obtain the output of the multi-head attention module, wherein the dimensionality of a full connection layer mapping matrix is 768 multiplied by 768.
1.4) building a rear-end classification network of a Chinese event extraction network BERT-FF:
the back-end classification network of the Chinese event extraction network BERT-FF is divided into two parts, namely an event detection back-end network and an event argument extraction back-end network.
The structural relationship of the event detection back-end network is as follows in sequence: input layer → fully connected layer → multi-label classifier → output layer.
The specific parameters and implementation modes of the modules are as follows:
the input of the input layer is a feature vector of a [ CLS ] label in the fused feature vector, and the dimension is 1 × 768.
The fully connected layer mapping matrix has dimensions of 768 × n _ events, which is the total number of event types.
The multi-label classifier consists of n _ events Sigmoid functions, and the final output is the probability distribution of the event types contained in the current input text. And if the probability is greater than 0.5, the corresponding event type is considered to be contained in the text, otherwise, the corresponding event type is considered not to be contained.
The structure relationship of the event argument extraction back-end network is as follows: input layer → fully connected layer → conditional random field → output layer.
The specific parameters and implementation of each module are as follows:
the input of the input layer is a fusion feature vector with a dimension of sequence length1×768。
The dimensionality of the fully-connected layer mapping matrix is 768 × n _ labels, the n _ labels are the total number of the labeled labels of the event argument sequences, and the labeling method is a BIO labeling method.
The conditional random field is a linear chain element random field, and a sequence labeling result of event arguments is finally output. Conditional Random Field (CRF) is a discriminative probabilistic model, which is a probabilistic undirected graph model of a Random variable Y given the Random variable X. A special conditional random field defined on the Linear Chain, called Linear Chain component random field (Linear Chain CRF), is defined as:
let X ═ X1,X2,…,Xn),Y=(Y1,Y2,…,Yn) All random variable sequences are represented by linear chains, and if the conditional probability distribution P (Y | X) of Y satisfies Markov property under the condition of given X
P(Yi|X,Y1,...,Yi-1,Yi+1,…,Yn)=P(Yi|X,Yi-1,Yi+1),i=1,2,...,n#(4)
Then P (Y | X) is called the random field of the linear chain element.
And 2, constructing a training data set. The training data set consists of texts which are crawled from an open network and are used for describing events and annotation files which are in one-to-one correspondence with the texts.
In the embodiment of the invention, the construction process is as follows:
2.1) crawling at least 5000 texts describing the events from the open network.
2.2) manually labeling the event type, the event argument and the argument role contained in each text, and generating annotation files corresponding to the crawled texts one by one. The event type refers to an event type contained in the text; event arguments refer to elements in the text that are related to an event, typically named entities; argument roles refer to the role an event argument plays in an event.
In the news text "also security officer 25 days, a landmine explosion event occurred in the port city in south west of the country on the day of mooha, resulting in the death of at least 4 civilians. For example, according to a predefined event pattern, the event extraction system should recognize that the text contains two events of "disaster/accident-explosion" and "life-death", and extract that "the time" when the disaster/accident-explosion "occurs is" the day "," the location "is" the city mooha at the port in the south west of the country ", the number of the resulting" death "is" at least 4 ", and" the time "when the life-death" occurs is "the day", "the location" is "the city mooha at the port in the south west of the country", and "the dead" is "the civilian".
In the above examples, "disaster/accident-explosion" and "life-death" are event types; "the current day", "the port city of southwest of the country", "at least 4" and "civilian" are the event arguments; "time", "place", "number of dead", and "dead" are the corresponding argument roles.
2.3) composing the crawled text and the annotation files into a training data set.
And 3, training the event extraction network BERT-FF.
3.1) configuring an event extraction network BERT-FF operating environment, comprising different software such as CUDA11.0, cuDNN 8.0.4, Python 3.7, PyTorch 1.7.0 and the like.
3.2) downloading the pre-training parameter file (Chinese _ BERT _ wwm. bin) of the pre-training language model BERT to a local hard disk.
And 3.3) optimizing the pre-training parameters of the BERT by using a contrast learning method to obtain an optimized pre-training parameter file.
And 3.4) loading the optimized pre-training parameter file in the word level feature extraction network by using a transfer learning method to obtain the loaded word level feature extraction network. The method is characterized in that model parameters with strong semantic representation capability are obtained in related pre-training tasks and are loaded into a word level feature extraction network, so that the convergence speed of model training is increased, the training time is saved, and the risks of under-fitting and over-fitting are reduced.
3.5) training the training data set by using the BERT-FF of the loaded word level feature extraction network to obtain a trained Chinese event extraction network BERT-FF, wherein the training is realized as follows:
the iteration times (Epoch) of the network training are set to be 50 rounds, so that the network weight is sufficiently iterated and converged to the optimal value. And testing on the test sample after each round of training is finished, and storing the best test result and the network weight by taking the F1 score extracted by the event argument as a standard. The Batch Size (Batch Size) during training is set to 24, and the computational resources of the GPU are fully utilized. The network optimizer selects an adaptive momentum estimation (Adam) algorithm. Learning Rate (Learning Rate) the hierarchical Learning Rate is used: the learning rate of the word-level feature extraction network was set to 0.00002, the learning rate of the conditional random field was set to 0.002, and the learning rate of the other portions of the model was set to 0.0002. The word-level feature extraction network is based on a pre-training language model, and can be converged only by fine adjustment, so that the learning rate is set to be small, and other parts of the model need to be fully trained from the beginning. The conditional random field based on the probabilistic graphical model is difficult to converge, and the learning rate is set to be larger.
And 4, crawling a text describing the event from the open network, inputting the text serving as a test data set into a trained Chinese event extraction network BERT-FF for event extraction, outputting structured event information, namely an event extraction result, and calculating the event extraction accuracy and recall rate. Illustratively, to ensure effectiveness, the crawled text is at least 500 pieces.
To enter the news text "also security officer 25 days, a landmine explosion event occurred in the city in port in south west of the country on the day, resulting in the death of at least 4 civilians. "for example, the final event extraction result can be organized in a structured form of key-value pairs, see FIG. 4. This structured form may be conveniently stored in a json file or database for the user to obtain event information or a visual presentation.
The effect of the present invention will be further described with reference to simulation experiments.
1. Conditions of the experiment
The hardware test platform of the simulation experiment of the invention is as follows: CPU is Intel (R) Xeon (R) Silver 4310, the main frequency is 2.1Ghz, the memory is 32GB, and the GPU is a single NVIDIA GeForce RTX 3080 Ti; the software platform is as follows: ubuntu 18.04 system.
2. Analysis of Experimental Contents and results
The simulation experiment of the invention adopts the method of the invention to train the constructed Chinese event extraction network BERT-FF on the constructed event extraction training data set. And performing event extraction on the input test text by using the trained Chinese event extraction network BERT-FF, outputting structured event information to obtain an event extraction result, and calculating the accuracy and the recall rate of the event extraction.
Table 1 shows the performance of event detection on a test set after training a constructed data set using the method of the present invention.
TABLE 1 comparison of event detection Performance for different methods
Model | Precision | Recall | F1 |
LSTM-CRF | 0.8315 | 0.7266 | 0.7755 |
BERT(baseline) | 0.8863 | 0.9270 | 0.9062 |
BERT-FF | 0.9356 | 0.9203 | 0.9279 |
Table 1 shows the performance of the method of the invention in detecting events on the test set, while the other two methods were used as control groups. LSTM-CRF represents a classical event extraction method based on LSTM-CRF sequence labeling, BERT represents an event extraction method without using the feature fusion method in the invention, and BERT-FF represents the method in the invention. Precision represents Precision, Recall represents Recall, and F1 represents F1 score.
In contrast to BERT, it can be observed that BERT-FF, after using the feature fusion method of the present invention, is 5.56% more accurate than BERT in event detection, while there is only less than 1% reduction in recall. Finally, the F1 score is calculated, and BERT-FF is improved by 2.39% compared with BERT. In comprehensive comparison, the event detection performance of BERT-FF is improved considerably compared with BERT.
Table 2 shows the performance of event argument extraction on the test set after training the constructed data set by the method of the present invention.
TABLE 2 comparison of event argument extraction Performance for different methods
Model | Precision | Recall | F1 |
LSTM-CRF | 0.6826 | 0.4989 | 0.5765 |
BERT(baseline) | 0.7446 | 0.7465 | 0.7456 |
BERT-FF | 0.7570 | 0.7801 | 0.7684 |
Table 2 shows the performance of the method of the present invention in extracting event arguments on the test set, while the other two methods were used as control groups to perform the experiment.
Compared with BERT, the characteristic fusion method provided by the invention has the advantages that the BERT-FF is improved in each index, the accuracy is improved by 1.67%, the recall rate is improved by 4.50%, and the final calculation F1 score is improved by 3.04%. Compared with BERT, the performance of the event argument extraction of BERT-FF is greatly improved compared with BERT.
In conclusion, after the feature fusion method is added, the event detection performance and the event argument extraction performance of the model are improved, so that the fused word-level features and the word-level features really play a certain role in optimizing the semantic representation capability of the model, and the effectiveness of the feature fusion method provided by the chapter on the event extraction task is proved.
The above is only one specific example of the present invention for the convenience of those skilled in the art to understand the present invention, but the present invention is not limited to the scope of the specific example, and it is obvious to those skilled in the art that various changes are possible as long as they are within the spirit and scope of the present invention defined and determined by the appended claims, and all the inventions utilizing the inventive concept are protected by the present invention.
Claims (10)
1. The Chinese event extraction method based on feature fusion is characterized by comprising the following steps of:
step 1, constructing a Chinese event extraction network BERT-FF
The Chinese event extraction network BERT-FF comprises a word level feature extraction network, a feature fusion network and a back-end classification network;
the word level feature extraction network is based on a BERT pre-training language model and is used for extracting word level features of an input text; the word level feature extraction network is used for extracting word level features of an input text; the feature fusion network fuses the extracted word-level features and the word-level features through an attention mechanism so as to enhance the semantic representation capability of the model and obtain fusion feature vectors; the back-end classification network is used for respectively inputting the fusion feature vector into an event detection back-end network and an event argument extraction back-end network to obtain a final event extraction result;
step 2, constructing a training data set
The training data set consists of texts which are crawled from an open network and describe events and annotation files which are in one-to-one correspondence with the texts;
step 3, training the Chinese event extraction network BERT-FF
And 4, crawling a text describing the event from the open network, inputting the text as a test data set into a trained Chinese event extraction network BERT-FF for event extraction, outputting structured event information, obtaining an event extraction result, and calculating the event extraction accuracy and recall rate.
2. The method for extracting chinese events based on feature fusion according to claim 1, wherein the word-level feature extraction network has the following structural relationships in sequence: input layer → word embedding layer → position coding → N cascaded semantic coders → output layer;
the input of the input layer is a token sequence obtained after text word segmentation, the maximum length of the token sequence during training is set to be 128, if the maximum length exceeds 128, truncation is performed, the length of the token sequence in each batch is kept equal, and if the maximum length is not equal, the length of the token sequence is sequence length according to the longest token sequence length in each batch1;
The embedding dimension embeddedsize of the word embedding layer is 768, that is, the word vector of each token is a column vector of 768 dimensions;
the position coding method adopts Sinusoid position coding, and is shown as a formula (1) and a formula (2):
wherein pos refers to the position of the current token in the sequence, and the numeric area is [0, sequence length ]1) I refers to the dimension serial number of the word vector, the value range is [0, embedded size/2), namely the dimension of the position code is consistent with the dimension of the word vector, and d refers to the dimension of the word vector; the formula (1) and the formula (2) are respectively calculation formulas of position coding when the word vector dimension serial number i is even number and odd number, so that different periodic changes are generated; with the increase of i, the frequency of periodic variation is lower and lower, and finally, a unique texture containing position information is generated at each different position and is used as a position code to be added into a word vector, so that a model can learn the dependency relationship between the positions and the time sequence characteristic of a natural language;
the N cascaded semantic encoders are main bodies of a word-level feature extraction network, and each semantic encoder consists of two parts: a multi-headed self-attention module containing a residual network and a forward propagation module containing a residual network.
3. The feature fusion-based Chinese event extraction method of claim 2, wherein the multi-head self-attention module comprising the residual error network is formed by splicing a multi-head attention module and a residual error module, and an input of the multi-head attention module1And input2All the word vectors are binding position codes, three inputs of each Attention Head are respectively a query vector sequence Q, a key vector sequence K and a value vector sequence V, and the number of the Attention heads is 12; q is input1Obtained through the full connection layer, K and V are input2The method comprises the steps that all-connection layers are obtained, the dimensionality of each all-connection layer mapping matrix is 768 x 64, each Attention Head generates a group of Q, K and V, each group of Q, K and V inputs a scaling dot product Attention module to be calculated to obtain context eigenvectors, finally, the obtained multiple groups of context eigenvectors are spliced, and then the multiple groups of context eigenvectors are input into all-connection layers to obtain multi-Head AttentionThe dimensionality of a fully-connected layer mapping matrix is 768 x 768 for the output of the module, and the input and the output of a front module are added by a residual module and subjected to layer normalization;
the forward propagation module comprising the residual error network is formed by splicing two full connection layers, a GeLU activation function and a residual error module, wherein the GeLU activation function is positioned between the two full connection layers, the dimensionality of a mapping matrix of the first full connection layer is 768 multiplied by 2048, the dimensionality of a mapping matrix of the second full connection layer is 2048 multiplied by 768, and the residual error module adds the input and the output of a front-end module and performs layer normalization; the GeLU activation function is expressed as:
where x represents the output of the pre-stage block and erf (—) represents the gaussian error computation function.
4. The method for extracting chinese events based on feature fusion as claimed in claim 2, wherein the word-level feature extraction network has the following structural relationships: input layer → word embedding layer → position coding → full-link layer → output layer;
the input of the input layer is a token sequence obtained after text word segmentation, the maximum length of the token sequence during training is set to be 128, if the maximum length exceeds 128, truncation is carried out, the length of the token sequence in each batch is kept equal, and if the maximum length is not equal, the length of the token sequence is sequence length according to the longest length Padding of the token sequence in each batch2;
The embedding dimension embeddedsize of the word embedding layer is 128, that is, the word vector of each token is a 128-dimensional column vector;
the position coding method adopts Sinusoid position coding.
The dimension of the fully-connected layer mapping matrix is 128 × 768.
5. The method for extracting Chinese events based on feature fusion as claimed in claim 4, wherein the feature fusion network has the following structural relationships: input layer → multi-headed attention module → output layer;
the input of the input layer is composed of two parts, namely a word-level feature vector input1And word-level feature vector input2,input1Has a dimension of sequence length1×768,input2Has a dimension of sequence length2×768;
The three inputs of each Attention Head of the multi-Head Attention module are respectively a query vector sequence Q, a key vector sequence K and a value vector sequence V, the number of the Attention heads is 24, and Q is input by input1Obtained through a full connection layer, K and V are input2The method comprises the steps that all-connection layers are obtained, the dimensionality of each all-connection layer mapping matrix is 768 x 32, each Attention Head generates a group of Q, K and V, each group of Q, K and V is input into a scaling dot product Attention module to be calculated to obtain context eigenvectors, finally, the obtained multiple groups of context eigenvectors are spliced, then the obtained multiple groups of context eigenvectors are input into all-connection layers to obtain the output of a multi-Head Attention module, and the dimensionality of the all-connection layer mapping matrix is 768 x 768.
6. The feature fusion-based Chinese event extraction method according to claim 5, wherein the back-end classification network is divided into two parts, namely an event detection back-end network and an event argument extraction back-end network;
the structural relationship of the event detection back-end network is as follows: input layer → fully connected layer → multi-label classifier → output layer;
the input of the input layer is a feature vector of a [ CLS ] label in the fusion feature vector, and the dimensionality is 1 multiplied by 768;
the dimensionality of the mapping matrix of the full connection layer is 768 multiplied by n _ events, and the n _ events is the total number of the event types;
the multi-label classifier consists of n _ events Sigmoid functions, the final output is probability distribution of event types contained in the current input text, if the probability is greater than 0.5, the text is considered to contain the corresponding event types, otherwise, the text is considered not to contain the event types;
the structural relationship of the event argument extraction back-end network is as follows in sequence: input layer → fully connected layer → conditional random field → output layer;
the input of the input layer is a fusion feature vector with a dimension of sequence length1×768;
The dimensionality of the mapping matrix of the full connection layer is 768 multiplied by n _ labels, the n _ labels are the total number of the event argument sequences labeled labels, and the labeling method is a BIO labeling method;
the conditional random field is a linear chain element random field, and a sequence labeling result of event arguments is finally output.
7. The method for extracting Chinese events based on feature fusion according to claim 1, wherein in the step 2, at least 5000 texts describing the events are crawled from the open network, the event types, the event arguments and the argument roles contained in each text are manually labeled, and annotation files corresponding to the crawled texts one to one are generated.
8. The method for extracting chinese events based on feature fusion according to claim 1, wherein the step 3 comprises:
step 3a, configuring an event extraction network BERT-FF environment
Step 3b, downloading a pre-training parameter file (Chinese _ BERT _ wwm. bin) of a pre-training language model BERT;
step 3c, optimizing the pre-training parameters of the BERT by using a contrast learning method to obtain an optimized pre-training parameter file;
step 3d, loading the optimized pre-training parameter file in the word level feature extraction network by using a transfer learning method to obtain a loaded word level feature extraction network;
and 3e, extracting the BERT-FF of the network by using the loaded character level features, and training the training data set to obtain a trained Chinese event extraction network BERT-FF.
9. The feature fusion-based Chinese event extraction method of claim 8, wherein in step 3d, model parameters having strong semantic representation capability are obtained in a relevant pre-training task and loaded into a word-level feature extraction network, so as to accelerate convergence speed during model training, save training time, and reduce under-fitting and over-fitting risks.
10. The method for extracting chinese events based on feature fusion of claim 8, wherein in step 3e, the number of iterations of network training is set to 50 rounds, testing is performed on a test sample after each round of training is finished, the best test result and network weight are stored with F1 score extracted from event arguments as the standard, the batch size during training is set to 24, the network optimizer selects the adaptive momentum estimation algorithm, and the learning rate is a layered learning rate: the learning rate of the word-level feature extraction network was set to 0.00002, the learning rate of the conditional random field was set to 0.002, and the learning rate of the other portions of the model was set to 0.0002.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210354653.4A CN114780677B (en) | 2022-04-06 | 2022-04-06 | Chinese event extraction method based on feature fusion |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210354653.4A CN114780677B (en) | 2022-04-06 | 2022-04-06 | Chinese event extraction method based on feature fusion |
Publications (2)
Publication Number | Publication Date |
---|---|
CN114780677A true CN114780677A (en) | 2022-07-22 |
CN114780677B CN114780677B (en) | 2024-07-16 |
Family
ID=82427172
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202210354653.4A Active CN114780677B (en) | 2022-04-06 | 2022-04-06 | Chinese event extraction method based on feature fusion |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN114780677B (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116861901A (en) * | 2023-07-04 | 2023-10-10 | 广东外语外贸大学 | Chinese event detection method and system based on multitask learning and electronic equipment |
CN117236338A (en) * | 2023-08-29 | 2023-12-15 | 北京工商大学 | Named entity recognition model of dense entity text and training method thereof |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111310458A (en) * | 2020-03-20 | 2020-06-19 | 广东工业大学 | Subjective question automatic scoring method based on multi-feature fusion |
CN111897908A (en) * | 2020-05-12 | 2020-11-06 | 中国科学院计算技术研究所 | Event extraction method and system fusing dependency information and pre-training language model |
CN111967268A (en) * | 2020-06-30 | 2020-11-20 | 北京百度网讯科技有限公司 | Method and device for extracting events in text, electronic equipment and storage medium |
CN112836046A (en) * | 2021-01-13 | 2021-05-25 | 哈尔滨工程大学 | Four-risk one-gold-field policy and regulation text entity identification method |
-
2022
- 2022-04-06 CN CN202210354653.4A patent/CN114780677B/en active Active
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111310458A (en) * | 2020-03-20 | 2020-06-19 | 广东工业大学 | Subjective question automatic scoring method based on multi-feature fusion |
CN111897908A (en) * | 2020-05-12 | 2020-11-06 | 中国科学院计算技术研究所 | Event extraction method and system fusing dependency information and pre-training language model |
CN111967268A (en) * | 2020-06-30 | 2020-11-20 | 北京百度网讯科技有限公司 | Method and device for extracting events in text, electronic equipment and storage medium |
CN112836046A (en) * | 2021-01-13 | 2021-05-25 | 哈尔滨工程大学 | Four-risk one-gold-field policy and regulation text entity identification method |
Non-Patent Citations (3)
Title |
---|
CAN TIAN等: "A Chinese Event Relation Extraction Model Based on BERT", 《2019 2ND INTERNATIONAL CONFERENCE ON ARTIFICIAL INTELLIGENCE AND BIG DATA (ICAIBD)》, 16 September 2019 (2019-09-16), pages 271 - 276 * |
柯欣飞等: "基于BERT的端到端中文事件抽取方法研究", 《CNKI 西安电子科技大学硕士学位论文》, 6 June 2024 (2024-06-06), pages 1 - 79 * |
王朱君: "基于BERT预训练的事件抽取方法研究", 《中国优秀硕士学位论文全文数据库信息科技辑》, no. 08, 15 August 2021 (2021-08-15), pages 138 - 689 * |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116861901A (en) * | 2023-07-04 | 2023-10-10 | 广东外语外贸大学 | Chinese event detection method and system based on multitask learning and electronic equipment |
CN116861901B (en) * | 2023-07-04 | 2024-04-09 | 广东外语外贸大学 | Chinese event detection method and system based on multitask learning and electronic equipment |
CN117236338A (en) * | 2023-08-29 | 2023-12-15 | 北京工商大学 | Named entity recognition model of dense entity text and training method thereof |
CN117236338B (en) * | 2023-08-29 | 2024-05-28 | 北京工商大学 | Named entity recognition model of dense entity text and training method thereof |
Also Published As
Publication number | Publication date |
---|---|
CN114780677B (en) | 2024-07-16 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN110826336B (en) | Emotion classification method, system, storage medium and equipment | |
Xiang et al. | A convolutional neural network-based linguistic steganalysis for synonym substitution steganography | |
Gallant et al. | Representing objects, relations, and sequences | |
Ren et al. | Conversational query understanding using sequence to sequence modeling | |
CN113239700A (en) | Text semantic matching device, system, method and storage medium for improving BERT | |
CN110765240B (en) | Semantic matching evaluation method for multi-phase sentence pairs | |
CN114780677B (en) | Chinese event extraction method based on feature fusion | |
Ju et al. | An efficient method for document categorization based on word2vec and latent semantic analysis | |
CN116662582B (en) | Specific domain business knowledge retrieval method and retrieval device based on natural language | |
CN110990555B (en) | End-to-end retrieval type dialogue method and system and computer equipment | |
CN110851584B (en) | Legal provision accurate recommendation system and method | |
Gu et al. | Sentiment analysis via deep multichannel neural networks with variational information bottleneck | |
Grzegorczyk | Vector representations of text data in deep learning | |
CN115422369B (en) | Knowledge graph completion method and device based on improved TextRank | |
CN110955745B (en) | Text hash retrieval method based on deep learning | |
CN115600597A (en) | Named entity identification method, device and system based on attention mechanism and intra-word semantic fusion and storage medium | |
CN114564953A (en) | Emotion target extraction model based on multiple word embedding fusion and attention mechanism | |
Anass et al. | Deceptive opinion spam based on deep learning | |
CN109948163A (en) | The natural language semantic matching method that sequence dynamic is read | |
KR102462758B1 (en) | Method for document summarization based on coverage with noise injection and word association, recording medium and device for performing the method | |
CN111581365B (en) | Predicate extraction method | |
CN117932066A (en) | Pre-training-based 'extraction-generation' answer generation model and method | |
CN111767388B (en) | Candidate pool generation method | |
Jin et al. | An efficient machine reading comprehension method based on attention mechanism | |
Prajapati et al. | Automatic Question Tagging using Machine Learning and Deep learning Algorithms |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant |