CN113536799B - Medical named entity recognition modeling method based on fusion attention - Google Patents
Medical named entity recognition modeling method based on fusion attention Download PDFInfo
- Publication number
- CN113536799B CN113536799B CN202110927320.1A CN202110927320A CN113536799B CN 113536799 B CN113536799 B CN 113536799B CN 202110927320 A CN202110927320 A CN 202110927320A CN 113536799 B CN113536799 B CN 113536799B
- Authority
- CN
- China
- Prior art keywords
- character
- word
- medical
- lstm
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 36
- 230000004927 fusion Effects 0.000 title claims abstract description 8
- 239000013598 vector Substances 0.000 claims abstract description 19
- 230000011218 segmentation Effects 0.000 claims abstract description 10
- 238000004364 calculation method Methods 0.000 claims description 15
- 238000004422 calculation algorithm Methods 0.000 claims description 13
- 238000002372 labelling Methods 0.000 claims description 9
- 230000001413 cellular effect Effects 0.000 claims description 2
- 238000006243 chemical reaction Methods 0.000 claims description 2
- 238000010606 normalization Methods 0.000 claims description 2
- 239000011159 matrix material Substances 0.000 claims 1
- 210000004027 cell Anatomy 0.000 description 10
- 238000007711 solidification Methods 0.000 description 5
- 230000008023 solidification Effects 0.000 description 5
- 201000009267 bronchiectasis Diseases 0.000 description 4
- 238000013527 convolutional neural network Methods 0.000 description 4
- 238000011156 evaluation Methods 0.000 description 4
- 239000012634 fragment Substances 0.000 description 4
- 238000000605 extraction Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000015654 memory Effects 0.000 description 3
- 238000012216 screening Methods 0.000 description 3
- 238000012549 training Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 210000000621 bronchi Anatomy 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 210000003437 trachea Anatomy 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 241000282414 Homo sapiens Species 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000002457 bidirectional effect Effects 0.000 description 1
- 239000008280 blood Substances 0.000 description 1
- 210000004369 blood Anatomy 0.000 description 1
- 210000003123 bronchiole Anatomy 0.000 description 1
- 206010006451 bronchitis Diseases 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 239000012141 concentrate Substances 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000008034 disappearance Effects 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 230000008014 freezing Effects 0.000 description 1
- 238000007710 freezing Methods 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
- 238000007670 refining Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000006403 short-term memory Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
- G06F40/295—Named entity recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/237—Lexical tools
- G06F40/242—Dictionaries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- General Engineering & Computer Science (AREA)
- Biomedical Technology (AREA)
- Evolutionary Computation (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Machine Translation (AREA)
- Medical Treatment And Welfare Office Work (AREA)
Abstract
The medical named entity recognition modeling method based on the fusion attention comprises the following steps: performing Chinese word segmentation and indexing on the medical text sentence; obtaining a Bi-LSTM model by splicing forward LSTM and reverse LSTM; updating the output feature vector through an attention mechanism; and decoding the output characteristic vector by a conditional random field CRF to obtain the label of the medical entity type of the input medical text sentence. The invention adds the matched words in the dictionary on the basis of the character sequence, provides more guidance for the model through the dynamic control of the gate structure, and selects the most relevant characters and words from the medical corpus. Compared to character-based methods, multi-granular information is explicitly exploited by the model to obtain better recognition performance. And moreover, an attention mechanism is introduced, so that the model focuses on effective information, and the condition that the traditional Bi-LSTM-CRF model considers context information but ignores different characters and words with different importance in sentences is overcome.
Description
Technical Field
The invention relates to the technical field of natural language processing, in particular to a medical named entity recognition modeling method based on fusion attention.
Background
The continuous advancement of medical informatization has led to the explosive growth of massive data in the medical field, and the data forms are more and more diverse, which contain a great deal of useful information. Medical data is widely available and mainly comes from medical websites, hospital electronic medical records, medical books and the like. The medical knowledge map regulates the whole refining process of acquiring knowledge update from knowledge according to a complete set of criteria, so that massive medical knowledge can be efficiently managed and applied. For the construction of the medical knowledge map, the important foundation is laid for accurately identifying medical named entities from massive medical data. The extraction of specific ad hoc types of medical entities, such as "blood routine" entities belonging to the examination item type, "bronchitis" entities belonging to the disease type, is performed by medical named entity identification, a task also referred to as medical entity extraction.
Named entity recognition is a basic task of knowledge extraction, which is generally viewed as a sequence labeling task for predicting entity boundaries and entity class labels. Currently, the named entity recognition method is mainly a deep learning-based method. The neural network is applied to named entity recognition for the first time by carrying out named entity recognition through a one-way Short-Term Memory network (LSTM). Excellent recognition is achieved by using a Convolutional Neural Network (CNN) in combination with a Conditional Random Field (CRF). The performance of the CNN-CRF model may be enhanced by utilizing a character-based CNN. In the field of Chinese named entity recognition, the LSTM-CRF system structure is the mainstream at present and is adopted by a great deal of research institutes. Character features of manual design (Hand-crafted), character-based CNN, and character-based LSTM, etc. are used to represent character features.
Character sequence notation is the primary method of Chinese named entity recognition. By comparing word-based and character-based statistical methods in the named entity recognition task, the results indicate that the character-based method is relatively superior. There has been much interest from researchers regarding how to better utilize word information in a task of recognizing a named entity in chinese, such as using word segmentation information as a feature of the named entity recognition, combining word segmentation and named entity recognition through dual decomposition, and multi-task learning.
External information resources are used to assist named entity recognition, especially dictionaries are widely used. Such as multi-task learning on large raw text using word-level language models to enhance training of named entity recognition, word representation enhancement by pre-training character-level language models, and mining of cross-domain and cross-language knowledge through multi-task learning.
Disclosure of Invention
The invention aims to provide a medical named entity recognition modeling method based on fusion attention.
The technical scheme for realizing the purpose of the invention is as follows:
the medical named entity recognition modeling method based on the fusion attention comprises the following steps:
step 1: performing Chinese word segmentation and indexing on the medical text sentence s: matching the medical text sentence s with a dictionary to obtain a word sequence w 1 ,w 2 \8230, wn, wi is the ith word in the word sequence, i =1,2, \8230, n; indexing the kth character of the ith word by t (i, k), wherein k is the position of the character in the word;
the medical text sentence s = c 1 ,c 2 \8230, cm, cj is the jth character of the medical text sentence s, j =1,2, \8230, m;
words with index starting with b and ending with e are passed throughRepresenting, wherein b represents an index of a word start character, and e represents an index of a word end character;
step 2: obtaining a Bi-LSTM model by splicing the forward LSTM and the reverse LSTM;
wherein,and &>An input gate, a forgetting gate and an output gate respectively; />Information of new candidate cells; wcT and bc are model weight parameters and bias terms to be learned respectively; sigma is a Sigmoid function; as a hadamard product;
for an embedded representation of the character cj, ->ec represents a character embedding look-up table;
for the hidden state corresponding to the character cj>The cell state of the character corresponding to the character cj; />Is a previous character cj -1 Corresponding hidden state, is greater or less than>Is a previous character cj -1 The corresponding character cell state;
is an additional gate structure introduced for controlling all word cells ending with the character with index e>On tail character cells>Is based on the contribution of->WlT and bl are model weight parameters to be learned and bias terms respectively;
wherein,is a input door, is>To forget the door; />Cell information of new candidate words is obtained; wwT and bw are respectively a model weight parameter to be learned and a bias term; />Is a hidden state corresponding to the first character of the word;
is a word/phrase>Is shown in a non-volatile memory cell (c), device for combining or screening>ew represents a word embedding lookup table obtained by the dictionary conversion in the step 1;
reverse LSTM is similar to forward LSTM;
the method is respectively used for medical text sentences s to obtainAnd &>Splicing the two groups of vectors, and then performing final hidden vector(s) based on each character in s>The calculation formula is as follows:
and 3, step 3: the output of step 2 is assigned a weight α tj, corresponding thereto, by the attention mechanism, the eigenvectorAnd carrying out weighted summation with the corresponding weight alpha tj to obtain a new output vector ct, specifically:
wherein etj is used for measuring jth source end character and jth source end characterMatching degree of t target end characters; st -1 Hiding the layer state at the t-th moment;wa and Ua are weight matrixes;
and 4, step 4: performing comparison on feature vector c = { c output in step 3 through conditional random field CRF 1 ,c 2 Decoding \8230;, cm } to obtain the label of the medical entity type of the input medical text statement s, specifically:
P(y|c)=CRF(c,y);
wherein y is all possible output tag sequences of the input medical text sentence s, and P (y | c) is the conditional probability of the possible output tag sequences y;
during prediction, a labeling sequence with the highest score is found on an input sequence through a Viterbi Algorithm (Viterbi Algorithm) of dynamic programming to obtain a label y of the medical entity type of the input medical text sentence s,
y*=argmaxP(y|c)。
compared with the prior art, the invention has the beneficial effects that:
1. the medical named entity recognition accuracy is improved by utilizing the word information. Input of the invention on the basis of a character sequence
Adding the matched words in the dictionary, and providing more guidance for the model through the dynamic control of the gate structure, thereby selecting the most relevant characters and words from the medical corpus. Compared to character-based methods, multi-granular information is explicitly exploited by the model for better recognition performance.
2. An attention mechanism is introduced to ensure that the model focuses on effective information, thereby making up the consideration of the traditional Bi-LSTM-CRF model
Context information is ignored, but the situation that different characters and words have different importance in the sentence is ignored.
Drawings
Fig. 1 is a schematic diagram of a medical named entity recognition model architecture.
FIG. 2 is a schematic illustration of an Encode-Decoder using an attention mechanism.
FIG. 3 is a schematic diagram of an attention machine mechanism in a modeling method.
Detailed Description
In order to improve the recognition accuracy of the medical named entity by using word information, avoid or reduce recognition errors of unknown words caused by ambiguity due to certain particularity of medical professional terms, and simultaneously make up for the situation that although context information is considered in a traditional Bi-LSTM-CRF model, different characters and words have different importance in sentences, the invention provides a medical named entity recognition method Ng-BAC based on new word discovery and a fusion attention mechanism, wherein the model architecture is shown in figure 1. The model firstly acquires new words from medical corpus through N-grams algorithm to construct a medical related external dictionary; integrating potential word information into the character-based model as an extension, dynamically routing information from different paths to each character through a gate structure; different weights are distributed to the Bi-LSTM layer output by introducing an attention mechanism to improve the accuracy of the output; and finally, carrying out sequence labeling by a conditional random field CRF.
The specific implementation steps are as follows:
step 1: extracting new words by a new word discovery algorithm based on an N-grams model, wherein the new words are extracted mainly according to two indexes of solidity and word frequency:
(1) Degree of solidification: represents how close the word segments are, usually measured by mutual information;
(2) Word frequency: representing the number of times a word occurs in a corpus.
Different from a method only considering the solidity of adjacent characters, the method of the invention simultaneously considers the internal solidity of multiple characters in the corpus, segments the corpus through N-grams, and calculates the internal solidity, wherein a calculation formula of the internal solidity of a character string taking 3 characters as an example is as follows:
wherein a, b and c are three adjacent characters in the character string; p (a), p (b), p (c) represent the frequency of their individual occurrence; p (ab), p (bc), p (abc) represent the frequency of occurrence after their component words; taking the minimum value of the solidification degree as the internal solidification degree of the whole character string.
The new word discovery is to extract words meeting two indexes of a freezing degree threshold value and a word frequency, taking 4-grams as an example, the algorithm execution steps are as follows:
step 1.1: setting n as 4 characters (namely 4-grams) to segment the sentences, counting 2-grams, 3-grams and 4-grams, and calculating the internal solidification degree;
step 1.2: setting different thresholds as multiples of 5, if n is equal to 2, the threshold is set to 5, if n is equal to 3, the threshold is set to 25, and if n is equal to 4, the threshold is set to 125;
step 1.3: generating a set G to reserve the fragments with the internal solidification degree higher than the threshold value;
step 1.4: coarsely segmenting the corpus according to the setting of the grams in the step 1.1, and if only one segment exists in the set G, the segment is not segmented any more and the frequency of the segment is calculated;
step 1.5: after the step 1.4, backtracking inspection is needed according to the principle of 'peaceful placement and no cutting error'; when the word number of the word is less than or equal to 4, judging whether the word is in the set G, if so, keeping the word, otherwise, deleting the word; when the number of words is larger than 4 words, detecting whether each 4-word fragment is in the set G, if so, keeping the fragment, otherwise, deleting the fragment;
and after extracting the new words, comparing the new words with the internal Jieba word segmentation dictionary, and manually screening the new words after comparison and screening to further determine a final dictionary.
The invention introduces new word discovery on the basis of utilizing word information to avoid or reduce errors caused by unknown words. Because the professional terms have certain particularity, the recognition of the unknown words can be wrong due to ambiguity for the named entity recognition task in a special field. For example, in "bronchiectasis", since "trachea" is recognized erroneously as a common tool in the living field, such an error is particularly significant in the medical field. The invention obtains new words from the medical corpus through the N-grams algorithm to assist the Word segmentation algorithm to carry out Word segmentation operation, and then builds a medical related dictionary by using Word2Vec according to the result obtained after Word segmentation.
And 2, step: the invention sets the basic line network structure as Bi-LSTM as the basis, and introduces word information to improve the accuracy of medical named entity recognition.
Unlike the traditional character-based model, the method of the invention is applied to the character sequence c 1 ,c 2 8230cm, the words matched by all subsequences of the character sequence in a dictionary determined by the new word discovery algorithm based on the N-grams model mentioned in step 1 are input. Other dictionaries, such as a Jieba participle built-in dictionary, may also be used.
Formally, by s = c 1 ,c 2 \8230cmrepresents the input sentence and cj represents the jth character of the input sequence.
For the method of the invention, the input sentence s can also be regarded as a word sequence: s = w 1 ,w 2 And 8230wn, obtaining the word sequence by performing Chinese word segmentation on the input sentence, wherein the ith word in the sequence is wi.
Subsequence pass with index starting with b and ending with eIs shown, as shown in FIG. 1, in "bronchiectasis>Means "bronchus">Meaning "expand".
The specific character k of the specific word i is indexed by t (i, k), and as shown in fig. 1, if the sentence is divided into "bronchiectasis", t (2, 1) =4 (bronchiole) and t (1, 3) =3 (tubes).
And 3, step 3: compared with the traditional recurrent neural network, the LSTM model has the advantages that a gating mechanism is introduced, the neuron forgetting function is added to the LSTM model, and the method can be used for avoiding the gradient disappearance phenomenon.
Taking forward LSTM as an example, the algorithm performs the following steps:
the baseline model of the method is a model based on characters, and when the model is input, the embedded expression calculation formula of each character cj is as follows:
wherein ec represents a character embedding look-up table;
the basic cycle structure of the baseline model is based on the character cell stateHidden status ≥ corresponding to each character cj>Is constructed in which>For recording a stream of cycle information from the beginning of a period to the character cj, and->The method is used for sequence labeling on a CRF layer;
the main calculation in LSTM is as follows:
wherein,and &>Respectively representing an input gate, a forgetting gate and an output gate; wcT and bc are model parameters; σ represents Sigmoid function; an as high as the hadamard product.
Unlike the character-based model, the method of the present invention matches sentences with a dictionary to obtain a subsequenceIntroducing a status on cells->Is calculated such that, when entered into the model, each subsequence +>The embedding expression of (a) represents the calculation formula as follows:
where ew represents a term embedding look-up table.
Word cellFor indicating that start of a period->Is marked only at the character level, there is no output gate for the word cell, which is present here, or which is present there>The calculation formula of (a) is as follows:
Introduction of word cellsThen flows into each->Has more circular paths, as indicated by the "bronchiectasis" input in FIG. 1, and ` according to `>Includes->(tube),. Or (es)>(trachea) and->(bronchi).
The method of the invention ends all characters with index eIs connected to the cell->And an additional door structure is introduced>To control each sub-sequence of cells->Is paired and/or matched>The calculation formula is as follows:
therefore, the cell state calculation formula of the model after introducing the word information is as follows:
after word information is introduced, vectors are hiddenThe calculation is still performed as in the character-based model.
During model training, the loss values are propagated back to the parameters, constantly optimizing the parameters, so that the model dynamically focuses on more relevant words during annotation.
The Bi-LSTM layer applies the above method to each corpus in forward and reverse order, resulting inAndsplicing the two groups of vectors, and obtaining the final hidden vector corresponding to each character in the corpusThe calculation formula is as follows:
and 4, step 4: as shown in FIG. 2, the method of the present invention introduces an attention mechanism to simulate the processing mode of human beings for focus of attention, and let the model ignore useless information and concentrate on target information needing important attention in the input medical corpus and information related to the current output, so as to improve the output efficiency and quality;
as shown in FIG. 3, the weights corresponding to the output of the Bi-LSTM layer are assigned by the attention mechanism to assign the feature vectors of the preamble model outputAnd after weighted summation with the corresponding weight alpha tjObtaining a new output vector ct, wherein the calculation formula is as follows:
the etj is used for measuring the matching degree of the jth source terminal character and the tth target terminal character;
after introducing the attention mechanism into the conventional Decoder, the Encoder does not need to encode all the information in a fixed length any more, and in this way, the Decoder can select the information propagated in the annotation sequence.
And 5: finally, sequence labeling is carried out by using standard CRF, and the CRF can predict the relationship between labels by using the dependency information between labels in the labeling process so as to obtain a globally optimal labeling sequence;
the eigenvector c = { c } output in step 4 by the conditional random field CRF 1 ,c 2 Decoding \8230;, cm } to obtain the label of the medical entity type of the input medical text statement s, specifically:
P(y|c)=CRF(c,y);
wherein y is all possible output tag sequences of the input medical text sentence s, and P (y | c) is the conditional probability of the possible output tag sequences y;
during prediction, a labeling sequence with the highest score is found on an input sequence through a Viterbi Algorithm (Viterbi Algorithm) of dynamic programming, so as to obtain a label y of the medical entity type of the input medical text statement s, wherein the calculation formula is as follows:
y*=argmax P(y|c)。
in order to verify the effectiveness of the invention, the invention carries out comparison experiments on 1200 electronic medical records with artificial labels in a CCKS2017-Task2 public data set and classical models in 3 named entity recognition tasks, wherein the 3 comparison models are as follows:
(1) LSTM-CRF: introducing a conditional random field in the unidirectional long-short term memory network;
(2) Bi-LSTM-CRF: introducing a conditional random field in a bidirectional long-short term memory network;
(3) Lattice-LSTM: word-based units are introduced in character-based LSTM-CRF.
The evaluation index was evaluated by using Precision (Precision), recall (Recall) and F1 value (F1-Score), and the calculation formula was as follows:
depending on the desired output and the actual output of the model, the results can be divided into TP (True Positive), FP (False Positive), TN (True Negative) and FN (False Negative). In the named entity recognition task, the meaning is as follows:
(1) TP: correctly identifying the entity as a number of entities;
(2) FP: identifying non-entity errors as a number of entities;
(3) TN: correctly judging the non-entities as the number of the non-entities;
(4) FN: and judging the entity error as the number of non-entities.
The precision rate represents the ratio of real entities in all samples identified as entities, i.e., precision rate. The recall rate represents the rate at which all entities in the sample are correctly identified, i.e., the recall rate. And comprehensively considering the two evaluation indexes, and performing weighted harmonic averaging on the two evaluation indexes to obtain an F1 value so as to avoid the mutual restriction of the two evaluation indexes.
The results of the experiment are shown in table 1:
table 1 model comparative experimental results
Experimental results show that the Ng-BAC model provided by the invention is excellent in performance in the CCKS2017-Task2 data set, the accuracy rate reaches 89.86%, the recall rate reaches 92.05%, the F1 value reaches 90.94%, and the model is improved to a certain extent compared with a classical model in a named entity recognition Task.
Claims (1)
1. The medical named entity recognition modeling method based on fusion attention is characterized by comprising the following steps:
step 1: performing Chinese word segmentation and indexing on the medical text sentence s: matching the medical text sentence s with the dictionary to obtain a word sequence w 1 ,w 2 ,...,w n ,w i Is the ith word in the sequence of words, i =1, 2. Indexing the kth character of the ith word by t (i, k), wherein k is the position of the character in the word;
the medical text sentence s = c 1 ,c 2 ,...,c m ,c j J =1, 2., m, which is the jth character of the medical text sentence s;
words with index starting with b and ending with e are passed throughRepresenting, wherein b represents an index of a word start character, and e represents an index of a word end character;
step 2: obtaining a Bi-LSTM model by splicing the forward LSTM and the reverse LSTM;
wherein,and &>An input gate, a forgetting gate and an output gate respectively; />Information of new candidate cells; />And b c Respectively are a model weight parameter to be learned and a bias term; sigma is a Sigmoid function; as a hadamard product;
is a character c j Corresponding hidden state in the interior of the housing>Is a character c j The corresponding character cell state; />Is a preceding character c j-1 Corresponding hidden state, is greater or less than>Is a previous character c j-1 The corresponding character cell state;
character cell state after introduction of word informationThe method comprises the following steps:
is an additional gate structure introduced for controlling all word cells ending with the character with index e>For the tail character cell->Is based on the contribution of-> And b l Respectively are a model weight parameter to be learned and a bias term;
wherein,is a input door, is>To forget the door; />Cell information of new candidate words; />And b w Respectively are a model weight parameter to be learned and a bias term; />Is a hidden state corresponding to the first character of the word;
is a word/phrase>Is embedded means that->e w Indicating that the words obtained by the dictionary conversion in the step 1 are embedded into a lookup table;
the reverse LSTM is similar to the forward LSTM;
respectively applying the methods to medical text sentences s to obtainAnd &>Two sets of vectors, and then combining the two sets of vectorsSplicing is performed, and the final hidden vector corresponding to each character in s->The calculation formula is as follows:
and step 3: assigning a weight α corresponding to the output of step 2 by the attention mechanism tj Feature vectorAnd a weight α corresponding thereto tj Carrying out weighted summation to obtain a new output vector c t The method specifically comprises the following steps:
wherein e is tj The matching degree of the jth source terminal character and the tth target terminal character is measured; s t-1 Hiding the layer state at the t-th moment;W a and U a Is a weight matrix;
and 4, step 4: performing comparison on feature vector c = { c output in step 3 through conditional random field CRF 1 ,c 2 ,...,c m Decoding is carried out, and a label of the medical entity type of the input medical text sentence s is obtained, specifically:
P(y|c)=CRF(c,y);
wherein y is all possible output tag sequences of the input medical text sentence s, and P (y | c) is the conditional probability of the possible output tag sequences y;
during prediction, a labeling sequence with the highest score is found on an input sequence through a Viterbi Algorithm (Viterbi Algorithm) of dynamic programming, and a label y of the medical entity type of the input medical text sentence s is obtained * ,
y * =arg max P(y|c)。
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110927320.1A CN113536799B (en) | 2021-08-10 | 2021-08-10 | Medical named entity recognition modeling method based on fusion attention |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110927320.1A CN113536799B (en) | 2021-08-10 | 2021-08-10 | Medical named entity recognition modeling method based on fusion attention |
Publications (2)
Publication Number | Publication Date |
---|---|
CN113536799A CN113536799A (en) | 2021-10-22 |
CN113536799B true CN113536799B (en) | 2023-04-07 |
Family
ID=78122434
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202110927320.1A Active CN113536799B (en) | 2021-08-10 | 2021-08-10 | Medical named entity recognition modeling method based on fusion attention |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN113536799B (en) |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114818721B (en) * | 2022-06-30 | 2022-11-01 | 湖南工商大学 | Event joint extraction model and method combined with sequence labeling |
CN115146644B (en) * | 2022-09-01 | 2022-11-22 | 北京航空航天大学 | Alarm situation text-oriented multi-feature fusion named entity identification method |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106569998A (en) * | 2016-10-27 | 2017-04-19 | 浙江大学 | Text named entity recognition method based on Bi-LSTM, CNN and CRF |
CN110032739A (en) * | 2019-04-18 | 2019-07-19 | 清华大学 | Chinese electronic health record name entity abstracting method and system |
CN110866401A (en) * | 2019-11-18 | 2020-03-06 | 山东健康医疗大数据有限公司 | Chinese electronic medical record named entity identification method and system based on attention mechanism |
CN111079377A (en) * | 2019-12-03 | 2020-04-28 | 哈尔滨工程大学 | Method for recognizing named entities oriented to Chinese medical texts |
CN111737991A (en) * | 2020-07-01 | 2020-10-02 | 携程计算机技术(上海)有限公司 | Text sentence break position identification method and system, electronic device and storage medium |
CN111783466A (en) * | 2020-07-15 | 2020-10-16 | 电子科技大学 | Named entity identification method for Chinese medical records |
CN112597774A (en) * | 2020-12-14 | 2021-04-02 | 山东师范大学 | Chinese medical named entity recognition method, system, storage medium and equipment |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10606846B2 (en) * | 2015-10-16 | 2020-03-31 | Baidu Usa Llc | Systems and methods for human inspired simple question answering (HISQA) |
-
2021
- 2021-08-10 CN CN202110927320.1A patent/CN113536799B/en active Active
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106569998A (en) * | 2016-10-27 | 2017-04-19 | 浙江大学 | Text named entity recognition method based on Bi-LSTM, CNN and CRF |
CN110032739A (en) * | 2019-04-18 | 2019-07-19 | 清华大学 | Chinese electronic health record name entity abstracting method and system |
CN110866401A (en) * | 2019-11-18 | 2020-03-06 | 山东健康医疗大数据有限公司 | Chinese electronic medical record named entity identification method and system based on attention mechanism |
CN111079377A (en) * | 2019-12-03 | 2020-04-28 | 哈尔滨工程大学 | Method for recognizing named entities oriented to Chinese medical texts |
CN111737991A (en) * | 2020-07-01 | 2020-10-02 | 携程计算机技术(上海)有限公司 | Text sentence break position identification method and system, electronic device and storage medium |
CN111783466A (en) * | 2020-07-15 | 2020-10-16 | 电子科技大学 | Named entity identification method for Chinese medical records |
CN112597774A (en) * | 2020-12-14 | 2021-04-02 | 山东师范大学 | Chinese medical named entity recognition method, system, storage medium and equipment |
Also Published As
Publication number | Publication date |
---|---|
CN113536799A (en) | 2021-10-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN109657239B (en) | Chinese named entity recognition method based on attention mechanism and language model learning | |
Yu et al. | Named entity recognition as dependency parsing | |
CN111241294B (en) | Relationship extraction method of graph convolution network based on dependency analysis and keywords | |
CN108460013B (en) | Sequence labeling model and method based on fine-grained word representation model | |
He et al. | Decoding with value networks for neural machine translation | |
Jiang et al. | A BERT-BiLSTM-CRF model for Chinese electronic medical records named entity recognition | |
CN107168955B (en) | Utilize the Chinese word cutting method of the word insertion and neural network of word-based context | |
CN103853710B (en) | A kind of bilingual name entity recognition method based on coorinated training | |
CN111738007B (en) | Chinese named entity identification data enhancement algorithm based on sequence generation countermeasure network | |
CN109800411A (en) | Clinical treatment entity and its attribute extraction method | |
CN109800434B (en) | Method for generating abstract text title based on eye movement attention | |
CN106980608A (en) | A kind of Chinese electronic health record participle and name entity recognition method and system | |
CN110110324A (en) | A kind of biomedical entity link method that knowledge based indicates | |
CN113536799B (en) | Medical named entity recognition modeling method based on fusion attention | |
Gao et al. | Named entity recognition method of Chinese EMR based on BERT-BiLSTM-CRF | |
CN112541356A (en) | Method and system for recognizing biomedical named entities | |
CN114818717A (en) | Chinese named entity recognition method and system fusing vocabulary and syntax information | |
CN116341557A (en) | Diabetes medical text named entity recognition method | |
Li et al. | Biomedical named entity recognition based on the two channels and sentence-level reading control conditioned LSTM-CRF | |
Liu et al. | Chinese clinical entity recognition via attention-based CNN-LSTM-CRF | |
Luo et al. | Loss prediction: End-to-end active learning approach for speech recognition | |
Cao et al. | Knowledge guided short-text classification for healthcare applications | |
Hu et al. | A simple yet effective subsequence-enhanced approach for cross-domain NER | |
CN116757195B (en) | Implicit emotion recognition method based on prompt learning | |
CN113986345B (en) | Pre-training enhanced code clone detection method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |