CN112883161A - Transliteration name recognition rule generation method, transliteration name recognition rule generation device, transliteration name recognition rule generation equipment and storage medium - Google Patents
Transliteration name recognition rule generation method, transliteration name recognition rule generation device, transliteration name recognition rule generation equipment and storage medium Download PDFInfo
- Publication number
- CN112883161A CN112883161A CN202110242748.2A CN202110242748A CN112883161A CN 112883161 A CN112883161 A CN 112883161A CN 202110242748 A CN202110242748 A CN 202110242748A CN 112883161 A CN112883161 A CN 112883161A
- Authority
- CN
- China
- Prior art keywords
- text
- transliteration
- transliteration name
- name
- recognition rule
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000012549 training Methods 0.000 claims abstract description 10
- 238000004590 computer program Methods 0.000 claims description 19
- 238000013145 classification model Methods 0.000 claims description 8
- 238000012545 processing Methods 0.000 claims description 7
- 238000000605 extraction Methods 0.000 claims description 5
- 230000007246 mechanism Effects 0.000 claims description 5
- 238000012216 screening Methods 0.000 claims description 5
- 238000003058 natural language processing Methods 0.000 abstract description 2
- 230000006870 function Effects 0.000 description 7
- 238000012795 verification Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000003550 marker Substances 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000007711 solidification Methods 0.000 description 1
- 230000008023 solidification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
- G06F40/295—Named entity recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/3331—Query processing
- G06F16/334—Query execution
- G06F16/3344—Query execution using natural language analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
- G06F40/216—Parsing using statistical methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Computation (AREA)
- Evolutionary Biology (AREA)
- Databases & Information Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Machine Translation (AREA)
Abstract
The embodiment of the application is suitable for the field of natural language processing, and provides a method for generating a transliteration name recognition rule, which comprises the following steps: acquiring a text sample to be processed, wherein the text sample to be processed is marked with a first transliteration name text; extracting a first transliteration name text; and generating a transliteration name recognition rule according to the first transliteration name text, wherein the transliteration name recognition rule is used for recognizing a second transliteration name text in the text to be processed. Therefore, the transliteration name recognition rule is generated through training sample data, the names of foreigners containing transliterations in the Chinese text are recognized according to the transliteration name recognition rule, and the recognition accuracy rate of the named entity recognition of the Chinese at the present stage is improved.
Description
Technical Field
The present application relates to the field of natural language processing, and in particular, to a method and an apparatus for generating transliterated name recognition rules, a generation device, and a storage medium.
Background
The Chinese named entity recognition mainly realizes the entity recognition of the types of names, place names, organizations and the like at the present stage, wherein the recognition of the names has relatively mature means for the Chinese name recognition result,
but the recognition results are relatively poor when the chinese text contains the transliterated foreign person name. The foreign names of the transliterated names have variable lengths and no obvious boundary marker words, so that the foreign names containing the transliterated names cannot be identified in the current stage of the named entity identification of the Chinese, and the problem of low identification accuracy is caused.
Disclosure of Invention
In view of this, the embodiment of the present application provides a method, an apparatus, a device and a storage medium for generating a transliterated name recognition rule, so as to solve the problem of low recognition accuracy due to the fact that a foreign person name including a transliteration cannot be recognized in the present stage of the named entity recognition of chinese.
A first aspect of an embodiment of the present application provides a method for generating a transliteration name recognition rule, including:
acquiring a text sample to be processed, wherein the text sample to be processed is marked with a first transliteration name text;
extracting the first transliteration name text;
and generating the transliteration name recognition rule according to the first transliteration name text, wherein the transliteration name recognition rule is used for recognizing a second transliteration name text in the text to be processed.
In a possible implementation manner of the first aspect, the generating the transliteration name recognition rule according to the transliteration name text includes:
constructing a transliteration name database according to the first transliteration name text;
determining the frequency corresponding to each word in the first transliteration name text in the transliteration name database;
and generating a transliteration name recognition rule according to the frequency corresponding to each word in the first transliteration name text.
In a possible implementation manner of the first aspect, constructing a transliteration name recognition rule according to a frequency corresponding to each word in the first transliteration name text includes:
obtaining a frequency threshold according to the frequency corresponding to each word in the first transliteration name text;
the transliteration name recognition rule is obtained according to the following formula:
where i =1, 2.,. n, min (pi) denotes the minimum frequency of words in the phrase of the text to be processed, PThreshold valueAnd representing a frequency threshold, and when the minimum frequency of the words in the phrases of the text to be processed is greater than the frequency threshold, determining that the phrases in the text to be processed are second transliterated name texts.
In a possible implementation manner of the first aspect, before extracting the first transliteration name text, the method further includes:
dividing the text sample to be processed into a first text sample and a second text sample, wherein the first text sample contains a name of a person, and the second text sample does not contain the name of the person;
inputting the first text sample into a preset classification model for training to obtain a name recognition model;
the person name recognition model is used as a pre-screening rule corresponding to the transliteration name recognition rule, and before the transliteration name recognition rule recognizes a second transliteration name text in the text to be processed, the person name in the text to be processed is screened out.
In a possible implementation manner of the first aspect, after generating the transliteration name recognition rule according to the first transliteration name text, the method further includes:
extracting left adjacent characters and right adjacent characters of the first transliteration name from the text sample to be processed;
forming a prefix corpus according to the left adjacent characters;
forming a suffix corpus according to the right adjacent characters;
and generating a check rule corresponding to the transliteration name identification rule based on the prefix corpus and the suffix corpus, wherein the check rule is used for checking a second transliteration name text in the text to be processed after the transliteration name identification mechanism identifies the second transliteration name text.
A second aspect of the embodiments of the present application provides an apparatus for generating a transliteration name recognition rule, including:
the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a text sample to be processed, and the text sample to be processed is marked with a first transliteration name text;
the extraction module is used for extracting the first transliteration name text;
and the generating module is used for generating the transliteration name recognition rule according to the first transliteration name text, and the transliteration name recognition rule is used for recognizing a second transliteration name text in the text to be processed.
In one possible implementation, the generating module includes:
the construction unit is used for constructing a transliteration name database according to the first transliteration name text;
the determining unit is used for determining the frequency corresponding to each word in the first transliteration name text in the transliteration name database;
and the generating unit is used for generating a transliteration name recognition rule according to the frequency corresponding to each word in the first transliteration name text.
In one possible implementation, the generating unit includes:
the calculation subunit is used for obtaining a frequency threshold value according to the frequency corresponding to each word in the first transliteration name text;
a rule generating subunit, configured to obtain the transliteration name identification rule according to the following formula:
where i =1, 2.,. n, min (pi) denotes the minimum frequency of words in the phrase of the text to be processed, PThreshold valueAnd representing a frequency threshold, and when the minimum frequency of the words in the phrases of the text to be processed is greater than the frequency threshold, determining that the phrases in the text to be processed are second transliterated name texts.
In one possible implementation, the apparatus further includes:
the dividing module is used for dividing the text sample to be processed into a first text sample and a second text sample, wherein the first text sample contains a name of a person, and the second text sample does not contain the name of the person;
the training module is used for inputting the first text sample into a preset classification model for training to obtain a name recognition model;
the person name recognition model is used as a pre-screening rule corresponding to the transliteration name recognition rule, and before the transliteration name recognition rule recognizes a second transliteration name text in the text to be processed, the person name in the text to be processed is screened out.
In one possible implementation, the apparatus further includes:
the extraction module is used for extracting left adjacent characters and right adjacent characters of the first transliteration name from the text sample to be processed;
the first forming submodule is used for forming a prefix corpus according to the left adjacent characters;
a second forming submodule for forming a suffix corpus from the right glyph;
and the verification generation submodule is used for generating a verification rule corresponding to the transliteration name identification rule based on the prefix corpus and the suffix corpus, and the verification rule is used for verifying a second transliteration name text in the text to be processed after the transliteration name identification mechanism identifies the second transliteration name text.
A third aspect of an embodiment of the present application provides a generation apparatus, including: a memory, a processor, an image capture device, and a computer program stored in the memory and executable on the processor, the processor implementing the steps of the method as described above when executing the computer program.
A fourth aspect of embodiments of the present application provides a readable storage medium, which stores a computer program that, when executed by a processor, implements the steps of the method as described above.
Compared with the prior art, the embodiment of the application has the advantages that:
according to the method and the device, the transliteration name recognition rule is generated through training sample data, so that the names of foreigners containing transliterations in the Chinese text are recognized according to the transliteration name recognition rule, and the recognition accuracy rate of the named entity recognition of Chinese at the present stage is improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
Fig. 1 is a schematic flowchart of a method for generating a transliteration name recognition rule according to an embodiment of the present application;
fig. 2 is a schematic structural diagram of a transliteration name recognition rule generating device according to an embodiment of the present application;
fig. 3 is a schematic diagram of a generating device provided in an embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
In order to explain the technical solution described in the present application, the following description will be given by way of specific examples.
Referring to fig. 1, a schematic flow chart of a transliteration name recognition rule provided in an embodiment of the present application is applied to a generating device, where the generating device includes a server or a terminal device, the server may be a computing device such as a cloud server, and the terminal device may be a computing device such as a desktop computer, a notebook computer, and a palm computer, and the method includes the following steps:
and step S101, obtaining a text sample to be processed.
The text sample to be processed is marked with a first transliteration name text.
It can be understood that, according to the transliteration name type to be identified, the embodiment of the present application collects corresponding corpora, where the corpora include a large number of text samples to be processed, and each text sample to be processed includes a labeled transliteration name text.
And S102, extracting a first transliteration name text.
It can be understood that the transliteration name text marked out in the text sample to be processed is directly extracted.
Step S103, generating a transliteration name recognition rule according to the first transliteration name text,
and the transliteration name identification rule is used for identifying a second transliteration name text in the text to be processed.
In a specific application, generating a transliteration name recognition rule according to a first transliteration name text comprises the following steps:
firstly, a transliteration name database is constructed according to the first transliteration name text.
And secondly, determining the frequency corresponding to each character in the first transliteration name text in the transliteration name database.
And thirdly, establishing a transliteration name recognition rule according to the frequency corresponding to each word in the first transliteration name text.
It is understood that a word library of common names is constructed from the transliterated name corpus in step S101. Firstly, removing special characters in a transliteration name; for example, if the English transliteration name has a word representing the title, such as 'or' -and the like, and the Burma transliteration name has a word representing the title, such as 'Du' or 'Wu', special characters need to be removed according to the definition in different transliteration names; secondly, counting the characters appearing in the transliteration name; then, counting common words in the transliteration name, such as 'Mike', 'Michael' and the like; and finally, forming a word stock of common names by the counted words and words.
Names of different countries have different naming rules, for example, the names of Burma people are only names without surnames, the names have at least one character, and the names can be as long as six or seven characters, but before the Burma people title, women will add 'du', 'ma' and the like, men will add 'Guo', 'Wu' and the like, and the social status according to the ages needs to be modified appropriately, so the rules need to be considered when the Burma people name is identified; when the english name is transliterated, a connector such as ". or" - "may appear in the middle of the name, so that the influence of the special character needs to be considered when the english name is transliterated and identified. According to the name transliteration characteristics of different countries, corresponding identification rules are established, although the purpose of full identification cannot be achieved, the rules can identify partial effective names, and the error rate is reduced.
Illustratively, constructing a transliteration name recognition rule according to the frequency corresponding to each word in the first transliteration name text comprises the following steps:
1. obtaining a frequency threshold according to the frequency corresponding to each word in the first transliteration name text;
2. the transliteration name recognition rule is obtained according to the following formula:
wherein i =1, 2.. once, n, min (pi) represents the minimum frequency of words in the text to be processed, PThreshold valueAnd representing a frequency threshold, and when the minimum frequency of the words in the phrases of the text to be processed is greater than the frequency threshold, determining that the phrases are second transliteration name texts.
It can be understood that, in the embodiment of the present application, the recognition of the transliteration name is implemented by using the degree of solidification of the phrase, and when the minimum frequency of the words in the phrase of the text to be processed is greater than the frequency threshold, it indicates that the phrase is the transliteration name text.
In an optional implementation manner, before extracting the first transliteration name text, the method further includes:
dividing a text sample to be processed into a first text sample and a second text sample, wherein the first text sample contains a name of a person, and the second text sample does not contain the name of the person;
and secondly, inputting the first text sample into a preset classification model for training to obtain a name recognition model.
The person name recognition model is used as a pre-screening rule corresponding to the transliteration name recognition rule, and before the transliteration name recognition rule recognizes a second transliteration name text in the text to be processed, the person name in the text to be processed is screened out.
It is understood that the text sample to be processed can be divided into two types including a person name and a non-person name according to the corpus. The name recognition model is mainly used for reducing the ambiguity or the problem of false extraction of the transliterated names in the sentences, for example, the transliterated name ' bell ' represents the name of a person in the sentence ' bell has said ' xxx ', and does not represent the name of the person in the sentence ' bell city is a beautiful city ', so whether the words represent the name of the person and the context of the sentences is large. A classification model is introduced into the project, whether the sentence contains the name of a person or not is judged, and then further transliteration name recognition is carried out.
Common classification models include an SVM, a maximum entropy model, a Bayesian model and the like, which can be adopted. Establishing a corresponding classification model whether the name of the person is contained or not according to the corresponding corpus;
in another alternative embodiment, after generating the transliteration name recognition rule based on the first transliteration name text, the method further includes:
firstly, extracting left adjacent characters and right adjacent characters of a first transliteration name from a text sample to be processed;
and secondly, forming a prefix corpus according to the left adjacent characters.
And thirdly, forming a suffix corpus according to the right adjacent characters.
It can be understood that the corresponding transliteration name is obtained by manual collection according to the data in the corpus, and the corresponding words or characters before and after the transliteration name are obtained according to the word segmentation; taking the English transliteration name as an example, "Ohio State Long Mike Germany, law enforcement agencies continue to worry about possible violent incidents in the next few days, and will continue to maintain the state party building alert level. "wherein" Mike. Dewen "is the name of a person, the word preceding it is" State Long ", and the word following it is" say ". Collecting a large number of transliteration names according to the corpus data to form a transliteration name corpus; forming a prefix language database according to the words or characters before the transliteration name; and forming a suffix corpus according to the words or characters after the transliteration name.
And fourthly, generating a check rule corresponding to the transliteration name recognition rule based on the prefix corpus and the suffix corpus, wherein the check rule is used for checking the second transliteration name text after the transliteration name recognition mechanism recognizes the second transliteration name text in the text to be processed.
It will be appreciated that the set of left and right neighbors of the transliteration name to be determined may be calculated from the prefix corpus and the suffix corpus. The part of speech of the prefix name of the transliterated name to be determined can be nouns, verbs and the like, but certainly not adverbs; the suffix part of the transliterated name to be determined may be a verb, an assistant, etc. The data in the corpus is analyzed for part of speech based on the prefix corpus and the suffix corpus, and the analysis should be done simultaneously in step1 since the part of speech needs to be combined with the context. And judging whether the prefix words and the suffix words of the transliteration name identified by the current sentence meet the requirements or not according to the data parts of speech in the prefix pre-material library and the suffix corpus, judging the transliteration name when the conditions are met, and judging the transliteration name not when the conditions are not met.
In the embodiment of the application, the transliteration name recognition rule is generated through training sample data, so that the names of foreigners containing transliterations in the Chinese text are recognized according to the transliteration name recognition rule, and the recognition accuracy rate of the named entity recognition of Chinese at the present stage is improved.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.
The following describes a transliteration name recognition rule generation apparatus provided in an embodiment of the present application.
The transliteration name recognition rule generation device of the present embodiment corresponds to the transliteration name recognition rule generation method described above.
Fig. 2 is a schematic structural diagram of an apparatus for generating a transliteration name recognition rule according to an embodiment of the present application, where the apparatus may be specifically integrated in a generating device, and the apparatus may include:
the obtaining module 21 is configured to obtain a text sample to be processed, where the text sample to be processed is marked with a first transliteration name text;
an extracting module 22, configured to extract the first transliteration name text;
the generating module 23 is configured to generate the transliteration name recognition rule according to the first transliteration name text, where the transliteration name recognition rule is used to recognize a second transliteration name text in the text to be processed.
In one possible implementation, the generating module includes:
the construction unit is used for constructing a transliteration name database according to the first transliteration name text;
the determining unit is used for determining the frequency corresponding to each word in the first transliteration name text in the transliteration name database;
and the generating unit is used for generating a transliteration name recognition rule according to the frequency corresponding to each word in the first transliteration name text.
In one possible implementation, the generating unit includes:
the calculation subunit is used for obtaining a frequency threshold value according to the frequency corresponding to each word in the first transliteration name text;
a rule generating subunit, configured to obtain the transliteration name identification rule according to the following formula:
where i =1, 2.,. n, min (pi) denotes the minimum frequency of words in the phrase of the text to be processed, PThreshold valueAnd representing a frequency threshold, and when the minimum frequency of the words in the phrases of the text to be processed is greater than the frequency threshold, determining that the phrases in the text to be processed are second transliterated name texts.
In one possible implementation, the apparatus further includes:
the dividing module is used for dividing the text sample to be processed into a first text sample and a second text sample, wherein the first text sample contains a name of a person, and the second text sample does not contain the name of the person;
the training module is used for inputting the first text sample into a preset classification model for training to obtain a name recognition model;
the person name recognition model is used as a pre-screening rule corresponding to the transliteration name recognition rule, and before the transliteration name recognition rule recognizes a second transliteration name text in the text to be processed, the person name in the text to be processed is screened out.
In one possible implementation, the apparatus further includes:
the extraction module is used for extracting left adjacent characters and right adjacent characters of the first transliteration name from the text sample to be processed;
the first forming submodule is used for forming a prefix corpus according to the left adjacent characters;
a second forming submodule for forming a suffix corpus from the right glyph;
and the verification generation submodule is used for generating a verification rule corresponding to the transliteration name identification rule based on the prefix corpus and the suffix corpus, and the verification rule is used for verifying a second transliteration name text in the text to be processed after the transliteration name identification mechanism identifies the second transliteration name text.
It should be noted that, for the information interaction, execution process, and other contents between the above-mentioned devices/units, the specific functions and technical effects thereof are based on the same concept as those of the embodiment of the method of the present application, and specific reference may be made to the part of the embodiment of the method, which is not described herein again.
Fig. 3 is a schematic diagram of a generating device 3 provided in an embodiment of the present application. As shown in fig. 3, the generation device 3 of this embodiment includes: a processor 30, a memory 31 and a computer program 32 stored in said memory 31 and executable on said processor 30. The steps in the above-described method embodiments are implemented when the computer program 32 is executed by the processor 30. Alternatively, the processor 30 implements the functions of the modules/units in the above-described device embodiments when executing the computer program 32.
Illustratively, the computer program 32 may be partitioned into one or more modules/units that are stored in the memory 31 and executed by the processor 30 to accomplish the present application. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution of the computer program 32 in the generating device 3.
The generating device 3 may be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The generating device 3 may include, but is not limited to, a processor 30, a memory 31. It will be appreciated by those skilled in the art that fig. 3 is only an example of the generating device 3, and does not constitute a limitation of the generating device 3, and may comprise more or less components than those shown, or some components may be combined, or different components, for example, the generating device 3 may further comprise an input-output device, a network access device, a bus, etc.
The Processor 30 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 31 may be an internal storage unit of the generating device 3, such as a hard disk or a memory of the generating device 3. The memory 31 may also be an external storage device of the generating device 3, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the generating device 3. Further, the memory 31 may also include both an internal storage unit of the generating device 3 and an external storage device. The memory 31 is used for storing the computer program and other programs and data required by the generating device 3. The memory 31 may also be used to temporarily store data that has been output or is to be output.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed terminal device and method may be implemented in other ways. For example, the above-described terminal device embodiments are merely illustrative, and for example, the division of the modules or units is only one logical function division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated modules/units, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. Based on such understanding, all or part of the flow in the method of the embodiments described above can be realized by a computer program, which can be stored in a readable storage medium, where the readable storage medium can be a computer readable storage medium, and when the computer program is executed by a processor, the steps of the method embodiments described above can be realized. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.
Claims (8)
1. A method for generating a transliterated name recognition rule is characterized by comprising the following steps:
acquiring a text sample to be processed, wherein the text sample to be processed is marked with a first transliteration name text;
extracting the first transliteration name text;
and generating the transliteration name recognition rule according to the first transliteration name text, wherein the transliteration name recognition rule is used for recognizing a second transliteration name text in the text to be processed.
2. The method for generating a transliteration name recognition rule according to claim 1, wherein generating the transliteration name recognition rule based on the transliteration name text comprises:
constructing a transliteration name database according to the first transliteration name text;
determining the frequency corresponding to each word in the first transliteration name text in the transliteration name database;
and generating a transliteration name recognition rule according to the frequency corresponding to each word in the first transliteration name text.
3. The method for generating a transliteration name recognition rule according to claim 2, wherein constructing a transliteration name recognition rule according to the frequency corresponding to each word in the first transliteration name text comprises:
obtaining a frequency threshold according to the frequency corresponding to each word in the first transliteration name text;
the transliteration name recognition rule is obtained according to the following formula:
where i =1, 2.,. n, min (pi) denotes the minimum frequency of words in the phrase of the text to be processed, PThreshold valueAnd representing a frequency threshold, and when the minimum frequency of the words in the phrases of the text to be processed is greater than the frequency threshold, determining that the phrases in the text to be processed are second transliterated name texts.
4. The method for generating a transliteration name recognition rule according to claim 1, wherein before extracting the first transliteration name text, the method further comprises:
dividing the text sample to be processed into a first text sample and a second text sample, wherein the first text sample contains a name of a person, and the second text sample does not contain the name of the person;
inputting the first text sample into a preset classification model for training to obtain a name recognition model;
the person name recognition model is used as a pre-screening rule corresponding to the transliteration name recognition rule, and before the transliteration name recognition rule recognizes a second transliteration name text in the text to be processed, the person name in the text to be processed is screened out.
5. The transliteration name recognition rule generating method of claim 1, wherein after generating the transliteration name recognition rule based on the first transliteration name text, further comprising:
extracting left adjacent characters and right adjacent characters of the first transliteration name from a text sample to be processed;
forming a prefix corpus according to the left adjacent characters;
forming a suffix corpus according to the right adjacent characters;
and generating a check rule corresponding to the transliteration name identification rule based on the prefix corpus and the suffix corpus, wherein the check rule is used for checking the second transliteration name text after the transliteration name identification mechanism identifies the second transliteration name text in the text to be processed.
6. A method for generating a transliterated name recognition rule is characterized by comprising the following steps:
the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a text sample to be processed, and the text sample to be processed is marked with a first transliteration name text;
the extraction module is used for extracting the first transliteration name text;
and the generating module is used for generating the transliteration name recognition rule according to the first transliteration name text, and the transliteration name recognition rule is used for recognizing a second transliteration name text in the text to be processed.
7. Generating device comprising a memory, a processor, an image pick-up means and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1 to 5 when executing the computer program.
8. Storage medium, storing a computer program, characterized in that the computer program, when being executed by a processor, carries out the steps of the method according to any of the claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110242748.2A CN112883161A (en) | 2021-03-05 | 2021-03-05 | Transliteration name recognition rule generation method, transliteration name recognition rule generation device, transliteration name recognition rule generation equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110242748.2A CN112883161A (en) | 2021-03-05 | 2021-03-05 | Transliteration name recognition rule generation method, transliteration name recognition rule generation device, transliteration name recognition rule generation equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112883161A true CN112883161A (en) | 2021-06-01 |
Family
ID=76055430
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110242748.2A Pending CN112883161A (en) | 2021-03-05 | 2021-03-05 | Transliteration name recognition rule generation method, transliteration name recognition rule generation device, transliteration name recognition rule generation equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112883161A (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101145166A (en) * | 2007-11-13 | 2008-03-19 | 北京中搜在线软件有限公司 | Syllable drive based transliterated entity name computer automatic identification method |
| CN102033879A (en) * | 2009-09-27 | 2011-04-27 | 腾讯科技（深圳）有限公司 | Method and device for identifying Chinese name |
| CN102955775A (en) * | 2012-06-14 | 2013-03-06 | 华东师范大学 | Automatic foreign name identification and control method based on context semantics |
| CN104182423A (en) * | 2013-05-27 | 2014-12-03 | 华东师范大学 | Conditional random field-based automatic Chinese personal name recognition method |
| CN104657343A (en) * | 2013-11-15 | 2015-05-27 | 富士通株式会社 | Method and device for recognizing transliteration name |
| CN105868184A (en) * | 2016-05-10 | 2016-08-17 | 大连理工大学 | Chinese name recognition method based on recurrent neural network |
| CN108681537A (en) * | 2018-05-08 | 2018-10-19 | 中国人民解放军国防科技大学 | Chinese entity linking method based on neural network and word vector |
| CN111222336A (en) * | 2019-12-25 | 2020-06-02 | 北京明略软件***有限公司 | Method and device for identifying unknown entity |
-
2021
- 2021-03-05 CN CN202110242748.2A patent/CN112883161A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101145166A (en) * | 2007-11-13 | 2008-03-19 | 北京中搜在线软件有限公司 | Syllable drive based transliterated entity name computer automatic identification method |
| CN102033879A (en) * | 2009-09-27 | 2011-04-27 | 腾讯科技（深圳）有限公司 | Method and device for identifying Chinese name |
| CN102955775A (en) * | 2012-06-14 | 2013-03-06 | 华东师范大学 | Automatic foreign name identification and control method based on context semantics |
| CN104182423A (en) * | 2013-05-27 | 2014-12-03 | 华东师范大学 | Conditional random field-based automatic Chinese personal name recognition method |
| CN104657343A (en) * | 2013-11-15 | 2015-05-27 | 富士通株式会社 | Method and device for recognizing transliteration name |
| CN105868184A (en) * | 2016-05-10 | 2016-08-17 | 大连理工大学 | Chinese name recognition method based on recurrent neural network |
| CN108681537A (en) * | 2018-05-08 | 2018-10-19 | 中国人民解放军国防科技大学 | Chinese entity linking method based on neural network and word vector |
| CN111222336A (en) * | 2019-12-25 | 2020-06-02 | 北京明略软件***有限公司 | Method and device for identifying unknown entity |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109359175B (en) | Electronic device, litigation data processing method, and storage medium | |
| WO2020224219A1 (en) | Chinese word segmentation method and apparatus, electronic device and readable storage medium | |
| CN109299280B (en) | Short text clustering analysis method and device and terminal equipment | |
| CN111460820B (en) | Network space security domain named entity recognition method and device based on pre-training model BERT | |
| CN110083832B (en) | Article reprint relation identification method, device, equipment and readable storage medium | |
| CN111767714B (en) | Text smoothness determination method, device, equipment and medium | |
| CN110929520B (en) | Unnamed entity object extraction method and device, electronic equipment and storage medium | |
| CN110032734B (en) | Training method and device for similar meaning word expansion and generation of confrontation network model | |
| CN114186061A (en) | Statement intention prediction method, device, storage medium and computer equipment | |
| CN115840808A (en) | Scientific and technological project consultation method, device, server and computer-readable storage medium | |
| CN111177375A (en) | Electronic document classification method and device | |
| CN112783825A (en) | Data archiving method, data archiving device, computer device and storage medium | |
| KR102166102B1 (en) | Device and storage medium for protecting privacy information | |
| CN114722832A (en) | Abstract extraction method, device, equipment and storage medium | |
| CN116402166B (en) | Training method and device of prediction model, electronic equipment and storage medium | |
| CN112307175B (en) | Text processing method, text processing device, server and computer readable storage medium | |
| CN114842982B (en) | Knowledge expression method, device and system for medical information system | |
| CN116757207A (en) | ICD automatic coding method based on artificial intelligence and related equipment | |
| CN112182448A (en) | Page information processing method, device and equipment | |
| CN115630643A (en) | Language model training method and device, electronic equipment and storage medium | |
| CN115544214A (en) | Event processing method and device and computer readable storage medium | |
| CN112883161A (en) | Transliteration name recognition rule generation method, transliteration name recognition rule generation device, transliteration name recognition rule generation equipment and storage medium | |
| CN113158693A (en) | Uygur language keyword generation method and device based on Chinese keywords, electronic equipment and storage medium | |
| CN113836297A (en) | Training method and device for text emotion analysis model | |
| CN111611379A (en) | Text information classification method, device, equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210601 |