CN110046344B - Method for adding separator and terminal equipment - Google Patents
Method for adding separator and terminal equipment Download PDFInfo
- Publication number
- CN110046344B CN110046344B CN201910184608.7A CN201910184608A CN110046344B CN 110046344 B CN110046344 B CN 110046344B CN 201910184608 A CN201910184608 A CN 201910184608A CN 110046344 B CN110046344 B CN 110046344B
- Authority
- CN
- China
- Prior art keywords
- target
- matrix
- word
- separator
- preset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/166—Editing, e.g. inserting or deleting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/284—Lexical analysis, e.g. tokenisation or collocates
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Machine Translation (AREA)
Abstract
The invention is suitable for the technical field of artificial intelligence, and provides a method for adding separators and terminal equipment, wherein a plurality of target words are obtained by performing Word segmentation processing on target sentences, a position matrix corresponding to the target words is generated according to the positions of the target words in a preset Word set, and the position matrix of the target words is converted into Word vectors through a Word2Vec model; converting word vectors of a plurality of target words contained in the target statement into a mixing matrix corresponding to the target statement through a preset neural network model; and inputting the mixed matrix into a preset classifier model, outputting the probability of each target word corresponding to each separator, and adding the separator with the highest probability corresponding to the target word behind the target word so as to add the separator to the target sentence, so that the target sentence is separated by the separators of different types, and a user can read and understand the target sentence conveniently.
Description
Technical Field
The invention belongs to the field of artificial intelligence, and particularly relates to a method for adding separators and terminal equipment.
Background
In recent years, more and more voice recognition software can convert voice into characters, but after voice is converted into characters, the generated characters are often difficult to read smoothly by users because the voice recognition cannot add separators such as punctuation marks to the characters. Especially, when a large voice with a very short pause time is converted into a character at one time, the user is more difficult to read.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method for adding a separator and a terminal device, so as to solve the problem in the prior art that it is difficult for a user to read a character due to difficulty in automatically adding a separator.
A first aspect of the embodiments of the present invention provides a method for adding a separator, including:
acquiring a target sentence to which a separator is to be added, and performing word segmentation processing on the target sentence to generate a plurality of target words; generating a position matrix corresponding to the target Word and used for representing the position of the target Word in the Word set according to a preset Word set, and converting the position matrix of the target Word into a Word vector of the target Word through a preset Word2Vec model; respectively inputting word vectors of the target words into a preset neural network model according to the sequence of the target words from front to back and the sequence of the target words from back to front in the target sentences, generating a forward matrix and a backward matrix corresponding to the target sentences, and splicing the forward matrix and the backward matrix to generate a mixed matrix corresponding to the target sentences; and inputting the mixing matrix into a preset classifier model, outputting the probability of each target word corresponding to each separator, and adding the separator with the highest probability corresponding to the target word behind the target word to add the separator for the target sentence.
A second aspect of an embodiment of the present invention provides an apparatus for adding a separator, including: the acquisition module is used for acquiring a target sentence to be added with a separator, and performing word segmentation processing on the target sentence to generate a plurality of target words; the conversion module is used for generating a position matrix which is corresponding to the target Word and used for representing the position of the target Word in the Word set according to a preset Word set, and converting the position matrix of the target Word into a Word vector of the target Word through a preset Word2Vec model; a calculation module, configured to input word vectors of the target words into a preset neural network model according to a sequence from front to back and a sequence from back to front of the target words in the target sentences, respectively, generate a forward matrix and a backward matrix corresponding to the target sentences, and splice the forward matrix and the backward matrix to generate a mixed matrix corresponding to the target sentences; and the adding module is used for inputting the mixing matrix into a preset classifier model, outputting the probability of each separator corresponding to each target word, and adding the separator with the highest probability corresponding to the target word behind the target word so as to add the separator to the target sentence.
A third aspect of the present invention provides a terminal device, including a memory and a processor, where the memory stores a computer program that is executable on the processor, and when the computer program is executed by the processor, the steps of the method provided in the first aspect of the present invention are implemented.
A fourth aspect of embodiments of the present invention provides a computer-readable storage medium, in which a computer program is stored, which, when executed by a processor, performs the steps of the method provided by the first aspect of embodiments of the present invention.
In the embodiment of the invention, a plurality of target words are obtained by performing Word segmentation processing on a target sentence, a position matrix corresponding to the target words is generated according to the positions of the target words in a preset Word set, and the position matrix of the target words is converted into Word vectors through a Word2Vec model; converting word vectors of a plurality of target words contained in the target statement into a mixing matrix corresponding to the target statement through a preset neural network model; and inputting the mixed matrix into a preset classifier model, outputting the probability of each target word corresponding to each separator, and adding the separator with the highest probability corresponding to the target word behind the target word so as to add the separator to the target sentence, so that the target sentence is separated by the separators of different types, and a user can read and understand the target sentence conveniently.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
FIG. 1 is a flow chart of an implementation of a method for adding delimiters according to an embodiment of the present invention;
fig. 2 is a flowchart illustrating a specific implementation of the method S104 for adding a separator according to an embodiment of the present invention;
FIG. 3 is a block diagram of an apparatus for adding delimiters according to an embodiment of the present invention;
fig. 4 is a schematic diagram of a terminal device according to an embodiment of the present invention.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.
In order to explain the technical means of the present invention, the following description will be given by way of specific examples.
Fig. 1 shows an implementation flow of a method for adding a separator according to an embodiment of the present invention, where the method flow includes steps S101 to S106. The specific implementation principle of each step is as follows.
S101: and acquiring a target sentence to be added with a separator, and performing word segmentation processing on the target sentence to generate a plurality of target words.
In an embodiment of the present invention, the target sentence is a sentence lacking a separator. For example, after a long-length speech is converted into a long-length sentence through the speech recognition module, the sentence often lacks punctuation marks, and in order to make it easier for a user to read the sentence, punctuation marks may be added to the sentence according to an embodiment of the present invention, where the sentence is a target sentence to which a separator is to be added in the embodiment of the present invention, and various kinds of punctuation marks are separators to be added. It should be understood that the separator in the embodiment of the present invention is not limited to punctuation marks, and any coincidence used for dividing a sentence can be used as the separator in the embodiment of the present invention, and the type of the separator may be different in different application scenarios.
In the embodiment of the present invention, the target sentence needs to be segmented by the existing segmentation toolkit, for example, the jieba chinese segmentation tool, to obtain a plurality of target words constituting the target sentence.
S102, generating a position matrix corresponding to the target Word and used for representing the position of the target Word in the Word set according to a preset Word set, and converting the position matrix of the target Word into a Word vector of the target Word through a preset Word2Vec model.
It is understood that the target words in text form cannot be brought into the subsequent data calculation, and thus each target word needs to be converted into a word vector. In the embodiment of the invention, because the neural network model is required to be used in the subsequent steps, in order to keep the neural network model consistent in the training process and the actual calculation process of the target sentence, the target word and the vocabulary of the training sentence required to be used in the training process need to be positioned through the same word set, and after the positioning, the target word and the vocabulary are converted into the word vector according to the positioning condition.
Optionally, in the embodiment of the present invention, a dictionary containing a large number of words may be prepared in advance as a preset word set, and obviously, each word in the word set is arranged in sequence and has a corresponding position in the word set. According to the embodiment of the invention, a large number of articles are collected in advance, words contained in the articles are counted, the occurrence frequency corresponding to each word is calculated, and finally the words contained in the articles are stored in the dictionary according to the sequence of the occurrence frequency from large to small to generate the word set in the embodiment of the invention. For example, assuming that a word set contains 5 words (of course, the number of words in the word set actually used is far more than 5, and this is only for convenience of description), the word in the word set that is ranked first is used as: [1,0,0,0,0] indicates that the word ranked second is: [0,1,0,0,0] and so on.
It can be understood that, by the above method, individual target words can be converted into a position matrix for representing the positions of the target words in the word set, which is beneficial to converting the target words in the target sentences and the words of the training sentences needed in the training process into a matrix form in a unified manner, and improving the reliability of subsequent calculation.
In the embodiment of the present invention, each position matrix needs to be converted into a Word vector through the existing Word2Vec model, so that the target Word can be brought into the subsequent neural network model for further calculation, and the process of generating the Word vector through the Word2Vec model is the prior art, and therefore is not described herein again.
S103, respectively inputting the word vectors of the target words into a preset neural network model according to the sequence of the target words from front to back and the sequence of the target words from back to front in the target sentences, generating a forward matrix and a backward matrix corresponding to the target sentences, and splicing the forward matrix and the backward matrix to generate a mixed matrix corresponding to the target sentences.
In the embodiment of the present invention, it is considered that the corresponding segmenters of the same word at different positions in the target sentence may be different, so that the word vectors of the target words cannot be analyzed separately, and one target word must be analyzed in combination with the context thereof. Alternatively, embodiments of the present invention employ long and short term memory networks that incorporate a mechanism of attention. Wherein the long-term and short-term memory network comprises: input layer, hidden layer, attention mechanism layer, and output gate processing.
It can be understood that, after the word vectors of the target words are input into the preset neural network model according to the sequence of the target words from front to back in the target sentence, h is used in the embodiment of the present invention due to different input times of the word vectorstRepresenting the word vector entering the input layer at time t.
Optionally, in the hidden layer, by the formula: y ist＝relu((tanh(ht-1·Wp+Bp)+tanh(ht·Wq+Bq)·Wy+By) Computing an output result of a hidden layer corresponding to the word vector input at the time t; wherein, ytOutput result of the hidden layer corresponding to the word vector input for time t, Wp、WqAnd WyAre respectively 3 preset weight matrixes, Bp、BqAnd ByRespectively 3 preset bias matrixes. Relu is a preset linear rectification function, tanh is a hyperbolic tangent function, htRepresenting the word vector input into the input layer at time t, which is an integer greater than 2.
Alternatively, in the attention floor process, an attention mechanism is introduced. In particular, by the formula
Calculating an attention reference matrix corresponding to the time t, wherein alphatFor attention reference matrix, y, corresponding to time ttAnd outputting the output result of the hidden layer corresponding to the word vector input at the moment t. Then, through the formula: rt＝tanh(ht*αt)·Wa+BaCalculating an attention matrix corresponding to the time t, wherein RtFor the attention matrix corresponding to time t, WaIs a predetermined weight matrix, BaIs a predetermined bias matrix, htRepresenting the word vector input into the input layer at time t. Denotes convolution operation.
Optionally, in the output layer, htAnd performing point multiplication on the word vector input into the input layer at the moment t and the attention matrix corresponding to the moment t to generate output layer results corresponding to the word vector input into the input layer at the moment t, and splicing the output layer results corresponding to the word vectors input into the input layer at all moments to generate a forward matrix corresponding to the target sentence.
It is understood that, in the above description, the word vector of each target word is input to the preset neural network model according to the sequence of the target word from front to back in the target sentence, so that the forward matrix is finally generated, and similarly, when the word vector of each target word is input to the preset neural network model according to the sequence of the target word from back to front in the target sentence, a backward matrix is finally generated.
In the embodiment of the invention, in order to more accurately and comprehensively describe the characteristics of the target statement, the forward matrix and the backward matrix are spliced to generate the mixed matrix corresponding to the target statement. It will be appreciated that in the subsequent computation process, the target statement is characterized using the mixing matrix.
It can be understood that before obtaining the target sentence to which the separator is to be added, the above-mentioned weight matrix and the bias matrix need to be obtained through a training process of training data, so as to generate the above-mentioned preset neural network model, where a training process of the neural network includes:
the method comprises the following steps: and acquiring a plurality of training sentence matrixes and training mixed matrixes corresponding to the training sentence matrixes.
Preferably, the number of the training sentence matrixes is more than 5000 so as to improve the recognition accuracy of the neural network.
Notably, the formats of the training sentence matrixes are the same, so that it can be ensured that neurons corresponding to elements in the weight matrixes and the bias matrix are fixed when the neural network is trained, so as to ensure the accuracy of the neural network.
Step two, repeatedly executing the following steps until the adjusted long-term and short-term memory network meets the preset convergence condition: and taking the training sentence matrix as the input of a long-short term memory network, taking the training mixed matrix as the output of the long-short term memory network, and updating the weights corresponding to the neural units in the long-short term memory network by a back propagation method.
Step three: and outputting the adjusted long-term and short-term memory network as the preset neural network model.
And S104, inputting the mixing matrix into a preset classifier model, outputting the probability of each separator corresponding to each target word, and adding the separator with the highest probability corresponding to the target word behind the target word to add the separator for the target sentence.
Optionally, the preset classifier model is represented by a formula:
calculating the probability corresponding to the mixing matrixA matrix; the sigma (j) is a probability value corresponding to the jth element in the probability matrix; z is a radical ofjThe parameter is a parameter corresponding to the jth element in a preset parameter matrix; m is the number of elements in the parameter matrix, xiAnd e is a natural constant for the ith element in the mixing matrix.
Further, according to the position of the target word in the target sentence, the probability of each separator corresponding to each target word is read from the probability matrix.
In the embodiment of the present invention, each row element in the probability matrix corresponds to the same target word, and each column element corresponds to the same delimiter, so that one element in the probability matrix represents: the probability that a certain target word corresponds to a certain delimiter.
Further, a separator with the highest probability corresponding to the target word is added after the target word, so as to add a separator for the target sentence.
Illustratively, assume that the target statement is: "you are asking you to ask you where you are going to work" in the safe bank, "it is assumed that, according to each element in the probability matrix, it is known that" you are "the highest-probability separator" corresponding to "you is" punctuation-free "," i "the highest-probability separator" punctuation-free ", that" the safe bank "corresponds to" punctuation-free "," work "corresponds to" the highest-probability separator ",", "," "asking" the corresponding highest-probability separator "is", "," "you" corresponds to "the highest-probability separator" punctuation-free "," where "corresponds to" the highest-probability separator "is" punctuation-free ", and" going to work "corresponds to" the highest-probability separator "is"? ". So finally, the target sentence after the addition of the separators is output as: "do you work at a safe bank, ask for a question, where do you work? ".
In the embodiment of the invention, a plurality of target words are obtained by performing Word segmentation processing on a target sentence, a position matrix corresponding to the target words is generated according to the positions of the target words in a preset Word set, and the position matrix of the target words is converted into Word vectors through a Word2Vec model; converting word vectors of a plurality of target words contained in the target statement into a mixing matrix corresponding to the target statement through a preset neural network model; and inputting the mixed matrix into a preset classifier model, outputting the probability of each target word corresponding to each separator, and adding the separator with the highest probability corresponding to the target word behind the target word so as to add the separator to the target sentence, so that the target sentence is separated by the separators of different types, and a user can read and understand the target sentence conveniently.
In the above embodiment, one method of calculating the probability that each target word corresponds to each separator in S104 is described, and other methods of calculating the probability that each target word corresponds to each separator exist, as another embodiment of the present invention, as shown in fig. 2, S104 further includes:
s1041, inputting the mixed matrix into a preset conditional random field model, and outputting the score value of each separator corresponding to each target word.
Optionally by means of a formula
Calculating the score value of each target word corresponding to each separator, wherein score (i, j) is the score value of target word i corresponding to separator j, m is the total amount of the separators, n is the total amount of the target words, τ is a preset coefficient, and f isi(i, j) is a characteristic function corresponding to the target word i, and the characteristic function is used for representing the distribution situation of the target word corresponding to each separator based on the training data, xiIs the ith element in the mixing matrix.
And S1042, calculating an index value corresponding to each score value according to a preset index function, and performing normalization processing on the index value to serve as the probability that each target word corresponds to each separator.
Optionally by formula
And calculating the probability of each target word corresponding to each separator, wherein p (i, j) is the probability of the target word i corresponding to the separator j, and score (i, j) is the fraction value of the target word i corresponding to the separator j.
It can be understood that, in the embodiment of the present invention, before obtaining the target sentence to which the separator is to be added, the conditional random field model needs to be trained, and the training process includes: acquiring a plurality of random field training sentences, wherein the random field training sentences comprise a plurality of training words, and each training word corresponds to a score value of more than one separator; obviously, a feature function corresponding to the target word i can be fitted according to the plurality of random field training sentences by using the conventional maximum likelihood estimation method, so that the preset conditional random field model is generated.
Fig. 3 is a block diagram illustrating a structure of a separator adding device according to an embodiment of the present invention, which corresponds to the method for adding separators described in the foregoing embodiments.
Referring to fig. 3, the apparatus includes:
an obtaining module 301, configured to obtain a target sentence to which a separator is to be added, and perform word segmentation processing on the target sentence to generate a plurality of target words;
a conversion module 302, configured to generate, according to a preset Word set, a position matrix corresponding to the target Word and used for representing the position of the target Word in the Word set, and convert the position matrix of the target Word into a Word vector of the target Word through a preset Word2Vec model;
a calculating module 303, configured to input word vectors of the target words into a preset neural network model according to a sequence from front to back and a sequence from back to front of the target words in the target sentence, respectively, generate a forward matrix and a backward matrix corresponding to the target sentence, and splice the forward matrix and the backward matrix to generate a mixed matrix corresponding to the target sentence;
an adding module 304, configured to input the mixing matrix into a preset classifier model, output probabilities of the target words corresponding to the separators, and add the separator with the highest probability corresponding to the target word after the target word, so as to add a separator to the target sentence.
Optionally, the neural network model is a long-short term memory network containing an attention mechanism;
the device further comprises:
the training acquisition module is used for acquiring a plurality of training sentence matrixes and training mixed matrixes corresponding to the training sentence matrixes;
the loop module is used for repeatedly executing the following steps until the adjusted long-term and short-term memory network meets the preset convergence condition: taking the training statement matrix as the input of a long-short term memory network, taking the training mixed matrix as the output of the long-short term memory network, and updating the weights corresponding to each neural unit in the long-short term memory network by a back propagation method;
and the output module is used for outputting the adjusted long-term and short-term memory network as the preset neural network model.
Optionally, the calculation module is specifically configured to:
by the formula:
calculating a probability matrix corresponding to the mixing matrix; the sigma (j) is a probability value corresponding to the jth element in the probability matrix; z is a radical ofjThe parameter is a parameter corresponding to the jth element in a preset parameter matrix; m is the number of elements in the parameter matrix, xiThe ith element in the mixing matrix is shown, and e is a natural constant;
and reading the probability of each separator corresponding to each target word from the probability matrix according to the position of the target word in the target sentence.
Optionally, the inputting the mixing matrix into a preset classifier model and outputting the probability that each target word corresponds to each separator includes:
inputting the mixed matrix into a preset conditional random field model, and outputting score values of the target words corresponding to the separators; and calculating an index value corresponding to each score value according to a preset index function, and performing normalization processing on the index value to serve as the probability that each target word corresponds to each separator. Optionally, before the obtaining the target statement to which the separator is to be added, the method further includes: acquiring a plurality of random field training sentences, wherein the random field training sentences comprise a plurality of training words, and each training word corresponds to a score value of more than one separator; and fitting the preset conditional random field model according to the plurality of random field training sentences by a maximum likelihood estimation method.
In the embodiment of the invention, a plurality of target words are obtained by performing Word segmentation processing on a target sentence, a position matrix corresponding to the target words is generated according to the positions of the target words in a preset Word set, and the position matrix of the target words is converted into Word vectors through a Word2Vec model; converting word vectors of a plurality of target words contained in the target statement into a mixing matrix corresponding to the target statement through a preset neural network model; and inputting the mixed matrix into a preset classifier model, outputting the probability of each target word corresponding to each separator, and adding the separator with the highest probability corresponding to the target word behind the target word so as to add the separator to the target sentence, so that the target sentence is separated by the separators of different types, and a user can read and understand the target sentence conveniently.
Fig. 4 is a schematic diagram of a terminal device according to an embodiment of the present invention. As shown in fig. 4, the terminal device 4 of this embodiment includes: a processor 40, a memory 41 and a computer program 42 stored in said memory 41 and executable on said processor 40, such as a separator adding program. The processor 40, when executing the computer program 42, implements the steps in each of the above-described embodiments of the method of adding delimiters, such as the steps 101 to 104 shown in fig. 1. Alternatively, the processor 40, when executing the computer program 42, implements the functions of the modules/units in the above-mentioned device embodiments, such as the functions of the units 301 to 304 shown in fig. 3.
Illustratively, the computer program 42 may be partitioned into one or more modules/units that are stored in the memory 41 and executed by the processor 40 to implement the present invention. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program 42 in the terminal device 4.
The terminal device 4 may be a desktop computer, a notebook, a palm computer, a cloud server, or other computing devices. The terminal device may include, but is not limited to, a processor 40, a memory 41. Those skilled in the art will appreciate that fig. 4 is merely an example of a terminal device 4 and does not constitute a limitation of terminal device 4 and may include more or fewer components than shown, or some components may be combined, or different components, e.g., the terminal device may also include input-output devices, network access devices, buses, etc.
The Processor 40 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 41 may be an internal storage unit of the terminal device 4, such as a hard disk or a memory of the terminal device 4. The memory 41 may also be an external storage device of the terminal device 4, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the terminal device 4. Further, the memory 41 may also include both an internal storage unit and an external storage device of the terminal device 4. The memory 41 is used for storing the computer program and other programs and data required by the terminal device. The memory 41 may also be used to temporarily store data that has been output or is to be output.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
The integrated modules/units, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. Based on such understanding, all or part of the flow of the method according to the above embodiments may be implemented by a computer program, which may be stored in a computer-readable storage medium, to instruct related hardware.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present invention, and are intended to be included within the scope of the present invention.
Claims (8)
1. A method of adding delimiters, comprising:
acquiring a target sentence to be added with a separator, and performing word segmentation processing on the target sentence to generate a plurality of target words;
generating a position matrix corresponding to the target Word and used for representing the position of the target Word in the Word set according to a preset Word set, and converting the position matrix of the target Word into a Word vector of the target Word through a preset Word2Vec model;
respectively inputting word vectors of the target words into a preset neural network model according to the sequence of the target words from front to back and the sequence of the target words from back to front in the target sentences, generating a forward matrix and a backward matrix corresponding to the target sentences, and splicing the forward matrix and the backward matrix to generate a mixed matrix corresponding to the target sentences;
inputting the mixing matrix into a preset classifier model, outputting the probability of each target word corresponding to each separator, and adding the separator with the highest probability corresponding to the target word behind the target word to add the separator to the target sentence;
the inputting the mixing matrix into a preset classifier model and outputting the probability that each target word corresponds to each separator comprises:
by the formula:
calculating a probability matrix corresponding to the mixing matrix; the sigma (j) is a probability value corresponding to the jth element in the probability matrix; z is a radical of formulajThe parameter is a parameter corresponding to the jth element in a preset parameter matrix; m is the number of elements in the parameter matrix, xiThe ith element in the mixing matrix is shown, and e is a natural constant;
and reading the probability of each separator corresponding to each target word from the probability matrix according to the position of the target word in the target sentence.
2. The method of adding delimiters of claim 1, wherein said neural network model is a long short term memory network including attention mechanism;
before the obtaining the target sentence to which the separator is to be added, the method further includes:
acquiring a plurality of training sentence matrixes and training mixed matrixes corresponding to the training sentence matrixes;
repeatedly executing the following steps until the adjusted long-term and short-term memory network meets the preset convergence condition:
taking the training statement matrix as the input of a long-short term memory network, taking the training mixed matrix as the output of the long-short term memory network, and updating the weights corresponding to each neural unit in the long-short term memory network by a back propagation method;
and outputting the adjusted long-term and short-term memory network as the preset neural network model.
3. The method for adding separators according to claim 1, wherein said inputting the mixing matrix into a preset classifier model and outputting the probability of each separator corresponding to each target word comprises:
inputting the mixed matrix into a preset conditional random field model, and outputting score values of the target words corresponding to the separators;
and calculating an index value corresponding to each score value according to a preset index function, and performing normalization processing on the index value to serve as the probability that each target word corresponds to each separator.
4. A method of adding delimiters as claimed in claim 3, wherein before said obtaining a target sentence to which a delimiter is to be added, further comprising:
acquiring a plurality of random field training sentences, wherein the random field training sentences comprise a plurality of training words, and each training word corresponds to a score value of more than one separator;
and fitting the preset conditional random field model according to the plurality of random field training sentences by a maximum likelihood estimation method.
5. An apparatus for adding delimiters, the apparatus comprising:
the acquisition module is used for acquiring a target sentence to be added with a separator, and performing word segmentation processing on the target sentence to generate a plurality of target words;
the conversion module is used for generating a position matrix corresponding to the target Word and used for representing the position of the target Word in the Word set according to a preset Word set, and converting the position matrix of the target Word into a Word vector of the target Word through a preset Word2Vec model;
a calculation module, configured to input word vectors of the target words into a preset neural network model according to a sequence from front to back and a sequence from back to front of the target words in the target sentences, respectively, generate a forward matrix and a backward matrix corresponding to the target sentences, and splice the forward matrix and the backward matrix to generate a mixed matrix corresponding to the target sentences;
the adding module is used for inputting the mixing matrix into a preset classifier model, outputting the probability of each separator corresponding to each target word, adding the separator with the highest probability corresponding to the target word behind the target word, and adding the separator to the target sentence;
the adding module is specifically configured to:
by the formula:
calculating a probability matrix corresponding to the mixing matrix; the sigma (j) is a probability value corresponding to the jth element in the probability matrix; z is a radical ofjThe parameter is a parameter corresponding to the jth element in a preset parameter matrix; m is the number of elements in the parameter matrix, xiThe ith element in the mixing matrix is shown, and e is a natural constant;
and reading the probability of each separator corresponding to each target word from the probability matrix according to the position of the target word in the target sentence.
6. The apparatus for adding delimiters as in claim 5, wherein said neural network model is a long-short term memory network including attention mechanism;
the device further comprises:
the training acquisition module is used for acquiring a plurality of training sentence matrixes and training mixed matrixes corresponding to the training sentence matrixes;
the loop module is used for repeatedly executing the following steps until the adjusted long-term and short-term memory network meets the preset convergence condition: taking the training statement matrix as the input of a long-short term memory network, taking the training mixed matrix as the output of the long-short term memory network, and updating the weights corresponding to each neural unit in the long-short term memory network by a back propagation method;
and the output module is used for outputting the adjusted long-term and short-term memory network as the preset neural network model.
7. A terminal device comprising a memory and a processor, the memory having stored therein a computer program operable on the processor, wherein the processor, when executing the computer program, implements the steps of:
acquiring a target sentence to be added with a separator, and performing word segmentation processing on the target sentence to generate a plurality of target words;
generating a position matrix corresponding to the target Word and used for representing the position of the target Word in the Word set according to a preset Word set, and converting the position matrix of the target Word into a Word vector of the target Word through a preset Word2Vec model;
respectively inputting word vectors of the target words into a preset neural network model according to the sequence of the target words from front to back and the sequence of the target words from back to front in the target sentences, generating a forward matrix and a backward matrix corresponding to the target sentences, and splicing the forward matrix and the backward matrix to generate a mixed matrix corresponding to the target sentences;
inputting the mixing matrix into a preset classifier model, outputting the probability of each target word corresponding to each separator, and adding the separator with the highest probability corresponding to the target word behind the target word to add the separator to the target sentence;
the inputting the mixing matrix into a preset classifier model and outputting the probability that each target word corresponds to each separator comprises:
by the formula:
calculating a probability matrix corresponding to the mixing matrix; the sigma (j) is a probability value corresponding to the jth element in the probability matrix; z is a radical ofjThe parameter is a parameter corresponding to the jth element in a preset parameter matrix; m is the number of elements in the parameter matrix, xiThe ith element in the mixing matrix is shown, and e is a natural constant;
and reading the probability of each separator corresponding to each target word from the probability matrix according to the position of the target word in the target sentence.
8. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910184608.7A CN110046344B (en) | 2019-03-12 | 2019-03-12 | Method for adding separator and terminal equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910184608.7A CN110046344B (en) | 2019-03-12 | 2019-03-12 | Method for adding separator and terminal equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110046344A CN110046344A (en) | 2019-07-23 |
| CN110046344B true CN110046344B (en) | 2022-05-06 |
Family
ID=67274738
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910184608.7A Active CN110046344B (en) | 2019-03-12 | 2019-03-12 | Method for adding separator and terminal equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110046344B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112988921A (en) * | 2019-12-13 | 2021-06-18 | 北京四维图新科技股份有限公司 | Method and device for identifying map information change |
| CN111261162B (en) * | 2020-03-09 | 2023-04-18 | 北京达佳互联信息技术有限公司 | Speech recognition method, speech recognition apparatus, and storage medium |
| CN113779964A (en) * | 2021-09-02 | 2021-12-10 | 中联国智科技管理(北京)有限公司 | Statement segmentation method and device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106528530A (en) * | 2016-10-24 | 2017-03-22 | 北京光年无限科技有限公司 | Method and device for determining sentence type |
| CN108932226A (en) * | 2018-05-29 | 2018-12-04 | 华东师范大学 | A kind of pair of method without punctuate text addition punctuation mark |
| CN109410949A (en) * | 2018-10-11 | 2019-03-01 | 厦门大学 | Content of text based on weighted finite state converter adds punctuate method |
-
2019
- 2019-03-12 CN CN201910184608.7A patent/CN110046344B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106528530A (en) * | 2016-10-24 | 2017-03-22 | 北京光年无限科技有限公司 | Method and device for determining sentence type |
| CN108932226A (en) * | 2018-05-29 | 2018-12-04 | 华东师范大学 | A kind of pair of method without punctuate text addition punctuation mark |
| CN109410949A (en) * | 2018-10-11 | 2019-03-01 | 厦门大学 | Content of text based on weighted finite state converter adds punctuate method |
Non-Patent Citations (2)
| Title |
|---|
| Punctuation Prediction using a Bidirectional Recurrent Neural Network with Part-of-Speech Tagging;Chin Char Juin et al.;《TENCON 2017 - 2017 IEEE Region 10 Conference》;20171105;第1806-1811页 * |
| 基于改进的多层BLSTM的中文分词和标点预测;李雅昆 等;《计算机应用》;20180510;第38卷(第5期);第1278-1282页，第1314页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110046344A (en) | 2019-07-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111695352A (en) | Grading method and device based on semantic analysis, terminal equipment and storage medium | |
| US11232263B2 (en) | Generating summary content using supervised sentential extractive summarization | |
| CN110765785B (en) | Chinese-English translation method based on neural network and related equipment thereof | |
| CN109299280B (en) | Short text clustering analysis method and device and terminal equipment | |
| CN105022754B (en) | Object classification method and device based on social network | |
| CN108763535B (en) | Information acquisition method and device | |
| CN110046344B (en) | Method for adding separator and terminal equipment | |
| CN109117474B (en) | Statement similarity calculation method and device and storage medium | |
| CN109684476B (en) | Text classification method, text classification device and terminal equipment | |
| US10685012B2 (en) | Generating feature embeddings from a co-occurrence matrix | |
| CN112434131B (en) | Text error detection method and device based on artificial intelligence and computer equipment | |
| CN111368037A (en) | Text similarity calculation method and device based on Bert model | |
| CN110941951B (en) | Text similarity calculation method, text similarity calculation device, text similarity calculation medium and electronic equipment | |
| CN110263328B (en) | Discipline capability type labeling method and device, storage medium and terminal equipment | |
| CN113158687B (en) | Semantic disambiguation method and device, storage medium and electronic device | |
| US20220058349A1 (en) | Data processing method, device, and storage medium | |
| CN111339775A (en) | Named entity identification method, device, terminal equipment and storage medium | |
| CN110874528A (en) | Text similarity obtaining method and device | |
| CN114840662A (en) | Event information extraction method and device and electronic equipment | |
| CN112307738B (en) | Method and device for processing text | |
| CN110929499B (en) | Text similarity obtaining method, device, medium and electronic equipment | |
| CN112559711A (en) | Synonymous text prompting method and device and electronic equipment | |
| CN114238611B (en) | Method, apparatus, device and storage medium for outputting information | |
| CN116187301A (en) | Model generation method, entity identification device, electronic equipment and storage medium | |
| CN111401069A (en) | Intention recognition method and intention recognition device for conversation text and terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |