CN113468289A - Training method and device of event detection model - Google Patents
Training method and device of event detection model Download PDFInfo
- Publication number
- CN113468289A CN113468289A CN202110840976.XA CN202110840976A CN113468289A CN 113468289 A CN113468289 A CN 113468289A CN 202110840976 A CN202110840976 A CN 202110840976A CN 113468289 A CN113468289 A CN 113468289A
- Authority
- CN
- China
- Prior art keywords
- information
- description
- numerical
- description information
- event
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 127
- 238000000034 method Methods 0.000 title claims abstract description 56
- 230000011218 segmentation Effects 0.000 claims description 87
- 238000010801 machine learning Methods 0.000 claims description 19
- 238000004590 computer program Methods 0.000 claims description 10
- 238000010586 diagram Methods 0.000 description 15
- 230000006870 function Effects 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 230000003287 optical effect Effects 0.000 description 4
- 239000000835 fiber Substances 0.000 description 2
- 238000011478 gradient descent method Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/31—Indexing; Data structures therefor; Storage structures
- G06F16/316—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/237—Lexical tools
- G06F40/242—Dictionaries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/258—Heading extraction; Automatic titling; Numbering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- General Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Databases & Information Systems (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Medical Informatics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a training method and device of an event detection model. One embodiment of the method comprises: acquiring a training sample set, wherein training samples in the training sample set comprise description information and title information corresponding to the same event; obtaining numerical description information corresponding to the description information and numerical title information corresponding to the title information in each training sample based on a preset dictionary; and taking the numerical description information as input, taking the numerical header information corresponding to the input numerical description information as expected output, and training to obtain the event detection model. The application provides a training method of an event detection model, and accuracy of the event detection model is improved.
Description
Technical Field
The embodiment of the application relates to the technical field of computers, in particular to a training method and a device of an event detection model, an event detection method and a device, and under a municipal administration scene.
Background
Event detection is used to classify events by their detailed description information. Under the urban management scene, the fine-grained event types can reach tens of thousands, and the fine-grained event types are difficult to specify manually. Currently, event detection is generally performed based on clustering, classification, and other methods. Under the urban management scene, the types of events can reach tens of thousands, and the accuracy of event detection is generally lower.
Disclosure of Invention
The embodiment of the application provides a training method and device of an event detection model, and an event detection method and device.
In a first aspect, an embodiment of the present application provides a method for training an event detection model, including: acquiring a training sample set, wherein training samples in the training sample set comprise description information and title information corresponding to the same event; obtaining numerical description information corresponding to the description information and numerical title information corresponding to the title information in each training sample based on a preset dictionary; and training to obtain the event detection model by using a machine learning algorithm and taking the numerical description information as input and the numerical header information corresponding to the input numerical description information as expected output.
In some embodiments, the obtaining, based on the preset dictionary, the digitized description information corresponding to the description information and the digitized heading information corresponding to the heading information in each training sample includes: based on a preset dictionary, segmenting the description information and the title information in each training sample respectively to obtain description segmentation information and title segmentation information; obtaining numerical description information according to index information of each participle in the description participle information in a preset dictionary; and obtaining the numerical header information according to the index information of each participle in the header participle information in a preset dictionary.
In some embodiments, the segmenting the description information and the title information in each training sample based on the preset dictionary to obtain the description segmentation information and the title segmentation information respectively includes: based on a preset dictionary, segmenting the description information and the title information in each training sample respectively to obtain an event description segmentation result and an event title segmentation result; and respectively processing the event description word segmentation result and the event title word segmentation result to obtain description word segmentation information with a first preset length and title word segmentation information with a second preset length.
In some embodiments, the training of obtaining the event detection model by using the digitized description information as an input and the digitized header information corresponding to the input digitized description information as an expected output includes: and training to obtain the event detection model by using a machine learning algorithm, wherein the numerical description information is used as the input of an encoder in the initial event detection model, the output of the encoder and the numerical header information corresponding to the input numerical description information are used as the input of a decoder in the initial event detection model, and the numerical header information corresponding to the input numerical description information is used as the expected output of the decoder.
In a second aspect, an embodiment of the present application provides an event detection method, including: acquiring description information; obtaining numerical description information corresponding to the description information based on a preset dictionary; and obtaining title information of the event represented by the description information based on the numerical description information and the event detection model, wherein the event detection model is obtained by training through a method described in any one implementation mode of the first aspect.
In some embodiments, the obtaining of the numerical description information corresponding to the description information based on the preset dictionary includes: segmenting the description information based on a preset dictionary to obtain description segmentation information; and obtaining the numerical description information according to the index information of each participle in the description participle information in a preset dictionary.
In some embodiments, the above method further comprises: determining the similarity between the title information and each category information in a preset event category set; and determining the category to which the event represented by the description information belongs according to the similarity between the title information and each category information in the preset event category set.
In a third aspect, an embodiment of the present application provides a training apparatus for an event detection model, including: a first obtaining unit configured to obtain a training sample set, wherein training samples in the training sample set include description information and header information corresponding to a same event; the first digitizing unit is configured to obtain digitized description information corresponding to the description information and digitized title information corresponding to the title information in each training sample based on a preset dictionary; and the training unit is configured to train the event detection model by using a machine learning algorithm and taking the numerical description information as input and taking the numerical header information corresponding to the input numerical description information as expected output.
In some embodiments, the first digitizing unit is further configured to: based on a preset dictionary, segmenting the description information and the title information in each training sample respectively to obtain description segmentation information and title segmentation information; obtaining numerical description information according to index information of each participle in the description participle information in a preset dictionary; and obtaining the numerical header information according to the index information of each participle in the header participle information in a preset dictionary.
In some embodiments, the first digitizing unit is further configured to: based on a preset dictionary, segmenting the description information and the title information in each training sample respectively to obtain an event description segmentation result and an event title segmentation result; and respectively processing the event description word segmentation result and the event title word segmentation result to obtain description word segmentation information with a first preset length and title word segmentation information with a second preset length.
In some embodiments, the training unit is further configured to: and training to obtain the event detection model by using a machine learning algorithm, wherein the numerical description information is used as the input of an encoder in the initial event detection model, the output of the encoder and the numerical header information corresponding to the input numerical description information are used as the input of a decoder in the initial event detection model, and the numerical header information corresponding to the input numerical description information is used as the expected output of the decoder.
In a fourth aspect, an embodiment of the present application provides an event detection apparatus, including: a second acquisition unit configured to acquire the description information; the second digitizing unit is configured to obtain digitized description information corresponding to the description information based on a preset dictionary; and the obtaining unit is configured to obtain the title information of the event represented by the description information based on the digitized description information and an event detection model, wherein the event detection model is obtained by training through the method described in any one of the implementation manners of the first aspect.
In some embodiments, the second digitizing unit is further configured to: segmenting the description information based on a preset dictionary to obtain description segmentation information; and obtaining the numerical description information according to the index information of each participle in the description participle information in a preset dictionary.
In some embodiments, the above apparatus further comprises: the determining unit is configured to determine similarity between the title information and each category information in the preset event category set; and determining the category to which the event represented by the description information belongs according to the similarity between the title information and each category information in the preset event category set.
In a fifth aspect, the present application provides a computer-readable medium, on which a computer program is stored, where the program, when executed by a processor, implements the method as described in any implementation manner of the first aspect and the second aspect.
In a sixth aspect, an embodiment of the present application provides an electronic device, including: one or more processors; a storage device, on which one or more programs are stored, which, when executed by one or more processors, cause the one or more processors to implement the method as described in any of the implementations of the first and second aspects.
According to the training method and device for the event detection model, a training sample set is obtained, wherein the training samples in the training sample set comprise description information and title information corresponding to the same event; obtaining numerical description information corresponding to the description information and numerical title information corresponding to the title information in each training sample based on a preset dictionary; and taking the numerical description information as input, taking the numerical header information corresponding to the input numerical description information as expected output, and training to obtain the event detection model, thereby providing a training method of the event detection model and improving the accuracy of the event detection model. Furthermore, event detection is performed based on the event detection model, and the accuracy of the obtained title information is improved.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
FIG. 1 is an exemplary system architecture diagram in which one embodiment of the present application may be applied;
FIG. 2 is a flow diagram of one embodiment of a method for training an event detection model according to the present application;
fig. 3 is a schematic diagram of an application scenario of a training method of an event detection model according to the present embodiment;
FIG. 4 is a flow diagram of yet another embodiment of a training method of an event detection model according to the present application;
FIG. 5 is a flow diagram for one embodiment of an event detection method according to the present application;
FIG. 6 is a block diagram of one embodiment of a training apparatus for an event detection model according to the present application;
FIG. 7 is a block diagram of one embodiment of an event detection device according to the present application;
FIG. 8 is a block diagram of a computer system suitable for use in implementing embodiments of the present application.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
Fig. 1 illustrates an exemplary architecture 100 to which the training method and apparatus of the event detection model of the present application, and the event detection method and apparatus, may be applied.
As shown in fig. 1, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The communication connections between the terminal devices 101, 102, 103 form a topological network, and the network 104 serves to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
The terminal devices 101, 102, 103 may be hardware devices or software that support network connections for data interaction and data processing. When the terminal devices 101, 102, 103 are hardware, they may be various electronic devices supporting network connection, information acquisition, interaction, display, processing, and the like, including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers, desktop computers, and the like. When the terminal apparatuses 101, 102, 103 are software, they can be installed in the electronic apparatuses listed above. It may be implemented, for example, as multiple software or software modules to provide distributed services, or as a single software or software module. And is not particularly limited herein.
The server 105 may be a server providing various services, for example, a background server receiving description information sent by the terminal devices 101, 102, and 103 and performing event detection through an event detection model. Optionally, the server may feed back the event detection result to the terminal device. The server may also train the event detection model. As an example, the server 105 may be a cloud server.
The server may be hardware or software. When the server is hardware, it may be implemented as a distributed server cluster formed by multiple servers, or may be implemented as a single server. When the server is software, it may be implemented as multiple pieces of software or software modules (e.g., software or software modules used to provide distributed services), or as a single piece of software or software module. And is not particularly limited herein.
It should be further noted that the training method and the event detection method of the event detection model provided in the embodiments of the present application may be executed by a server, or may be executed by a terminal device, or may be executed by the server and the terminal device in cooperation with each other. Accordingly, the training device of the event detection model and each part (for example, each unit) included in the event detection device may be all disposed in the server, may be all disposed in the terminal device, and may be disposed in the server and the terminal device, respectively.
It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation. When the training method of the event detection model, the electronic device on which the event detection method operates, does not need to perform data transmission with other electronic devices, the system architecture may include only the training method of the event detection model, the electronic device (e.g., server or terminal device) on which the event detection method operates.
With continued reference to FIG. 2, a flow 200 of one embodiment of a method of training an event detection model is shown, comprising the steps of:
In this embodiment, an executing subject (for example, a terminal device or a server in fig. 1) of the training method of the event detection model may obtain the training sample set from a remote location or from a local location in a wired manner or a wireless manner. Wherein the training samples in the training sample set comprise description information and header information corresponding to the same event. There is a one-to-one correspondence between training samples and events.
The description information represents specific explanatory information for the event content, and the title information represents a brief sentence indicating the key content of the event, which is summarized from the description information.
The event in this embodiment may be an event for each field, or an event for a certain target field. As an example, the event is an event of a municipal administration domain, and the training sample set may be generated based on description information and title information of the event collected by a service hotline of the municipal administration domain.
In this embodiment, the execution main body may obtain the digitized description information corresponding to the description information and the digitized header information corresponding to the header information in each training sample based on a preset dictionary.
The preset dictionary may be a variety of electronic dictionaries. As an example, the preset dictionary is a dictionary corresponding to an open-source roberta _ chip _ close _ tiny pre-training language model.
In this embodiment, the execution subject may first obtain the description information and the key participles in the heading information in each training sample based on a preset dictionary; further, the number of keywords in the description information is converted into numerical description information, and the number of keywords in the header information is converted into numerical header information.
In some optional implementations of this embodiment, the executing main body may execute the step 202 by:
firstly, based on a preset dictionary, segmenting the description information and the title information in each training sample respectively to obtain description segmentation information and title segmentation information.
Secondly, obtaining the numerical description information according to the index information of each participle in the description participle information in a preset dictionary.
In this implementation manner, the execution main body first determines index information of each participle in the description participle information in a preset dictionary, and then combines the index information corresponding to each participle according to the sequence of each participle in the description information to obtain the numerical description information.
Thirdly, obtaining the numerical header information according to the index information of each participle in the header participle information in a preset dictionary.
In this implementation manner, similar to the process of obtaining the digitized description information, the execution main body first determines the index information of each participle in the header participle information in the preset dictionary, and then combines the index information corresponding to each participle according to the sequence of each participle in the header information to obtain the digitized header information.
The execution main body may quickly generate the numerical description information and the numerical heading information based on index information of the participles in the description information and the heading information in a preset dictionary.
In order to unify the lengths of the respective pieces of digitized description information and the lengths of the respective pieces of digitized header information, which is convenient for the subsequent training process of the event detection model and the accuracy of the trained event detection model, the execution main body unifies the lengths of the respective pieces of description participle information and the lengths of the respective pieces of header participle information. Specifically, the executing body may execute the first step by:
firstly, based on a preset dictionary, segmenting the description information and the title information in each training sample respectively to obtain an event description segmentation result and an event title segmentation result.
In this implementation manner, the event description word segmentation result includes all the words in the corresponding description information, and the event title word segmentation result includes all the words in the corresponding title information.
Then, the event description word segmentation result and the event title word segmentation result are processed respectively to obtain description word segmentation information with a first preset length and title word segmentation information with a second preset length.
When the length of the event description word segmentation result is smaller than a first preset length, filling the event description word segmentation result through preset characters to obtain description word segmentation information of the first preset length; when the length of the event description word segmentation result is greater than the first preset length, a word segmentation with the first preset length can be selected from the event description word segmentation result, and description word segmentation information with the first preset length is obtained. The selecting method includes, but is not limited to, selecting preceding participles with a first preset length, and selecting key participles with the first preset length.
Similarly, when the length of the event title word segmentation result is smaller than a second preset length, the event title word segmentation result can be filled with preset characters to obtain title word segmentation information of the second preset length; when the length of the event title word segmentation result is greater than the second preset length, a word segmentation with the second preset length can be selected from the event title word segmentation result to obtain the title word segmentation information with the second preset length. The selecting method includes, but is not limited to, selecting a preceding participle with a second preset length, and selecting a key participle with the second preset length.
The first preset length and the second preset length can be flexibly set according to actual conditions, and are not limited herein.
And 203, training to obtain an event detection model by using a machine learning algorithm and taking the numerical description information as input and the numerical header information corresponding to the input numerical description information as expected output.
In this embodiment, the executing body may use a machine learning algorithm to input the numerical description information, output the numerical header information corresponding to the input numerical description information as an expected output, and train to obtain the event detection model.
Specifically, the executing entity may calculate a loss value between an actual output and an expected output of the event detection model through a preset loss function; and calculating a gradient according to the loss value, updating parameters of the initial event detection model by using a gradient descent method, a random gradient descent method and the like until a preset end condition is reached, and determining the trained initial event detection model as the event detection model. Wherein the predetermined loss function is used to constrain the consistency between the actual output and the expected output of the event detection model.
The preset ending condition can be flexibly set according to the actual training condition, including but not limited to the fact that the training times exceed the preset time threshold, the training time exceeds the preset time threshold, and the loss value tends to converge.
In some optional implementations of the present embodiment, the event detection model includes an encoder and a decoder. The executing agent may execute the step 203 by: and training to obtain the event detection model by using a machine learning algorithm, wherein the numerical description information is used as the input of an encoder in the initial event detection model, the output of the encoder and the numerical header information corresponding to the input numerical description information are used as the input of a decoder in the initial event detection model, and the numerical header information corresponding to the input numerical description information is used as the expected output of the decoder.
In the implementation mode, the numerical header information as the label is used as the input and the expected output of the decoder, so that the training speed and the accuracy of the event detection model are further improved. To even further increase the training speed, the encoder and decoder in the initial event detection model may be pre-trained.
With continued reference to fig. 3, fig. 3 is a schematic diagram 300 of an application scenario of the training method of the event detection model according to the present embodiment. In the application scenario of fig. 3, the server 301 first obtains a training sample set 302. Wherein the training samples in the training sample set 302 include description information and header information corresponding to the same event. The training sample set is generated according to the description information and the title information of the events collected by the service hotline in the municipal administration field. Then, based on the preset dictionary 303, the server 301 obtains the numerical description information corresponding to the description information and the numerical heading information corresponding to the heading information in each training sample. And finally, based on a machine learning algorithm, the server selects untrained corresponding numerical description information and numerical header information, takes the selected numerical description information as input, takes the selected numerical header information as expected output, and iterates the training process of the initial event detection model in a circulating manner until the event detection model is obtained.
In the method provided by the above embodiment of the present application, a training sample set is obtained, where training samples in the training sample set include description information and header information corresponding to the same event; obtaining numerical description information corresponding to the description information and numerical title information corresponding to the title information in each training sample based on a preset dictionary; and taking the numerical description information as input, taking the numerical header information corresponding to the input numerical description information as expected output, and training to obtain the event detection model, thereby providing a training method of the event detection model and improving the accuracy of the event detection model.
With continuing reference to FIG. 4, an exemplary flow 400 of one embodiment of a method for training an event detection model according to the present application is shown, comprising the steps of:
Wherein the training samples in the training sample set comprise description information and header information corresponding to the same event.
And step 402, based on a preset dictionary, performing word segmentation on the description information and the title information in each training sample respectively to obtain an event description word segmentation result and an event title word segmentation result.
And step 404, obtaining the numerical description information according to the index information of each participle in the description participle information in a preset dictionary.
As can be seen from this embodiment, compared with the embodiment corresponding to fig. 2, the flow 400 of the training method of the event detection model in this embodiment specifically illustrates the process of digitizing the description information and the header information, and the training process of the model, which further improves the accuracy of the obtained event detection model.
With continued reference to FIG. 5, a flow 500 of one embodiment of an event detection method is shown, comprising the steps of:
In this embodiment, an execution subject (for example, a terminal device or a server in fig. 1) of the event detection method may obtain the description information from a remote location or from a local location in a wired manner or a wireless manner.
The description information is description information for the event to be detected. The descriptive information may be in the form of text, speech, etc. When the description information is information in a voice form, it may be converted into information in a text form through a voice-to-text technique.
In this implementation manner, the execution main body may obtain the numerical description information corresponding to the description information based on a preset dictionary.
As an example, the execution main body may first obtain a keyword of the description information based on a preset dictionary, and then, digitize the keyword to obtain digitized description information.
In some optional implementations of this embodiment, the executing main body may execute the step 502 by:
firstly, segmenting the description information based on a preset dictionary to obtain description segmentation information.
In the implementation mode, the execution main body performs word segmentation on the description information based on a preset dictionary to obtain an event description word segmentation result; and processing the event description word segmentation result to obtain description word segmentation information with a first preset length.
Secondly, obtaining the numerical description information according to the index information of each participle in the description participle information in a preset dictionary.
In this implementation manner, the execution main body first determines index information of each participle in the description participle information in a preset dictionary, and then combines the index information corresponding to each participle according to the sequence of each participle in the description information to obtain the numerical description information.
And step 203, obtaining the title information of the event represented by the description information based on the numerical description information and the event detection model.
In this embodiment, the execution body may obtain the title information of the event represented by the description information based on the digitized description information and the event detection model. The event detection model is obtained by training the embodiments 200 and 400.
Specifically, the execution main body inputs the digitized description information into the event detection model to obtain digitized title information corresponding to the title information, and the digitized title information represents index information of the participles in the title in the preset dictionary. Furthermore, according to the preset dictionary, the title information in the text form corresponding to the numerical title information can be determined.
The description segmentation information with the first preset length may be obtained by filling preset characters in the description segmentation result, and in this case, the preset characters in the obtained header information need to be removed to obtain final header information.
In some optional implementation manners of this embodiment, the execution main body may further determine similarity between the header information and each category information in the preset event category set; and determining the category to which the event represented by the description information belongs according to the similarity between the title information and each category information in the preset event category set.
The similarity may be determined based on cosine similarity, euclidean distance, or the like, for example.
The execution main body may sort the category information in the preset event category set based on a similarity between the header information and the category information in the preset event category set, and determine the category information ranked first as a category to which the event represented by the fixed description information belongs.
In the method provided by the above embodiment of the present application, the description information is obtained; obtaining numerical description information corresponding to the description information based on a preset dictionary; based on the numerical description information and the event detection model, the title information of the event represented by the description information is obtained, and the accuracy of the obtained title information is improved.
With continuing reference to fig. 6, as an implementation of the method shown in the above-mentioned figures, the present application provides an embodiment of an apparatus for training an event detection model, where the embodiment of the apparatus corresponds to the embodiment of the method shown in fig. 2, and the apparatus may be applied to various electronic devices.
As shown in fig. 6, the training apparatus of the event detection model includes: a first obtaining unit 601 configured to obtain a training sample set, where training samples in the training sample set include description information and header information corresponding to a same event; a first digitizing unit 602, configured to obtain digitized description information corresponding to the description information and digitized header information corresponding to the header information in each training sample based on a preset dictionary; a training unit 603 configured to train the event detection model by using a machine learning algorithm, taking the digitized description information as an input, and taking the digitized header information corresponding to the input digitized description information as an expected output.
In some embodiments, the first digitizing unit 602 is further configured to: based on a preset dictionary, segmenting the description information and the title information in each training sample respectively to obtain description segmentation information and title segmentation information; obtaining numerical description information according to index information of each participle in the description participle information in a preset dictionary; and obtaining the numerical header information according to the index information of each participle in the header participle information in a preset dictionary.
In some embodiments, the first digitizing unit 602 is further configured to: based on a preset dictionary, segmenting the description information and the title information in each training sample respectively to obtain an event description segmentation result and an event title segmentation result; and respectively processing the event description word segmentation result and the event title word segmentation result to obtain description word segmentation information with a first preset length and title word segmentation information with a second preset length.
In some embodiments, the training unit 603 is further configured to: and training to obtain the event detection model by using a machine learning algorithm, wherein the numerical description information is used as the input of an encoder in the initial event detection model, the output of the encoder and the numerical header information corresponding to the input numerical description information are used as the input of a decoder in the initial event detection model, and the numerical header information corresponding to the input numerical description information is used as the expected output of the decoder.
In this embodiment, a first obtaining unit in a training apparatus of an event detection model obtains a training sample set, where training samples in the training sample set include description information and header information corresponding to a same event; the first digitization unit obtains digitized description information corresponding to the description information in each training sample and digitized title information corresponding to the title information based on a preset dictionary; the training unit takes the numerical description information as input by using a machine learning algorithm, takes the numerical title information corresponding to the input numerical description information as expected output, and trains to obtain the event detection model, so that a training device of the event detection model is provided, and the accuracy of the event detection model is improved.
With continuing reference to fig. 7, as an implementation of the method shown in the above-mentioned figures, the present application provides an embodiment of an event detection apparatus, which corresponds to the embodiment of the method shown in fig. 5, and which can be applied to various electronic devices.
As shown in fig. 7, the event detecting apparatus includes: a second acquisition unit 701 configured to acquire description information; a second digitizing unit 702 configured to obtain digitized description information corresponding to the description information based on a preset dictionary; the obtaining unit 703 is configured to obtain, based on the digitized description information and an event detection model, header information of an event represented by the description information, where the event detection model is obtained by training in the embodiments 200 and 400.
In some embodiments, the second digitizing unit 702 is further configured to: segmenting the description information based on a preset dictionary to obtain description segmentation information; and obtaining the numerical description information according to the index information of each participle in the description participle information in a preset dictionary.
In some embodiments, the above apparatus further comprises: a determining unit (not shown in the figure) configured to determine similarity of the header information and each category information in the preset event category set; and determining the category to which the event represented by the description information belongs according to the similarity between the title information and each category information in the preset event category set.
In this embodiment, a second obtaining unit in the training apparatus of the event detection model obtains the description information; the second digitization unit obtains digitized description information corresponding to the description information based on a preset dictionary; the obtaining unit obtains the title information of the event represented by the description information based on the numerical description information and the event detection model, and improves the accuracy of the obtained title information.
Referring now to FIG. 8, shown is a block diagram of a computer system 800 suitable for use in implementing devices of embodiments of the present application (e.g., devices 101, 102, 103, 105 shown in FIG. 1). The apparatus shown in fig. 8 is only an example, and should not bring any limitation to the function and the scope of use of the embodiments of the present application.
As shown in fig. 8, a computer system 800 includes a processor (e.g., CPU, central processing unit) 801 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)802 or a program loaded from a storage section 808 into a Random Access Memory (RAM) 803. In the RAM803, various programs and data necessary for the operation of the system 800 are also stored. The processor 801, the ROM802, and the RAM803 are connected to each other by a bus 804. An input/output (I/O) interface 805 is also connected to bus 804.
The following components are connected to the I/O interface 805: an input portion 806 including a keyboard, a mouse, and the like; an output section 807 including a signal such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 808 including a hard disk and the like; and a communication section 809 including a network interface card such as a LAN card, a modem, or the like. The communication section 809 performs communication processing via a network such as the internet. A drive 810 is also connected to the I/O interface 805 as necessary. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 810 as necessary, so that a computer program read out therefrom is mounted on the storage section 808 as necessary.
In particular, according to embodiments of the application, the processes described above with reference to the flow diagrams may be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 809 and/or installed from the removable medium 811. The computer program, when executed by the processor 801, performs the above-described functions defined in the methods of the present application.
It should be noted that the computer readable medium of the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the client computer, partly on the client computer, as a stand-alone software package, partly on the client computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the client computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present application may be implemented by software or hardware. The described units may also be provided in a processor, and may be described as: a processor comprises a first acquisition unit, a first numeralization unit and a training unit; or a processor comprising a second obtaining unit, a second digitizing unit and a deriving unit. For example, the training unit may be described as a unit that trains an event detection model by using a machine learning algorithm and taking numerical description information as input and taking numerical header information corresponding to the input numerical description information as desired output.
As another aspect, the present application also provides a computer-readable medium, which may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by the apparatus, cause the computer device to: acquiring a training sample set, wherein training samples in the training sample set comprise description information and title information corresponding to the same event; obtaining numerical description information corresponding to the description information and numerical title information corresponding to the title information in each training sample based on a preset dictionary; and training to obtain the event detection model by using a machine learning algorithm and taking the numerical description information as input and the numerical header information corresponding to the input numerical description information as expected output. Or cause the computer device to: acquiring description information; obtaining numerical description information corresponding to the description information based on a preset dictionary; and obtaining the title information of the event represented by the description information based on the numerical description information and the event detection model.
The above description is only a preferred embodiment of the application and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention herein disclosed is not limited to the particular combination of features described above, but also encompasses other arrangements formed by any combination of the above features or their equivalents without departing from the spirit of the invention. For example, the above features may be replaced with (but not limited to) features having similar functions disclosed in the present application.
Claims (16)
1. A method of training an event detection model, comprising:
acquiring a training sample set, wherein training samples in the training sample set comprise description information and title information corresponding to the same event;
obtaining numerical description information corresponding to the description information and numerical title information corresponding to the title information in each training sample based on a preset dictionary;
and training to obtain the event detection model by using a machine learning algorithm and taking the numerical description information as input and the numerical header information corresponding to the input numerical description information as expected output.
2. The method according to claim 1, wherein the obtaining of the numerical description information corresponding to the description information and the numerical heading information corresponding to the heading information in each training sample based on the preset dictionary comprises:
based on the preset dictionary, segmenting the description information and the title information in each training sample respectively to obtain description segmentation information and title segmentation information;
obtaining numerical description information according to index information of each participle in the description participle information in the preset dictionary;
and obtaining the numerical header information according to the index information of each participle in the header participle information in the preset dictionary.
3. The method according to claim 2, wherein the segmenting the description information and the heading information in each training sample based on the preset dictionary to obtain description segmentation information and heading segmentation information respectively comprises:
based on the preset dictionary, segmenting the description information and the title information in each training sample respectively to obtain an event description segmentation result and an event title segmentation result;
and respectively processing the event description word segmentation result and the event title word segmentation result to obtain description word segmentation information with a first preset length and title word segmentation information with a second preset length.
4. The method of claim 1, wherein the training the event detection model using a machine learning algorithm with digitized description information as input and digitized header information corresponding to the input digitized description information as desired output comprises:
and training to obtain the event detection model by using a machine learning algorithm, wherein the numerical description information is used as the input of an encoder in an initial event detection model, the output of the encoder and the numerical header information corresponding to the input numerical description information are used as the input of a decoder in the initial event detection model, and the numerical header information corresponding to the input numerical description information is used as the expected output of the decoder.
5. An event detection method, comprising:
acquiring description information;
obtaining numerical description information corresponding to the description information based on a preset dictionary;
obtaining the title information of the event represented by the description information based on the numerical description information and an event detection model, wherein the event detection model is obtained by training according to any one of claims 1-4.
6. The method according to claim 5, wherein the obtaining of the numerical description information corresponding to the description information based on a preset dictionary comprises:
segmenting the description information based on the preset dictionary to obtain description segmentation information;
and obtaining the numerical description information according to the index information of each participle in the description participle information in the preset dictionary.
7. The method of claim 5, further comprising:
determining the similarity between the title information and each category information in a preset event category set;
and determining the category to which the event represented by the description information belongs according to the similarity between the header information and each category information in the preset event category set.
8. An apparatus for training an event detection model, comprising:
a first obtaining unit configured to obtain a training sample set, wherein training samples in the training sample set include description information and header information corresponding to a same event;
the first digitizing unit is configured to obtain digitized description information corresponding to the description information and digitized title information corresponding to the title information in each training sample based on a preset dictionary;
and the training unit is configured to train the event detection model by using a machine learning algorithm and taking the numerical description information as input and taking the numerical header information corresponding to the input numerical description information as expected output.
9. The apparatus of claim 8, wherein the first digitizing unit is further configured to:
based on the preset dictionary, segmenting the description information and the title information in each training sample respectively to obtain description segmentation information and title segmentation information; obtaining numerical description information according to index information of each participle in the description participle information in the preset dictionary; and obtaining the numerical header information according to the index information of each participle in the header participle information in the preset dictionary.
10. The apparatus of claim 9, the first digitizing unit further configured to:
based on the preset dictionary, segmenting the description information and the title information in each training sample respectively to obtain an event description segmentation result and an event title segmentation result; and respectively processing the event description word segmentation result and the event title word segmentation result to obtain description word segmentation information with a first preset length and title word segmentation information with a second preset length.
11. The apparatus of claim 8, wherein the training unit is further configured to:
and training to obtain the event detection model by using a machine learning algorithm, wherein the numerical description information is used as the input of an encoder in an initial event detection model, the output of the encoder and the numerical header information corresponding to the input numerical description information are used as the input of a decoder in the initial event detection model, and the numerical header information corresponding to the input numerical description information is used as the expected output of the decoder.
12. An event detection device comprising:
a second acquisition unit configured to acquire the description information;
the second digitization unit is configured to obtain digitized description information corresponding to the description information based on a preset dictionary;
a deriving unit configured to derive title information of an event characterized by the description information based on the quantified description information and an event detection model, wherein the event detection model is trained by any one of claims 1 to 4.
13. The apparatus of claim 12, wherein the second digitizing unit is further configured to:
segmenting the description information based on the preset dictionary to obtain description segmentation information; and obtaining the numerical description information according to the index information of each participle in the description participle information in the preset dictionary.
14. The apparatus of claim 12, further comprising:
a determining unit configured to determine similarity between the title information and each category information in a preset event category set; and determining the category to which the event represented by the description information belongs according to the similarity between the header information and each category information in the preset event category set.
15. A computer-readable medium, on which a computer program is stored, wherein the program, when executed by a processor, implements the method of any one of claims 1-7.
16. An electronic device, comprising:
one or more processors;
a storage device having one or more programs stored thereon,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110840976.XA CN113468289A (en) | 2021-07-23 | 2021-07-23 | Training method and device of event detection model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110840976.XA CN113468289A (en) | 2021-07-23 | 2021-07-23 | Training method and device of event detection model |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113468289A true CN113468289A (en) | 2021-10-01 |
Family
ID=77882386
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110840976.XA Pending CN113468289A (en) | 2021-07-23 | 2021-07-23 | Training method and device of event detection model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113468289A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109299477A (en) * | 2018-11-30 | 2019-02-01 | 北京字节跳动网络技术有限公司 | Method and apparatus for generating text header |
| CN111046662A (en) * | 2018-09-26 | 2020-04-21 | 阿里巴巴集团控股有限公司 | Training method, device and system of word segmentation model and storage medium |
| CN111563163A (en) * | 2020-04-29 | 2020-08-21 | 厦门市美亚柏科信息股份有限公司 | Text classification model generation method and device and data standardization method and device |
| CN111898369A (en) * | 2020-08-17 | 2020-11-06 | 腾讯科技（深圳）有限公司 | Article title generation method, model training method and device and electronic equipment |
| CN112116075A (en) * | 2020-09-18 | 2020-12-22 | 厦门安胜网络科技有限公司 | Event extraction model generation method and device and text event extraction method and device |
| CN112446207A (en) * | 2020-12-01 | 2021-03-05 | 平安科技（深圳）有限公司 | Title generation method and device, electronic equipment and storage medium |
| CN112559747A (en) * | 2020-12-15 | 2021-03-26 | 北京百度网讯科技有限公司 | Event classification processing method and device, electronic equipment and storage medium |
-
2021
- 2021-07-23 CN CN202110840976.XA patent/CN113468289A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111046662A (en) * | 2018-09-26 | 2020-04-21 | 阿里巴巴集团控股有限公司 | Training method, device and system of word segmentation model and storage medium |
| CN109299477A (en) * | 2018-11-30 | 2019-02-01 | 北京字节跳动网络技术有限公司 | Method and apparatus for generating text header |
| CN111563163A (en) * | 2020-04-29 | 2020-08-21 | 厦门市美亚柏科信息股份有限公司 | Text classification model generation method and device and data standardization method and device |
| CN111898369A (en) * | 2020-08-17 | 2020-11-06 | 腾讯科技（深圳）有限公司 | Article title generation method, model training method and device and electronic equipment |
| CN112116075A (en) * | 2020-09-18 | 2020-12-22 | 厦门安胜网络科技有限公司 | Event extraction model generation method and device and text event extraction method and device |
| CN112446207A (en) * | 2020-12-01 | 2021-03-05 | 平安科技（深圳）有限公司 | Title generation method and device, electronic equipment and storage medium |
| CN112559747A (en) * | 2020-12-15 | 2021-03-26 | 北京百度网讯科技有限公司 | Event classification processing method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107066449B (en) | Information pushing method and device | |
| CN110046254B (en) | Method and apparatus for generating a model | |
| CN107301170B (en) | Method and device for segmenting sentences based on artificial intelligence | |
| CN108628830B (en) | Semantic recognition method and device | |
| CN110019742B (en) | Method and device for processing information | |
| CN111078887B (en) | Text classification method and device | |
| CN107437417B (en) | Voice data enhancement method and device based on recurrent neural network voice recognition | |
| CN111159220B (en) | Method and apparatus for outputting structured query statement | |
| CN108121699B (en) | Method and apparatus for outputting information | |
| CN109582825B (en) | Method and apparatus for generating information | |
| CN109190123B (en) | Method and apparatus for outputting information | |
| CN111563163A (en) | Text classification model generation method and device and data standardization method and device | |
| CN111767740A (en) | Sound effect adding method and device, storage medium and electronic equipment | |
| CN112988753A (en) | Data searching method and device | |
| CN115798661A (en) | Knowledge mining method and device in clinical medicine field | |
| CN112364653A (en) | Text analysis method, apparatus, server and medium for speech synthesis | |
| CN113033707B (en) | Video classification method and device, readable medium and electronic equipment | |
| CN112711943A (en) | Uygur language identification method, device and storage medium | |
| CN113111167A (en) | Method and device for extracting vehicle model of alarm receiving and processing text based on deep learning model | |
| CN114141236B (en) | Language model updating method and device, electronic equipment and storage medium | |
| CN115620726A (en) | Voice text generation method, and training method and device of voice text generation model | |
| CN113468289A (en) | Training method and device of event detection model | |
| CN111079185B (en) | Database information processing method and device, storage medium and electronic equipment | |
| CN113111169A (en) | Deep learning model-based alarm receiving and processing text address information extraction method and device | |
| CN112131378A (en) | Method and device for identifying categories of civil problems and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |