CN112035641A - Intention extraction model verification method and device, computer equipment and storage medium - Google Patents
Intention extraction model verification method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN112035641A CN112035641A CN202010897652.5A CN202010897652A CN112035641A CN 112035641 A CN112035641 A CN 112035641A CN 202010897652 A CN202010897652 A CN 202010897652A CN 112035641 A CN112035641 A CN 112035641A
- Authority
- CN
- China
- Prior art keywords
- result
- intention
- data
- model
- online
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000605 extraction Methods 0.000 title claims abstract description 145
- 238000012795 verification Methods 0.000 title claims abstract description 64
- 238000000034 method Methods 0.000 title claims abstract description 44
- 238000012360 testing method Methods 0.000 claims abstract description 75
- 238000004364 calculation method Methods 0.000 claims abstract description 59
- 238000002372 labelling Methods 0.000 claims description 68
- 238000004458 analytical method Methods 0.000 claims description 30
- 238000004590 computer program Methods 0.000 claims description 25
- 238000010200 validation analysis Methods 0.000 claims description 12
- 238000011084 recovery Methods 0.000 claims description 2
- 238000013473 artificial intelligence Methods 0.000 abstract description 4
- 206010028813 Nausea Diseases 0.000 description 10
- 238000010586 diagram Methods 0.000 description 10
- 230000008693 nausea Effects 0.000 description 10
- 239000011159 matrix material Substances 0.000 description 7
- 238000011156 evaluation Methods 0.000 description 6
- 230000006399 behavior Effects 0.000 description 4
- 206010019233 Headaches Diseases 0.000 description 3
- 206010047700 Vomiting Diseases 0.000 description 3
- 210000001015 abdomen Anatomy 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 3
- 231100000869 headache Toxicity 0.000 description 3
- 208000024891 symptom Diseases 0.000 description 3
- 230000008673 vomiting Effects 0.000 description 3
- 208000004998 Abdominal Pain Diseases 0.000 description 2
- 230000036528 appetite Effects 0.000 description 2
- 235000019789 appetite Nutrition 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 208000002193 Pain Diseases 0.000 description 1
- 238000007635 classification algorithm Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 230000036461 convulsion Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000010992 reflux Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/332—Query formulation
- G06F16/3329—Natural language query formulation or dialogue systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/3331—Query processing
- G06F16/3332—Query translation
- G06F16/3338—Query expansion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/237—Lexical tools
- G06F40/247—Thesauruses; Synonyms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/02—Marketing; Price estimation or determination; Fundraising
- G06Q30/0281—Customer communication at a business location, e.g. providing product or service information, consulting
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
- G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
- G16H40/67—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for remote operation
Abstract
The application relates to the technical field of artificial intelligence, and provides an intention extraction model verification method, an intention extraction model verification device, computer equipment and a storage medium. The method comprises the following steps: obtaining model test data, wherein the model test data comprises first reply data generated based on a preset slot rule and second reply data which is generated based on the first reply data and carries an expected intention marking result, the expected intention marking result is determined based on a preset data grade division principle, calling an intention extraction model to be verified to perform intention extraction on the model test data to obtain an intention extraction result, performing classification calculation on the intention extraction result and the expected intention marking result to obtain a classification result, and generating a model verification report based on the classification result. By adopting the method, the efficiency of verifying the intention extracted by the intention extraction model can be improved.
Description
Technical Field
The present application relates to the field of artificial intelligence technologies, and in particular, to a method and an apparatus for verifying an intention extraction model, a computer device, and a storage medium.
Background
With the rapid development of artificial intelligence technology, the integration of traditional industries such as finance, entertainment, transportation, medical treatment and the like and artificial intelligence is rapidly developed. In recent years, medical software and mobile terminal equipment are closely combined, so that convenience and quickness in medical treatment are provided for patients, and an intelligent inquiry scheme and system appear.
The intelligent inquiry refers to that a patient can reply to a problem provided by an online doctor (customer service robot) on line, intelligent intention extraction is carried out according to reply data of the patient, and after multiple rounds of conversation, relevant conditions of the patient and certain disease symptoms are obtained, so that intelligent diagnosis is realized. In the whole process, the accuracy of extracting the patient intention in the patient reply data is very critical, so the accuracy of the extracted patient intention needs to be verified.
Currently, the method for verifying the accuracy of patient intention extraction is to acquire patient reply data corresponding to an online doctor question, manually label an expected intention, extract a patient intention by using an intention extraction model, and then verify whether the patient intention extracted by the model is accurate. However, in the above scheme, a large amount of labor cost is consumed for manually labeling the separation data, and a lot of situations that the patient replies disorderly, answers questions randomly, or the algorithm extraction difficulty is large exist in the online pulled patient reply data through manual inspection, so that the efficiency and accuracy of the model extraction of the patient intention are greatly reduced, and the efficiency of verifying the patient intention extracted by the model is further influenced.
Disclosure of Invention
In view of the above, it is necessary to provide an efficient intent extraction model verification method, apparatus, computer device and storage medium for solving the above technical problems.
An intent extraction model verification method, the method comprising:
obtaining model test data, wherein the model test data comprises first reply data generated based on a preset slot position rule and second reply data which is generated based on the first reply data and carries an expected intention marking result, and the expected intention marking result is determined based on a preset data grade division principle;
calling an intention extraction model to be verified to extract the intention of the model test data to obtain an intention extraction result;
classifying and calculating the intention extraction result and the expected intention labeling result to obtain a classification result;
based on the classification results, a model validation report is generated.
In one embodiment, obtaining model test data comprises:
problem data are obtained;
generating first reply data corresponding to the problem data according to a preset slot position rule;
and expanding the first reply data, and marking the expected intention of the first reply data according to a preset data grade division principle to obtain second reply data.
In one embodiment, before generating the first reply data corresponding to the problem data according to the slot rule, the method further includes:
acquiring slot bit data;
and expanding a prefix rule, a suffix rule, a synonym rule and a synonym rule of the slot bit data to obtain a preset slot rule.
In one embodiment, the classifying calculation of the intention extraction result and the expected intention labeling result includes:
and performing two-classification calculation on the intention extraction result and the preset intention labeling result to obtain two-classification results.
In one embodiment, generating the model validation report based on the classification result comprises:
performing micro statistical calculation on the classification result according to the standard recall rate, the precision rate and the accuracy calculation logic to obtain a statistical result, wherein the statistical result comprises the standard recall rate, the precision rate and the accuracy corresponding to different data grades;
and generating a model verification report based on the standard recall rate, the precision rate and the accuracy rate corresponding to different data grades.
In one embodiment, after generating the model verification report based on the classification result, the method further includes:
when the standard recall rate, the precision rate and the accuracy rate in the model verification report are judged to meet the requirements corresponding to the expected intention labeling result, an intention extraction model is operated on line;
acquiring online reply data and an online intention extraction result of an intention extraction model;
and analyzing the online recovery data and the online intention extraction result to obtain a model online analysis report.
In one embodiment, analyzing the on-line iteration data and the on-line intent extraction result to obtain the model on-line analysis report includes:
carrying out expected intention labeling on the online return data according to a preset data grade division principle to obtain an online expected intention labeling result;
performing two-classification calculation on the online intention extraction result and the online preset intention marking result to obtain an online two-classification result;
performing micro statistical calculation on the two online classification results according to the standard recall rate, the precision rate and the accuracy calculation logic to obtain online statistical results;
and generating a model online analysis report according to the online statistical result.
An intent extraction model validation apparatus, the apparatus comprising:
the data acquisition module is used for acquiring model test data, the model test data comprises reply data generated based on a preset slot position rule and reply data carrying an expected intention labeling result, and the expected intention labeling result is determined based on a preset data grade division principle;
the intention extraction module is used for calling an intention extraction model to be verified to carry out intention extraction on the model test data to obtain an intention extraction result;
the classification calculation module is used for performing classification calculation on the intention extraction result and the expected intention labeling result to obtain a classification result;
and the model verification report generation module is used for generating a model verification report based on the classification result.
A computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program:
obtaining model test data, wherein the model test data comprises first reply data generated based on a preset slot position rule and second reply data which is generated based on the first reply data and carries an expected intention marking result, and the expected intention marking result is determined based on a preset data grade division principle;
calling an intention extraction model to be verified to extract the intention of the model test data to obtain an intention extraction result;
classifying and calculating the intention extraction result and the expected intention labeling result to obtain a classification result;
based on the classification results, a model validation report is generated.
A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:
obtaining model test data, wherein the model test data comprises first reply data generated based on a preset slot position rule and second reply data which is generated based on the first reply data and carries an expected intention marking result, and the expected intention marking result is determined based on a preset data grade division principle;
calling an intention extraction model to be verified to extract the intention of the model test data to obtain an intention extraction result;
classifying and calculating the intention extraction result and the expected intention labeling result to obtain a classification result;
based on the classification results, a model validation report is generated.
According to the intention extraction model verification method, the device, the computer equipment and the storage medium, the acquired model test data comprise the first reply data generated based on the preset slot rule and the second reply data carrying the expected intention labeling result, so that the steps of manually expanding the test data and manually labeling the expected intention are omitted in the model verification process, the labor cost is saved, the risk caused by manual error is avoided, the quality and the test efficiency of the test data are improved, in addition, the intention extraction result and the expected intention labeling result are classified and calculated, the obtained model test report comprises a plurality of dimensions, the effective and uniform model evaluation index is provided, and the intention extraction efficiency of the verification intention extraction model can be improved.
Drawings
FIG. 1 is a diagram of an application environment for a verification method intended for extraction of models, in one embodiment;
FIG. 2 is a schematic flow chart diagram illustrating a verification method for extracting models, according to one embodiment;
FIG. 3 is a schematic flow chart diagram illustrating a verification method for extracting an intent model according to another embodiment;
FIG. 4 is a flowchart illustrating the steps of obtaining model test data in one embodiment;
FIG. 5 is a diagram of a model validation report in one embodiment;
FIG. 6 is a diagram of an on-line analysis report for a model in one embodiment;
FIG. 7 is a block diagram of an embodiment of an apparatus for validating an extraction model;
FIG. 8 is a block diagram of an apparatus for validating an extraction model according to another embodiment;
FIG. 9 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The intention extraction model verification method provided by the application can be applied to the application environment shown in fig. 1. Wherein the terminal 102 communicates with the server 104 via a network. Specifically, the server 104 deploys an intention extraction model to be verified, a user uploads model test data to the server 104 through the terminal 102, then the user operates the terminal 102 to initiate a verification operation of the verification intention extraction model, generates and sends a model verification request to the server 104, the server 104 responds to the request to obtain the model test data, the model test data comprises first reply data generated based on a preset slot rule and second reply data generated based on the first reply data and carrying an expected intention labeling result, the expected intention labeling result is determined based on a preset data grade division principle, the intention extraction model to be verified is called to perform intention extraction on the model test data to obtain the intention extraction result, the intention extraction result and the expected intention labeling result are classified and calculated to obtain a classification result, and based on the classification result, and generating a model verification report. The terminal 102 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices, and the server 104 may be implemented by an independent server or a server cluster formed by a plurality of servers.
In one embodiment, as shown in fig. 2, there is provided an intention extraction model verification method, which is described by taking the method as an example applied to the server in fig. 1, and includes the following steps:
The model test data is data for testing a verification intention extraction model (hereinafter referred to as a model). In this embodiment, the scheme is described with reference to an application scenario of an intelligent inquiry session, where the model test data is doctor-patient session text data, and includes a question data list (hereinafter referred to as question data) provided by a doctor and reply data (hereinafter referred to as reply data) replied by a patient, where the reply data includes first reply data generated based on a preset slot rule after a series of processing and second reply data generated based on the first reply data and labeled with an expected intention by adopting a preset data classification level.
In practical application, because the effectiveness of real online data is low, the online data needs to be subjected to a series of processing such as cleaning, dumping and manual labeling, the labor cost is high, and all slots concerned by an intelligent questioning clinic (the purpose of doctor problems) cannot be covered, the model is verified in a mode of automatically constructing a test data set. Meanwhile, in the prior art, since the expected performance (i.e., an evaluation index) of the model needs to be formulated to verify the accuracy of the intention extraction model for intention extraction, the performance of the model is quantified according to the expected performance. Therefore, developers can abstract the data grading principle by analyzing the user behaviors of the historical doctor-patient dialogue data and combining the activity of the reply content of the patient and the difficulty grade supported by the algorithm. Specifically, the model requirements can be decomposed, the support degree of the reply intention of the patient is extracted as the key items of the test, the data level L1 is abstracted according to the behavior condition of the patient, and the data levels L2-L6 are abstracted according to the support degree condition of the algorithm. The data grade of the quick reply data of the click mode of the slot position (high heat and low heat) in the problem data (such as high heat or low heat during twitch) can be divided into L1, the reply data comprising the slot position primitive word can be divided into L2-L3, the reply data comprising synonyms or synonyms of the slot position primitive word can be divided into L4-L6, if the reply data is related to the problem but the reply data can not be classified into an option (low heat or high heat), the reply data can be divided into C1, and if the reply data is not related to the problem or the option, the reply data is divided into C2. The model test data is divided into a plurality of data levels (the data levels comprise L1-L6, C1 and C2) according to a data level division principle, expected performance, namely evaluation indexes of the model are formulated according to the dimension of the data levels, for example, the recall rate and the accuracy rate of reply data formulated into the data levels of L1-L3 are more than 95%, and the recall rate and the accuracy rate of reply data formulated into the data levels of L4-L6 are more than 85%. Correspondingly, the expected intention of the reply data may also be labeled according to the data level, and specifically, the expected intention labeling result includes the labeled slot and the data level corresponding to the slot, for example, the data level of the slot "abdominalgia" may be labeled as "+ _ L2", and the data level of the slot "nausea" and "vomiting" may be labeled as "+ _ L3".
And 204, calling an intention extraction model to be verified to extract the intention of the model test data to obtain an intention extraction result.
In practical application, after model test data are prepared and before the intention extraction model is called, whether the labeling result of reply data in the model test data covers all slot positions of problem data is checked, after the labeling result of the reply data covers all slot positions of problems, the model test data can be divided into a plurality of test tasks, after the data to be tested are selected, the intention extraction model to be verified is called by creating a scheduling task, a corresponding test task is executed, and intention extraction is performed on the data to obtain the intention extraction result.
And step 206, performing classification calculation on the intention extraction result and the expected intention labeling result to obtain a classification result.
In specific implementation, after the intention extraction result output by the model is obtained, whether the result of the model extraction is accurate or not is verified, the verification result is quantized, the intention extraction result and the expected intention labeling result can be compared and analyzed, and classification calculation is performed on the intention extraction result and the expected intention labeling result to obtain a classification result. Specifically, the classification calculation may employ a binary classification algorithm.
As shown in fig. 3, in one embodiment, the classifying calculation is performed on the intention extraction result and the expected intention labeling result, and obtaining the classification result includes: and 226, performing two-classification calculation on the intention extraction result and the preset intention labeling result to obtain two-classification results.
In this embodiment, the core of the classification calculation is a two-classification confusion matrix, and the two-classification calculation of the intention extraction result and the preset intention labeling result may be to automatically align the same slot, and classify the dropped slot as two classifications, so as to obtain tp (true positive), fn (false negative), fp (false positive), and tn (true negative). Specifically, TP means the prediction of positive class as positive class number, FN means the prediction of positive class as negative class number, FP means the prediction of negative class as positive class number, and TN means the prediction of negative class as negative class number. Specifically, if the extraction intention result and the predicted result are the same for the slot "nausea", the classification result may be TP in the classification matrix "nausea +", if the extraction intention result is null and the actual classification result is "nausea +", the classification matrix "vomiting +", the classification matrix "nausea +", the classification matrix "FP", and the classification matrix "nausea +", the classification matrix "TN.
Based on the classification result, a model verification report is generated, step 208.
After the classification result is obtained, a model verification report can be generated based on the classification result and by combining with the corresponding statistical calculation rule. Specifically, the model verification report may include recall, accuracy, and accuracy of slots of different data levels.
As shown in FIG. 3, in one embodiment, generating a model validation report based on the classification results includes: and 228, performing micro statistical calculation on the classification result according to the standard recall rate, the precision rate and the accuracy calculation logic to obtain a statistical result, wherein the statistical result comprises the standard recall rate, the precision rate and the accuracy rate corresponding to different data grades, and a model verification report is generated based on the standard recall rate, the precision rate and the accuracy rate corresponding to different data grades.
In this embodiment, after the two classification results are obtained, micro statistical calculation may be performed on the two classification results according to the standard recall rate, the precision rate, and the precision rate calculation logic, so as to calculate the sum of all TPs, the sum of all FPs, the sum of all FNs, and the sum of all TNs, and then the standard recall rate, the precision rate, and the precision rate calculation logic corresponding to different data levels are calculated according to the standard recall rate, the precision rate, and the precision rate calculation logic. Wherein TP/(FN + TP) is standard recall rate, TP/(FP + TP) is precision rate, and (TP + TN)/(TP + FN + FP + TN) is precision rate. After the corresponding recall rate, the precision rate and the accuracy rate are obtained according to the calculation logic, a model verification report can be generated. The model verification report comprises the standard recall rate, the precision rate and the accuracy rate statistical result corresponding to each data grade. Specifically, the model test report may be as shown in fig. 5. In this embodiment, a detailed and intuitive model verification report can be obtained by performing micro statistical calculation including standard recall rate, precision rate, and accuracy rate on the classification result.
In the intention extraction model verification method, the acquired model test data comprises first reply data generated based on the preset slot rule and second reply data carrying the expected intention labeling result, so that the steps of manually expanding the test data and manually labeling the expected intention are omitted in the model verification process, the labor cost is saved, the risk caused by manual error is avoided, the quality and the test efficiency of the test data are improved, in addition, the obtained model test report comprises a plurality of dimensions by performing classification calculation on the intention extraction result and the expected intention labeling result, an effective and uniform model evaluation index is provided, the verification effectiveness is ensured, and the efficiency of the intention extracted by the verification intention extraction model can be improved.
As shown in FIG. 4, in one embodiment, obtaining model test data comprises:
In this embodiment, the problem data includes a doctor problem data list, and specifically, the problem data may be obtained by writing a problem utterance for a slot/option by a dentist and a user experience team, and sorting the written problem utterance. After the problem data is obtained, the data may be expanded according to a preset slot rule to generate patient reply data (i.e., first reply data) corresponding to the problem data of the doctor, the patient reply data (i.e., first reply data) is expanded according to the question stem of the problem data, and the first reply data is subjected to expected intention labeling according to a data grade division principle to obtain second reply data. As illustrated by the foregoing embodiments, slots "belly" may be labeled with a data level of "+ _ L2" for slot "belly" and "+ _ L3" for slot "nausea" and "vomiting. Furthermore, the content replied by the patient can be generated, and manual marking can be further performed, so that the quality of the test data is further improved, and the objective accuracy of the model evaluation index is ensured. In this embodiment, after the slot rule has a certain scale, the subsequent model test data can be automatically expanded through the slot rule, and the work center of gravity for preparing the model test data is transferred from the data replied by the patient to the maintenance of the slot rule, so that the method is convenient for flexibly dealing with the change of the doctor question and the change of the content of the slot dictionary (for example, poor appetite is changed from poor appetite), the testing period after the test is greatly shortened, and the virtuous cycle of the testing efficiency which is higher and higher is achieved.
In one embodiment, before generating the reply data corresponding to the question data according to the preset slot rule, the method further includes: step 240, obtaining slot bit data, and expanding a prefix rule, a suffix rule, a synonym rule and a synonym rule of the slot bit data to obtain a preset slot rule.
The slot data is the purpose of the doctor question data, and if the doctor question data is "ask for whether there are symptoms of abdominal pain, nausea and headache? "the slot data is" abdominal pain, nausea and headache ", and in addition to this, option data may be constructed based on the slot data, and the option data returned to the patient may be" there is belly pain/feeling headache/nausea/unclear/none at all ". In this embodiment, the slot bit data and/or the option data may be acquired, and then the slot bit data and/or the option data may be expanded according to the preset data expansion. Specifically, the extension process includes: according to a general dictionary such as a Hardsword forest, the slot data synonym/synonym rule is expanded, according to a medical word congruence/synonym dictionary, the slot data synonym/synonym rule is expanded from a professional perspective, the prefix and suffix universal rule of different symptom words is expanded manually, and a manual classification slot/option data list, and fine classification of suffixes and suffixes is assisted according to classified and online analyzed data. According to the data expansion basis, after the slot position rule is obtained, secondary quality inspection and re-expansion are carried out manually, the validity of the rules of the intended slot position such as prefix/suffix and near/synonym can be matched is ensured, and the quality of the rules is ensured.
As shown in fig. 3, in one embodiment, after generating the model verification report based on the classification result, the method further includes:
In specific implementation, whether the model meets the service requirements can be verified according to various evaluation indexes (standard recall rate, precision rate and accuracy rate) in the model verification report, if the standard recall rate and precision rate of the data grade L1-L3 reach 95%, and the standard recall rate and precision rate of the data grade L4-L6 reach more than 85%, the intention extraction model is considered to reach the requirements, and the intention extraction model can be released and operated on line. After the intention extraction model runs for a certain time on line, on-line reply data and on-line intention extraction results can be timely pulled to perform user behavior analysis, and user input behaviors are analyzed, specifically, on-line expected intention labeling is performed on the on-line reply data according to a data grade division principle, namely, on the basis of the data grade division principle, L1-L3 grade data are preferentially searched out in a regular matching mode to perform data grade labeling, after data with the data grades of L4-L6, C1 and C2 are cleaned and screened, manual intervention analysis (namely, manually labeling the two classification results and the data grades) is performed, then, the on-line intention extraction results and the on-line preset intention labeling results are subjected to two classification calculation, and on-line two classification results are obtained; and performing micro statistical calculation on the online two-classification result according to the standard recall rate, the accuracy rate and the accuracy calculation logic to obtain an online statistical result, and further obtaining a model online analysis report (the model online report can be shown in fig. 6) according to the online statistical result, wherein the online statistical result comprises the online standard recall rate, the accuracy rate and the accuracy rate. Meanwhile, the rapid reply (click) utilization rate, the free input recall rate and the like are counted, and finally the trend graphs of a plurality of model versions are obtained, so that the model effect is conveniently tracked. The data after manual analysis can be imported into the model test data, and the standard test data set is enlarged to form a closed loop. In this embodiment, online reflux analysis is performed on the intention extraction model, so that whether model test data and a model verification scheme are reasonable or not can be effectively verified.
In one embodiment, after online reflow and analysis of the intent extraction model and obtaining an online analysis report of the model, the method further includes: and 212, comparing the on-line analysis report and the model verification report of the analysis model to obtain a comparison analysis result, and optimizing the intention extraction model according to the comparison analysis result.
In practical application, after the on-line analysis report of the model is obtained, the standard recall rate, the precision rate and the accuracy rate in the on-line analysis report of the model and the standard recall rate and the accuracy rate in the model verification report can be compared and analyzed, the on-line standard recall rate, the precision rate and the accuracy rate, the difference of the precision rate and the accuracy rate in the on-line standard recall rate, the standard recall rate and the accuracy rate in the model verification report and the like are obtained, the reason for generating the difference is analyzed, the model test data are continuously optimized, and then the intention extraction model is optimized along with the optimized model test.
It should be understood that although the various steps in the flow charts of fig. 2-4 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 2-4 may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed in turn or alternately with other steps or at least some of the other steps.
In one embodiment, as shown in fig. 7, there is provided an intention extraction model verifying apparatus including: a data acquisition module 510, an intent extraction module 520, a classification calculation module 530, and a model validation report generation module 540, wherein:
the data obtaining module 510 is configured to obtain model test data, where the model test data includes first reply data generated based on a preset slot rule and second reply data generated based on the first reply data and carrying an expected intention labeling result, and the expected intention labeling result is determined based on a preset data classification rule.
And the intention extraction module 520 is used for calling an intention extraction model to be verified to perform intention extraction on the model test data to obtain an intention extraction result.
And the classification calculation module 530 is configured to perform classification calculation on the intention extraction result and the expected intention labeling result to obtain a classification result.
And a model verification report generation module 540, configured to generate a model verification report based on the classification result.
In one embodiment, the data obtaining module 510 is further configured to obtain the problem data, generate first reply data corresponding to the problem data according to a preset slot rule, expand the first reply data, and perform expected intention labeling on the first reply data according to a preset data classification rule to obtain second reply data.
As shown in fig. 8, in one embodiment, the apparatus further includes a slot position rule generating module 550, configured to obtain slot position data, and expand a prefix rule, a suffix rule, a synonym rule, and a synonym rule of the slot position data to obtain a preset slot position rule.
In one embodiment, the classification calculation module 530 is further configured to perform a binary classification calculation on the intention extraction result and the preset intention labeling result to obtain a binary classification result.
In one embodiment, the model verification report generation module 540 is further configured to perform micro statistical calculation on the classification result according to the standard recall rate, the accuracy rate, and the accuracy rate calculation logic to obtain a statistical result, where the statistical result includes the standard recall rate, the accuracy rate, and the accuracy rate corresponding to different data levels, and generate the model verification report based on the standard recall rate, the accuracy rate, and the accuracy rate corresponding to different data levels.
As shown in fig. 8, in one embodiment, the apparatus further includes an online analysis module 560, configured to run the intention extraction model online when it is determined that the standard recall rate, the accuracy rate, and the accuracy rate in the model verification report meet the requirements corresponding to the expected intention labeling result, obtain online reply data and an online intention extraction result of the intention extraction model, and analyze the online reply data and the online intention extraction result to obtain an online analysis report of the model.
In one embodiment, the online analysis module 560 is further configured to perform expected intention labeling on the online repeated data according to a preset data grade division principle to obtain an online expected intention labeling result, perform binary computation on the online intention extraction result and the online preset intention labeling result to obtain an online binary result, perform micro statistical computation on the online binary result according to the standard recall rate, the precision rate and the accuracy computation logic to obtain an online statistical result, and generate a model online analysis report according to the online statistical result.
For specific limitations of the intent extraction model verification apparatus, reference may be made to the above limitations of the intent extraction model verification method, which are not described herein again. The various modules in the above-described intent extraction model verification apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 9. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer equipment is used for storing data such as model test data, model on-line analysis reports and the like. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement an intent extraction model validation method.
Those skilled in the art will appreciate that the architecture shown in fig. 9 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program: obtaining model test data, wherein the model test data comprises first reply data generated based on a preset slot rule and second reply data which is generated based on the first reply data and carries an expected intention marking result, the expected intention marking result is determined based on a preset data grade division principle, calling an intention extraction model to be verified to perform intention extraction on the model test data to obtain an intention extraction result, performing classification calculation on the intention extraction result and the expected intention marking result to obtain a classification result, and generating a model verification report based on the classification result.
In one embodiment, the processor, when executing the computer program, further performs the steps of: obtaining problem data, generating first reply data corresponding to the problem data according to a preset slot rule, expanding the first reply data, and marking the first reply data with expected intentions according to a preset data grade division principle to obtain second reply data.
In one embodiment, the processor, when executing the computer program, further performs the steps of: and acquiring slot bit data, and expanding a prefix rule, a suffix rule, a synonym rule and a synonym rule of the slot bit data to obtain a preset slot position rule.
In one embodiment, the processor, when executing the computer program, further performs the steps of: and performing two-classification calculation on the intention extraction result and the preset intention labeling result to obtain two-classification results.
In one embodiment, the processor, when executing the computer program, further performs the steps of: and carrying out micro statistical calculation on the classification result according to the standard recall rate, the precision rate and the accuracy calculation logic to obtain a statistical result, wherein the statistical result comprises the standard recall rate, the precision rate and the accuracy rate corresponding to different data grades, and a model verification report is generated based on the standard recall rate, the precision rate and the accuracy rate corresponding to different data grades.
In one embodiment, the processor, when executing the computer program, further performs the steps of: when the standard recall rate, the precision rate and the accuracy rate in the model verification report are judged to meet the requirements corresponding to the expected intention labeling results, the intention extraction model is operated on line, on-line reply data and on-line intention extraction results of the intention extraction model are obtained, the on-line reply data and the on-line intention extraction results are analyzed, and the model on-line analysis report is obtained.
In one embodiment, the processor, when executing the computer program, further performs the steps of: and performing expected intention labeling on the online repeated data according to a preset data grade division principle to obtain an online expected intention labeling result, performing binary calculation on the online intention extraction result and the online preset intention labeling result to obtain an online binary result, performing micro statistical calculation on the online binary result according to standard recall rate, precision rate and accuracy calculation logic to obtain an online statistical result, and generating a model online analysis report according to the online statistical result.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which when executed by a processor performs the steps of: obtaining model test data, wherein the model test data comprises first reply data generated based on a preset slot rule and second reply data which is generated based on the first reply data and carries an expected intention marking result, the expected intention marking result is determined based on a preset data grade division principle, calling an intention extraction model to be verified to perform intention extraction on the model test data to obtain an intention extraction result, performing classification calculation on the intention extraction result and the expected intention marking result to obtain a classification result, and generating a model verification report based on the classification result.
In one embodiment, the computer program when executed by the processor further performs the steps of: obtaining problem data, generating first reply data corresponding to the problem data according to a preset slot rule, expanding the first reply data, and marking the first reply data with expected intentions according to a preset data grade division principle to obtain second reply data.
In one embodiment, the computer program when executed by the processor further performs the steps of: and acquiring slot bit data, and expanding a prefix rule, a suffix rule, a synonym rule and a synonym rule of the slot bit data to obtain a preset slot position rule.
In one embodiment, the computer program when executed by the processor further performs the steps of: and performing two-classification calculation on the intention extraction result and the preset intention labeling result to obtain two-classification results.
In one embodiment, the computer program when executed by the processor further performs the steps of: and carrying out micro statistical calculation on the classification result according to the standard recall rate, the precision rate and the accuracy calculation logic to obtain a statistical result, wherein the statistical result comprises the standard recall rate, the precision rate and the accuracy rate corresponding to different data grades, and a model verification report is generated based on the standard recall rate, the precision rate and the accuracy rate corresponding to different data grades.
In one embodiment, the computer program when executed by the processor further performs the steps of: when the standard recall rate, the precision rate and the accuracy rate in the model verification report are judged to meet the requirements corresponding to the expected intention labeling results, the intention extraction model is operated on line, on-line reply data and on-line intention extraction results of the intention extraction model are obtained, the on-line reply data and the on-line intention extraction results are analyzed, and the model on-line analysis report is obtained.
In one embodiment, the computer program when executed by the processor further performs the steps of: and performing expected intention labeling on the online repeated data according to a preset data grade division principle to obtain an online expected intention labeling result, performing binary calculation on the online intention extraction result and the online preset intention labeling result to obtain an online binary result, performing micro statistical calculation on the online binary result according to standard recall rate, precision rate and accuracy calculation logic to obtain an online statistical result, and generating a model online analysis report according to the online statistical result.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database or other medium used in the embodiments provided herein can include at least one of non-volatile and volatile memory. Non-volatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical storage, or the like. Volatile Memory can include Random Access Memory (RAM) or external cache Memory. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. An intent extraction model validation method, the method comprising:
obtaining model test data, wherein the model test data comprises first reply data generated based on a preset slot rule and second reply data which is based on the first reply data and carries an expected intention marking result, and the expected intention marking result is determined based on a preset data grade division principle;
calling an intention extraction model to be verified to extract the intention of the model test data to obtain an intention extraction result;
classifying and calculating the intention extraction result and the expected intention labeling result to obtain a classification result;
and generating a model verification report based on the classification result.
2. The method of claim 1, wherein the obtaining model test data comprises:
problem data are obtained;
generating first reply data corresponding to the problem data according to the preset slot position rule;
and expanding the first reply data, and carrying out expected intention labeling on the first reply data according to the preset data grade division principle to obtain second reply data.
3. The method of claim 2, wherein before generating the first recovery data corresponding to the problem data according to the default slot rule, the method further comprises:
acquiring slot bit data;
and expanding a prefix rule, a suffix rule, a synonym rule and a synonym rule of the slot bit data to obtain the preset slot position rule.
4. The method according to claim 1, wherein the classifying the intention extraction result and the expected intention labeling result to obtain a classification result comprises:
and performing two-classification calculation on the intention extraction result and the preset intention labeling result to obtain two-classification results.
5. The method of claim 1, wherein generating a model validation report based on the classification result comprises:
performing micro statistical calculation on the classification result according to standard recall rate, precision rate and accuracy calculation logic to obtain a statistical result, wherein the statistical result comprises standard recall rate, precision rate and accuracy rate corresponding to different data grades;
and generating a model verification report based on the standard recall rate, the precision rate and the accuracy rate corresponding to the different data grades.
6. The method according to any one of claims 1 to 5, wherein after generating a model validation report based on the classification result, the method further comprises:
when the standard recall rate, the precision rate and the accuracy rate in the model verification report are judged to meet the requirements corresponding to the expected intention labeling result, an intention extraction model is operated on line;
acquiring online reply data and an online intention extraction result of the intention extraction model;
and analyzing the online reply data and the online intention extraction result to obtain a model online analysis report.
7. The method of claim 6, wherein analyzing the online reply data and the online intent extraction result to obtain a model online analysis report comprises:
carrying out expected intention labeling on the online reply data according to the preset data grade division principle to obtain an online expected intention labeling result;
performing two-classification calculation on the online intention extraction result and the online preset intention marking result to obtain an online two-classification result;
performing micro statistical calculation on the online binary classification result according to standard recall rate, precision and accuracy calculation logic to obtain an online statistical result;
and generating a model online analysis report according to the online statistical result.
8. An intent extraction model verification apparatus, characterized in that the apparatus comprises:
the data acquisition module is used for acquiring model test data, wherein the model test data comprises reply data generated based on a preset slot position rule and reply data carrying an expected intention labeling result, and the expected intention labeling result is determined based on a preset data grade division principle;
the intention extraction module is used for calling an intention extraction model to be verified to carry out intention extraction on the model test data to obtain an intention extraction result;
the classification calculation module is used for performing classification calculation on the intention extraction result and the expected intention labeling result to obtain a classification result;
and the model verification report generation module is used for generating a model verification report based on the classification result.
9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010897652.5A CN112035641A (en) | 2020-08-31 | 2020-08-31 | Intention extraction model verification method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010897652.5A CN112035641A (en) | 2020-08-31 | 2020-08-31 | Intention extraction model verification method and device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112035641A true CN112035641A (en) | 2020-12-04 |
Family
ID=73586024
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010897652.5A Pending CN112035641A (en) | 2020-08-31 | 2020-08-31 | Intention extraction model verification method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112035641A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018109806A1 (en) * | 2016-12-12 | 2018-06-21 | 三菱電機株式会社 | Model learning device and model learning method |
| US20180351819A1 (en) * | 2017-05-31 | 2018-12-06 | Cisco Technology, Inc. | Semantic analysis to detect shadowing of rules in a model of network intents |
| CN110377911A (en) * | 2019-07-23 | 2019-10-25 | 中国工商银行股份有限公司 | Intension recognizing method and device under dialogue frame |
| CN110532374A (en) * | 2019-09-04 | 2019-12-03 | 深圳市慧择时代科技有限公司 | The processing method and processing device of insurance information |
| CN111143561A (en) * | 2019-12-26 | 2020-05-12 | 北京百度网讯科技有限公司 | Intention recognition model training method and device and electronic equipment |
| CN111198937A (en) * | 2019-12-02 | 2020-05-26 | 泰康保险集团股份有限公司 | Dialog generation device, computer-readable storage medium, and electronic device |
-
2020
- 2020-08-31 CN CN202010897652.5A patent/CN112035641A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018109806A1 (en) * | 2016-12-12 | 2018-06-21 | 三菱電機株式会社 | Model learning device and model learning method |
| US20180351819A1 (en) * | 2017-05-31 | 2018-12-06 | Cisco Technology, Inc. | Semantic analysis to detect shadowing of rules in a model of network intents |
| CN110377911A (en) * | 2019-07-23 | 2019-10-25 | 中国工商银行股份有限公司 | Intension recognizing method and device under dialogue frame |
| CN110532374A (en) * | 2019-09-04 | 2019-12-03 | 深圳市慧择时代科技有限公司 | The processing method and processing device of insurance information |
| CN111198937A (en) * | 2019-12-02 | 2020-05-26 | 泰康保险集团股份有限公司 | Dialog generation device, computer-readable storage medium, and electronic device |
| CN111143561A (en) * | 2019-12-26 | 2020-05-12 | 北京百度网讯科技有限公司 | Intention recognition model training method and device and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Miu et al. | Predicting the execution time of workflow activities based on their input features | |
| Wang et al. | Software defect prediction model based on LASSO–SVM | |
| CN111258767A (en) | Intelligent cloud computing resource allocation method and device for complex system simulation application | |
| US20090217246A1 (en) | Evaluating Software Programming Skills | |
| CN112199512B (en) | Scientific and technological service-oriented case map construction method, device, equipment and storage medium | |
| Barash et al. | Bridging the gap between ml solutions and their business requirements using feature interactions | |
| CN112632252B (en) | Dialogue response method, dialogue response device, computer equipment and storage medium | |
| CN110674636A (en) | Power utilization behavior analysis method | |
| US8650180B2 (en) | Efficient optimization over uncertain data | |
| CN113158435B (en) | Complex system simulation running time prediction method and device based on ensemble learning | |
| CN114201328A (en) | Fault processing method and device based on artificial intelligence, electronic equipment and medium | |
| US20210150358A1 (en) | System and method for controlling confidential information | |
| CN113761193A (en) | Log classification method and device, computer equipment and storage medium | |
| Ramadhina et al. | Software defect prediction using process metrics systematic literature review: Dataset and granularity level | |
| CN116560968A (en) | Simulation calculation time prediction method, system and equipment based on machine learning | |
| CN112131587A (en) | Intelligent contract pseudo-random number security inspection method, system, medium and device | |
| CN112035641A (en) | Intention extraction model verification method and device, computer equipment and storage medium | |
| Zamani | A prediction model for software requirements change impact | |
| CN115344386A (en) | Method, device and equipment for predicting cloud simulation computing resources based on sequencing learning | |
| Han et al. | Two-phase assessment approach to improve the efficiency of refactoring identification | |
| CN113610225A (en) | Quality evaluation model training method and device, electronic equipment and storage medium | |
| Djedović et al. | Model business process improvement by statistical analysis of the users' conduct in the process | |
| CN114548353A (en) | Model training method, electronic device and storage medium | |
| CN111309572B (en) | Test analysis method and device, computer equipment and storage medium | |
| CN115203057B (en) | Low code test automation method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201204 |