CN113095563A - Method and device for reviewing prediction result of artificial intelligence model - Google Patents
Method and device for reviewing prediction result of artificial intelligence model Download PDFInfo
- Publication number
- CN113095563A CN113095563A CN202110374361.2A CN202110374361A CN113095563A CN 113095563 A CN113095563 A CN 113095563A CN 202110374361 A CN202110374361 A CN 202110374361A CN 113095563 A CN113095563 A CN 113095563A
- Authority
- CN
- China
- Prior art keywords
- review
- prediction result
- prediction
- reviewing
- artificial intelligence
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 62
- 238000013473 artificial intelligence Methods 0.000 title claims abstract description 51
- 238000012552 review Methods 0.000 claims abstract description 219
- 238000005457 optimization Methods 0.000 claims abstract description 26
- 238000001514 detection method Methods 0.000 claims description 26
- 230000015654 memory Effects 0.000 claims description 21
- 238000012795 verification Methods 0.000 claims description 6
- 238000013398 bayesian method Methods 0.000 claims description 3
- 230000007547 defect Effects 0.000 abstract description 13
- 230000005540 biological transmission Effects 0.000 description 8
- 238000010200 validation analysis Methods 0.000 description 5
- 238000007689 inspection Methods 0.000 description 4
- 238000013145 classification model Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000010295 mobile communication Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 230000002950 deficient Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000007429 general method Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/04—Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem"
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/10—Office automation; Time management
- G06Q10/103—Workflow collaboration or project management
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Strategic Management (AREA)
- Human Resources & Organizations (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Entrepreneurship & Innovation (AREA)
- Data Mining & Analysis (AREA)
- Economics (AREA)
- Physics & Mathematics (AREA)
- Marketing (AREA)
- Operations Research (AREA)
- General Business, Economics & Management (AREA)
- Tourism & Hospitality (AREA)
- Quality & Reliability (AREA)
- Life Sciences & Earth Sciences (AREA)
- Game Theory and Decision Science (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Computation (AREA)
- Development Economics (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a review method and a review device for a prediction result of an artificial intelligence model, wherein the review method comprises the following steps: obtaining a prediction result of the artificial intelligence model and a prediction category and a confidence coefficient corresponding to the prediction result; determining a review sequence and a review range according to the prediction category and the confidence coefficient; and carrying out the reexamination on the prediction result based on the preset optimization condition, the reexamination sequence and the reexamination range to obtain a reexamination result. By implementing the method and the device, the prediction result of the artificial intelligent model is reviewed, the process of manual review is avoided, the defect of manual review of the sample based on a preset scheme can be effectively overcome, and the review result is more accurate.
Description
Technical Field
The invention relates to the technical field of equipment detection, in particular to a method and a device for reviewing a prediction result of an artificial intelligence model.
Background
At present, artificial intelligence models are used in a number of industries. For example, in the field of power transmission inspection, an unmanned aerial vehicle shoots equipment such as a power transmission tower, and possible problems in a shot picture are screened out based on a target detection model; in the field of security protection, a possible criminal suspect escaping is found based on a face detection and face comparison model; in the Internet field, based on a text classification model, user speeches related to yellow, politics and violence are found; in the industrial field, defective products are found based on a picture target detection model. However, since the artificial intelligence model is not completely accurate, after the artificial intelligence model is applied to find potential problems, a manual review or a manual review is often required, and therefore, a decision needs to be made as to which samples are to be reviewed. The current general method is as follows: and estimating the false detection rate and the missed detection rate of the model under different confidence coefficient conditions based on the verification set, and setting a confidence coefficient threshold value of manual review/manual review by combining actual requirements.
However, when the system of model prediction and manual review is actually applied, because the sample collection mode, the sample source, the sample characteristics, and the like are different from the training set, the false detection rate and the false detection rate of the model prediction result at different confidence levels are different from the value estimated based on the verification set, so the manual review scheme based on the preset confidence threshold causes the waste of manpower: on one hand, the number of samples of the manual review is possibly beyond/lower than the expected number, so that the review cost is beyond the expected cost or the problem detection number is lower; on the other hand, for multi-class prediction results, the workload of manual review may be concentrated on a few classes that are frequently misreported, and the number of problem samples found by review or the number of filtered false-reported samples is low due to the mismatch of review resources.
Disclosure of Invention
In view of this, embodiments of the present invention provide a review method and an apparatus for a prediction result of an artificial intelligence model, so as to solve the problems of low review number, low review accuracy and the like in the prior art when a manual review mode is used to review the prediction result of the artificial intelligence model.
The embodiment of the invention provides a review method for a prediction result of an artificial intelligence model, which comprises the following steps: obtaining a prediction result of an artificial intelligence model and a prediction type and a confidence coefficient corresponding to the prediction result; determining a review sequence and a review range according to the prediction category and the confidence coefficient; and reviewing the prediction result based on preset optimization conditions, the review sequence and the review range to obtain a review result.
Optionally, the review method further includes: and adjusting the review sequence and/or review range of the prediction result according to the review result.
Optionally, the adjusting the review order and/or the review range of the prediction result according to the review result includes: determining a prediction result sample which is subjected to the review according to the review result; calculating the number of correct and wrong samples predicted in a preset range and the corresponding proportion in the prediction result samples after the review; the preset range includes: and at least one of a preset confidence interval, a preset prediction type result and a type marked by the prediction result sample.
Optionally, the preset optimization condition includes at least one of the following three items: under the condition that the total review quantity is determined, the detection quantity of the problem samples reaches the maximum; when the detection number of the problem samples is determined, the total review number reaches the minimum; and under the condition that the total review quantity is determined, the false alarm quantity of the artificial intelligent model determined by the review reaches the maximum.
Optionally, the reviewing the prediction result based on the preset optimization condition and the review order includes: based on the prediction confidence of the artificial intelligence model, carrying out a review from high to low, or carrying out a review from low to high from a preset threshold value; and when the number of the review reaches the preset number or the number of the detected problems reaches the preset number, terminating the review.
Optionally, the reviewing the prediction result based on the preset optimization condition and the review order includes: setting prior distribution of accuracy rates of the artificial intelligence model in different confidence degrees of each category by using verification set data based on a Bayesian method; determining the review sequence based on a multi-arm slot machine method, and calculating posterior distribution of model accuracy in the review process; determining a termination condition of the review based on the optimization condition, wherein the termination condition comprises: when the number of the review reaches the preset number or the number of the detected problems reaches the preset number, terminating the review; and reviewing the prediction result based on the prior distribution, the posterior distribution and the termination condition.
Optionally, the reviewing the prediction result based on the preset optimization condition and the review order includes: and selecting a prediction result sample with the highest confidence coefficient of the non-review in the prediction category for review.
Optionally, the review method further includes: for the prediction result samples outside the review range, judging the prediction result samples with the confidence coefficient lower than that of the manual review samples as being problem-free; the predicted result samples with higher confidence than the manually reviewed samples are considered problematic.
The embodiment of the invention also provides a review device for the prediction result of the artificial intelligence model, which comprises: the prediction result acquisition module is used for acquiring the prediction result of the artificial intelligence model and the prediction type and the confidence coefficient corresponding to the prediction result; a review order and range determining module for determining a review order and a review range according to the prediction category and the confidence; and the review module is used for reviewing the prediction result based on preset optimization conditions, the review sequence and the review range to obtain a review result.
An embodiment of the present invention further provides an electronic device/mobile terminal/server, including: the artificial intelligence model prediction review method comprises a memory and a processor, wherein the memory and the processor are connected in a communication mode, the memory stores computer instructions, and the processor executes the computer instructions to execute the artificial intelligence model prediction result review method in the first aspect or any one implementation manner of the first aspect.
An embodiment of the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores a computer instruction, where the computer instruction is configured to enable the computer to execute the method for reviewing the artificial intelligence model prediction result in the first aspect or any implementation manner of the first aspect.
The embodiment of the invention has the advantages that the review method of the embodiment of the invention carries out review aiming at the prediction result of the artificial intelligent model, avoids the process of artificial review, can effectively avoid the defect of carrying out sample artificial review based on the preset scheme, has more accurate review result, can adjust the review strategy according to the data characteristics in real time, can more accurately estimate the review workload and more efficiently search problem samples or false-alarm samples.
Drawings
The features and advantages of the present invention will be more clearly understood by reference to the accompanying drawings, which are illustrative and not to be construed as limiting the invention in any way, and in which:
FIG. 1 is a flow chart of a review method for the prediction results of an artificial intelligence model according to an embodiment of the present invention;
FIG. 2 is a schematic structural diagram of a review device for artificial intelligence model prediction results according to an embodiment of the present invention;
fig. 3 shows a hardware configuration diagram of a computer device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The embodiment of the invention provides a review method for a prediction result of an artificial intelligence model, which comprises the following steps of:
step S101: and obtaining a prediction result of the artificial intelligence model and a prediction type and a confidence degree corresponding to the prediction result. In this embodiment, the review method is mainly a scheme of reviewing the prediction result of the artificial intelligence model, and therefore, the prediction result of the artificial intelligence model and the prediction type and the confidence degree corresponding to the prediction result need to be obtained first. Optionally, in this embodiment, an object to which a prediction result of the artificial intelligence model is directed is not limited, and the prediction result may be, for example, a defect detection result applied in the power transmission inspection field, or an identification result applied in the internet field and used for identifying data, information, and the like appearing in the network.
Step S102: and determining a review sequence and a review range according to the prediction category and the confidence coefficient.
Step S103: and reviewing the prediction result based on preset optimization conditions, the review sequence and the review range to obtain a review result. In this embodiment, the preset optimization condition may mainly include at least one of the following three items: under the condition that the total review quantity is determined, the detection quantity of the problem samples reaches the maximum; when the detection number of the problem samples is determined, the total review number reaches the minimum; and under the condition that the total review quantity is determined, the false alarm quantity of the artificial intelligent model determined by the review reaches the maximum.
By means of the review method, the defects of manual review of samples based on the preset scheme can be effectively overcome by reviewing the prediction result of the artificial intelligent model, the review result is more accurate, the review strategy can be adjusted in real time according to the data characteristics, the review workload can be estimated more accurately, and problem samples or false-reported samples can be searched more efficiently.
Optionally, in some embodiments of the present invention, the review method further includes: and adjusting the review sequence and/or review range of the prediction result according to the review result. Specifically, this adjustment process mainly includes:
determining a prediction result sample which is subjected to the review according to the review result; calculating the number of correct and wrong samples predicted in a preset range and the corresponding proportion in the prediction result samples after the review; the preset range includes: and at least one of a preset confidence interval, a preset prediction type result and a type marked by the prediction result sample.
Through the process, the defect of sample review based on the preset scheme can be effectively overcome, the review strategy is adjusted in real time according to the data characteristics, the review workload is accurately estimated, and problem samples or false-reported samples are effectively searched.
Optionally, in some embodiments of the present invention, in the step S103, a process of reviewing the prediction result based on a preset optimization condition and the review order includes: based on the prediction confidence of the artificial intelligence model, carrying out a review from high to low, or carrying out a review from low to high from a preset threshold value; and when the number of the review reaches the preset number or the number of the detected problems reaches the preset number, terminating the review. Based on this process, the optimization conditions can be achieved: when the detection number of the problem samples is determined, the total review number reaches the minimum; and under the condition that the total review quantity is determined, the false alarm quantity of the artificial intelligent model determined by the review reaches the maximum.
Optionally, in some embodiments of the present invention, in the step S103, a process of reviewing the prediction result based on a preset optimization condition and the review order includes: setting prior distribution of accuracy rates of the artificial intelligence model in different confidence degrees of each category by using verification set data based on a Bayesian method; determining the review sequence based on a multi-arm slot machine method, and calculating posterior distribution of model accuracy in the review process; determining a termination condition of the review based on the optimization condition, wherein the termination condition comprises: when the number of the review reaches the preset number or the number of the detected problems reaches the preset number, terminating the review; and reviewing the prediction result based on the prior distribution, the posterior distribution and the termination condition.
In this embodiment, the validation set data is used to set a prior distribution of the model at different confidence levels for each class (assuming that the ratio of the model accuracy to the validation set accuracy is constant at different confidence levels for the different classes, and assuming that the prior of the model accuracy at confidence level P0 obeys Beta (tA, (1-t) a), where t is the validation set accuracy and a is a predetermined integer, typically 10 to 50); based on the multi-arm slot machine method, a review sequence is determined (the category with the maximum estimation value of 90% quantile points on the accuracy is selected, the sample which is not reviewed in the category and has the maximum confidence coefficient is used as the next sample to be reviewed), and the posterior distribution of the model accuracy is calculated in the review process (for each category, the picture with the maximum confidence coefficient in the unexamined samples is selected, the confidence coefficient is recorded as P, and the upper 90% quantile points of the model accuracy when the confidence coefficient of the category is P are calculated according to the reviewed pictures).
Through this process, the optimization conditions can be achieved: under the condition that the total review quantity is determined, the detection quantity of the problem samples reaches the maximum; and when the detection number of the problem samples is determined, the total review number reaches the minimum.
Optionally, in some embodiments of the present invention, in the step S103, a process of reviewing the prediction result based on a preset optimization condition and the review order includes: and selecting a prediction result sample with the highest confidence coefficient of the non-review in the prediction category for review.
For the multi-class problem, n (n > ═ 1) conditions are preset, wherein the condition i is continuous ki (ki > ═ 1) review samples of the class, and the number of samples with correct model prediction results does not exceed ri. Firstly, selecting a category, and selecting a sample with the highest confidence coefficient of the non-review in the category for review. If at least one condition is met, switching to the next category; otherwise, the review of the category is continued. And if the total review amount or the number of detected problem/defect samples reaches a given value, stopping the review. For each preset condition, ki and ri are preset positive integers, for example: two conditions, k 1-3, r 1-0, k 2-10, and r 2-4, may be preset, i.e., if for a certain category, there is no problem sample in the first 3 samples reviewed (r 1-0), or the number of problem samples in 10 consecutive samples does not exceed 4, then switching to the next category.
Through this process, the optimization conditions can be achieved: under the condition that the total review quantity is determined, the detection quantity of the problem samples reaches the maximum; and when the detection number of the problem samples is determined, the total review number reaches the minimum.
Further, for the optimization conditions: and under the condition that the total review quantity is determined, the false alarm quantity of the artificial intelligent model determined by the review reaches the maximum. The false alarm condition can be found out as much as possible by repeating the test from low to high according to the confidence coefficient, at this time, n (n > -1) conditions are preset, wherein the condition i is continuous ki (ki > -1) review samples of the category, and the number of samples with correct model prediction results is not less than ri.
Optionally, in some embodiments of the present invention, the review method further includes: for the prediction result samples outside the review range, judging the prediction result samples with the confidence coefficient lower than that of the manual review samples as being problem-free; the predicted result samples with higher confidence than the manually reviewed samples are considered problematic.
The review method for the artificial intelligence model prediction result of the embodiment of the invention is further explained by combining the specific application example.
Application example 1: the intelligent defect detection system is applied to the field of power transmission inspection and carries out intelligent defect detection on power transmission equipment. The system mainly comprises the following modules:
(1) and the picture acquisition module. The picture to be detected can be input into the system through modes of remote manual uploading of a user, uploading of the image acquired by the unmanned aerial vehicle through wifi or a mobile communication network, copying by using a mobile storage device and the like. The picture to be detected can be a picture shot for the inspection of all or part of the towers of one power transmission line.
(2) And a defect intelligent detection module. And predicting the position, the category and the confidence coefficient of a target frame of the hidden danger of the power transmission equipment possibly contained in the picture by using a target detection model (namely the artificial intelligence model in the embodiment of the invention) to obtain a prediction result.
(3) And a review module. And obtaining the accuracy of each category of the intelligent defect detection model on different confidence levels based on the verification set. Assuming that the ratio of model accuracy to validation set accuracy is constant for different classes at different confidence levels, and assuming that the prior of model accuracy at confidence level P0 obeys Beta (tA, (1-t) A), where t is the validation set accuracy and A is a predetermined integer, typically 10 to 50. When the picture is reviewed, if the ratio of the number of reviewed samples (or the number of false alarms in the reviewed samples) to the total picture number reaches a given value, the review is finished; otherwise, firstly, selecting the picture with the maximum confidence level in the unexamined samples for each category, recording the confidence level as P, and calculating the upper 90% quantile point of the model accuracy rate when the confidence level of the category is P according to the reviewed pictures; selecting the category with the maximum estimation value of 90% quantile points on the accuracy rate, returning an unexamined sample with the maximum confidence coefficient in the category as the next sample to be reviewed, and displaying the sample to a user; and after the user finishes the review, collecting and recording the review result of the user on the sample.
(4) And a report generation module. And obtaining a defect list based on the review result, and generating a defect report.
Application example 2: the data auditing system is applied to the field of the Internet and is used for network transmission data. The system comprises the following modules:
(1) and a primary examination module. The initial examination module calls an AI classification model (namely an artificial intelligence model), predicts the input data analysis request, determines the requests of violation categories such as yellow and storm related and with confidence coefficient exceeding a corresponding category given threshold 2 as not passing the examination according to the classification result and the confidence coefficient, determines the requests of the violation categories with confidence coefficient not lower than the corresponding threshold 1 and not higher than the corresponding threshold 2 as needing manual review, and determines the classification results of other categories as passing the examination. And returning the initial examination result, and sending a request to be reexamined to a reexamination module to wait for reexamination.
(2) And a review module. And providing the text contained in the received review request to a review module, and placing the text in a to-be-reviewed list. Calculating (a) review speed every fixed time; (b) estimating the generation speed of the review request of each confidence interval of each category; (c) and estimating the model false alarm rate of each confidence interval of each category. The length of the confidence interval is a preset value, and may be 1%, 5%, or the like. And calculating the maximum review workload according to the (a) and the (b) and a preset condition. Then, the threshold 2 of each category is calculated according to the maximum review workload, so that the number of model false reports of requests of each category between the thresholds 1 and 2 reaches the maximum. And resetting the threshold 2 of the initial review module according to the calculation result, moving the sample with the confidence coefficient exceeding the threshold 2 out of the list to be reviewed in the request to be reviewed, and setting the state of the sample to be reviewed as pass. And displaying the samples in the list to be reviewed to the reviewers in sequence according to the time sequence of entering the list, and setting the state of the corresponding sample as approved or not approved according to the judgment result of the reviewers on the samples.
The embodiment of the present invention further provides a review device for the prediction result of the artificial intelligence model, as shown in fig. 2, the review device includes:
the prediction result obtaining module 1 is used for obtaining a prediction result of the artificial intelligence model and a prediction type and a confidence coefficient corresponding to the prediction result; for details, reference may be made to the related description of step S101 in the above method embodiment, and details are not repeated herein.
A review order and range determining module 2, configured to determine a review order and a review range according to the prediction category and the confidence; for details, reference may be made to the related description of step S102 in the above method embodiment, and details are not repeated herein.
And the review module 3 is used for reviewing the prediction result based on preset optimization conditions, the review sequence and the review range to obtain a review result. For details, reference may be made to the related description of step S103 in the above method embodiment, and details are not repeated here.
By means of the review device, review is conducted according to the prediction result of the artificial intelligent model, the defect that manual review of the sample is conducted based on the preset scheme can be effectively overcome, the review result is more accurate, the review strategy can be adjusted in real time according to the data characteristics, the review workload can be estimated more accurately, and the problem sample or the false-reported sample can be found more efficiently.
An embodiment of the present invention further provides a computer device, as shown in fig. 3, the computer device may include a processor 31 and a memory 32, where the processor 31 and the memory 32 may be connected by a bus or in another manner, and fig. 3 takes the example of being connected by a bus as an example.
The processor 31 may be a Central Processing Unit (CPU). The Processor 31 may also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, or combinations thereof.
The memory 32 is a non-transitory computer readable storage medium, and can be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules corresponding to the review method for the artificial intelligence model prediction results in the embodiment of the present invention (for example, the prediction result obtaining module 1, the review order and scope determining module 2, and the review module 3 shown in fig. 2). The processor 31 executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory 32, namely, implements the review method of the prediction result of the artificial intelligence model in the above method embodiment.
The memory 32 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created by the processor 31, and the like. Further, the memory 32 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory 32 may optionally include memory located remotely from the processor 31, and these remote memories may be connected to the processor 31 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The one or more modules are stored in the memory 32 and, when executed by the processor 31, perform a review method of artificial intelligence model predictions as in the embodiment of FIG. 1.
The details of the computer device may be understood by referring to the corresponding descriptions and effects in the embodiments shown in fig. 1 to fig. 2, and are not described herein again.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory (Flash Memory), a Hard Disk (Hard Disk Drive, abbreviated as HDD), a Solid State Drive (SSD), or the like; the storage medium may also comprise a combination of memories of the kind described above.
Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations fall within the scope defined by the appended claims.
Claims (11)
1. A review method for the prediction result of an artificial intelligence model is characterized by comprising the following steps:
obtaining a prediction result of an artificial intelligence model and a prediction type and a confidence coefficient corresponding to the prediction result;
determining a review sequence and a review range according to the prediction category and the confidence coefficient;
and reviewing the prediction result based on preset optimization conditions, the review sequence and the review range to obtain a review result.
2. The method for reviewing the prediction results of the artificial intelligence model according to claim 1, wherein the reviewing method further comprises:
and adjusting the review sequence and/or review range of the prediction result according to the review result.
3. The method for reviewing the artificial intelligence model prediction result of claim 2, wherein the adjusting the review order and/or the review range of the prediction result according to the review result comprises:
determining a prediction result sample which is subjected to the review according to the review result;
calculating the number of correct and wrong samples predicted in a preset range and the corresponding proportion in the prediction result samples after the review; the preset range includes: and at least one of a preset confidence interval, a preset prediction type result and a type marked by the prediction result sample.
4. The method for reviewing the prediction result of the artificial intelligence model according to claim 1, wherein the preset optimization condition comprises at least one of the following three items:
under the condition that the total review quantity is determined, the detection quantity of the problem samples reaches the maximum;
when the detection number of the problem samples is determined, the total review number reaches the minimum;
and under the condition that the total review quantity is determined, the false alarm quantity of the artificial intelligent model determined by the review reaches the maximum.
5. The method for reviewing the prediction result of the artificial intelligence model according to claim 1 or 4, wherein reviewing the prediction result based on the preset optimization condition and the review order comprises:
based on the prediction confidence of the artificial intelligence model, carrying out a review from high to low, or carrying out a review from low to high from a preset threshold value;
and when the number of the review reaches the preset number or the number of the detected problems reaches the preset number, terminating the review.
6. The method for reviewing the prediction result of the artificial intelligence model according to claim 1 or 4, wherein reviewing the prediction result based on the preset optimization condition and the review order comprises:
setting prior distribution of accuracy rates of the artificial intelligence model in different confidence degrees of each category by using verification set data based on a Bayesian method;
determining the review sequence based on a multi-arm slot machine method, and calculating posterior distribution of model accuracy in the review process;
determining a termination condition of the review based on the optimization condition, wherein the termination condition comprises: when the number of the review reaches the preset number or the number of the detected problems reaches the preset number, terminating the review;
and reviewing the prediction result based on the prior distribution, the posterior distribution and the termination condition.
7. The method for reviewing the prediction result of the artificial intelligence model according to claim 1 or 4, wherein reviewing the prediction result based on the preset optimization condition and the review order comprises:
and selecting a prediction result sample with the highest confidence coefficient of the non-review in the prediction category for review.
8. The method for reviewing the prediction results of the artificial intelligence model according to claim 1, wherein the reviewing method further comprises:
for the prediction result samples outside the review range, judging the prediction result samples with the confidence coefficient lower than that of the manual review samples as being problem-free; the predicted result samples with higher confidence than the manually reviewed samples are considered problematic.
9. A review device for the prediction result of an artificial intelligence model is characterized by comprising:
the prediction result acquisition module is used for acquiring the prediction result of the artificial intelligence model and the prediction type and the confidence coefficient corresponding to the prediction result;
a review order and range determining module for determining a review order and a review range according to the prediction category and the confidence;
and the review module is used for reviewing the prediction result based on preset optimization conditions, the review sequence and the review range to obtain a review result.
10. A computer device, comprising:
a memory and a processor, the memory and the processor being communicatively connected to each other, the memory having stored therein computer instructions, the processor executing the computer instructions to perform the method for reviewing the artificial intelligence model prediction result according to any one of claims 1-8.
11. A computer-readable storage medium storing computer instructions for causing a computer to perform the method of reviewing artificial intelligence model predictors according to any one of claims 1 to 8.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110374361.2A CN113095563A (en) | 2021-04-07 | 2021-04-07 | Method and device for reviewing prediction result of artificial intelligence model |
| PCT/CN2021/123302 WO2022213565A1 (en) | 2021-04-07 | 2021-10-12 | Review method and apparatus for prediction result of artificial intelligence model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110374361.2A CN113095563A (en) | 2021-04-07 | 2021-04-07 | Method and device for reviewing prediction result of artificial intelligence model |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113095563A true CN113095563A (en) | 2021-07-09 |
Family
ID=76674770
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110374361.2A Pending CN113095563A (en) | 2021-04-07 | 2021-04-07 | Method and device for reviewing prediction result of artificial intelligence model |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN113095563A (en) |
| WO (1) | WO2022213565A1 (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114612752A (en) * | 2022-03-02 | 2022-06-10 | 捻果科技(深圳)有限公司 | Method and system for intelligently reviewing recognition result of video analysis technology |
| WO2022213565A1 (en) * | 2021-04-07 | 2022-10-13 | 全球能源互联网研究院有限公司 | Review method and apparatus for prediction result of artificial intelligence model |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117579625B (en) * | 2024-01-17 | 2024-04-09 | 中国矿业大学 | Inspection task pre-distribution method for double prevention mechanism |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107196953A (en) * | 2017-06-14 | 2017-09-22 | 上海丁牛信息科技有限公司 | A kind of anomaly detection method based on user behavior analysis |
| CN107729924A (en) * | 2017-09-25 | 2018-02-23 | 平安科技（深圳）有限公司 | Picture review probability interval generation method and picture review decision method |
| CN109658042A (en) * | 2018-10-08 | 2019-04-19 | 平安科技（深圳）有限公司 | Review method, apparatus, equipment and storage medium based on artificial intelligence |
| CN110689531A (en) * | 2019-09-23 | 2020-01-14 | 云南电网有限责任公司电力科学研究院 | Automatic power transmission line machine inspection image defect identification method based on yolo |
| CN110992329A (en) * | 2019-11-28 | 2020-04-10 | 上海微创医疗器械（集团）有限公司 | Product surface defect detection method, electronic device and readable storage medium |
| CN111597958A (en) * | 2020-05-12 | 2020-08-28 | 西安网算数据科技有限公司 | Highly automated bill classification method and system |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10558920B2 (en) * | 2017-10-02 | 2020-02-11 | Servicenow, Inc. | Machine learning classification with confidence thresholds |
| CN108805173A (en) * | 2018-05-16 | 2018-11-13 | 苏州迈为科技股份有限公司 | Solar battery sheet aberration method for separating |
| CN108875963A (en) * | 2018-06-28 | 2018-11-23 | 北京字节跳动网络技术有限公司 | Optimization method, device, terminal device and the storage medium of machine learning model |
| CN112508044A (en) * | 2019-09-16 | 2021-03-16 | 华为技术有限公司 | Artificial intelligence AI model evaluation method, system and equipment |
| CN111124863B (en) * | 2019-12-24 | 2024-02-13 | 北京安兔兔科技有限公司 | Intelligent device performance testing method and device and intelligent device |
| CN111753914B (en) * | 2020-06-29 | 2024-04-16 | 北京百度网讯科技有限公司 | Model optimization method and device, electronic equipment and storage medium |
| CN113095563A (en) * | 2021-04-07 | 2021-07-09 | 全球能源互联网研究院有限公司 | Method and device for reviewing prediction result of artificial intelligence model |
-
2021
- 2021-04-07 CN CN202110374361.2A patent/CN113095563A/en active Pending
- 2021-10-12 WO PCT/CN2021/123302 patent/WO2022213565A1/en active Application Filing
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107196953A (en) * | 2017-06-14 | 2017-09-22 | 上海丁牛信息科技有限公司 | A kind of anomaly detection method based on user behavior analysis |
| CN107729924A (en) * | 2017-09-25 | 2018-02-23 | 平安科技（深圳）有限公司 | Picture review probability interval generation method and picture review decision method |
| CN109658042A (en) * | 2018-10-08 | 2019-04-19 | 平安科技（深圳）有限公司 | Review method, apparatus, equipment and storage medium based on artificial intelligence |
| CN110689531A (en) * | 2019-09-23 | 2020-01-14 | 云南电网有限责任公司电力科学研究院 | Automatic power transmission line machine inspection image defect identification method based on yolo |
| CN110992329A (en) * | 2019-11-28 | 2020-04-10 | 上海微创医疗器械（集团）有限公司 | Product surface defect detection method, electronic device and readable storage medium |
| CN111597958A (en) * | 2020-05-12 | 2020-08-28 | 西安网算数据科技有限公司 | Highly automated bill classification method and system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022213565A1 (en) * | 2021-04-07 | 2022-10-13 | 全球能源互联网研究院有限公司 | Review method and apparatus for prediction result of artificial intelligence model |
| CN114612752A (en) * | 2022-03-02 | 2022-06-10 | 捻果科技(深圳)有限公司 | Method and system for intelligently reviewing recognition result of video analysis technology |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2022213565A1 (en) | 2022-10-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9129228B1 (en) | Robust and fast model fitting by adaptive sampling | |
| CN109726763B (en) | Information asset identification method, device, equipment and medium | |
| CN111210399B (en) | Imaging quality evaluation method, device and equipment | |
| CN113095563A (en) | Method and device for reviewing prediction result of artificial intelligence model | |
| CN111309539A (en) | Abnormity monitoring method and device and electronic equipment | |
| CN112528801B (en) | Abnormal event detection method, model training method and device | |
| CN112613569B (en) | Image recognition method, training method and device for image classification model | |
| CN110869942B (en) | Self-feed deep learning method and system | |
| US20200349364A1 (en) | Method and apparatus for training lane line identifying model, device, and storage medium | |
| KR20200123584A (en) | Apparatus and method for predicting error of annotation | |
| CN113377484A (en) | Popup window processing method and device | |
| CN112307900A (en) | Method and device for evaluating facial image quality and electronic equipment | |
| CN113516251B (en) | Machine learning system and model training method | |
| CN114218505A (en) | Abnormal space-time point identification method and device, electronic equipment and storage medium | |
| CN109960626A (en) | Recognition methods, device, equipment and the medium of port exception | |
| CN112434717A (en) | Model training method and device | |
| CN116863288A (en) | Target detection and alarm method, device and equipment based on deep learning | |
| CN111783883A (en) | Abnormal data detection method and device | |
| CN112149698A (en) | Method and device for screening difficult sample data | |
| CN112152968B (en) | Network threat detection method and device | |
| CN113807209A (en) | Parking space detection method and device, electronic equipment and storage medium | |
| CN111523826A (en) | Data acquisition method, device and equipment | |
| CN111191593A (en) | Image target detection method and device, storage medium and sewage pipeline detection device | |
| CN115861321B (en) | Production environment detection method and system applied to industrial Internet | |
| CN116450632B (en) | Geographic sample data quality evaluation method, device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |