CN117764427A - Electric power artificial intelligent model success feedback evaluation method and system - Google Patents
Electric power artificial intelligent model success feedback evaluation method and system Download PDFInfo
- Publication number
- CN117764427A CN117764427A CN202311370267.5A CN202311370267A CN117764427A CN 117764427 A CN117764427 A CN 117764427A CN 202311370267 A CN202311370267 A CN 202311370267A CN 117764427 A CN117764427 A CN 117764427A
- Authority
- CN
- China
- Prior art keywords
- model
- feedback
- feedback information
- user
- confidence
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000011156 evaluation Methods 0.000 title claims description 83
- 230000006399 behavior Effects 0.000 claims abstract description 87
- 230000000694 effects Effects 0.000 claims abstract description 36
- 238000000034 method Methods 0.000 claims abstract description 26
- 238000013473 artificial intelligence Methods 0.000 claims description 18
- 238000004873 anchoring Methods 0.000 claims description 17
- 238000004590 computer program Methods 0.000 claims description 6
- 238000005457 optimization Methods 0.000 claims description 6
- 238000012216 screening Methods 0.000 claims description 4
- 230000005611 electricity Effects 0.000 claims 1
- 238000012549 training Methods 0.000 description 5
- 238000012545 processing Methods 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 3
- 238000012937 correction Methods 0.000 description 3
- 230000008094 contradictory effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011867 re-evaluation Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a method and a system for evaluating the success feedback of an electric artificial intelligent model, comprising the following steps: acquiring operation behavior data and corresponding model effect feedback information of a user during each use of a model, wherein the user operation behavior data comprises keyboard operation behaviors, mouse operation behaviors and page operation behaviors; evaluating the confidence level of the effect feedback information of the model according to the similarity of the operation behavior data of each time and the operation habit model of the user; the operation habit model is built based on historical operation behavior data of a user; all the effect feedback information aiming at a certain target model is obtained, weighting is carried out according to the confidence coefficient of each piece of feedback information, and a corrected effect feedback result is obtained.
Description
Technical Field
The invention belongs to the technical field of artificial intelligence, and particularly relates to a method and a system for evaluating the success feedback of an electric artificial intelligence model.
Background
The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art.
A good-use and excellent artificial intelligence model has good prediction capability on not only known data but also unknown data, and a large amount of data is needed for training the model. However, training by simple feeding data also causes poor generalization ability of the model and deviation of output results, so that evaluation and correction of feedback results of the model are required.
For the artificial intelligent model in the power industry, model evaluation is performed by using aspects such as training result evaluation (training result accuracy, recall rate and the like), calling result evaluation (model calling success rate and the like), model processing result evaluation (model identification accuracy, processing result recall rate) and the like. Through the evaluation indexes, the trained model algorithm can be reasonably and correctly evaluated, so that the working result of the user is correctly applied to the business task.
However, when the algorithm model is currently trained and corrected, most of the results output by the algorithm model are judged by technicians, and then evaluation is given for correcting the model, so that time and effort of developers are extremely occupied, manpower is consumed, and the obtained data volume is limited. The electric artificial intelligent model system provides an artificial intelligent model suitable for operation state identification and fault identification under various scenes for users, the users can identify or predict target data through online calling, if the users can give feedback for model correction in the using process, a large amount of data can be obtained, a large amount of manpower can be saved, however, the prior art can easily obtain feedback evaluation of the users on the model, but the validity of the feedback evaluation cannot be guaranteed, sometimes after the users use the model, the users can not carefully submit feedback, and have quite probability to choose feedback options for submission by hand, if the user feedback is taken as the basis of model calibration, data pollution is likely to be caused, the final effect of the model is affected, and therefore, the online operation flow and experience results of the users are required to be analyzed, the user evaluation feedback is accurately quantized, and the user feedback rationality is judged.
Disclosure of Invention
In order to overcome the defects in the prior art, the invention provides a method and a system for evaluating the performance feedback of an electric artificial intelligent model. Confidence evaluation can be carried out on the success feedback information of the model, and guarantee is provided for optimization of the follow-up model.
To achieve the above object, one or more embodiments of the present invention provide the following technical solutions:
the method for evaluating the success feedback of the electric artificial intelligent model comprises the following steps:
acquiring operation behavior data and corresponding model effect feedback information of a user during each use of a model, wherein the user operation behavior data comprises keyboard operation behaviors, mouse operation behaviors and page operation behaviors;
evaluating the confidence level of the effect feedback information of the model according to the similarity of the operation behavior data of each time and the operation habit model of the user; the operation habit model is built based on historical operation behavior data of a user;
and obtaining all the effect feedback information aiming at a certain target model, weighting according to the confidence coefficient of each piece of feedback information, and obtaining a corrected effect feedback result which is used for learning optimization of the target model.
Further, evaluating the confidence of the model performance feedback information includes:
selecting a plurality of operation behaviors as evaluation indexes, and distributing weights for each operation behavior;
and comparing the similarity of each operation behavior corresponding to the sub-feedback information with the similarity of the operation behaviors corresponding to the operation habit model, and carrying out weighted summation on the similarity of each operation behavior to obtain the confidence coefficient of the sub-feedback information.
Further, after evaluating the confidence coefficient of the performance feedback information of the current model, the method further comprises the following steps of: setting a plurality of anchoring indexes in the achievement feedback questionnaire, judging whether the feedback information is reliable or not according to the feedback condition of a user aiming at the plurality of anchoring indexes for certain feedback information to be evaluated, and if the feedback information is unreliable, the confidence is low.
Further, after evaluating the confidence level of each model success feedback information, the feedback evaluation confidence level of each user is also evaluated: setting a confidence threshold to distinguish whether each feedback evaluation is effective; and calculating the feedback evaluation confidence coefficient of each user according to the number of the effective feedback evaluation of each user.
Further, if the feedback contents of the same index are inconsistent in the model achievement feedback information of different users, the confidence coefficient of the feedback contents of the index is corrected by combining the feedback evaluation confidence coefficient of the users.
Further, evaluating the confidence of the model performance feedback information includes:
selecting a plurality of operation behaviors as evaluation indexes, and distributing weights for each operation behavior;
and comparing the similarity of each operation behavior corresponding to the sub-feedback information with the similarity of the operation behaviors corresponding to the operation habit model, and carrying out weighted summation on the similarity of each operation behavior to obtain the confidence coefficient of the sub-feedback information.
Further, after evaluating the confidence coefficient of the performance feedback information of the current model, the method further comprises the following steps of: setting a plurality of anchoring indexes in the achievement feedback questionnaire, judging whether the feedback information is reliable or not according to the feedback condition of a user aiming at the plurality of anchoring indexes for certain feedback information to be evaluated, and if the feedback information is unreliable, the confidence is low.
One or more embodiments provide a power artificial intelligence model performance feedback evaluation system comprising:
the data acquisition module is used for acquiring operation behavior data and corresponding model effect feedback information during each use of the model by a user, wherein the user operation behavior data comprises a keyboard, a mouse operation behavior and a page operation behavior;
the confidence evaluation module is used for evaluating the confidence of the effect feedback information of the model according to the similarity between the operation behavior data of each time and the operation habit model of the user; the operation habit model is built based on historical operation behavior data of a user;
and the feedback information screening module is used for acquiring all the effect feedback information aiming at a certain target model, weighting according to the confidence coefficient of each piece of feedback information, and obtaining corrected effect feedback results which are used for learning and optimizing the target model.
One or more embodiments provide an electronic device including a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the method of power artificial intelligence model performance feedback evaluation when executing the program.
One or more embodiments provide a computer readable storage medium having stored thereon a computer program which when executed by a processor implements the method of power artificial intelligence model performance feedback evaluation.
The invention has the following beneficial effects:
the invention creatively provides an evaluation method for the effect feedback information of the electric artificial intelligent model, which can judge whether the operation of the user meets the daily habit or not based on the operation behavior of the user by establishing the operation habit model of the user, so as to measure whether the model effect feedback information corresponding to the operation is credible or not, thereby realizing the credibility evaluation of the user feedback information and improving the feedback evaluation efficiency.
The invention innovatively provides a method for carrying out multistage evaluation on the feedback information of the user, by setting the anchoring index, the overall evaluation can be given to the feedback information of the user based on whether the feedback content of the anchoring index is correct or whether the feedback content of the anchoring index is contradictory, the re-evaluation of the feedback information of the user is realized through the information cross authentication of two layers, and the accuracy of the evaluation of the feedback information of the user is improved.
The invention innovatively provides a method for evaluating the overall reliability of a user based on each piece of model achievement feedback information, namely, the feedback evaluation confidence coefficient of the user, when the feedback contents of different users aiming at the same index are inconsistent, the weight is adjusted according to the feedback evaluation confidence coefficient of the user, and the feedback information is screened in two dimensions of each piece of feedback information and the overall user confidence coefficient, so that the forward optimization of the model is promoted, and the workload of developers aiming at model optimization is saved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention.
FIG. 1 is a flow diagram of an overall modeling performance feedback assessment method in accordance with one or more embodiments of the present invention.
Detailed Description
It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the invention. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of exemplary embodiments according to the present invention. As used herein, the singular is also intended to include the plural unless the context clearly indicates otherwise, and furthermore, it is to be understood that the terms "comprises" and/or "comprising" when used in this specification are taken to specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof.
Embodiments of the invention and features of the embodiments may be combined with each other without conflict.
Example 1
The artificial intelligence model system comprises various models in the technical direction of artificial intelligence, and comprises models such as image recognition, video recognition, voice recognition, text recognition and the like which are commonly used in the electric power field. The accuracy of these models depends largely on the user's feedback on how the model outputs the results correctly. The embodiment discloses a power artificial intelligent model achievement feedback evaluation method from the dimensions of online operation behaviors of users, feedback conditions of known achievement models and the like, and the feedback confidence of the users is evaluated to realize screening of user feedback information so as to guide correction optimization of the models. Referring to fig. 1, the method in this embodiment specifically includes the following steps:
step 1: and during each login of the user to the artificial intelligent model system, acquiring operation behavior data and model effect feedback information, and establishing an operation habit model for each user.
The user uses the model in the system to log in the system, and each user is allocated with a unique account (UID) in the system, so that the user is taken as the main distinguishing basis of the user identity, and the operation behavior and feedback information of the user are tracked based on the account information.
User operation behavior features are mainly divided into two aspects of hardware behavior and software behavior. The hardware aspect mainly refers to the operation of a keyboard and a mouse, including single click frequency and double click frequency of the mouse, the number and distance of dragging, the key-press frequency and the knocking frequency of the keyboard and the like. The behavior characteristics of the software mainly study the behavior characteristics of the user operating various software and corresponding operations in the software, such as page stay time, reading speed according to page text quantity, page turning habit, interface setting and the like. Those skilled in the art will appreciate that the operational activities include all activities after a user logs into the system, covering operations during use of the model.
Specifically, the preliminary operating habits of the user may be established by training the user to use the system a plurality of times during the course guidance phase. After the preliminary operation habit is established, the details of each operation are analyzed, and the operation habit model of the user can be obtained after long-term use by the user.
In the artificial intelligent model system, after a user performs operations such as identification or classification by using a model each time, feedback information of the user about model evaluation is collected. As a specific implementation manner, the feedback information is collected in a form of a questionnaire, and the questionnaire includes a plurality of evaluation indexes, and each evaluation index corresponds to a question.
After the operation behavior data and the model effect feedback information are obtained, data cleaning is further carried out, the data record with incomplete feedback information is regarded as invalid data, and subsequent analysis is not adopted.
And performing associated storage on the operation behavior data and the model effect feedback information after the user logs in each time.
The model of the user's operational habits includes a plurality of operational behaviors and quantized values of the user for the plurality of operational behaviors. For example, the plurality of operation behaviors include a mouse click frequency, a keyboard key press frequency, a page turning speed, and the like, and a typical operation behavior for a user is preferable depending on the user's historical operation behaviors. The operation habit model can be obtained by adopting the existing user behavior portrait construction method.
Step 2: acquiring operation behavior data and corresponding model effect feedback information of a user during each use of the model; and evaluating the confidence level of the effect feedback information of the model according to the similarity of the operation behavior data of each time and the historical operation habit model of the user.
In general, users using models carefully have larger differences in various aspects such as mouse moving speed, moving distance, clicking frequency, page stay, page switching, etc., than users using models informally such as trial use and browsing, and feedback evaluation given by the former is more credible than the latter. For the same user, if the same person uses the system, the account numbers of the user can be logged in, the account numbers uid, the corresponding operation habits, the interface settings and the like are all in one-to-one correspondence, if one or more items of the account numbers uid, the corresponding operation habits, the interface settings and the like are not corresponding, the user is considered to be problematic, the situation that the user logs in may exist, and no serious operation and feedback may be caused by the use scenes such as displaying or browsing to the user. When the behavior habit revealed by a user in a single operation is inconsistent with the consistent performance, the feedback confidence given by the operation is considered to be lower.
Based on the above principle, in this embodiment, for each feedback information of a user, according to an operation corresponding to the feedback information, comparison is performed based on an operation habit model of the user, and a confidence level of each feedback information is determined.
Specifically, multiple operation behaviors are selected as evaluation indexes, weights are distributed for each operation behavior, each operation behavior corresponding to certain feedback information to be evaluated is compared with the similarity of the operation behavior corresponding to the operation habit model, and the similarity corresponding to each operation behavior is weighted and summed to obtain the confidence coefficient of the feedback information.
Those skilled in the art can understand that the selected operation behavior and the weight coefficient can be adjusted according to the actual needs, so long as the confidence evaluation calculation formulas of all users are the same during each evaluation.
In addition, reevaluation is performed for each feedback information of the user: by setting a plurality of anchor indexes in the achievement feedback questionnaire, for certain feedback information to be evaluated, judging whether the feedback information is reliable or not according to feedback conditions of a user aiming at the anchor indexes, and if not, the confidence is low.
The plurality of anchor indexes may be indexes with clear and obvious answers, indexes with the same questions and options but different expressions, indexes with the same questions but different options sequences, and the like, and are not limited herein. For the indexes with clear and obvious answers, if the answer selected by the user is incorrect, the confidence coefficient of the feedback information is considered to be lower, and for the latter two anchoring indexes, if the answer selected is contradictory, the confidence coefficient of the feedback information is considered to be lower.
The confidence evaluation through the anchoring index is used as a one-ticket overrule item for secondary test, for example, different pages describe the same questions by using different descriptions, if the users answer seriously and normally, the two answers of the users are the same answer, if the contradiction between the users and the users appears, the users can consider that the users do not answer seriously, and the feedback is not credible; some answer answers are obvious and unique, and their answers are not in agreement with facts.
As an example, if the model is a mature model, several indexes with more outstanding performance, such as several results with extremely high calling success rate and extremely high result accuracy rate, can be selected as anchoring indexes. For these anchoring indexes, a plurality of feedback evaluation questions with different question methods and identical meanings are set for each index, and the questions are scattered in a user feedback evaluation page, so that in general, the feedback evaluation obtained by the questions can be predicted, if the feedback evaluation given by the user has a larger error with the predicted feedback evaluation or the answer to the same anchoring index has a contradiction, the feedback evaluation given by the user can be considered to have a lower confidence.
The confidence threshold is determined based on the confidence value range calculated in the step 2, and the accuracy of the confidence is determined by the accuracy of the user operation habit model, so that the more the effective feedback times of the user are, the more sufficient the sample data of the user operation habit model is, and the more reliable the obtained confidence is.
The embodiment realizes the judgment of whether the feedback evaluation is reliable or not based on whether the operation behavior corresponding to the feedback evaluation of the user accords with the behavior habit of the user or not. In addition, the feedback evaluation is reevaluated based on the anchoring index mode, so that cross authentication of various modes is realized, and the accuracy of the feedback evaluation is further ensured.
Step 3: and obtaining all the effect feedback information aiming at a certain target model, and weighting according to the confidence coefficient of each piece of feedback information to obtain a corrected effect feedback result for optimizing the target model.
All feedback information may relate to feedback information of multiple users, and there may be situations where the feedback information of users is different for certain evaluation indexes, for example: for the recognition result of a certain model on a certain image, some user feedback is correct in recognition, and some user feedback is incorrect. In this case, the present embodiment also performs the auxiliary discrimination in combination with the degree of reliability of the user.
In this embodiment, the reliability degree of the user is measured by using feedback evaluation confidence, and the feedback evaluation confidence of the user is evaluated according to the confidence of each feedback information of the user. An archive is established for each user for recording the validity of each feedback evaluation. The more times the user feedback evaluation is valid, the higher the feedback confidence and the higher the weight. Specifically, a confidence threshold is set so as to distinguish whether each feedback evaluation is valid or not; and calculating the feedback evaluation confidence coefficient of each user according to the number of the effective feedback evaluation of each user. When one user uses more times than other users and gives more effective feedback times than other users, the feedback content weight is considered to be higher.
Specifically, the initial confidence corresponding to each evaluation index in each piece of feedback information is the same, and the initial confidence is the confidence of the piece of feedback information; and integrating the same evaluation indexes in all feedback information. For the evaluation indexes with inconsistent feedback contents, the confidence level of the feedback contents is adjusted according to the feedback evaluation confidence level of the user; and for each evaluation index, weighting the feedback content to obtain comprehensive feedback data of the evaluation index, and optimizing the target model for the evaluation index.
And for a plurality of evaluation indexes with consistent feedback content in each piece of feedback information, determining the priority of the evaluation index to be optimized according to the feedback evaluation confidence of the user, and particularly determining the priority according to the proportion of the user with high feedback evaluation confidence.
For example, for index a, user feedback with high confidence is not satisfied one hundred times, user feedback with low confidence is not satisfied twenty times; for index B, user feedback with high confidence is not satisfied twenty times, and user feedback with low confidence is one hundred times, which processes index A preferentially.
To facilitate understanding, the user's feedback evaluation confidence is reduced to two levels: the member user and the common user, wherein the feedback evaluation confidence of the member user is higher than that of the common user;
as an example, there is a speech recognition model, and most users feed back a conclusion to be more accurate with respect to the index of recognition rate. In this case, if the feedback recognition rate of the common user is low, the feedback is considered to be optional, and only recording is performed without processing; if the feedback recognition rate of the member user is low, the situation that the recognition rate is low is considered to be truly generated, and the reason that the recognition rate is low by the record analysis used by the user at the time is called out, for example, the accent of the dialect of the user is heavy.
It will be appreciated that this example is for ease of understanding only and is not necessarily divided in terms of this in actual use; more precisely, each user establishes corresponding confidence weight along with the use, such as the credit of certain software, the higher the credit, the higher the authority of the user in the system, and the more credible the feedback; meanwhile, this example is also merely for explaining "when feedback contents of a certain index are inconsistent, filtering can be performed according to the confidence level of the user", and a person skilled in the art can perform specific processing according to the actual application scenario.
Example two
It is an object of this embodiment to provide an electrical artificial intelligence model performance feedback evaluation system, the system comprising:
the data acquisition module is used for acquiring operation behavior data and corresponding model effect feedback information during each use of the model by a user, wherein the user operation behavior data comprises a keyboard, a mouse operation behavior and a page operation behavior;
the confidence evaluation module is used for evaluating the confidence of the effect feedback information of the model according to the similarity between the operation behavior data of each time and the operation habit model of the user; the operation habit model is built based on historical operation behavior data of a user;
and the feedback information screening module is used for acquiring all the effect feedback information aiming at a certain target model, weighting according to the confidence coefficient of each piece of feedback information, and obtaining corrected effect feedback results which are used for learning and optimizing the target model.
Example III
An object of the present embodiment is to provide an electronic apparatus.
An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method as in embodiment one when executing the program.
Example IV
An object of the present embodiment is to provide a computer-readable storage medium.
A computer readable storage medium having stored thereon a computer program which when executed by a processor implements the method as described in embodiment one.
The steps involved in the second to fourth embodiments correspond to the first embodiment of the method, and the detailed description of the second embodiment refers to the relevant description of the first embodiment.
It will be appreciated by those skilled in the art that the modules or steps of the invention described above may be implemented by general-purpose computer means, alternatively they may be implemented by program code executable by computing means, whereby they may be stored in storage means for execution by computing means, or they may be made into individual integrated circuit modules separately, or a plurality of modules or steps in them may be made into a single integrated circuit module. The present invention is not limited to any specific combination of hardware and software.
While the foregoing description of the embodiments of the present invention has been presented in conjunction with the drawings, it should be understood that it is not intended to limit the scope of the invention, but rather, it is intended to cover all modifications or variations within the scope of the invention as defined by the claims of the present invention.
Claims (10)
1. The method for evaluating the success feedback of the electric artificial intelligent model is characterized by comprising the following steps of:
acquiring operation behavior data and corresponding model effect feedback information of a user during each use of a model, wherein the user operation behavior data comprises keyboard operation behaviors, mouse operation behaviors and page operation behaviors;
evaluating the confidence level of the effect feedback information of the model according to the similarity of the operation behavior data of each time and the operation habit model of the user; the operation habit model is built based on historical operation behavior data of a user;
and obtaining all the effect feedback information aiming at a certain target model, weighting according to the confidence coefficient of each piece of feedback information, and obtaining a corrected effect feedback result which is used for learning optimization of the target model.
2. The method for evaluating the performance feedback of an electric artificial intelligence model according to claim 1, wherein evaluating the confidence of the performance feedback information of the model comprises:
selecting a plurality of operation behaviors as evaluation indexes, and distributing weights for each operation behavior;
and comparing the similarity of each operation behavior corresponding to the sub-feedback information with the similarity of the operation behaviors corresponding to the operation habit model, and carrying out weighted summation on the similarity of each operation behavior to obtain the confidence coefficient of the sub-feedback information.
3. The method for evaluating the performance feedback of the electric power artificial intelligence model according to claim 1 or 2, wherein after evaluating the confidence of the performance feedback information of the model, the method further comprises the steps of: setting a plurality of anchoring indexes in the achievement feedback questionnaire, judging whether the feedback information is reliable or not according to the feedback condition of a user aiming at the plurality of anchoring indexes for certain feedback information to be evaluated, and if the feedback information is unreliable, the confidence is low.
4. The method for evaluating the feedback of the performance of an electric artificial intelligence model according to claim 1, wherein after evaluating the confidence of the feedback information of each model performance, the feedback evaluation confidence of each user is also evaluated: setting a confidence threshold to distinguish whether each feedback evaluation is effective; and calculating the feedback evaluation confidence coefficient of each user according to the number of the effective feedback evaluation of each user.
5. The method for evaluating the performance feedback of an electric artificial intelligence model according to claim 4, wherein if the feedback contents of the same index are inconsistent in the model performance feedback information of different users, the confidence level of the feedback contents of the index is corrected by combining the feedback evaluation confidence level of the users.
6. The utility model relates to an electric power artificial intelligence model achievement feedback evaluation system which is characterized by comprising:
the data acquisition module is used for acquiring operation behavior data and corresponding model effect feedback information during each use of the model by a user, wherein the user operation behavior data comprises a keyboard, a mouse operation behavior and a page operation behavior;
the confidence evaluation module is used for evaluating the confidence of the effect feedback information of the model according to the similarity between the operation behavior data of each time and the operation habit model of the user; the operation habit model is built based on historical operation behavior data of a user;
and the feedback information screening module is used for acquiring all the effect feedback information aiming at a certain target model, weighting according to the confidence coefficient of each piece of feedback information, and obtaining corrected effect feedback results which are used for learning and optimizing the target model.
7. The system for evaluating the performance feedback of an artificial intelligence model of claim 6, wherein evaluating the confidence of the performance feedback information of the model comprises:
selecting a plurality of operation behaviors as evaluation indexes, and distributing weights for each operation behavior;
and comparing the similarity of each operation behavior corresponding to the sub-feedback information with the similarity of the operation behaviors corresponding to the operation habit model, and carrying out weighted summation on the similarity of each operation behavior to obtain the confidence coefficient of the sub-feedback information.
8. The system for evaluating the performance feedback of an electric power artificial intelligence model according to claim 6 or 7, wherein after evaluating the confidence of the performance feedback information of the model, the system further evaluates: setting a plurality of anchoring indexes in the achievement feedback questionnaire, judging whether the feedback information is reliable or not according to the feedback condition of a user aiming at the plurality of anchoring indexes for certain feedback information to be evaluated, and if the feedback information is unreliable, the confidence is low.
9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the method for evaluating the performance feedback of an artificial intelligence model of electricity as claimed in any one of claims 1 to 5 when the program is executed by the processor.
10. A computer readable storage medium having stored thereon a computer program, which when executed by a processor implements the method for evaluating the performance feedback of an electric artificial intelligence model according to any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311370267.5A CN117764427A (en) | 2023-10-20 | 2023-10-20 | Electric power artificial intelligent model success feedback evaluation method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311370267.5A CN117764427A (en) | 2023-10-20 | 2023-10-20 | Electric power artificial intelligent model success feedback evaluation method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117764427A true CN117764427A (en) | 2024-03-26 |
Family
ID=90311234
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311370267.5A Pending CN117764427A (en) | 2023-10-20 | 2023-10-20 | Electric power artificial intelligent model success feedback evaluation method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117764427A (en) |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105718805A (en) * | 2016-01-25 | 2016-06-29 | 福建北卡科技有限公司 | Cloud-computing trust management method based on evaluation confidence degree |
| EP3065133A1 (en) * | 2015-03-06 | 2016-09-07 | ZETES Industries S.A. | Method and system for generating an optimised solution in speech recognition |
| CN106355414A (en) * | 2015-07-15 | 2017-01-25 | 阿里巴巴集团控股有限公司 | Method and apparatus for processing user feedback information |
| US20180357571A1 (en) * | 2017-06-13 | 2018-12-13 | Beijing Baidu Netcom Science And Technology Co., Ltd. | Conversation processing method and apparatus based on artificial intelligence, device and computer-readable storage medium |
| CN109242573A (en) * | 2018-09-26 | 2019-01-18 | 湖南人文科技学院 | Evaluation method, device, equipment and the storage medium of APP |
| CN109255028A (en) * | 2018-08-28 | 2019-01-22 | 西安交通大学 | Quality of instruction integrated evaluating method based on teaching evaluation data reliability |
| WO2020119030A1 (en) * | 2018-12-11 | 2020-06-18 | 平安科技（深圳）有限公司 | Model training method, device and equipment used for answering questions and storage medium |
| CN113162923A (en) * | 2021-04-12 | 2021-07-23 | 广州杰赛科技股份有限公司 | User reliability evaluation method and device based on user behaviors and storage medium |
| CN113360512A (en) * | 2021-06-21 | 2021-09-07 | 特赞（上海）信息科技有限公司 | Model updating method and device based on user feedback and storage medium |
| CN114388108A (en) * | 2020-10-22 | 2022-04-22 | 浙江远图互联科技股份有限公司 | User feedback analysis method based on multi-task learning |
| CN115293238A (en) * | 2022-07-07 | 2022-11-04 | 支付宝(杭州)信息技术有限公司 | Credibility judging method and system |
| CN115562960A (en) * | 2022-09-20 | 2023-01-03 | 中国银行股份有限公司 | Application optimization method and system of intelligent equipment |
-
2023
- 2023-10-20 CN CN202311370267.5A patent/CN117764427A/en active Pending
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3065133A1 (en) * | 2015-03-06 | 2016-09-07 | ZETES Industries S.A. | Method and system for generating an optimised solution in speech recognition |
| CN106355414A (en) * | 2015-07-15 | 2017-01-25 | 阿里巴巴集团控股有限公司 | Method and apparatus for processing user feedback information |
| CN105718805A (en) * | 2016-01-25 | 2016-06-29 | 福建北卡科技有限公司 | Cloud-computing trust management method based on evaluation confidence degree |
| US20180357571A1 (en) * | 2017-06-13 | 2018-12-13 | Beijing Baidu Netcom Science And Technology Co., Ltd. | Conversation processing method and apparatus based on artificial intelligence, device and computer-readable storage medium |
| CN109255028A (en) * | 2018-08-28 | 2019-01-22 | 西安交通大学 | Quality of instruction integrated evaluating method based on teaching evaluation data reliability |
| CN109242573A (en) * | 2018-09-26 | 2019-01-18 | 湖南人文科技学院 | Evaluation method, device, equipment and the storage medium of APP |
| WO2020119030A1 (en) * | 2018-12-11 | 2020-06-18 | 平安科技（深圳）有限公司 | Model training method, device and equipment used for answering questions and storage medium |
| CN114388108A (en) * | 2020-10-22 | 2022-04-22 | 浙江远图互联科技股份有限公司 | User feedback analysis method based on multi-task learning |
| CN113162923A (en) * | 2021-04-12 | 2021-07-23 | 广州杰赛科技股份有限公司 | User reliability evaluation method and device based on user behaviors and storage medium |
| CN113360512A (en) * | 2021-06-21 | 2021-09-07 | 特赞（上海）信息科技有限公司 | Model updating method and device based on user feedback and storage medium |
| CN115293238A (en) * | 2022-07-07 | 2022-11-04 | 支付宝(杭州)信息技术有限公司 | Credibility judging method and system |
| CN115562960A (en) * | 2022-09-20 | 2023-01-03 | 中国银行股份有限公司 | Application optimization method and system of intelligent equipment |
Non-Patent Citations (2)
| Title |
|---|
| 熊丽荣;雷静之;刘坚;汤颖;: "用户主观评估和客观质量评估相结合的云服务评价方法", 小型微型计算机***, no. 02, 15 February 2017 (2017-02-15) * |
| 黄龙超: "社会媒体中产品评论的可信度预测方法与应用", 《万方数据库》, 26 October 2018 (2018-10-26) * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3401856A1 (en) | System, device and method for releasing vehicle insurance surveying task, and readable storage medium | |
| CN111860658A (en) | Transformer fault diagnosis method based on cost sensitivity and integrated learning | |
| CN111160582B (en) | Equipment fault identification method and device and computer readable storage medium | |
| CN112633962B (en) | Service recommendation method and device, computer equipment and storage medium | |
| CN111178675A (en) | LR-Bagging algorithm-based electric charge recycling risk prediction method, system, storage medium and computer equipment | |
| CN109508429B (en) | Individualized self-adaptive learning recommendation method based on big data analysis of education platform | |
| CN111797320A (en) | Data processing method, device, equipment and storage medium | |
| CN110634060A (en) | User credit risk assessment method, system, device and storage medium | |
| CN115934490A (en) | Server performance prediction model training method, device, equipment and storage medium | |
| CN113642652A (en) | Method, device and equipment for generating fusion model | |
| CN111061948B (en) | User tag recommendation method and device, computer equipment and storage medium | |
| WO2018036402A1 (en) | Method and device for determining key variable in model | |
| CN115049019A (en) | Method and device for evaluating arsenic adsorption performance of metal organic framework and related equipment | |
| CN111191722A (en) | Method and device for training prediction model through computer | |
| CN113762401A (en) | Self-adaptive classification task threshold adjusting method, device, equipment and storage medium | |
| CN114912549B (en) | Training method of risk transaction identification model, and risk transaction identification method and device | |
| CN111311025B (en) | Load prediction method based on meteorological similar days | |
| CN113722098A (en) | Server capacity prediction method, device, equipment and medium based on artificial intelligence | |
| US10984305B1 (en) | Synthetic clickstream testing using a neural network | |
| CN116774986A (en) | Automatic evaluation method and device for software development workload, storage medium and processor | |
| CN117764427A (en) | Electric power artificial intelligent model success feedback evaluation method and system | |
| CN116739649A (en) | User response potential evaluation method and device | |
| WO2022222230A1 (en) | Indicator prediction method and apparatus based on machine learning, and device and storage medium | |
| CN109408531B (en) | Method and device for detecting slow-falling data, electronic equipment and storage medium | |
| CN117929655B (en) | Smell intensity detection method, device, equipment, storage medium and vehicle |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |