CN109559020A - A kind of quality testing method mutually commented based on colleague - Google Patents
A kind of quality testing method mutually commented based on colleague Download PDFInfo
- Publication number
- CN109559020A CN109559020A CN201811325273.8A CN201811325273A CN109559020A CN 109559020 A CN109559020 A CN 109559020A CN 201811325273 A CN201811325273 A CN 201811325273A CN 109559020 A CN109559020 A CN 109559020A
- Authority
- CN
- China
- Prior art keywords
- participant
- achievement
- colleague
- quality
- reliability
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q10/06393—Score-carding, benchmarking or key performance indicator [KPI] analysis
Landscapes
- Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- Engineering & Computer Science (AREA)
- Strategic Management (AREA)
- Development Economics (AREA)
- Economics (AREA)
- Entrepreneurship & Innovation (AREA)
- Educational Administration (AREA)
- Operations Research (AREA)
- Marketing (AREA)
- Game Theory and Decision Science (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- Physics & Mathematics (AREA)
- General Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a kind of quality testing methods mutually commented based on colleague, after participant submits solution to a certain task, several other participants are distributed in each submission to score to its quality of data, each iteration includes: (i) to adjust the difference between the scoring of evaluated person and the grade of the evaluated person speculated according to estimator the evaluation reliability of each estimator；(ii) the reliability based on estimator updates the quality of data of each evaluated person in weighted fashion；(iii) since each user is both estimator and evaluated person, then each participant's grade evaluates reliability by it and data quality two indices weighted average obtains.Advantage: the present invention explicitly distinguishes the reliability of the evaluation of participant and data quality, and the ability correctly evaluated and encouraging appraisal person that can measure participant correctly score to evaluated person.
Description
Technical field
The present invention relates to a kind of quality testing method mutually commented based on colleague, belongs to artificial intelligence and big data is excavated
Technical field.
Background technique
Based on network, for extensive online Open Course (MOOCs, Massive the Online Open of general population
Courses) fast-developing, although current MOOC can support the functions such as video classes, forum, test and evaluation, to
The evaluation of raw learning effect and the ability for giving feedback are still restricted.A key challenge of MOOC is student's assessment: a large amount of
It is infeasible that student, which makes coach or assiatant (TA) carry out scoring to all tasks,.Colleague/companion mutually comments and (student is allowed mutually to evaluate)
It is the effective method for solving extensive evaluation problem.But companion has mutually commented several root problems.Firstly, how be
Does participant, which provides an excitation, allow them correctly to evaluate the companion of oneself? second, since companion may be to the correctness of evaluation
Pay no attention to, how to compensate companion and mutually comment any intentional deviation that may be introduced?
Traditional companion similar to Pagerank mutually comments algorithm, it is according to the scoring that participant proposes other companions
Participant constructs achievement.Similar PeerRank method makes two basic assumptions to the achievement for how combining companion.Firstly, it is false
Can the achievement of a fixed participant be the ability measuring them and correctly scoring.Second, participant should correctly be scored
Reward, this is that participant provides the accurate companion motivation mutually commented.But this method, which has the drawback that, can not accurately predict to join
It is larger with the error of person's achievement, result prediction.
Summary of the invention
The technical problem to be solved by the present invention is to overcome the deficiencies of existing technologies, a kind of number mutually commented based on colleague is provided
According to quality evaluating method.
In order to solve the above technical problems, the present invention provides a kind of quality testing method mutually commented based on colleague, it is special
Sign is, includes the following steps:
Step 1: assuming that there is N number of participant, each participant needs to submit a task solution of oneself, then
The solution of each participant is randomly assigned to m other participants to score, obtains a rating matrix GN×N, should
Matrix GN×NThe i-th row indicate the scoring that the solution of participant i obtains, jth column indicate the scoring that provides of participant j, if most
Big scoring is c；
Step 2: calculating the initial prediction achievement of participant iIt indicates mentioning for participant i
Hand over the mean value of all scorings obtained, S→iIndicate the set for the companion j that all couples of participant i score, Gi←jIt indicates to participate in
Scoring of the person j to companion i；
Step 3: iterating to calculate the achievement of each participant iT is the number of iterations,Consist of two parts: participant i
Evaluation reliabilityWith the quality of data of the participant i of supposition
Step 4: step 3 is repeated, until the prediction achievement of all participants restrains.
Further, in the step 3, the evaluation reliability for the participant i that when the t times iteration deduces uses such as lower section
Method calculates:
Si→Indicate all participant j evaluated by participant i
Set, | Si→| it is participant's number in set,It is that participant j speculates in the t-1 times iteration and obtains achievement, β is
Exponential factor, the i.e. reliability of participant i are participant i to the pre- of the scoring of m evaluated person and these evaluated persons itself
The summation of the difference of achievement is surveyed, c indicates permitted maximum scores value, Gj←iIndicate evaluation of the participant i to participant j, i.e. square
Battle array GN×NIn (j, i) a element value.
Further, the value of the β takes 1.2.
Further, in the step 3, thus it is speculated that the quality of data of participant i calculate with the following method: The quality of data of the participant i deduced when being the t times iteration,
It is the evaluation reliability that the participant j evaluated i is obtained in upper primary iteration.
Further, in the step 3, the achievement of the participant i of the t times iterationAre as follows:Wherein α is slide coefficient, 0 < 1.
Further, in the step 4, the prediction convergent condition of achievement is:Wherein T is institute
The small threshold value of setting.
Further, the T=10-4。
Advantageous effects of the invention:
1) present invention provides the evaluation reliability of participant to the quality of data with its own and explicitly separates, wherein participant
Evaluation reliability be participant to other companions (evaluated person) scoring and the difference of the prediction achievement of these evaluated persons
It determines；And the quality for itself providing scheme of participant is determined by the evaluation reliability weighting of estimator, i.e. the evaluation of participant
Reliability influences each other with own tier, and reliability measures the ability that participant correctly scores, and grade/achievement of participant is
Its sliding average for evaluating reliability and data quality；
2) the evaluation reliability of each participant influences each other with own tier, and reliability measures what participant correctly scored
Ability, own tier accurately score for participant and provide excitation；
3) present invention can either more accurately predict participant's achievement, reduce the error of result prediction, and can be participant
Accurate scoring provides excitation.
Detailed description of the invention
Fig. 1 is the overall flow schematic diagram of the method for the present invention.
Specific embodiment
The invention will be further described below in conjunction with the accompanying drawings.Following embodiment is only used for clearly illustrating the present invention
Technical solution, and not intended to limit the protection scope of the present invention.
The present invention is by the evaluation reliability of participant and itself provides the quality of data and explicitly separates, and as predicting each ginseng
With the two indices of person's achievement, and participant's the final result be evaluation the sum of reliability and the weighting of data quality.Therefore
Evaluation reliability and the own tier of participant influences each other, and reliability measures the ability that participant correctly scores, own tier
It accurately scores for participant and provides excitation, therefore the present invention can more accurately predict participant's achievement, reduce result prediction
Error.
The symbol and its meaning that the present invention uses:
As shown in Figure 1, method flow:
Step 1: assuming that there is N number of participant, each participant needs to submit a certain task the solution of oneself, then
The submission of each participant is randomly assigned to m other participants (companion) to score, obtains a rating matrix GN×N；
If maximum scores are c.
Matrix G in step of the present inventionN×NThe i-th row indicate participant i submission obtain scoring, jth column indicate participant
The scoring that j is provided.
Step 2: calculate initial forecast ratings/achievement of participant i:Indicate participant i
Submission obtain all scorings mean value.Wherein S→iIndicate the companion j's (estimator) that all couples of participant i score
Set, Gi←jIndicate scoring of the participant j to companion i,The forecast ratings of participant i when being initial, it is new pre- to generate
Survey achievement.
Step 3: calculating the evaluation reliability of participant i are as follows:
Wherein Si→Indicate all colleague's collection (i.e. evaluated person collects) evaluated by participant i,It is the evaluation of the t times iteration of participant i
Reliability,It is the achievement that companion j is obtained in the t-1 times iteration, β is exponential factor, usually takes 1.2.That is, participant i
Reliability be participant i to other m companion scoring and the summation of the difference of the prediction achievement of other companions itself.
Step 4: the evaluation reliability of other companions (estimator) of gained being utilized to be weighted the participant i scoring obtained
It is average, calculate the quality of data of participant i:WhereinIt is participant i t
The quality of data speculated when secondary iteration, S→iIndicate colleague's collection (i.e. estimator gathers) that all couples of participant i are evaluated,It is the reliability that estimator j is obtained in upper primary iteration, i.e., the data deduced in the t times iteration of participant i
Quality is the weighted average of scoring of other companions to this participant.
Step 5: it calculates participant i and predicts achievement:Wherein,It is participant i t
The prediction achievement of secondary iteration, α are slide coefficient, 0 < α < 1.
Step 6: calculatingWhether threshold T (such as T=10 is less than-4), such as less than, this outputAchievement as final each user；Otherwise, repetition step 3,4,5 and 6.
Inventive method assumes that there is N number of participant, each participant needs to submit a task solution of oneself,
Then the submission of each participant is randomly assigned to m other participants (companion) to score, obtains a rating matrix
GN×N, middle matrix GN×NThe i-th row indicate the scoring that the submission of participant i obtains, jth column indicate the scoring that provides of participant j.
If maximum scores are c.Then initial prediction achievement is calculatedIt is the acquisition of the submission of participant i
All scorings mean value, we willAs the initial value of following iterative process, so as to generate new estimation really at
Achievement.Then the reliability of calculating participant i isIndicate participant i's
Reliability is scoring and the summation of the difference of the prediction achievement of other companions itself of the participant i to other m companion, in this way may be used
To motivate participant accurately to score the submission of other companions.Then scoring of the resulting reliability to participant i is utilized
It is weighted and averaged, calculates the quality of data of the participant i of supposition are as follows:Ginseng
The t times iteration grade with person i is the weighted average of scoring of other companions to this participant.Subsequently calculate prediction achievementThe step of finally computing repeatedly reliability, own tier and prediction achievement, until predicting achievementConvergence, wherein prediction achievement convergence refers toThat is prediction achievement no longer changes with additional iteration, receives
The prediction achievement held back is the prediction achievement of final output.
The above is only a preferred embodiment of the present invention, it is noted that for the ordinary skill people of the art
For member, without departing from the technical principles of the invention, several improvement and deformations can also be made, these improvement and deformations
Also it should be regarded as protection scope of the present invention.
Claims (7)
1. a kind of quality testing method mutually commented based on colleague, which comprises the steps of:
Step 1: assuming that there is N number of participant, each participant needs to submit a task solution of oneself, then will be every
The solution of a participant is randomly assigned to m other participants and scores, and obtains a rating matrix GN×N, the matrix
GN×NThe i-th row indicate the scoring that the solution of participant i obtains, jth column indicate the scoring that provides of participant j, if maximum is commented
It is divided into c；
Step 2: calculating the initial prediction achievement of participant iIt indicates that the submission of participant i obtains
All scorings mean value, S→iIndicate the set for the companion j that all couples of participant i score, Gi←jIndicate participant j to same
With the scoring of i；
Step 3: iterating to calculate the achievement of each participant iT is the number of iterations,Consist of two parts: participant i's comments
Valence reliabilityWith the quality of data of the participant i of supposition
Step 4: step 3 is repeated, until the prediction achievement of all participants restrains.
2. the quality testing method according to claim 1 mutually commented based on colleague, which is characterized in that the step 3
In, the evaluation reliability for the participant i that when the t times iteration deduces calculates with the following method:
Si→It indicates by the collection of the participant i all participant j evaluated
It closes, | Si→| it is participant's number in set,It is that participant j speculates in the t-1 times iteration and obtains achievement, β is index
The factor, i.e. the reliability of participant i be participant i to the prediction of the scoring of m evaluated person and these evaluated persons itself at
The summation of the difference of achievement, c indicate permitted maximum scores value, Gj←iIndicate evaluation of the participant i to participant j, i.e. matrix
GN×NIn (j, i) a element value.
3. the quality testing method according to claim 2 mutually commented based on colleague, which is characterized in that the value of the β
Take 1.2.
4. the quality testing method according to claim 1 mutually commented based on colleague, which is characterized in that the step 3
In, thus it is speculated that the quality of data of participant i calculate with the following method: It is
The quality of data of the participant i deduced when the t times iteration,It is the participant j that is evaluated i in upper primary iteration
The evaluation reliability of acquisition.
5. the quality testing method according to claim 1 mutually commented based on colleague, which is characterized in that the step 3
In, the achievement of the participant i of the t times iterationAre as follows:Wherein α is slide coefficient, 0 < α <
1。
6. the quality testing method according to claim 1 mutually commented based on colleague, which is characterized in that the step 4
In, the prediction convergent condition of achievement is:Wherein T is set small threshold value.
7. the quality testing method according to claim 6 mutually commented based on colleague, which is characterized in that the T=10-4。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811325273.8A CN109559020A (en) | 2018-11-08 | 2018-11-08 | A kind of quality testing method mutually commented based on colleague |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811325273.8A CN109559020A (en) | 2018-11-08 | 2018-11-08 | A kind of quality testing method mutually commented based on colleague |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109559020A true CN109559020A (en) | 2019-04-02 |
Family
ID=65865878
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811325273.8A Pending CN109559020A (en) | 2018-11-08 | 2018-11-08 | A kind of quality testing method mutually commented based on colleague |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109559020A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110837554A (en) * | 2019-10-25 | 2020-02-25 | 天津大学 | User evaluation reliability judgment method based on multi-source data |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103377296A (en) * | 2012-04-19 | 2013-10-30 | 中国科学院声学研究所 | Data mining method for multi-index evaluation information |
| CN105976070A (en) * | 2016-05-27 | 2016-09-28 | 北京交通大学 | Key-element-based matrix decomposition and fine tuning method |
| CN107766316A (en) * | 2016-08-15 | 2018-03-06 | 株式会社理光 | The analysis method of evaluating data, apparatus and system |
-
2018
- 2018-11-08 CN CN201811325273.8A patent/CN109559020A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103377296A (en) * | 2012-04-19 | 2013-10-30 | 中国科学院声学研究所 | Data mining method for multi-index evaluation information |
| CN105976070A (en) * | 2016-05-27 | 2016-09-28 | 北京交通大学 | Key-element-based matrix decomposition and fine tuning method |
| CN107766316A (en) * | 2016-08-15 | 2018-03-06 | 株式会社理光 | The analysis method of evaluating data, apparatus and system |
Non-Patent Citations (1)
| Title |
|---|
| HUI FANG: "RankwithTA: A robust and accurate peer grading mechanism for MOOCs", 《2017 IEEE 6TH INTERNATIONAL CONFERENCE ON TEACHING, ASSESSMENT, AND LEARNING FOR ENGINEERING (TALE)》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110837554A (en) * | 2019-10-25 | 2020-02-25 | 天津大学 | User evaluation reliability judgment method based on multi-source data |
| CN110837554B (en) * | 2019-10-25 | 2023-10-10 | 天津大学 | User evaluation reliability judging method based on multi-source data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Lockwood et al. | Uncertainty in rank estimation: Implications for value-added modeling accountability systems | |
| CN108399453A (en) | A kind of Electric Power Customer Credit Rank Appraisal method and apparatus | |
| CN105354260B (en) | The Mobile solution of a kind of mosaic society's network and item characteristic recommends method | |
| Chen et al. | Bid evaluation in civil construction under uncertainty: A two-stage LSP-ELECTRE III-based approach | |
| CN105843829A (en) | Big data credibility measurement method based on layering model | |
| CN110197340A (en) | A kind of learner's H-NTLA method towards online education platform | |
| CN105809510A (en) | Multi-faceted social trust based collaborative recommendation method | |
| CN110826164A (en) | Complex network node importance evaluation method based on local and global connectivity | |
| Palley et al. | Boosting the wisdom of crowds within a single judgment problem: Weighted averaging based on peer predictions | |
| Ayala-Gaytán et al. | Attitudes and causes of cheating among Mexican college students: An exploratory research | |
| Mair et al. | Robust statistical methods using WRS2 | |
| Abu-Ismail et al. | A new approach to measuring the middle class: Egypt | |
| CN110069750A (en) | Non-precision analysis method based on four parameters of Weibull | |
| Bhat et al. | Clustering trust dynamics in a human-robot sequential decision-making task | |
| CN109559020A (en) | A kind of quality testing method mutually commented based on colleague | |
| CN113268976B (en) | Microblog-oriented topic influence assessment method | |
| Petscher et al. | Testing the Importance of Individual Growth Curves in Predicting Performance on a High-Stakes Reading Comprehension Test in Florida. REL 2014-006. | |
| Hu et al. | Prediction of MVP attribution in NBA regular match based on BP neural network model | |
| Brett et al. | Advancing equity in the Australian higher education system | |
| Harris | Conceptualizing uncertainty: the IPCC, model robustness and the weight of evidence | |
| Zhou et al. | Static strategic game approach for multiple attribute decision making problems without weight information | |
| Banerjee et al. | Roles of education in productivity growth in Australia, 1860–1939 | |
| dos Reis et al. | Hierarchical modelling of power law processes for the analysis of repairable systems with different truncation times: An empirical Bayes approach | |
| Kulkarni | Development of performance prediction models using expert opinions | |
| Gordon et al. | Social choice, optimal inference and figure skating |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190402 |
|
| RJ01 | Rejection of invention patent application after publication |