CN107590176B

CN107590176B - Evaluation index obtaining method and device and electronic equipment

Info

Publication number: CN107590176B
Application number: CN201710638936.0A
Authority: CN
Inventors: 赵晓萌; 胡军
Original assignee: Beijing QIYI Century Science and Technology Co Ltd
Current assignee: Beijing QIYI Century Science and Technology Co Ltd
Priority date: 2017-07-31
Filing date: 2017-07-31
Publication date: 2021-01-15
Anticipated expiration: 2037-07-31
Also published as: CN107590176A

Abstract

The embodiment of the invention provides an evaluation index obtaining method, an evaluation index obtaining device and electronic equipment, wherein the method comprises the following steps: acquiring a first time length of each file corresponding to a preset query word played after a user clicks; obtaining a relevance value of each file according to the first time length, a preset corresponding relation between the first time length and a first preset threshold value and a preset corresponding relation between the first preset threshold value and the relevance value, wherein the relevance value is a quantitative parameter of the relevance degree of each file and the preset query word; and calculating the normalized accumulated discount information gain value of the sorting result according to the correlation value, and taking the normalized accumulated discount information gain value as an evaluation index of the sorting result. By applying the embodiment of the invention, the accuracy of the relevance of the file can be improved, and the accuracy of the evaluation of the sequencing result is further improved.

Description

Evaluation index obtaining method and device and electronic equipment

Technical Field

The invention relates to the technical field of file retrieval, in particular to an evaluation index obtaining method and device and electronic equipment.

Background

As more and more files are available on the network, when a user is presented with files corresponding to query terms, the relevant files need to be sorted and then displayed. When the related files corresponding to the query words are sorted, a certain sorting rule is required, so that various sorting models are promoted. Because there are many kinds of ranking models based on relevance, different ranking results may be obtained by applying different ranking models, and in order to screen a more optimized ranking model, the ranking results need to be evaluated.

Currently, NDCG @ K (normalized divided Cu006 dual relative Gain) is commonly used to evaluate whether the sorting result is reasonable. The idea of the NDCG @ K is to multiply a discount by a correlation value of each first K files of a current query term, the smaller the discount corresponding to the file ranked more forward is, the larger the discount corresponding to the file ranked more backward is, and the NDCG @ K can be obtained by quotient of the sum of the Discounted correlation values and IDCG (Ideal discrete temporal Gain value). When the evaluation is carried out, the formula is firstly utilized,

calculating a dynamic accumulated discount information gain value, wherein the DCG is the dynamic accumulated discount information gain value; k is the first K files in the sequencing result, represents the number of the files which are estimated by the professional and possibly viewed by the user, and assumes that the user views the K files, the user clicks according to the relevance of each file; i is the sequence number of the file in the sequencing result; rel_iThe relevance value of the file with the sequence number i in the sequencing result is shown. Reuse formula

Calculating a normalized dynamic accumulated discount information gain value, wherein the NDCG is the normalized dynamic accumulated discount information gain value; DCG is a dynamic accumulated discount information gain value; IDCG is an artificially determined ideal cumulative discount information gain value. If more files with high correlation are contained in the top-ranked files, the NDCG @ K of the ranking result is larger, correspondingly, if less files with high correlation are contained in the top-ranked K files, the NDCG @ K value of the ranking result is smaller, and therefore the files with higher correlation are ranked in the front when the ranking result is returned by the search engine.

The relevance value of each file in the prior art is determined according to whether the user clicks the file or not. However, in practical applications, titles of some files appear to be related, but after clicking, the relevance between the files and the query words is found to be small or no, so that the value of NDCG @ K determined according to the relevance is not accurate enough, and thus the prior art has a technical problem that the evaluation of the ranking result is not accurate enough.

Disclosure of Invention

The embodiment of the invention aims to provide an evaluation index obtaining method, an evaluation index obtaining device and electronic equipment so as to improve the accuracy of sequencing result evaluation.

In order to achieve the above object, an embodiment of the present invention provides a method for obtaining an evaluation index, where the method includes:

acquiring a first time length for playing each file after clicking by a user, wherein each file is in a sequencing result corresponding to a preset query word;

obtaining a relevance value of each file according to the first time length, a preset corresponding relation between the first time length and a first preset threshold value and a preset corresponding relation between the first preset threshold value and the relevance value, wherein the relevance value is a quantitative parameter of the relevance degree of each file and the preset query word;

and calculating the normalized accumulated discount information gain value of the sorting result according to the correlation value, and taking the normalized accumulated discount information gain value as an evaluation index of the sorting result.

Optionally, before calculating the normalized cumulative discount information gain value of the sorting result according to the correlation value, the method further includes:

obtaining a sorting result of the file corresponding to the preset query word;

obtaining a file which is sequenced last in the sequencing result and is clicked by the user according to the click record of the user, and taking the file as a target file;

establishing a model aiming at the clicking behavior of the user according to the historical clicking record of the user by using a preset mathematical model;

starting from the first file after the target file in the sequencing result, and aiming at each file, calculating the user viewing probability of the current file by using the model;

obtaining a relevance value of the current file according to the user viewing probability of the current file, whether the current file is clicked or not or the playing duration of the current file by a user;

judging whether the user checking probability is greater than or equal to a second preset threshold value or not;

setting the next file of the current file as the current file under the condition that the user viewing probability is greater than or equal to a second preset threshold value, and executing the step of calculating the user viewing probability of the current file by using the model;

the obtaining of the first time length for playing the preset query word corresponding to each file after the user clicks includes:

and aiming at each file before the last clicked file in the sorting result, obtaining the first time length for playing each file by the user.

Optionally, the obtaining a relevance value of the current file according to the user viewing probability of the current file, whether the current file is clicked, or the playing duration of the user for the current file includes:

judging whether the user checking probability is greater than or equal to a third preset threshold value or not; under the condition that the user viewing probability is larger than or equal to the third preset threshold, obtaining a relevance value of the file according to whether the file is clicked or not; or, under the condition that the user viewing probability is smaller than the third preset threshold, calculating a second duration corresponding to the file according to the total historical playing duration of the file and the historical playing times of the user of the file, and obtaining a relevance value of the file according to the second duration, the corresponding relation between the second duration and the first preset threshold, and the corresponding relation between the first preset threshold and the relevance value.

Optionally, the calculating a second duration corresponding to the file according to the total historical playing duration of the file and the historical playing times of the user of the file includes:

and calculating the average playing time corresponding to the file according to the quotient of the total historical playing time of the file and the user historical playing times of the file, and taking the average playing time as a second time.

Optionally, the calculating a normalized cumulative discount information gain value of the sorting result according to the correlation value includes:

according to the file before the last clicked file in the sorting result and the relevance value of each file with the user viewing probability not less than a second preset threshold, utilizing a formula:

calculating a dynamic cumulative discount information gain value, wherein,

DDCG is a dynamic accumulated discount information gain value; n _ est is the sum of the number of files before the last clicked file is sorted and the number of files with the user viewing probability not less than a second preset threshold; i is the sequence number of the file in the sequencing result; w is a_iThe weight of the file with the sequence number i; rel_iThe relevance value of the file with the sequence number of i before the last clicked file in the sorting result is sorted; avg _ rel_iThe relevance value of the file with the sequence number of i after the last clicked file in the sorting result is sorted;

using formulas

Calculating a normalized dynamic cumulative discount information gain value, wherein,

DNDCG is a normalized dynamic accumulated discount information gain value; DDCG is a dynamic accumulated discount information gain value; IDCG is an artificially determined ideal cumulative discount information gain value.

In order to achieve the above object, an embodiment of the present invention further provides an apparatus for obtaining an evaluation index, where the apparatus includes: a first obtaining module, a second obtaining module, and a first calculating module, wherein,

the first obtaining module is used for obtaining a first time length for playing each file after the user clicks, wherein each file is in a sequencing result corresponding to a preset query word;

the second obtaining module is configured to obtain a relevance value of each file according to the first duration, a preset correspondence between the first duration and a first preset threshold, and a preset correspondence between the first preset threshold and the relevance value, where the relevance value is a quantitative parameter of a degree of association between each file and the preset query term;

the first calculating module is used for calculating the normalized accumulated discount information gain value of the sorting result according to the correlation value and taking the normalized accumulated discount information gain value as the evaluation index of the sorting result.

Optionally, the apparatus further comprises: a third obtaining module, a fourth obtaining module, a constructing module, a second calculating module, a fifth obtaining module, a judging module, a setting module,

The third obtaining module is used for obtaining the sequencing result of the file corresponding to the preset query word;

the fourth obtaining module is configured to obtain a file which is ranked last in the ranking result and is clicked by the user according to the click record of the user, and use the file as a target file;

the building module is used for building a model aiming at the clicking behavior of the user according to the historical clicking record of the user by using a preset mathematical model;

the second calculation module is used for calculating the user viewing probability of the current file by using the model aiming at each file from the first file after the target file in the sequencing result;

the fifth obtaining module is configured to obtain a relevance value of the current file according to the user viewing probability of the current file, whether the current file is clicked, or the playing duration of the current file by the user;

the judging module is used for judging whether the user checking probability is greater than or equal to a second preset threshold value;

the setting module is used for setting the next file of the current file as the current file under the condition that the user viewing probability is larger than or equal to a second preset threshold value, and executing the step of calculating the user viewing probability of the current file by using the model;

the first obtaining module is further configured to:

Optionally, the fifth obtaining module is further configured to:

judging whether the user checking probability is greater than or equal to a third preset threshold value or not; under the condition that the user viewing probability is larger than or equal to the third preset threshold, obtaining a relevance value of the file according to whether the file is clicked or not; under the condition that the user viewing probability is smaller than the third preset threshold, calculating a second time length corresponding to the file according to the total historical playing time length of the file and the user historical playing times of the file, and obtaining a correlation value of the file according to the second time length, the corresponding relation between the second time length and a first preset threshold, and the corresponding relation between the first preset threshold and the correlation value.

Optionally, the fifth obtaining module is further configured to:

Optionally, the first computing module includes: a first computing unit and a second computing unit, wherein,

the first calculating unit is configured to, according to the relevance value of the file before the last clicked file in the sorting result and the relevance value of each file of which the user viewing probability is not less than a second preset threshold, utilize a formula:

calculating a dynamic cumulative discount information gain value, wherein,

DDCG is a dynamic accumulated discount information gain value; n _ est is the sum of the number of files before the last clicked file is sorted and the number of files with the user viewing probability not less than a second preset threshold; i is the sequence number of the file in the sequencing result; w is a_iThe weight of the file with the sequence number i; rel_iThe relevance value of the file with the sequence number of i before the last clicked file in the sorting result is sorted; avg _ rel_iTo sequence inThe relevance value of the file with the sequence number of i after the last clicked file is sequenced in the result;

the second calculation unit is used for utilizing a formula

In order to achieve the above object, an embodiment of the present invention further provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;

a memory for storing a computer program;

a processor for implementing any of the above method steps when executing a program stored in the memory.

In order to achieve the above object, an embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method for obtaining an evaluation index is implemented as any one of the above methods.

In another aspect of the present invention, the present invention further provides a computer program product including instructions, which when run on an electronic device, causes the electronic device to execute any one of the above-mentioned methods for obtaining an evaluation index.

By applying the method, the device and the electronic equipment for obtaining the evaluation index, provided by the embodiment of the invention, the relevance of the file is determined by utilizing the first time length for playing the file in the sorting result after the user clicks, and in general, the longer the first time length is, the higher the relevance of the file and the query word is, compared with the relevance determined by only using whether the user clicks in the prior art, the accuracy of the relevance of the file is improved, and the accuracy of the evaluation of the sorting result is further improved. Of course, it is not necessary for any product or method of practicing the invention to achieve all of the above-described advantages at the same time.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.

FIG. 1 is a schematic diagram of a scenario in which an embodiment of the present invention is applied;

fig. 2 is a schematic flow chart of a method for obtaining an evaluation index according to an embodiment of the present invention;

fig. 3 is a schematic flow chart of another method for obtaining an evaluation index according to an embodiment of the present invention;

fig. 4 is a flowchart illustrating a method for obtaining a correlation value according to an embodiment of the present invention;

FIG. 5 is a schematic diagram illustrating a calculation process of a normalized cumulative discount information gain value according to an embodiment of the present invention;

fig. 6 is a schematic structural diagram of an evaluation index obtaining apparatus according to an embodiment of the present invention;

fig. 7 is a schematic structural diagram of another evaluation index obtaining apparatus according to an embodiment of the present invention;

fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

In the prior art, a relevance value between each file and a query word is determined according to whether a user clicks each file in a ranking result to be evaluated, for example, if the user clicks a file a in the ranking result, the file a is relevant to the query word, and the relevance value of the file a is 1; if the user clicks the A file, the A file is irrelevant to the query word, and the relevance value of the A file is 0. However, in practical application, the titles of some files are related to query words, but after a user clicks, the relevance value of the file determined by the prior art is not accurate enough, so that the NDCG @ K value is not accurate enough. In order to solve the technical problem that the evaluation of the sequencing result is not accurate enough in the prior art, the embodiment of the invention determines the relevance value of each file according to the playing time of each file in the sequencing result played by a user. In general, the longer the playing time of the file a in the user playing sorting result is, the higher the correlation between the file a and the query word is, so that the correlation value of each file determined by using the playing time of the user is more accurate, and compared with the prior art, the more accurate the NDCG @ K value is.

As shown in fig. 1, fig. 1 is a schematic view of a scenario applied in the embodiment of the present invention, where an original data storage server 11 stores query results corresponding to query terms of users, a log of clicks of the query results by the users, and a log of files corresponding to each query record played by the users. For example, the query word "ping-pong course" input by the user, the server presents 100 search results corresponding to the query word to the user, the user clicks the first search result, the playing starts from the time 00:08:10, and the playing ends at the time 00:08: 15. The evaluation index obtaining device 12 obtains the original data from the original data storage server 11 according to the preset query term, and evaluates the ranking result corresponding to the preset query term according to the obtained original data. And finally, the evaluation index is sent to the downstream device 13.

Fig. 2 is a schematic flow chart of a method for obtaining an evaluation index according to an embodiment of the present invention, as shown in fig. 2, the method includes:

s101: and acquiring a first time length for playing each file corresponding to the preset query word after the user clicks, wherein each file is in the sequencing result corresponding to the preset query word.

Illustratively, a preset query word input by a user is a ping-pong tutorial, a search engine searches for files related to the preset query word according to the query word, and sorts the searched files according to a preset sorting model in the search engine to obtain a sorting result. If the files contained in the sequencing result are the file X-1, the file X-2, the file X-3, the file X-4 and the file X-5. Then, a first time length corresponding to each file played by the user is obtained, and the first time lengths corresponding to the file X-1, the file X-2, the file X-3, the file X-4 and the file X-5 are assumed to be 1 minute, 5 minutes, 0 minute, 12 minutes and 9 minutes respectively.

In practical applications, the user may only play a part of the files in the ranking result, and therefore, the ranking result may also include files that the user did not click, or files that the user clicked but did not play. In general, the file may be a file corresponding to a query term when a user inputs the query term at the current time.

The obtaining mode of the first duration for playing each file by the user may be: the document X-1 will be described as an example. For example, if the user starts playing the file at time 00:00:00 and stops playing the file at time 00:00:10, the first duration for the user to play the file X-1 is 10 seconds. In practical application, the above-mentioned each time can be obtained from the click log of the user.

S102: and obtaining a relevance value of each file according to the first time length, a preset corresponding relation between the first time length and a first preset threshold value and a preset corresponding relation between the first preset threshold value and the relevance value, wherein the relevance value is a quantitative parameter of the relevance degree of each file and the preset query word.

Illustratively, table 1 is a table of the correspondence between the first duration and the first preset threshold, and the correspondence between the first preset threshold and the correlation value, see table 1, where the first threshold includes, but is not limited to, 10 minutes, 5 minutes, and 2 minutes. And corresponding the playing time length corresponding to each file obtained in the step S101 to the first threshold in table 1, and determining a correlation value corresponding to each file.

For example, for the file X-1, the time period for the user to play the file X-1 is 1 minute, which is less than the first preset threshold value by 2 minutes, that is, the correlation value corresponding to the file X-1 is "0"; according to the method, the corresponding correlation values of the file X-2, the file X-3, the file X-4 and the file X-5 are determined to be 2, 0, 3 and 2 respectively.

It is understood that the relevance value is a quantitative parameter of the degree of association of the file with the preset query term. Generally, after a user inputs a query term, a search engine displays all documents corresponding to the query term in sequence in the form of an entry. For a certain file, after a user clicks a link and enters a page where the file is located, the higher the association degree between the file and a preset query word is, the longer the time for the user to view or browse the file after clicking is, and the larger the correlation value of the file is. If the title of the link corresponding to the file is highly associated with the query term, but after the user clicks and enters the page where the file is located, the content of the file is found to be inconsistent with the title through viewing or browsing, the association degree of the file with the query term is small, and the user generally closes the page or leaves the page. At this time, the behavior of the user viewing or browsing the file is finished, the duration of the user viewing or browsing the file is relatively short, and the relevance value of the file is relatively small.

TABLE 1

Play Time (Play Time)	Correlation value (rel)
		Greater than or equal to 10 minutes	3
Less than 10 minutes and greater than or equal to 5 minutes	2
		Less than 5 minutes and greater than or equal to 2 minutes	1
Less than 2 minutes	0
		Non-click or no-play duration	0

It should be noted that the corresponding relationship between the first duration of the file played by the user and the first preset threshold, and the corresponding relationship between the first preset threshold and the correlation value may be preset.

S103: and calculating the normalized accumulated discount information gain value of the sorting result according to the correlation value, and taking the normalized accumulated discount information gain value as an evaluation index of the sorting result.

Illustratively, in one aspect, a formula may be utilized,

the DCG value of the sorted result is calculated, wherein,

the DCG is the accumulated discount information gain value corresponding to the sorting result: k is the number of evaluation files participating in the sequencing result; rel_iThe relevance value of the file with the sequence number i in the sequencing result is obtained; i is the position of the order of the file in the sorting result.

Reuse formula

Calculating an NDCG @5 value (normalized cumulative discount information gain value) of the sorting result, wherein the NDCG is the NDCG @5 value (normalized cumulative discount information gain value) of the sorting result; DCG is the accumulated discount information gain value of the sequencing result; IDCG can be used after all files in the sorting result are sorted according to relevanceThe maximum accumulated discount information gain value can be obtained, and the IDCG can also be determined by a human-defined ideal accumulated discount information gain value.

If the IDCG is 10, the NDCG of the above sorting result is:

the evaluation index finally obtained was 0.607.

By applying the technical scheme provided by the embodiment of the invention shown in fig. 2, the first time length for playing the file in the sequencing result after the user clicks is utilized to determine the relevance of the file, in general, the longer the first time length is, the higher the relevance of the file and the query word is, compared with the relevance determined by only whether the user clicks in the prior art, the more accurate the relevance of the file and the query word can be obtained, and the accuracy of evaluating the sequencing result can be further improved.

Fig. 3 is a schematic flow chart of another method for obtaining an evaluation index according to an embodiment of the present invention, as shown in fig. 3, based on the embodiment of the present invention shown in fig. 2, before step S103, there are further added:

s104: and obtaining a sequencing result of the file corresponding to the preset query word.

Illustratively, the sequencing result of the file corresponding to the preset query word "ping-pong course" input by the user is as follows: file X-1, file X-2, file X-3, file X-4, file X-5, file X-6, file X-7, file X-8, file X-9, and file X-10.

It is understood that the ranking results are sequences in which the search engine or the search software displays the searched results in sequence for the query words input by the user. Generally, a search engine displays corresponding results in sequence according to the sequence of the relevance between the searched results and query words input by a user from large to small. In the embodiment of the present invention, the file is a kind of search result, and in practical applications, the search result includes, but is not limited to, a file, a text, a page, a picture, a code, or a link.

S105: and obtaining the file which is sequenced finally in the sequencing result and is clicked by the user according to the click record of the user, and taking the file as a target file.

Generally, when a user uses a search engine or search software, each operation of the search engine or search software generates a corresponding log on a server, for example, a search may have a corresponding search log, a click may have a corresponding click log, and a play may have a corresponding play log.

Specifically, a click log of a current sorting result corresponding to a preset query word clicked by a user may be obtained from the log, the log is a click record of the user, and a clicked file which is the last in the sorting result corresponding to the query word input by the user and is clicked by the user is obtained. And sorting the last file X-5 clicked by the user in the determined sorting result, and taking the file X-5 as a target file.

S106: and establishing a model aiming at the clicking behavior of the user according to the historical clicking record of the user by using a preset mathematical model.

Taking the query word "ping-pong course" as an example, the historical click records of the user include when the user clicked which file, the position of the file in the sorting result, and the duration of time the user played the file. According to the information, the click behavior of the user can be simulated, and a click model is established. For example, the User Click behavior may be modeled by a preset mathematical Model, such as UBM (User browse Model), BBM (Bayesian browse Model), or DBN (Dynamic Bayesian Network Click Model).

The process of modeling the user click behavior by using a preset mathematical model is the prior art, and the embodiment of the invention is not described herein again.

The historical click records of the user are as follows: after the user inputs the preset query word each time, the search engine displays the sequencing result to the user, and the user clicks each sequencing result to generate a set of click records. For example, the user A inputs the query word "ping-pong course" in 1 month and 1 day, the search engine displays the ranking result-1 to the user A, and the user clicks on the ranking result; the user A inputs a query word 'ping-pong course' in 1 month and 5 days, the search engine displays a sequencing result-2 to the user A, and the user clicks the sequencing result; the user A inputs the query word "ping-pong course" in 1 month and 8 days, the search engine displays the ranking result-3 to the user A, and the user clicks on the ranking result. The click record generated by the user clicking the three sorting results is the historical click record of the user, and the historical click record of the user can also comprise the click record of the user clicking the current sorting result.

S107: and calculating the user viewing probability of the current file by using the model aiming at each file from the first file after the target file in the sequencing result.

Specifically, the model for the user click behavior established in step S106 is used, starting from the first file X-6 after the target file X-5 in the sorting result. For example, the location information of file X-6 in the ranked results may be entered into the model, which calculates the probability that file X-6 is viewed by the user based on the location information of file X-6 in the ranked results. It can be understood that it is prior art to calculate the probability of user viewing of each file in the ranking result using a model for user click behavior.

And calculating the user viewing probability corresponding to the files X-6, X-7, X-8, X-9 and X-10.

For example, the probability of a user view of file X-6 is 0.6.

It should be emphasized that, for a certain query of a user, the last clicked file in the ranking result corresponding to the query word is ranked, and the subsequent files are not clicked by the user, and this step is only to estimate the probability that the file may be viewed by the user, i.e. the user viewing probability of the file.

In addition, for the user, the user can browse the sorting result corresponding to the preset query word, the browsing process is the process of viewing the file in the sorting result by the user, and the user does not click the file. Then, the user looks up a certain file associated with the preset keyword, clicks, finds that the file is the file desired by the user after clicking, and plays the file.

S108: and obtaining the correlation value of the current file according to the user viewing probability of the current file and whether the current file is clicked or not, or according to the user viewing probability of the current file and the playing time of the user for the current file.

Illustratively, although the probability of file X-6 being viewed is 0.6, the file is not actually clicked on by the user, nor played by the user. We cannot know whether the user looked at file X-6 and thought that file X-6 is relevant to the query term, in which case we cannot think that file X-6 has no relevance to the query term.

Generally, documents with higher probability of being viewed have a higher probability of being related to the query term.

Therefore, for the query term, the document X-6 may not be arranged at a position too far forward, in which case, the real relevance value of the document X-6 may be used as its relevance value, that is, whether the user clicked the document X-6 to obtain the relevance value of the document X-6, for example, the user clicked the document, and the relevance value of the document is 1; if the user does not click, the relevance value of the file is 0. For example,

although the user does not play the file X-6, the probability of being viewed is higher than the preset threshold value of 0.4; if the user clicks the file X-6, the relevance value of the file X-6 is 1, and if the user does not click the file X-6, the relevance value of the file X-6 is 0. Since the user did not click on file X-6, the relevance value for file X-6 is 0.

In addition, if the probability that the file X-6 is checked is low, the number of times that the user plays the file and the total duration of time that the user plays the file may be obtained according to the historical click record of the user for the preset query word, so as to obtain the average playing duration of time that the user plays the file according to the number of times that the user plays the file and the total duration of time that the user plays the file, and further obtain the correlation value of the file with reference to table 1.

S109: judging whether the user checking probability is greater than or equal to a second preset threshold value or not, if so, executing S1010; if not, S103 is executed.

If the second preset threshold is 0.1 and the probability that the user views the file X-6 (i.e., the probability of being viewed) is 0.6, then when it is determined whether the probability that the user views the file X-6 is greater than the second threshold, it is obvious that the probability that the user views the file X-6 is greater than the second preset threshold.

S1010: the next file of the current file is set as the current file, and step S107 is performed.

Illustratively, since the probability that the user views the file X-6 is greater than the second preset threshold, the next file X-7 of the file X-6 is set as the current file, and the user viewing probability of the file X-7 is calculated using the model established in step S106. All files in the sorting result are arranged in sequence, the position of the file X-7 in the sorting result is adjacent to the file X-6 and is behind the file X-6, so that the file X-7 is the next file of the file X-6; similarly, file X-8 is the next file to file X-7.

Assuming that the user viewing probability for file X-7 is 0.4 and the file is not clicked on by the user, the relevance value for file X-7 is 0.

If the user viewing probability of the file X-8 is 0.05 and is smaller than the second preset threshold, the correlation value of the file X-9 and the file X-10 does not need to be calculated. Calculating a sequencing result by using the method in the step S103: evaluation indexes of the file X-1, the file X-2, the file X-3, the file X-4, the file X-5, the file X-6 and the file X-7.

Specifically, S101 may be S101A: and aiming at each file before the last clicked file in the sorting result, obtaining the first time length for playing each file by the user.

Illustratively, the sequencing result of the file corresponding to the preset query word "ping-pong course" input by the user is as follows: file X-1, file X-2, file X-3, file X-4, file X-5, file X-6, file X-7, file X-8, file X-9, and file X-10, wherein the last file clicked by the user is "file X-5". Then the step S103 is: the first time lengths corresponding to the file X-1, the file X-2, the file X-3, the file X-4 and the file X-5 played by the user are obtained, for example, the first time lengths corresponding to the file X-1, the file X-2, the file X-3, the file X-4 and the file X-5 are respectively 1 minute, 5 minutes, 0 minute, 12 minutes and 9 minutes.

Then, the correlation values corresponding to the file X-1, the file X-2, the file X-3, the file X-4 and the file X-5 are respectively: 0. 2, 0, 3 and 2.

It is understood that the steps S103 and S1010 are not necessarily performed in a sequential order.

In the prior art, NDCG @ K values are adopted to evaluate the sorting results, the K values are generally set manually, and a user is assumed to check K files and click according to the relevance of each file. However, in practice, when the ranking model ranks the files, it is uncertain how many files the user has viewed at all, and therefore, the specified K value is not favorable for accurately evaluating the ranking result.

In addition, five files are taken as an example, and the file X-1 and the file X-2 are related to a preset query word and are clicked by a user; the file X-3 and the file X-4 are not related to a preset query word, and a user does not click; file X-5 is associated with a predetermined query term and the user has not clicked. The user has viewed file X-1, file X-2, and file X-3.

As the user views the file X-1, the file X-2 and the file X-3, the ranking results of the ranking models are evaluated according to the positions of the three files in the ranking models and the relevance values of the three files.

At present, there are two document sorting models, and after the five documents are sorted, the following two sorting results are obtained:

sequencing result 1: file X-1, file X-2, file X-3, file X-4, file X-5;

sequencing result 2: file X-1, file X-5, file X-2, file X-3, file X-4;

and if K is 5, calculating the NDCG @ K value of the sorting result according to the correlation values of the file X-1, the file X-2, the file X-3, the file X-4 and the file X-5. Since the files X-3, X-4 and X-5 are not clicked by the user, the file sorting system considers that the files X-3, X-4 and X-5 have no correlation with the preset query words and punishments are carried out on the files X-3, X-4 and X-5. However, in practice, although the user does not click, the document X-5 is also related to the preset query word, and the document X-5 should not be "punished". Therefore, the value of K should not include the file X-5, thus indicating that the value of K is not accurate enough.

By applying the embodiment shown in fig. 3 of the invention, the files with higher user viewing probability are calculated according to the user click model, and the files which have been viewed by the user and have higher viewing probability are evaluated, so that the ranking result can be evaluated more accurately compared with the prior art.

Fig. 4 is a flowchart illustrating a method for obtaining a correlation value according to an embodiment of the present invention, and as shown in fig. 4, S108 may specifically include:

S108A: judging whether the user checking probability is greater than or equal to a third preset threshold value, if so, executing S108B; if not, go to S108C, where,

S108B: and obtaining the relevance value of the file according to whether the file is clicked or not.

S108C: calculating a second time length corresponding to the file according to the total historical playing time length of the file and the user historical playing times of the file, and obtaining a correlation value of the file according to the second time length, a preset corresponding relation between the second time length and a first preset threshold value, and a preset corresponding relation between the first preset threshold value and the correlation value.

Illustratively, file X-6 is taken as an example for illustration:

if the third preset threshold is 0.4, the user viewing probability of the file X-6 is 0.6, and obviously, the user viewing probability of the file X-6 is greater than the third preset threshold. Therefore, the relevance value of the file needs to be obtained according to whether the user clicks on the file X-6.

For example, the user has clicked on file X-6, which has a relevance value of 1; if the user does not click, the relevance value of the file is 0, and the specific manner is shown in step S108.

In addition, when the user viewing probability of the file X-6 is smaller than a third preset threshold, since the user viewing probability of the file is very small, we cannot obtain the relevance value of the file through the click behavior of the user, so the second duration corresponding to the file is calculated by taking the quotient of the total duration of the file played by the user and the total historical times of the file played by the user under the preset query term. In the history playing records of the user for the preset query term, the sum of the playing time lengths corresponding to each record can be used for obtaining the total time length of the user for playing the file. And then according to the number of records of clicking and playing actions in the history playing records, taking the number of times of the history playing of the file by the user as the total times of the file playing history.

For example, the second duration of the file X-6 played by the user is 2 minutes, and referring to table 1, the second duration belongs to the file "less than 5 minutes and greater than or equal to 2 minutes", and the corresponding correlation value of the file is 1, so the correlation value of the file is 1.

By applying the embodiment shown in FIG. 4 of the present invention, the relevance values are obtained by different methods for different files with checked probabilities, so that the obtained relevance values are more accurate.

The embodiment of the present invention further provides another method for obtaining the correlation value, and S108C may specifically include: and calculating the average playing time corresponding to the file according to the quotient of the total historical playing time of the file and the user historical playing times of the file, and taking the average playing time as a second time.

By applying the embodiment of the invention, the average playing time length is used as the basis for obtaining the correlation value of the file, and the correlation value of the file can be obtained under the condition that the first time length for the user to play the file cannot be obtained.

Fig. 5 is a schematic diagram illustrating a calculation flow of the normalized cumulative discount information gain value according to an embodiment of the present invention, as shown in fig. 5, S103 includes S103A and S103B, wherein,

S103A: according to the relevance value of the file before the last clicked file in the sorting result and the relevance value of each file with the user viewing probability not less than a second preset threshold, utilizing a formula:

calculating a dynamic cumulative discount information gain value, wherein,

S103B: using formulas

To calculate the ranking result in step S1010: evaluation indexes of the file X-1, the file X-2, the file X-3, the file X-4, the file X-5, the file X-6 and the file X-7 are taken as examples.

The correlation values corresponding to the file X-1, the file X-2, the file X-3, the file X-4 and the file X-5 are respectively as follows: 0. 2, 0, 3 and 2; the correlation value of the file X-6 is 0 and the correlation value of the file X-7 is 1.

And substituting the relevance value of each file in the sequencing result and the corresponding user viewing probability into a formula:

calculate theAnd calculating the normalized accumulated discount information gain value of the sorting result according to the DNDCG value of the file and taking the normalized accumulated discount information gain value as an evaluation index of the sorting result.

In the actual calculation, it is also possible to use a formula,

calculating DDCG values of all files before the last clicked file in the sorting result;

by means of the formula (I) and (II),

and calculating DCG values of all files after the last clicked file in the sorting result.

The DDCG value of the sort result is calculated again using the sum of DDCG1 and DDCG 2.

By applying the embodiment shown in fig. 5 of the present invention, the evaluation index of the ranking result can be calculated.

Corresponding to the embodiment of the invention shown in fig. 2, the embodiment of the invention also provides an obtaining device of the evaluation finger.

Fig. 6 is a schematic structural diagram of an evaluation index obtaining apparatus according to an embodiment of the present invention, and as shown in fig. 6, the apparatus includes: a first obtaining module 601, a second obtaining module 602, and a first calculating module 603, wherein,

the first obtaining module 601 is configured to obtain a first time length for playing each file corresponding to a preset query word after being clicked by a user;

the second obtaining module 602 is configured to obtain a relevance value of each file according to the first duration, a corresponding relationship between the first duration and a first preset threshold, and a corresponding relationship between the first preset threshold and the relevance value, where the relevance value is a quantitative parameter of a degree of association between each file and the preset query term;

the first calculating module 603 is configured to calculate a normalized cumulative discount information gain value of the sorting result according to the correlation value, and use the normalized cumulative discount information gain value as an evaluation index of the sorting result.

By applying the technical scheme provided by the embodiment of the invention shown in fig. 6, the first time length for playing the file in the ranking result after the user clicks is utilized to determine the relevance of the file, in general, the longer the first time length is, the higher the relevance of the file and the query word is, compared with the relevance determined by only whether the user clicks in the prior art, the more accurate the relevance of the file and the query word can be obtained, and the accuracy of evaluating the ranking result can be further improved.

The invention also provides another device for obtaining the evaluation index, which corresponds to the embodiment of the invention shown in FIG. 3.

Fig. 7 is a schematic structural diagram of another evaluation index obtaining apparatus provided in an embodiment of the present invention, as shown in fig. 7, in the embodiment shown in fig. 7, on the basis of the embodiment shown in fig. 6, there are further added: a third obtaining module 604, a fourth obtaining module 605, a constructing module 606, a second calculating module 607, a fifth obtaining module 608, a judging module 609, a setting module 6010,

The third obtaining module 604 is configured to obtain a ranking result of a file corresponding to a preset query word;

the fourth obtaining module 605 is configured to obtain, according to the click record of the user, a file that is clicked by the user and is sorted last in the sorting result, and use the file as a target file;

the building module 606 is configured to build a model for a user click behavior according to a user historical click record corresponding to the preset query term;

the second calculating module 607 is configured to calculate, for each file, a user viewing probability of a current file by using the model, starting from a first file after the target file in the sorting result;

the fifth obtaining module 608 is configured to obtain a relevance value of the current file according to the user viewing probability of the current file, whether the current file is clicked, or the playing duration of the current file by the user;

the judging module 609 is configured to judge whether the user viewing probability is greater than or equal to a second preset threshold;

the setting module 6010 is configured to, when the user viewing probability is greater than or equal to a second preset threshold, set a next file of the current file as the current file, and trigger the second calculating module 607;

the first obtaining module 601 is further configured to:

By applying the embodiment shown in fig. 7 of the invention, the files with higher user viewing probability are calculated according to the user click model, and the files which have been viewed by the user and have higher viewing probability are evaluated, so that the ranking result can be evaluated more accurately compared with the prior art.

Optionally, in a specific implementation manner of the embodiment of the present invention, the fifth obtaining module 608 includes: a determining unit 608A, a first obtaining unit 608B, and a second obtaining unit 608C, wherein,

a judging unit 608A (not shown in the figure) for judging whether the user viewing probability is greater than or equal to a third preset threshold.

A first obtaining unit 608B (not shown in the figure) for obtaining the relevance value of the file according to whether the file is clicked or not in case that the judgment result of the judging unit 608A is yes.

A second obtaining unit 608C (not shown in the figure), configured to, if the determination result of the determining unit 608A is negative, calculate a second duration corresponding to the file according to the total historical playing duration of the file and the historical playing frequency of the user of the file, and obtain a relevance value of the file according to the second duration, a corresponding relationship between the second duration and a first preset threshold, and a corresponding relationship between the first preset threshold and the relevance value.

By applying the embodiment of the invention, the relevance values are obtained by adopting different methods for the files with different checked probabilities, so that the obtained relevance values are more accurate.

Optionally, in a specific implementation manner of the embodiment of the present invention, the second obtaining unit 608C is further configured to:

Optionally, in a specific implementation manner of the embodiment of the present invention, the first calculating module 603 includes: a first computing unit 603A and a second computing unit 603B, wherein,

the first calculating unit 603A (not shown in the figure) is configured to, according to the relevance value of the file before the last clicked file in the sorting result and the relevance value of each file with the user viewing probability not less than a second preset threshold, utilize a formula:

calculating a dynamic cumulative discount information gain value, wherein,

the second computing unit 603B (not shown in the figure) for utilizingFormula (II)

By applying the embodiment of the invention, the evaluation index of the sequencing result can be calculated.

Corresponding to the embodiment shown in fig. 1 of the present invention, an electronic device is further provided in the embodiment of the present invention, fig. 8 is a schematic structural diagram of an electronic device provided in the embodiment of the present invention, as shown in fig. 8, the electronic device includes a processor 801, a communication interface 802, a memory 803 and a communication bus 804, where the processor 801, the communication interface 802 and the memory 803 complete communication with each other through the communication bus 804,

a memory 803 for storing a computer program;

the processor 801 is configured to implement the following steps when executing the program stored in the memory 803:

acquiring a first time length of each file corresponding to a preset query word played after a user clicks;

obtaining a relevance value of each file according to the first duration, the corresponding relation between the first duration and a first preset threshold value and the corresponding relation between the first preset threshold value and the relevance value, wherein the relevance value is a quantitative parameter of the relevance degree of each file and the preset query word;

The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.

The communication interface is used for communication between the electronic equipment and other equipment.

The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.

The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.

By applying the electronic device provided by the embodiment of fig. 8 of the present invention, the first time length for playing the file in the ranking result after the user clicks is utilized to determine the correlation of the file, in general, the longer the first time length is, the higher the correlation of the file and the query word is, compared with the correlation determined by only whether the user clicks in the prior art, the more accurate the correlation of the file and the query word can be obtained, and the accuracy of evaluating the ranking result can be further improved.

In still another embodiment of the present invention, a computer-readable storage medium is further provided, in which instructions are stored, and when the instructions are executed on a computer, the instructions cause the computer to execute a method for obtaining an evaluation index as described in any one of the above embodiments.

In another embodiment of the present invention, a computer program product containing instructions is provided, which when run on a computer, causes the computer to execute a method for obtaining an evaluation index as described in any of the above embodiments.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the apparatus, the electronic device, the computer-readable storage medium, and the computer program product embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and in relation to them, reference may be made to the partial description of the method embodiments.

The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for obtaining an evaluation index, the method comprising:

calculating a normalized accumulated discount information gain value of the sorting result according to the correlation value, and taking the normalized accumulated discount information gain value as an evaluation index of the sorting result;

before calculating a normalized cumulative discount information gain value of the ranking results according to the correlation value, the method further comprises:

obtaining a sorting result of the file corresponding to the preset query word;

obtaining a correlation value of the current file according to the user viewing probability of the current file and whether the current file is clicked or not, or according to the user viewing probability of the current file and the playing time of a user for the current file;

2. The method according to claim 1, wherein obtaining the relevance value of the current file according to the user viewing probability of the current file, whether the current file is clicked or not, or the playing duration of the current file by a user comprises:

judging whether the user checking probability is greater than or equal to a third preset threshold value or not;

under the condition that the user viewing probability is larger than or equal to the third preset threshold, obtaining a relevance value of the file according to whether the file is clicked or not;

or, under the condition that the user viewing probability is smaller than the third preset threshold, calculating a second duration corresponding to the file according to the total historical playing duration of the file and the historical playing times of the user of the file, and obtaining a relevance value of the file according to the second duration, the corresponding relation between the second duration and the first preset threshold, and the corresponding relation between the first preset threshold and the relevance value.

3. The method of claim 2, wherein calculating the second duration corresponding to the file according to the total duration of the historical playing of the file and the number of times of the historical playing of the user of the file comprises:

4. The method of claim 1, wherein calculating a normalized cumulative discount information gain value of the ranking result according to the correlation value comprises:

according to the relevance value of the file before the last clicked file in the sorting result and the relevance value of each file with the user viewing probability not less than a second preset threshold, utilizing a formula:

calculating a dynamic cumulative discount information gain value, wherein,

DDCG is a dynamic accumulated discount information gain value; n _ est is the sum of the number of files before the last clicked file is sorted and the number of files with the user viewing probability not less than a second preset threshold; i is the sequence number of the file in the sequencing result; w is a_iThe weight of the file with the sequence number i; rel_iBefore the last clicked document in the sorting result is sorted,and the correlation value of the file with the sequence number i; avg _ rel_iThe relevance value of the file with the sequence number of i after the last clicked file in the sorting result is sorted;

using formulas

5. An apparatus for obtaining an evaluation index, the apparatus comprising: a first obtaining module, a second obtaining module, and a first calculating module, wherein,

the first calculation module is used for calculating a normalized accumulated discount information gain value of the sorting result according to the correlation value and taking the normalized accumulated discount information gain value as an evaluation index of the sorting result;

the device further comprises: a third obtaining module, a fourth obtaining module, a constructing module, a second calculating module, a fifth obtaining module, a judging module, a setting module,

the first obtaining module is further configured to:

6. The apparatus of claim 5, wherein the fifth obtaining module is further configured to:

7. The apparatus of claim 6, wherein the fifth obtaining module is further configured to:

8. The apparatus of claim 5, wherein the first computing module comprises: a first computing unit and a second computing unit, wherein,

the first calculating unit is configured to, according to the relevance value between the file before the last clicked file in the ranking result and each file of which the user viewing probability is not less than a second preset threshold, use a formula:

calculating a dynamic cumulative discount information gain value, wherein,

the second calculation unit is used for utilizing a formula

9. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;

a memory for storing a computer program;

a processor for implementing the method steps of any of claims 1 to 4 when executing a program stored in the memory.