CN110781340A - Offline evaluation method, system and device for recall strategy of recommendation system and storage medium - Google Patents
Offline evaluation method, system and device for recall strategy of recommendation system and storage medium Download PDFInfo
- Publication number
- CN110781340A CN110781340A CN201910924987.9A CN201910924987A CN110781340A CN 110781340 A CN110781340 A CN 110781340A CN 201910924987 A CN201910924987 A CN 201910924987A CN 110781340 A CN110781340 A CN 110781340A
- Authority
- CN
- China
- Prior art keywords
- offline
- recall
- groups
- data
- evaluation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/60—Information retrieval; Database structures therefor; File system structures therefor of audio data
- G06F16/63—Querying
- G06F16/635—Filtering based on additional data, e.g. user or group profiles
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/60—Information retrieval; Database structures therefor; File system structures therefor of audio data
- G06F16/68—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/686—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using information manually generated, e.g. tags, keywords, comments, title or artist information, time, location or usage information, user ratings
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Library & Information Science (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses an offline evaluation method, system, device and storage medium for a recall strategy of a recommendation system, wherein the method comprises the steps of packaging an offline click rate estimation model in the recommendation system into a JAR file package; obtaining offline user data of M days, wherein M is a positive integer; calculating to obtain N groups of offline recall data sets based on N groups of offline recall strategy models according to the offline user data, wherein N is a positive integer greater than 1; converting the data formats of the N groups of offline recall data sets into data formats required by an offline click rate estimation model; operating a JAR file package to carry out click rate estimation on the N groups of offline recall data sets after data format conversion to obtain estimated scores of the corresponding N groups of offline recall strategy models, and storing the corresponding score data to a database; and performing offline evaluation on the N groups of offline recall strategy models based on the estimated scores of the N groups of offline recall strategy models. The method and the device realize the offline automatic evaluation of the recall strategy, and have the advantages of simple operation and reliable evaluation result.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to an offline evaluation method, system and device for a recall strategy of a recommendation system and a storage medium.
Background
The recall rough part in the recommendation system needs to be contrasted and evaluated against a plurality of recall strategies developed by algorithm engineers during off-line development. First, an algorithm engineer is typically relied upon to subjectively judge the quality of the recalled audio album. This requires a profound understanding of the business and operation by the relevant personnel, while there is inevitably a subjective assumption. Thirdly, some similar methods for comparing results such as recall data set coverage and the like are not capable of well distinguishing the results, and have a difference with a real online business actual scene, so that the situation of large deviation between an offline state and an online state is easy to occur. Even the opposite conclusion is reached. In addition, direct judgment basis can be obtained by directly online the new recall strategy as the ABtest, but the development cost is huge, and if the recall effect is not good, the online service is directly influenced negatively, so that huge risk exists.
Disclosure of Invention
The invention aims to at least solve one of the technical problems in the prior art, and particularly provides an offline evaluation method, system, device and storage medium for a recall strategy of a recommendation system, which can realize offline automatic evaluation of the recall strategy and have reliable evaluation results.
In order to achieve the above object of the present invention, according to a first aspect of the present invention, there is provided an offline evaluation method of a recommendation system recall policy, the method comprising the steps of:
packaging an offline click rate estimation model in a recommendation system into a JAR file package;
obtaining offline user data of M days, wherein M is a positive integer;
calculating to obtain N groups of offline recall data sets based on N groups of offline recall strategy models according to the offline user data, wherein N is a positive integer greater than 1;
converting the data format of the N groups of offline recall data sets into the data format required by the offline click rate estimation model;
operating the JAR file package to carry out click rate estimation on the N groups of offline recall data sets after data format conversion to obtain estimated scores of the corresponding N groups of offline recall strategy models, and storing the corresponding score data to a database;
and performing offline evaluation on the N groups of offline recall strategy models based on the estimated scores of the N groups of offline recall strategy models.
Preferably, the method further comprises:
and establishing the off-line click rate estimation model.
Preferably, the acquiring the offline user data for M days includes:
and acquiring user login information of M days from the database, and extracting the user ID of the login user.
Preferably, the offline evaluation of the N groups of offline recall policy models based on the estimated scores of the N groups of offline recall policy models includes:
and comparing the estimated scores of the N groups of offline recall strategy models, judging the quality of the N groups of offline recall strategy models according to the height of the estimated scores, and outputting an evaluation result, wherein the higher the estimated score is, the better the corresponding offline recall strategy model is judged to be.
According to a second aspect of the present invention, there is provided an offline evaluation system for recommending system recall policies, the system comprising:
the model packaging module is used for packaging the offline click rate estimation model in the recommendation system into a JAR file package;
the data acquisition module is used for acquiring offline user data of M days, wherein M is a positive integer;
the data processing module is used for calculating and obtaining N groups of offline recall data sets based on N groups of offline recall strategy models according to the offline user data, wherein N is a positive integer greater than 1;
the data conversion module is used for converting the data format of the N groups of offline recall data sets into the data format required by the offline click rate estimation model;
the estimation scoring module is used for operating the JAR file package to carry out click rate estimation on the N groups of offline recall data sets after data format conversion to obtain estimation scores of the corresponding N groups of offline recall strategy models, and storing the corresponding score data to a database;
and the offline evaluation module is used for performing offline evaluation on the N groups of offline recall strategy models based on the estimated scores of the N groups of offline recall strategy models.
Preferably, the system further comprises:
and the model establishing module is used for establishing the offline click rate estimation model.
Preferably, the data acquisition module is specifically configured to:
and acquiring user login information of M days from the database, and extracting the user ID of the login user.
Preferably, the offline evaluation module is specifically configured to:
and comparing the estimated scores of the N groups of offline recall strategy models, judging the quality of the N groups of offline recall strategy models according to the height of the estimated scores, and outputting an evaluation result, wherein the higher the estimated score is, the better the corresponding offline recall strategy model is judged to be.
According to a third aspect of the present invention, the present invention provides an offline evaluation apparatus for a recommended system recall policy, including a memory, a processor, and a computer program stored in the memory and operable on the processor, where the processor implements the steps of the offline evaluation method for a recommended system recall policy of the first aspect when executing the computer program.
According to a fourth aspect of the present invention, there is provided a computer-readable storage medium storing a computer program which, when executed by a processor, implements the steps of the method for offline evaluation of recommendation system recall policies of the first aspect.
According to the scheme, the invention provides an offline evaluation method, a system, a device and a storage medium for a recall strategy of a recommendation system, wherein the method comprises the steps of packaging an offline click rate estimation model in the recommendation system into a JAR file package; obtaining offline user data of M days, wherein M is a positive integer; calculating to obtain N groups of offline recall data sets based on N groups of offline recall strategy models according to the offline user data, wherein N is a positive integer greater than 1; converting the data format of the N groups of offline recall data sets into the data format required by the offline click rate estimation model; operating the JAR file package to carry out click rate estimation on the N groups of offline recall data sets after data format conversion to obtain estimated scores of the corresponding N groups of offline recall strategy models, and storing the corresponding score data to a database; and performing offline evaluation on the N groups of offline recall strategy models based on the estimated scores of the N groups of offline recall strategy models. According to the method, the offline recall strategy model can be automatically calculated and scored, the quality of the recall strategy is judged according to the scored height, and an algorithm engineer is not required to subjectively judge the quality of the recall strategy through experience; the offline automatic evaluation of the recall strategy is realized, the ABtest development work is not required, and the development cost is reduced; and packaging the off-line click rate estimation model into a JAR file package, and directly operating to obtain a scoring result for judging the high or low click rate of the recall set only by configuring and inputting the recall set data in a fixed format, so that the operation is simple and the evaluation result is reliable.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
The above and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
FIG. 1 is a flow chart illustrating an off-line evaluation method for a recommendation system recall strategy in accordance with a preferred embodiment of the present invention;
FIG. 2 is a schematic structural diagram of an offline evaluation system for recommending system recall policies in a preferred embodiment of the present invention;
fig. 3 is a schematic structural diagram of an offline evaluation apparatus for a recommendation system recall policy according to a preferred embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
In the description of the present invention, it is to be understood that the terms "longitudinal", "lateral", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", and the like, indicate orientations or positional relationships based on those shown in the drawings, and are used merely for convenience of description and for simplicity of description, and do not indicate or imply that the referenced devices or elements must have a particular orientation, be constructed in a particular orientation, and be operated, and thus, are not to be construed as limiting the present invention.
In the description of the present invention, unless otherwise specified and limited, it is to be noted that the terms "mounted," "connected," and "connected" are to be interpreted broadly, and may be, for example, a mechanical connection or an electrical connection, a communication between two elements, a direct connection, or an indirect connection via an intermediate medium, and specific meanings of the terms may be understood by those skilled in the art according to specific situations.
According to a first aspect of the present invention, the present invention provides an offline evaluation method for a recommendation system recall policy, as shown in fig. 1, the method may include the following steps:
s101, packaging an offline click rate estimation model in a recommendation system into a JAR file package;
the recall rough-scheduling part in the recommendation system needs to compare and evaluate various recall strategies developed by algorithm engineers during off-line development, so that the reliability of each recall strategy is judged, the algorithm engineers can conveniently know performance parameters such as recall accuracy of each recall strategy, and a recall strategy model meeting high-reliability requirements such as high recall accuracy is developed and used for the recommendation system. When multiple recall strategies developed by an algorithm engineer are evaluated, firstly, an algorithm model (namely an offline click rate estimation model) corresponding to an estimated offline click rate in a recommendation system is required to be subjected to data encapsulation, and the offline click rate estimation model is encapsulated into a JAR file package, so that reliable and intuitive data support is provided for the algorithm engineer to evaluate the quality of the recall strategies through the encapsulated JAR data package, the understanding requirements of the algorithm engineer on services can be reduced, and the recall data can be conveniently automatically configured to serve as input to be transmitted to the encapsulated JAR file package.
S102, obtaining offline user data of M days, wherein M is a positive integer;
in order to perform comparative evaluation on multiple recall strategies developed by an algorithm engineer, offline user data of M days are acquired as original input data of the evaluated recall strategies, that is, the offline user data of M days are input into the system, and after the offline user data are subjected to data processing, the offline user data are used as input data to evaluate and score a recall strategy model. The above M is a positive integer, and the larger the value of M, the larger the data amount as the original input data, so that a more accurate evaluation result can be obtained. However, the larger M is, the larger the input data size becomes, and the system operation time becomes longer accordingly, so that the value of M can be set as required, i.e. selected preferentially according to the configuration of the system and the requirement of evaluation accuracy. Specifically, in this embodiment, the offline user data is offline data stored in the database, and can be retrieved at any time when needed, and the user login information for M days is obtained from the database, and the user ID of the login user is extracted as the offline user data.
S103, calculating to obtain N groups of offline recall data sets based on N groups of offline recall strategy models according to the offline user data, wherein N is a positive integer greater than 1;
after obtaining the offline user data, inputting the offline user data into N groups of offline recall strategy models to be evaluated respectively for calculation, thereby obtaining N groups of offline recall data sets, wherein N is a positive integer greater than 1, and N is specifically the number of groups of the offline recall strategy models to be evaluated.
S104, converting the data format of the N groups of offline recall data sets into a data format required by an offline click rate estimation model;
and then, carrying out data format conversion on the N groups of offline recall data sets to convert the data formats into fixed format data formats required by the offline click rate estimation model, so that the offline click rate estimation model can identify and process the N groups of offline recall data sets.
S105, operating a JAR file package to carry out click rate estimation on the N groups of offline recall data sets after data format conversion to obtain estimated scores of the corresponding N groups of offline recall strategy models, and storing the corresponding score data to a database;
then, a JAR file package packaged by an offline click rate estimation model in a recommendation system is operated on the system, click rate estimation calculation is carried out on N groups of offline recall data sets after data format conversion through the offline click rate estimation model to obtain estimated scores of the N groups of corresponding offline recall strategy models, namely, each group of offline recall strategy models corresponds to one estimated score which represents the quality of the performance of the recall strategy corresponding to the offline recall strategy models, and meanwhile, corresponding score data are stored in a database to facilitate subsequent lookup and calling.
And S106, performing offline evaluation on the N groups of offline recall strategy models based on the estimated scores of the N groups of offline recall strategy models.
And finally, performing offline evaluation on the N groups of offline recall strategy models according to the estimated scores of the N groups of offline recall strategy models, namely comparing the scores to obtain the quality of the corresponding offline recall strategy. The method can be specifically carried out in the following way: and comparing the estimated values of the N groups of offline recall strategy models, judging the quality of the N groups of offline recall strategy models according to the height of the estimated values, and outputting an estimation result, wherein in the estimation result, the N groups of offline recall strategy models can be sequenced and output according to the height of the estimated values, wherein the higher the estimated value is, the higher the probability that a recall set is clicked by a user can be considered from the model angle, the higher the probability that the user likes the recall set is, the better the recall strategy is, and the better the corresponding offline recall strategy model is judged. The quality of each group of recall strategies can be visually displayed through the evaluation result. The automatic scoring estimation of the offline click rate model does not need to be carried out online, so that a large amount of engineering development time is saved, and negative business influence caused by the ABtest is avoided.
In this embodiment, the method may further include:
and establishing an off-line click rate estimation model.
And developing and establishing a corresponding off-line click rate estimation model in a modeling system by an algorithm engineer according to actual requirements.
According to the scheme, the invention provides an offline evaluation method, a system, a device and a storage medium for a recall strategy of a recommendation system, wherein the method comprises the steps of packaging an offline click rate estimation model in the recommendation system into a JAR file package; obtaining offline user data of M days, wherein M is a positive integer; calculating to obtain N groups of offline recall data sets based on N groups of offline recall strategy models according to the offline user data, wherein N is a positive integer greater than 1; converting the data formats of the N groups of offline recall data sets into data formats required by an offline click rate estimation model; operating a JAR file package to carry out click rate estimation on the N groups of offline recall data sets after data format conversion to obtain estimated scores of the corresponding N groups of offline recall strategy models, and storing the corresponding score data to a database; and performing offline evaluation on the N groups of offline recall strategy models based on the estimated scores of the N groups of offline recall strategy models. According to the method, the offline recall strategy model can be automatically calculated and scored, the quality of the recall strategy is judged according to the scored height, and an algorithm engineer is not required to subjectively judge the quality of the recall strategy through experience; the offline automatic evaluation of the recall strategy is realized, the ABtest development work is not required, and the development cost is reduced; and packaging the off-line click rate estimation model into a JAR file package, and directly operating to obtain a scoring result for judging the high or low click rate of the recall set only by configuring and inputting the recall set data in a fixed format, so that the operation is simple and the evaluation result is reliable.
According to a second aspect of the present invention, there is provided an offline evaluation system for recommending system recall policies, as shown in fig. 2, the system may include:
the model packaging module 201 is used for packaging the offline click rate estimation model in the recommendation system into a JAR file package;
the recall rough-scheduling part in the recommendation system needs to compare and evaluate various recall strategies developed by algorithm engineers during off-line development, so that the reliability of each recall strategy is judged, the algorithm engineers can conveniently know performance parameters such as recall accuracy of each recall strategy, and a recall strategy model meeting high-reliability requirements such as high recall accuracy is developed and used for the recommendation system. When multiple recall strategies developed by an algorithm engineer are evaluated, firstly, an algorithm model (namely an offline click rate estimation model) corresponding to an estimated offline click rate in a recommendation system is required to be subjected to data encapsulation, and the offline click rate estimation model is encapsulated into a JAR file package, so that reliable and intuitive data support is provided for the algorithm engineer to evaluate the quality of the recall strategies through the encapsulated JAR data package, the understanding requirements of the algorithm engineer on services can be reduced, and the recall data can be conveniently automatically configured to serve as input to be transmitted to the encapsulated JAR file package.
The data acquisition module 202 is configured to acquire offline user data for M days, where M is a positive integer;
in order to perform comparative evaluation on multiple recall strategies developed by an algorithm engineer, offline user data of M days are acquired as original input data of the evaluated recall strategies, that is, the offline user data of M days are input into the system, and after the offline user data are subjected to data processing, the offline user data are used as input data to evaluate and score a recall strategy model. The above M is a positive integer, and the larger the value of M, the larger the data amount as the original input data, so that a more accurate evaluation result can be obtained. However, the larger M is, the larger the input data size becomes, and the system operation time becomes longer accordingly, so that the value of M can be set as required, i.e. selected preferentially according to the configuration of the system and the requirement of evaluation accuracy. Specifically, in this embodiment, the offline user data is offline data stored in the database, and can be retrieved at any time when needed, and the user login information for M days is obtained from the database, and the user ID of the login user is extracted as the offline user data.
The data processing module 203 is configured to calculate, according to the offline user data, N groups of offline recall data sets based on N groups of offline recall policy models, where N is a positive integer greater than 1;
after obtaining the offline user data, inputting the offline user data into N groups of offline recall strategy models to be evaluated respectively for calculation, thereby obtaining N groups of offline recall data sets, wherein N is a positive integer greater than 1, and N is specifically the number of groups of the offline recall strategy models to be evaluated.
The data conversion module 204 is used for converting the data format of the N groups of offline recall data sets into a data format required by an offline click rate estimation model;
and then, carrying out data format conversion on the N groups of offline recall data sets to convert the data formats into fixed format data formats required by the offline click rate estimation model, so that the offline click rate estimation model can identify and process the N groups of offline recall data sets.
The estimation scoring module 205 is configured to run a JAR file package to perform click rate estimation on the N groups of offline recall data sets after data format conversion to obtain estimation scores of the corresponding N groups of offline recall policy models, and store the corresponding score data to a database;
then, a JAR file package packaged by an offline click rate estimation model in a recommendation system is operated on the system, click rate estimation calculation is carried out on N groups of offline recall data sets after data format conversion through the offline click rate estimation model to obtain estimated scores of the N groups of corresponding offline recall strategy models, namely, each group of offline recall strategy models corresponds to one estimated score which represents the quality of the performance of the recall strategy corresponding to the offline recall strategy models, and meanwhile, corresponding score data are stored in a database to facilitate subsequent lookup and calling.
And the offline evaluation module 206 is configured to perform offline evaluation on the N groups of offline recall policy models based on the estimated scores of the N groups of offline recall policy models.
And finally, performing offline evaluation on the N groups of offline recall strategy models according to the estimated scores of the N groups of offline recall strategy models, namely comparing the scores to obtain the quality of the corresponding offline recall strategy. The method can be specifically carried out in the following way: and comparing the estimated values of the N groups of offline recall strategy models, judging the quality of the N groups of offline recall strategy models according to the height of the estimated values, and outputting an estimation result, wherein in the estimation result, the N groups of offline recall strategy models can be sequenced and output according to the height of the estimated values, wherein the higher the estimated value is, the higher the probability that a recall set is clicked by a user can be considered from the model angle, the higher the probability that the user likes the recall set is, the better the recall strategy is, and the better the corresponding offline recall strategy model is judged. The quality of each group of recall strategies can be visually displayed through the evaluation result. The automatic scoring estimation of the offline click rate model does not need to be carried out online, so that a large amount of engineering development time is saved, and negative business influence caused by the ABtest is avoided.
In this embodiment, the system may further include:
and the model establishing module is used for establishing an offline click rate estimation model.
And developing and establishing a corresponding off-line click rate estimation model by an algorithm engineer through a model establishing module in the modeling system according to actual requirements.
According to the scheme, the invention provides an offline evaluation method, a system, a device and a storage medium for a recall strategy of a recommendation system, wherein the method comprises the steps of packaging an offline click rate estimation model in the recommendation system into a JAR file package; obtaining offline user data of M days, wherein M is a positive integer; calculating to obtain N groups of offline recall data sets based on N groups of offline recall strategy models according to the offline user data, wherein N is a positive integer greater than 1; converting the data formats of the N groups of offline recall data sets into data formats required by an offline click rate estimation model; operating a JAR file package to carry out click rate estimation on the N groups of offline recall data sets after data format conversion to obtain estimated scores of the corresponding N groups of offline recall strategy models, and storing the corresponding score data to a database; and performing offline evaluation on the N groups of offline recall strategy models based on the estimated scores of the N groups of offline recall strategy models. According to the method, the offline recall strategy model can be automatically calculated and scored, the quality of the recall strategy is judged according to the scored height, and an algorithm engineer is not required to subjectively judge the quality of the recall strategy through experience; the offline automatic evaluation of the recall strategy is realized, the ABtest development work is not required, and the development cost is reduced; and packaging the off-line click rate estimation model into a JAR file package, and directly operating to obtain a scoring result for judging the high or low click rate of the recall set only by configuring and inputting the recall set data in a fixed format, so that the operation is simple and the evaluation result is reliable.
According to a third aspect of the present invention, there is provided an offline evaluation apparatus 3 for recommending system recall policies, as shown in fig. 3, the apparatus comprising a memory 301, a processor 302 and a computer program 303 stored in the memory 301 and operable on the processor 302, wherein the processor 302 implements the steps of the automated code scanning method according to the first aspect of the present invention when executing the computer program 303.
According to a fourth aspect of the invention, there is provided a computer readable storage medium storing a computer program which, when executed by a processor, performs the steps of the automated code scanning method of the first aspect of the invention.
In this embodiment, the module/unit integrated with the offline evaluation system for recalling policies of the recommendation system, if implemented in the form of a software functional unit and sold or used as an independent product, may be stored in a computer-readable storage medium. Based on such understanding, all or part of the flow of the method according to the embodiments of the present invention may also be implemented by a computer program, which may be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method embodiments may be implemented. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.
Those skilled in the art will appreciate that while some embodiments herein include some features included in other embodiments, rather than others, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A method for offline evaluation of recommendation system recall policies, the method comprising:
packaging an offline click rate estimation model in a recommendation system into a JAR file package;
obtaining offline user data of M days, wherein M is a positive integer;
calculating to obtain N groups of offline recall data sets based on N groups of offline recall strategy models according to the offline user data, wherein N is a positive integer greater than 1;
converting the data format of the N groups of offline recall data sets into the data format required by the offline click rate estimation model;
operating the JAR file package to carry out click rate estimation on the N groups of offline recall data sets after data format conversion to obtain estimated scores of the corresponding N groups of offline recall strategy models, and storing the corresponding score data to a database;
and performing offline evaluation on the N groups of offline recall strategy models based on the estimated scores of the N groups of offline recall strategy models.
2. The method of offline evaluation of recommendation system recall policies according to claim 1, further comprising:
and establishing the off-line click rate estimation model.
3. The method of claim 1, wherein the obtaining of the offline user data for M days comprises:
and acquiring user login information of M days from the database, and extracting the user ID of the login user.
4. The method of any one of claims 1-3, wherein the offline evaluation of the N sets of offline recall policy models based on the estimated scores of the N sets of offline recall policy models comprises:
and comparing the estimated scores of the N groups of offline recall strategy models, judging the quality of the N groups of offline recall strategy models according to the height of the estimated scores, and outputting an evaluation result, wherein the higher the estimated score is, the better the corresponding offline recall strategy model is judged to be.
5. An offline evaluation system for recommending system recall policies, the system comprising:
the model packaging module is used for packaging the offline click rate estimation model in the recommendation system into a JAR file package;
the data acquisition module is used for acquiring offline user data of M days, wherein M is a positive integer;
the data processing module is used for calculating and obtaining N groups of offline recall data sets based on N groups of offline recall strategy models according to the offline user data, wherein N is a positive integer greater than 1;
the data conversion module is used for converting the data format of the N groups of offline recall data sets into the data format required by the offline click rate estimation model;
the estimation scoring module is used for operating the JAR file package to carry out click rate estimation on the N groups of offline recall data sets after data format conversion to obtain estimation scores of the corresponding N groups of offline recall strategy models, and storing the corresponding score data to a database;
and the offline evaluation module is used for performing offline evaluation on the N groups of offline recall strategy models based on the estimated scores of the N groups of offline recall strategy models.
6. The system of claim 5, further comprising:
and the model establishing module is used for establishing the offline click rate estimation model.
7. The offline evaluation system of recommendation system recall policies of claim 5, wherein the data acquisition module is specifically configured to:
and acquiring user login information of M days from the database, and extracting the user ID of the login user.
8. The offline evaluation system of recommendation system recall policies of any of claims 5-7, wherein the offline evaluation module is specifically configured to:
and comparing the estimated scores of the N groups of offline recall strategy models, judging the quality of the N groups of offline recall strategy models according to the height of the estimated scores, and outputting an evaluation result, wherein the higher the estimated score is, the better the corresponding offline recall strategy model is judged to be.
9. An apparatus for offline evaluation of recommended system recall policies, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the method for offline evaluation of recommended system recall policies according to any one of claims 1 to 4.
10. A computer-readable storage medium storing a computer program, wherein the computer program when executed by a processor implements the steps of the method for offline evaluation of recommendation system recall policies of any of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910924987.9A CN110781340B (en) | 2019-09-27 | 2019-09-27 | Offline evaluation method, system, device and storage medium for recall strategy of recommendation system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910924987.9A CN110781340B (en) | 2019-09-27 | 2019-09-27 | Offline evaluation method, system, device and storage medium for recall strategy of recommendation system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110781340A true CN110781340A (en) | 2020-02-11 |
| CN110781340B CN110781340B (en) | 2023-05-23 |
Family
ID=69384645
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910924987.9A Active CN110781340B (en) | 2019-09-27 | 2019-09-27 | Offline evaluation method, system, device and storage medium for recall strategy of recommendation system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110781340B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111444438A (en) * | 2020-03-24 | 2020-07-24 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for determining recall permission rate of recall strategy |
| CN111460384A (en) * | 2020-03-31 | 2020-07-28 | 北京百度网讯科技有限公司 | Policy evaluation method, device and equipment |
| CN113592543A (en) * | 2021-07-27 | 2021-11-02 | 北京达佳互联信息技术有限公司 | Method and device for evaluating live broadcast guide effect |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160147754A1 (en) * | 2014-11-21 | 2016-05-26 | Microsoft Corporation | Offline evaluation of ranking functions |
| US20190205477A1 (en) * | 2017-12-29 | 2019-07-04 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Method for Processing Fusion Data and Information Recommendation System |
| CN110275779A (en) * | 2019-06-20 | 2019-09-24 | 北京百度网讯科技有限公司 | A kind of resource acquiring method, device, equipment and storage medium |
-
2019
- 2019-09-27 CN CN201910924987.9A patent/CN110781340B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160147754A1 (en) * | 2014-11-21 | 2016-05-26 | Microsoft Corporation | Offline evaluation of ranking functions |
| US20190205477A1 (en) * | 2017-12-29 | 2019-07-04 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Method for Processing Fusion Data and Information Recommendation System |
| CN110275779A (en) * | 2019-06-20 | 2019-09-24 | 北京百度网讯科技有限公司 | A kind of resource acquiring method, device, equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| 李成;冯青青;: "推荐***准确度衡量方案――引入权重概念" * |
| 陈巧红;余仕敏;贾宇波;: "广告点击率预估技术综述" * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111444438A (en) * | 2020-03-24 | 2020-07-24 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for determining recall permission rate of recall strategy |
| CN111444438B (en) * | 2020-03-24 | 2023-09-01 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for determining quasi-recall rate of recall strategy |
| CN111460384A (en) * | 2020-03-31 | 2020-07-28 | 北京百度网讯科技有限公司 | Policy evaluation method, device and equipment |
| CN111460384B (en) * | 2020-03-31 | 2023-09-08 | 北京百度网讯科技有限公司 | Policy evaluation method, device and equipment |
| CN113592543A (en) * | 2021-07-27 | 2021-11-02 | 北京达佳互联信息技术有限公司 | Method and device for evaluating live broadcast guide effect |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110781340B (en) | 2023-05-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10726332B2 (en) | Generating a training model based on feedback | |
| CN109800220B (en) | Big data cleaning method, system and related device | |
| CN110765310B (en) | Audio album recommendation method and system based on parameter configuration | |
| CN110781340A (en) | Offline evaluation method, system and device for recall strategy of recommendation system and storage medium | |
| CN112087334A (en) | Alarm root cause analysis method, electronic device and storage medium | |
| CN113298121B (en) | Message sending method and device based on multi-data source modeling and electronic equipment | |
| CN112565422B (en) | Method, system and storage medium for identifying fault data of power internet of things | |
| CN108322917B (en) | Wireless network access fault positioning method, device, system and storage medium | |
| CN110968380A (en) | Data visualization method and system | |
| US20230034061A1 (en) | Method for managing proper operation of base station and system applying the method | |
| CN113191784A (en) | Abnormal enterprise identification method and device, electronic equipment and storage medium | |
| CN116957309A (en) | Textile scene dynamic monitoring method, system and medium | |
| CN106304085B (en) | Information processing method and device | |
| CN107341202A (en) | Appraisal procedure, device and the storage medium of business datum table amendment risk factor | |
| CN110070392B (en) | User loss early warning method and device | |
| CN109241249B (en) | Method and device for determining burst problem | |
| CN108229071A (en) | Cutting performance degradation assessment method and system based on AR models Yu SVDD algorithms | |
| CN109783559B (en) | Method and device for acquiring real estate transaction data, electronic equipment and storage medium | |
| CN114386468A (en) | Network abnormal flow detection method and device, electronic equipment and storage medium | |
| CN110868732B (en) | VoLTE radio access failure problem positioning method, system and equipment | |
| CN110909200A (en) | Offline and real-time combined audio album recommendation system recall method and system | |
| CN112783799A (en) | Software daemon test method and device | |
| CN110705627A (en) | Target detection method, target detection system, target detection device and readable storage medium | |
| CN118193503B (en) | Hierarchical management system for server center data | |
| CN117873828A (en) | Alarm processing method, device, equipment and medium of server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |