CN112435064A

CN112435064A - Method, device and equipment for evaluating recommendation information and computer readable storage medium

Info

Publication number: CN112435064A
Application number: CN202011362739.9A
Authority: CN
Inventors: 肖小范; 陈龙; 李宥壑
Original assignee: Beijing Wodong Tianjun Information Technology Co Ltd
Current assignee: Beijing Wodong Tianjun Information Technology Co Ltd
Priority date: 2020-11-27
Filing date: 2020-11-27
Publication date: 2021-03-02
Also published as: WO2022111291A1

Abstract

The application provides a recommendation information evaluation method, a recommendation information evaluation device, recommendation information evaluation equipment and a computer-readable storage medium, wherein the method comprises the following steps: acquiring recommendation information of an object to be recommended and object information of the object to be recommended from a recommendation information delivery platform; inputting the recommendation information and the object information into a trained case scoring model for evaluation to obtain a scoring result of the recommendation information in each dimension, wherein the dimension comprises themeness, compliance, attraction and compliance; and determining an evaluation result of the recommendation information based on the scoring result of the recommendation information in each dimension. Therefore, the recommendation information can be objectively and multi-dimensionally quantitatively evaluated, the evaluation efficiency and the evaluation accuracy can be improved, the evaluation time can be shortened, the evaluation cost can be reduced, and the evaluation risk can be reduced.

Description

Method, device and equipment for evaluating recommendation information and computer readable storage medium

Technical Field

The present application relates to the field of computer application technologies, and relates to, but is not limited to, a method, an apparatus, a device, and a computer-readable storage medium for evaluating recommendation information.

Background

The advertisement content is an important element of the advertisement and is related to the conversion rate of the commodity, the spread of the brand and the like. The advertising copy serves as a carrier of advertising content, and the importance degree of the advertising copy is very visible. In an integrated online shopping mall, over tens of thousands of brands and tens of millions of commodities are sold, millions of advertisements need to be put in every day, how to accurately and objectively evaluate millions of advertisement files in the system, and the important thing is to determine whether one advertisement file is an inferior advertisement so as to determine whether the advertisement needs to be filtered and removed, reduce unnecessary advertisement cost and reduce enterprise operation cost.

Disclosure of Invention

In view of this, embodiments of the present application provide a method, an apparatus, a device, and a computer-readable storage medium for evaluating recommendation information.

The technical scheme of the embodiment of the application is realized as follows:

the embodiment of the application provides an evaluation method of recommendation information, which comprises the following steps:

acquiring recommendation information of an object to be recommended and object information of the object to be recommended from a recommendation information delivery platform;

inputting the recommendation information and the object information into a trained case scoring model for evaluation to obtain a scoring result of the recommendation information in each dimension, wherein the dimension comprises themeness, compliance, attraction and compliance;

and determining an evaluation result of the recommendation information based on the scoring result of the recommendation information in each dimension.

In some embodiments, the method further comprises:

acquiring a theme sample set, a sensitive word set, an attraction sample set and a smoothness sample set;

respectively inputting the theme sample set, the attraction sample set and the smoothness sample set into a preset theme network model, a preset attraction network model and a preset smoothness network model to obtain a trained theme network model, a trained attraction network model and a trained smoothness network model;

constructing a dictionary tree-based search model according to the sensitive word set;

and constructing a trained pattern scoring model based on the trained theme network model, the trained attraction network model, the trained smoothness network model and the search model.

In some embodiments, inputting the theme sample set to a preset theme network model to obtain a trained theme network model, includes:

acquiring sample object information and sample recommendation information of each sample object in the theme sample set;

taking sample object information and sample recommendation information of the same sample object as a group of sample pairs, and acquiring marking information of the sample pairs, wherein the marking information represents the probability that the sample object information in the sample pairs is matched with the sample recommendation information;

and inputting each sample pair corresponding to each sample object in the theme sample set and the labeling information of each sample pair into a preset theme network model for training and learning to obtain a trained theme network model.

In some embodiments, constructing a dictionary tree based lookup model from the set of sensitive words includes:

constructing a dictionary tree according to each sensitive word in the sensitive word set;

and adding a query failure pointer to each node in the dictionary tree to obtain a dictionary tree-based search model.

In some embodiments, inputting the attraction sample set to a preset attraction network model to obtain a trained attraction network model, includes:

obtaining sample recommendation information of each sample object in the attraction sample set;

extracting information of the sample recommendation information of each sample object to obtain a characteristic information set of each sample object, wherein the characteristic information set comprises at least one of the name, the category, the preference and the attribute word of each sample object;

and inputting the characteristic information set of each sample object into a preset attraction network model to obtain a trained attraction network model.

In some embodiments, inputting the compliance sample set to a preset compliance network model to obtain a trained compliance network model, includes:

acquiring sample recommendation information of each sample object in the smoothness sample set;

performing word segmentation processing on the sample recommendation information of each sample object to obtain word segmentation of each sample recommendation information;

and inputting the word segmentation of the recommendation information of each sample into a preset smoothness network model to obtain a trained smoothness network model.

In some embodiments, the inputting the recommendation information and the object information into a trained case scoring model for evaluation to obtain a scoring result of the recommendation information in each dimension includes:

inputting the recommendation information and the object information as a group of evaluation pairs into a trained topic network model to obtain a topic scoring result of the recommendation information;

inputting the recommendation information into the search model to obtain a compliance grade result of the recommendation information;

inputting the recommendation information into a trained attraction network model to obtain an attraction scoring result of the recommendation information;

and inputting the recommendation information into a trained smoothness network model to obtain a smoothness scoring result of the recommendation information.

In some embodiments, the determining an evaluation result of the recommendation information based on the scoring results of the recommendation information in the dimensions includes:

determining a grading result of the recommendation information according to the thematic grading result, the compliance grading result, the attraction grading result and the compliance grading result;

when the scoring result of the recommendation information is larger than a first preset threshold value, determining that the evaluation result of the recommendation information is that the evaluation is passed;

and when the scoring result of the recommendation information is smaller than or equal to a first preset threshold value, determining that the evaluation result of the recommendation information is that the evaluation does not pass.

In some embodiments, the determining an evaluation result of the recommendation information based on the scoring results of the dimensions includes:

calculating the variance of the thematic score result, the compliance score result, the appeal score result and the compliance score result;

when the variance is smaller than a second preset threshold value and at least one of the thematic scoring result, the compliance scoring result, the attraction scoring result and the compliance scoring result is larger than a third preset threshold value, determining that the evaluation result of the recommendation information is that the evaluation is passed;

and when the variance is greater than or equal to a second preset threshold value, or the thematic scoring result, the compliance scoring result, the attraction scoring result and the compliance scoring result are all less than or equal to a third preset threshold value, determining that the evaluation result of the recommendation information is that the evaluation does not pass.

In some embodiments, the method further comprises:

and when the evaluation result is that the evaluation is failed, adjusting the recommendation information based on at least one of the thematic scoring result, the compliance scoring result, the attraction scoring result and the compliance scoring result.

In some embodiments, the method further comprises:

and sending the evaluation result to the recommendation information delivery platform so that the recommendation information delivery platform delivers the evaluation result as recommendation information which passes the evaluation.

The embodiment of the application provides an evaluation device of recommendation information, the device includes:

the system comprises a first acquisition module, a second acquisition module and a recommendation information delivery module, wherein the first acquisition module is used for acquiring recommendation information of an object to be recommended and object information of the object to be recommended from a recommendation information delivery platform;

the evaluation module is used for inputting the recommendation information and the object information into a trained case scoring model for evaluation to obtain a scoring result of the recommendation information in each dimension, and the dimensions comprise themeness, compliance, attraction and compliance;

and the determining module is used for determining the evaluation result of the recommendation information based on the grading result of the recommendation information in each dimension.

An embodiment of the present application provides an evaluation device for recommendation information, including:

a processor; and

a memory for storing a computer program operable on the processor;

wherein the computer program realizes the steps of the above recommendation information evaluation method when executed by a processor.

An embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions configured to perform the steps of the above-mentioned recommendation information evaluation method.

The embodiment of the application provides a method, a device, equipment and a computer readable storage medium for evaluating recommendation information, wherein the method comprises the following steps: acquiring recommendation information of an object to be recommended and object information of the object to be recommended from a recommendation information delivery platform; inputting the recommendation information and the object information into a trained case scoring model for evaluation to obtain a scoring result of the recommendation information in each dimension, wherein the dimension comprises themeness, compliance, attraction and compliance; and determining an evaluation result of the recommendation information based on the scoring result of the recommendation information in each dimension. Therefore, the recommendation information can be objectively and multi-dimensionally quantitatively evaluated, the evaluation efficiency and the evaluation accuracy can be improved, the evaluation time can be shortened, the evaluation cost can be reduced, and the evaluation risk can be reduced.

Drawings

In the drawings, which are not necessarily drawn to scale, like reference numerals may describe similar components in different views. The drawings illustrate generally, by way of example, but not by way of limitation, various embodiments discussed herein.

Fig. 1 is a schematic flow chart of an implementation of a method for evaluating recommendation information according to an embodiment of the present application;

fig. 2 is a schematic diagram of a dictionary tree constructed by the method for evaluating recommendation information according to the embodiment of the present application;

FIG. 3 is a diagram illustrating a dictionary tree-based search model constructed by a method for evaluating recommendation information according to an embodiment of the present application;

fig. 4 is a schematic flow chart of another implementation of the method for evaluating recommendation information according to the embodiment of the present application;

FIG. 5 is a schematic diagram illustrating an implementation principle of an evaluation method of an advertising creative document provided in an embodiment of the present application;

FIG. 6 is a schematic diagram illustrating training of a topic scoring model for an advertisement case according to an embodiment of the present application;

FIG. 7 is a schematic diagram of a training of a subject compliance model of an advertising copy provided in an embodiment of the present application;

FIG. 8 is a schematic diagram of training an attraction model for the topic of an advertisement case provided in an embodiment of the present application;

FIG. 9 is a schematic diagram of training a naturalness model of a topic of an advertisement document provided in an embodiment of the present application;

FIG. 10 is a diagram illustrating the scoring of 3 articles in each dimension under the ICAN model;

FIG. 11 is a radar map of scores of 3 articles in each dimension under the ICAN model;

fig. 12 is a schematic structural diagram illustrating a component of an evaluation apparatus for recommendation information according to an embodiment of the present application;

fig. 13 is a schematic structural diagram of an evaluation apparatus for recommendation information provided in an embodiment of the present application.

Detailed Description

In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be described in further detail with reference to the attached drawings, the described embodiments should not be considered as limiting the present application, and all other embodiments obtained by a person of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict.

In the following description, references to the terms "first \ second \ third" are only to distinguish similar objects and do not denote a particular order, but rather the terms "first \ second \ third" are used to interchange specific orders or sequences, where appropriate, so as to enable the embodiments of the application described herein to be practiced in other than the order shown or described herein.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the present application only and is not intended to be limiting of the application.

In order to better understand the embodiments of the present application, a method for evaluating recommendation information and the disadvantages thereof in the related art will be described first.

The following methods are mainly used for evaluating recommendation information in the related art:

the method comprises the following steps: the method is based on an evaluation method of an evaluator, and the method mainly depends on professional knowledge background, personal experience and the like of the evaluator to evaluate recommendation information, for example, an expert capable of representing the attitude of a consumer is selected to evaluate an advertisement file.

The method 2 comprises the following steps: the method comprises the steps of designing a questionnaire by combining the content of an object to be recommended based on a questionnaire survey evaluation method, screening a suitable interviewee group according to the audience attributes of the object to be recommended, evaluating the form, style, appeal point, understanding degree and the like of recommendation information by the interviewee, and selecting the recommendation information with a possibly ideal effect for actual delivery.

The method 3 comprises the following steps: the method comprises the steps of actually releasing recommendation information, evaluating the recommendation information based on relevant monitored indexes such as click times, display times and cost, and continuously modifying in an iterative mode to optimize the recommendation information and improve the recommendation effect.

The method for evaluating the recommendation information in the related art mainly has the following defects:

disadvantages of method 1: only depending on the preference, personal experience and intuition risk of the creative team is too large, the system is not responsible for putting on the market, and the manual item-by-item review of millions of pieces of recommended information in the system is unrealistic.

Disadvantages of method 2: in the traditional market research service, several months are generally needed from the design of questionnaires to the final report, and therefore, the questionnaire survey-based method is low in efficiency, long in time consumption and high in cost.

Disadvantages of method 3: compared with the method 1 and the method 2, the method belongs to the recommendation effect pre-test evaluation before the recommendation information is put in, and the method 3 belongs to the posterior evaluation. Because the evaluation method needs to put the recommendation information which is not evaluated actually, the evaluation method and the recommendation information score obtaining cost are higher, and meanwhile, higher risk exists.

It can be seen that the evaluation methods of the recommendation information in the related art either evaluate subjectively or indirectly evaluate the quality of the recommendation information according to the delivery effect of the recommendation information, and neither of these methods provides an objective quantitative evaluation for the recommendation information, and is difficult to migrate and expand to the evaluation scene of the recommendation information with a large number of objects to be recommended and a rich category.

Based on the above problems in the related art, an embodiment of the present application provides a method for evaluating recommendation information, which is applied to a device for evaluating recommendation information. The method provided by the embodiment of the application can be realized through a computer program, and when the computer program is executed, each step in the evaluation method of the recommendation information provided by the embodiment of the application is completed. In some embodiments, the computer program may be executable by a processor in an evaluation device for recommending information. Fig. 1 is a schematic flow chart of an implementation of an evaluation method for recommendation information provided in an embodiment of the present application, and as shown in fig. 1, the evaluation method for recommendation information includes the following steps:

step S101, obtaining recommendation information of an object to be recommended and object information of the object to be recommended from a recommendation information delivery platform.

In the embodiment of the present application, the recommendation information is taken as an example of an advertisement case, and a recommendation information delivery platform is explained with an advertisement delivery platform, so that the steps of the method provided by the embodiment of the present application can be implemented by an evaluation device of the advertisement case. The evaluation equipment of the advertisement case is connected with the advertisement putting platform. Before the advertisement is delivered by the advertisement delivery platform, in order to ensure the quality of the advertisement to be delivered, the advertisement to be delivered needs to be evaluated by the evaluation device of the advertisement file, so as to determine whether the advertisement file of the advertisement to be delivered is normally delivered according to the evaluation result.

The evaluation equipment of the advertisement case firstly obtains the advertisement case to be advertised and the object information corresponding to the advertisement case. Here, the advertisement document and the object information corresponding to the advertisement document may be information of the same object or information of different objects. When the advertisement file and the object information correspond to the same object, the object described by the advertisement file and the object described by the object information are shown to be the same object, namely the advertisement file is matched with the object information; when the advertisement file and the object information correspond to different objects, it is indicated that the object described by the advertisement file and the object described by the object information are different objects, that is, the advertisement file is not matched with the object information.

For example, if the advertisement pattern is "a teapot that dad likes", the object information is "transparent glass teapot chrysanthemum tea", and the described objects are all "teapots", at this time, the advertisement pattern and the object information corresponding to the advertisement pattern are information of the same object; if the advertisement case is 'teapot dad likes', the object information is 'special tea leaf Yunnan specialty before Ming dynasty', the object described by the advertisement case is 'teapot', the object described by the object information is 'tea leaf', and at the moment, the advertisement case and the object information corresponding to the advertisement case are information of different objects.

And S102, inputting the recommendation information and the object information into a trained case scoring model for evaluation to obtain a scoring result of the recommendation information in each dimension.

Here, the dimensions include themes, compliance, attractiveness, and compliance.

The embodiment of the application provides a document scoring model ICAN, which considers at least several dimensions including theme (I, Integrated), Compliance (C, company), attraction (A, App eal) and naturality (N, Natural) (also called Compliance), and trains the proposed document scoring model in advance to obtain the trained document scoring model ICAN. And inputting the information to be evaluated into a trained case scoring model ICAN to obtain scoring results of multiple dimensions of the theme I, the compliance C, the attraction A and the compliance N.

According to the embodiment of the application, the evaluation equipment of the advertisement file carries out multi-dimensional quantitative evaluation on the advertisement file to be advertised, objectivity of a grading result can be ensured, the multi-dimensional grading result is considered, evaluation accuracy can be improved, and compared with an evaluation method of the advertisement file in the related art, the evaluation efficiency can be improved, evaluation time consumption is shortened, evaluation cost is reduced, and evaluation risk is reduced.

Step S103, determining the evaluation result of the recommendation information based on the grading result of the recommendation information in each dimension.

In one implementation mode, after the scoring results of all dimensions are obtained, the scoring results of the recommendation information are determined according to all the scoring results; judging whether the grading result of the recommendation information is larger than a first preset threshold value or not; when the scoring result of the recommendation information is larger than a first preset threshold value, determining that the evaluation result of the recommendation information is that the evaluation is passed; and when the scoring result of the recommendation information is smaller than or equal to a first preset threshold value, determining that the evaluation result of the recommendation information is that the evaluation does not pass.

According to each scoring result, one implementation mode for determining the scoring result of the recommendation information is as follows: when the scoring result of the recommendation information is determined according to the scoring result of each dimension, the scoring result can be determined by combining a radar map, and the area of a quadrangle formed in the radar map by the scoring result of each dimension is determined as the scoring result of the recommendation information. The scoring result of the recommendation information determined by the implementation is determined based on the integrity of the multiple dimensions. And determining the scoring result of the recommendation information based on the integrity, so that the subsequent adjustment and optimization of the integrity of the recommendation information are facilitated, or the recommendation information with lower integral score is directly rejected.

In another implementation manner, after the scoring results of each dimension are obtained, the variance of the thematic scoring result, the compliance scoring result, the attraction scoring result and the naturalness scoring result is calculated; judging whether the variance is smaller than a second preset threshold value or not; when the variance is smaller than a second preset threshold value, further judging whether at least one of the theme scoring result, the compliance scoring result, the attraction scoring result and the currency scoring result is larger than a third preset threshold value; when at least one of the theme scoring result, the compliance scoring result, the attraction scoring result and the compliance scoring result is greater than a third preset threshold value, determining that the evaluation result of the recommendation information is that the evaluation is passed; and when the variance is greater than or equal to a second preset threshold value, or the thematic scoring result, the compliance scoring result, the attraction scoring result and the compliance scoring result are all less than or equal to a third preset threshold value, determining that the evaluation result of the recommendation information is that the evaluation does not pass.

Through the implementation mode, the recommendation information with too low score result in a certain dimension or too high score result in a certain dimension can be filtered, so that the recommendation information can be adjusted and optimized in a certain dimension in the follow-up process, or the recommendation information with large score difference in each dimension can be directly rejected.

According to the method for evaluating the recommendation information, the recommendation information of the object to be recommended and the object information of the object to be recommended are obtained from a recommendation information delivery platform; inputting the recommendation information and the object information into a trained case scoring model for evaluation to obtain a scoring result of the recommendation information in each dimension, wherein the dimension comprises themeness, compliance, attraction and compliance; and determining an evaluation result of the recommendation information based on the scoring result of the recommendation information in each dimension. Therefore, the recommendation information can be objectively and multi-dimensionally quantitatively evaluated, the evaluation efficiency and the evaluation accuracy can be improved, the evaluation time can be shortened, the evaluation cost can be reduced, and the evaluation risk can be reduced.

In some embodiments, before step S102 of the embodiment shown in fig. 1, the method for evaluating recommendation information further includes the following steps:

and step S11, acquiring a theme sample set, a sensitive word set, an attraction sample set and a smoothness sample set.

It should be noted that the acquired theme sample set, the acquired attraction sample set, and the compliance sample set may be the same sample set, and the sample recommendation information and the sample object information included in each sample set are the same, which is different from that when different models are trained, the input information of different models is different.

The recommendation information is exemplified by an advertisement case. The advertising copy is used as a carrier of the advertising content, and should convey the advertising content in a healthy and active representation mode to guide consumers to establish correct value view. The advertising copy should strictly comply with relevant laws and regulations such as "advertising laws of the people's republic of China", and put an end to advertising contents including obscency, pornography, gambling, confusion, terrorism, violence, etc. Based on this, the sensitive words which do not conform to the laws and regulations are combined into a sensitive word set. And when the sensitive words are judged to be included in the advertisement file, determining that the advertisement file is not in compliance.

And step S12, inputting the theme sample set, the attraction sample set and the smoothness sample set to a preset theme network model, a preset attraction network model and a preset smoothness network model respectively to obtain a trained theme network model, a trained attraction network model and a trained smoothness network model.

And inputting the theme sample set into a preset theme network model to obtain the trained theme network model. The trained topic network model is used for determining recommendation information of an object to be recommended and a topic score of the object information of the object to be recommended. In actual implementation, the topic of the recommendation information is determined as a first topic, the topic of the object information is determined as a second topic, then the probability of matching the first topic and the second topic is calculated, and the matching probability is determined as the topic score of the recommendation information. For example, when the topic matching probability of the topic of the advertisement document and the object information is 1, the topic score of the advertisement document is 1; when the topic matching probability of the topic of the advertisement file and the object information is 0.1, the topic score of the advertisement file is 0.1.

And inputting the attraction sample set into a preset attraction network model to obtain a trained attraction network model. The trained attraction network model is used for determining the attraction score of the recommendation information. The larger the amount of information delivered by the recommendation information, the more attractive the user, i.e. the more attractive the user. In informatics, a quantitative index for measuring the information amount is called "information entropy", and the information entropy of the recommendation information can be used in the embodiment of the application to determine the attraction. For example, when a commodity advertisement is recommended, firstly, a characteristic information set is preset, the characteristic information set comprises characteristic information of commodity categories, preference, attribute words and the like, then, an advertisement case is input into a trained attraction network model to obtain probability distribution of the advertisement case, information entropy is calculated according to the probability distribution, and the information entropy is determined as an attraction score of the advertisement case.

And inputting the smoothness sample set into a preset smoothness network model to obtain a trained smoothness network model. The trained smoothness network model is used for determining the naturalness (namely smoothness) score of the recommendation information. The language model confusion (PPL), which is the confusion, is an index for measuring the performance of a language model. In the embodiment of the application, the confusion degree can be used for quantifying the smoothness degree of the recommendation information, and the lower the confusion degree of the recommendation information is, the more smooth and natural the semantics of the recommendation information is, the higher the smoothness degree is; on the contrary, the recommendation information has the condition of unsmooth semantics or wrongly written words. In practical implementation, a preset calculation formula of PPL can be adopted to calculate the confusion degree, and then the calculated confusion degree of the recommendation information is subjected to weighted summation to obtain the compliance degree of the recommendation information based on the Chinese language model N-Gram.

And step S13, constructing a dictionary tree-based search model according to the sensitive word set.

In some embodiments, the step S13 may be implemented by:

and S131, constructing a dictionary tree according to the sensitive words in the sensitive word set.

In actual implementation, firstly, acquiring text data consisting of all sensitive words, and dividing different sensitive words into different lines; reading in the current line sensitive word, comparing the current character of the current line sensitive word with the child node of the current node, and searching the child node matched with the current character. If the searching is successful, the searched child node is used as the current node, and the next character of the sensitive word of the current line is continued; and if the search fails, inserting the newly-built child node into the current node, taking the newly-built child node as the current node, and continuing to use the next character of the sensitive word in the current row. And when all the sensitive words in the current row are searched, reading the sensitive words in the next row, and continuing to execute the same operation until the sensitive words in the next row are read and the last character of the sensitive words in the last row is searched and then is stopped. The tree constructed at this time is a dictionary tree, also called trie tree.

For example, the sensitive word set is { high h, high imitation, high interest, simulation gun, real person game }, and the constructed trie tree is shown in fig. 2.

And step S132, adding a query failure pointer to each node in the dictionary tree to obtain a dictionary tree-based search model.

Although the trie tree can be used for multi-pattern matching, each matching failure needs to be traced back, and if the pattern string is long, time is wasted, and based on this, after step S131, the embodiment of the present application continues to execute step S132, and introduces an Aho-coral automaton (multimode matching algorithm). The AC automaton is that a query failure pointer, namely a fail pointer, is added on the basis of a tie tree, and if the current node fails to be matched, the pointer is transferred to the position pointed by the fail pointer, so that the matching can be continued without backtracking.

In actual implementation, constructing an AC automaton may be implemented by the following pseudo code:

1) directing the fail of all child nodes of the root node to the root node, and then sequentially listing all the child nodes of the root node; here fail is the query failure pointer.

2) If the queue is not empty:

2.1) dequeuing, and marking the dequeued node as curr, failTo ═ curr. fail; here, failTo represents the node to which the fail of curr points.

2.2) a. determining whether curr. child [ i ] ═ failt to. child [ i ] holds;

the following holds true: curr. child [ i ]. fail ═ fail to. child [ i ];

the method is not true:

judging whether failTo is true or not;

the following holds true: curl, hair ═ root;

the method is not true: performing failTo. faill, continuing to perform step 2.2);

child [ i ] into a column, and continuing to execute the step 2);

3) if the queue is empty: and (6) ending.

Still by way of example, adding a sensitive word query failure pointer to the trie shown in fig. 2 results in a trie-based lookup model as shown in fig. 3.

Inputting the advertisement file into the search model shown in fig. 3 to obtain the result of scoring the compliance of the advertisement file, which can be implemented by the following pseudo code:

1) pointing the pointer of the current node to the root node of the AC automaton, namely curr is root;

2) reading the (next) character from the text string of the advertising copy;

3) searching all child nodes of the current node for a node matched with the character;

if the search is successful: and judging whether the current node and the node pointed by the fail of the current node indicate the end of one character string, if so, recording the index starting point of the text string in the corresponding character string storage result set (the index starting point is the length of the current index-character string and is + 1). Pointing curr to the child node, and continuing to execute the step 2);

if the search fails: step 4) is performed.

4) If fail is null (it is stated that no character string in the target character string is a prefix of the input character string, which is equivalent to a restart state machine), curr is root, and the step 2 is continuously executed);

otherwise, pointing the pointer of the current node to the fail node, and continuing to execute the step 3).

And step S14, constructing a trained case scoring model based on the trained theme network model, the trained attraction network model, the trained smoothness network model and the search model.

The embodiment of the application acquires a sample set, trains a pre-proposed case scoring model ICAN to obtain the trained case scoring model ICAN, and the trained case scoring model ICAN can carry out multidimensional quantitative evaluation on the recommendation information according to the theme I, the compliance C, the attraction A and the naturalness N, can ensure the objectivity of a scoring result, considers the multidimensional scoring result, can improve the evaluation accuracy, and can improve the evaluation efficiency, shorten the evaluation time, reduce the evaluation cost and reduce the evaluation risk compared with the evaluation method of the recommendation information in the related technology.

In some embodiments, the step S12 of inputting the theme sample set into a preset theme network model to obtain a trained theme network model may be implemented as the following steps:

step S121, obtaining sample object information and sample recommendation information of each sample object in the theme sample set.

The sample object information refers to description information of the sample object, and the sample recommendation information is recommendation content of the sample object. For example, the promoted commodity is "teapot", the sample object information is "transparent glass teapot chrysanthemum tea", and the sample recommendation information is "teapot dad would like".

And step S122, taking the sample object information and the sample recommendation information of the same sample object as a group of sample pairs, and acquiring the labeling information of the sample pairs.

The method is characterized in that a transparent glass teapot chrysanthemum tea and a teapot loved by dad are used as a group of sample pairs (also called sample sentence pairs), and before sample training, the sample pairs in a sample set are manually labeled in advance to obtain information of the sample pairs. During training, the evaluation device of the recommendation information may obtain the labeling information of the sample pair that is manually pre-labeled and stored from the storage device.

Here, the labeling information characterizes a probability that the sample object information in the sample pair matches the sample recommendation information. For example, if the sample recommendation information in the sample pair describes a "teapot" and the sample object information in the sample pair also describes a "teapot", the probability that the sample recommendation information matches the sample object information is 1; for another example, if the sample recommendation information in the sample pair describes a "teapot" and the sample object information in the sample pair describes a "mobile phone," the probability that the sample recommendation information matches the sample object information is 0.

And S123, inputting each sample pair corresponding to each sample object in the theme sample set and the labeling information of each sample pair into a preset theme network model for training and learning to obtain a trained theme network model.

Here, the trained topic network model is used for determining and outputting the labeling information of the evaluation pair based on the input evaluation pair so as to obtain the topic scoring result of the recommendation information.

When the method is implemented, sample object information and sample recommendation information in the theme sample set can be used as sample pairs and input into the preset theme network model, and the labeling information of the sample pairs is used as labeling data of the preset theme network model to perform transfer learning training to obtain a trained theme network model.

Here, the preset topic network model may be a natural language processing bert (bidirectional Encoder retrieval from transforms) model. After K rounds of training on the topic sample set (for example, let K equal to 10), a trained topic network model based on BERT is obtained and is denoted as model F.

And forming evaluation pairs by pairwise recommendation information of the object to be recommended and object information of the object to be recommended, and inputting the evaluation pairs into the trained topic network model to generate the topic score of the recommendation information. For example, suppose an advertising copy D_iThe generalized commodity is described as A_i(i.e., the object information is A)_i) Sentence pair D_i、A_iAfter model F is input, F outputs the probability r of the correlation of the two_i，r_iCan be used as an advertisement file D_iSubject score of (1).

In some embodiments, the step S12 of inputting the attraction sample set into a preset attraction network model to obtain a trained attraction network model may be implemented as the following steps:

step S124, obtaining sample recommendation information of each sample object in the attraction sample set.

In informatics, a quantization index that measures the amount of information is called "information entropy". After the user sees the recommendation information, the new information is received, and the information entropy of the cognition of the user is increased (namely the original ambiguous cognition becomes clear). For example, a user did not know that the teapot was at a reduced price before, seeing the advertisement "dad would like the teapot at 100 seventy-five folds" and learned the information that the teapot was at a reduced price. And the advertisement 'the old people like things and others do not tell him', the amount of information provided to the user is small. The user always wants to see the advertisement with information content, which is embodied on the advertisement file, and the concept is definite. Based on this, the embodiments of the present application use the information entropy of the recommendation information to evaluate its attractiveness.

And step S125, extracting information of the sample recommendation information of each sample object to obtain a characteristic information set of each sample object.

Here, the feature information set includes at least one of a name, a category, an offer, and an attribute word of the sample object.

For example, the following characteristic information is set: category, preference, attribute words.

The category is category information of the commodity, such as characteristics of 'mobile phone', 'fresh food', and the like;

offers, i.e., promotional information for the merchandise, such as "full reduction", "bonus", "discount", etc. features;

attribute words such as "red", "log", "import", "maos", "summer", etc.

Assume N pieces of feature information (C)₁，C₂，C₃，…，C_N) Advertisement writing case D_iThe probability of belonging to each feature is the vector P_i＝[p_i,1,p_i,2,p_i,3,…,p_i,N]Wherein p is_i,jFor advertising copy D_iBelong to the characteristic C_jThe probability of (c). Then, advertisement document D_iIs entropy of

If the advertising copy D_iNo explicit features, such as: d_i"good things everyone likes", then P_iWill be relatively even, E_iAnd will be relatively larger. When a user sees such an advertising copy, the user is likely to be unaware of the cloud and ignore it. On the contrary, if D_iClear concept, E_iWill be relatively smaller and it is more likely that users will be attracted to seeing such advertising copy.

And step S126, inputting the characteristic information set of each sample object into a preset attraction network model to obtain a trained attraction network model.

Here, the preset attraction network model may be a Magpie model for predicting a probability that certain recommended information belongs to each feature.

A piece of good recommendation information must be smooth and natural, and is concise and clear when being read. Therefore, when the recommendation information is quantitatively evaluated, the compliance is introduced and is used as one dimension of recommendation information evaluation. In some embodiments, the step S12 of inputting the smoothness sample set into a preset smoothness network model to obtain a trained smoothness network model may be implemented as the following steps:

and step S127, acquiring the sample recommendation information of each sample object in the smoothness sample set.

And step S128, performing word segmentation processing on the sample recommendation information of each sample object to obtain word segmentation of each sample recommendation information.

In the embodiment of the application, the confusion degree can be used for quantifying the smoothness degree of the recommendation information, and the lower the confusion degree of the recommendation information is, the more smooth and natural the semantics of the recommendation information is, the higher the smoothness degree is; on the contrary, the recommendation information has the condition of unsmooth semantics.

Obtaining the word segmentation of each sample recommendation information, for example, performing word segmentation processing on the sample advertisement document s to obtain s ═ (w)₁，w₂，…，w_n) Wherein w is_iThe ith word segmentation in the sample advertising copy s is shown, and n is the number of the word segmentation.

And S129, inputting the word segmentation of each sample recommendation information into a preset smoothness network model to obtain a trained smoothness network model.

The preset smoothness network model can be a Chinese language model N-Gram, the confusion degree of the word segmentation calculation of each sample recommendation information is calculated, and then the smoothness degree of the recommendation information is obtained by adopting weighted summation.

After word segmentation processing is performed on the recommendation information of the object to be recommended, calculating the confusion of the recommendation information, and calculating the confusion ppl(s) by adopting the following formula (1):

the smoothness of the recommendation information is obtained by adopting weighted summation, and when an N-Gram language model is used in the embodiment of the application, the values of N are taken as 2, 3 and 4 as examples. The formula for calculating the compliance f(s) of the recommendation information for different N-Gram models is shown in the following formula (2):

wherein alpha is_iAnd obtaining the smoothness of the recommendation information of the object to be recommended according to the weight values corresponding to the confusion degrees of the N in different values.

In some embodiments, the step S102 "inputting the recommendation information and the object information into a trained document scoring model for evaluation to obtain a scoring result of the recommendation information in each dimension" may be implemented as the following steps:

and step S1021, inputting the recommendation information and the object information as a group of evaluation pairs into the trained topic network model to obtain a topic scoring result of the recommendation information.

Step S1022, the recommendation information is input into the search model, and a compliance degree scoring result of the recommendation information is obtained.

And S1023, inputting the recommendation information into a trained attraction network model to obtain an attraction scoring result of the recommendation information.

And step S1024, inputting the recommendation information into a trained smoothness network model to obtain a smoothness scoring result of the recommendation information.

After obtaining recommendation information and object information of an object to be recommended in step S101, inputting the recommendation information and the object information as a set of evaluation pairs to a trained topic network model to obtain a topic score of the recommendation information, respectively inputting the recommendation information to a search model, the trained attraction network model, and a trained smoothness network model to respectively obtain a compliance score, an attraction score, and a naturalness score of the recommendation information, thereby obtaining a score result of each dimension.

When the step S103 "determining the evaluation result of the recommendation information based on the scoring result of the recommendation information in each dimension" is implemented, at least the following two implementation manners are included:

in a first implementation, the evaluation result of the recommendation information is determined based on the integrity of multiple dimensions. At this time, the step S103 "determining the evaluation result of the recommendation information based on the scoring result of the recommendation information in each dimension" may be implemented as the following steps:

step S103a1, determining the scoring result of the recommendation information according to the thematic scoring result, the compliance scoring result, the attraction scoring result and the smoothness scoring result.

Step S103a2, determining whether the scoring result of the recommendation information is greater than a first preset threshold.

When the scoring result of the recommendation information is greater than a first preset threshold value, indicating that the recommendation information meets the release requirement, and entering step S103a 3; when the scoring result of the recommendation information is less than or equal to the first preset threshold, it may be that the topic of the recommendation information does not match the topic of the object information, or that the recommendation information includes a sensitive word, or that the recommendation information includes too little information, or that there are defects that the recommendation information sentence is not smooth, there are wrongly written or mispronounced characters, and the like, and at this time, it is determined that the recommendation information does not satisfy the delivery requirement, and the process proceeds to step S103a 4.

Step S103a3, determining the evaluation result of the recommendation information as evaluation passing.

Step S103a4, determining the evaluation result of the recommendation information as that the evaluation is not passed.

In a second implementation, the evaluation result of the recommendation information is determined based on the stability of multiple dimensions. At this time, the step S103 "determining the evaluation result of the recommendation information based on the scoring result of the recommendation information in each dimension" may be implemented as the following steps:

step S103b1, calculating the variance of the theme score result, the compliance score result, the appeal score result and the naturalness score result.

Step S103b2, determining whether the variance is smaller than a second preset threshold.

When the variance is smaller than a second preset threshold value, the scoring results of all dimensions of the recommendation information are more average, and then the step S103b3 is executed; when the variance is greater than or equal to the second preset threshold, it indicates that the scoring result of the recommendation information in a certain dimension is too low or the scoring result in a certain dimension is too high, and at this time, it is determined that the recommendation information does not meet the release requirement, and the step S103b5 is entered.

Step S103b3, determining whether at least one of the theme scoring result, the compliance scoring result, the appeal scoring result, and the compliance scoring result is greater than a third preset threshold.

When at least one of the theme scoring result, the compliance scoring result, the attraction scoring result and the smoothness scoring result is greater than a third preset threshold value, indicating that the recommendation information meets the release result, then entering step S103b 4; when at least one of the theme scoring result, the compliance scoring result, the attraction scoring result and the compliance scoring result is not greater than a third preset threshold, that is, the theme scoring result, the compliance scoring result, the attraction scoring result and the compliance scoring result are all less than or equal to the third preset threshold, it is indicated that the scores of the recommendation information in each dimension are relatively average, but each scoring result is relatively low, at this time, the recommendation information is considered not to satisfy the delivery result, and the process proceeds to step S103b 5.

Step S103b4, determining the evaluation result of the recommendation information as evaluation passing.

Step S103b5, determining the evaluation result of the recommendation information as that the evaluation is not passed.

In some embodiments, in step S103a4 or step S103b5, when it is determined that the evaluation result of the recommendation information is not passed, the method may further include:

step S104, adjusting the recommendation information based on at least one of the thematic scoring result, the compliance scoring result, the attraction scoring result and the compliance scoring result.

And adjusting and optimizing the recommendation information which is not evaluated to pass, so that the evaluation result of the adjusted recommendation information is evaluation to pass.

In some embodiments, the method may further comprise:

and step S105, sending the evaluation result to the recommendation information delivery platform so that the recommendation information delivery platform delivers the evaluation result as the recommendation information passing the evaluation.

And the recommendation information evaluation device informs the recommendation information delivery platform of which recommendation information of the objects to be recommended can be directly delivered. For the adjusted recommendation information, the adjusted recommendation information needs to be sent to the recommendation information delivery platform at the same time, so that the recommendation information delivery platform delivers the adjusted recommendation information.

In some embodiments, the evaluation device of the recommendation information sends the evaluation result to the recommendation information delivery platform, and the recommendation information delivery platform can also send prompt information, so that a user who recommends an object to be recommended knows which recommendation information cannot be normally delivered.

An embodiment of the present application further provides an evaluation method of recommendation information, and fig. 4 is a schematic diagram of another implementation flow of the evaluation method of recommendation information provided in the embodiment of the present application, as shown in fig. 4, the method includes the following steps:

step S401, a theme sample set, a sensitive word set, an attraction sample set and a smoothness sample set are obtained.

Step S402, obtaining sample object information and sample recommendation information of each sample object in the theme sample set.

Step S403, using the sample object information and the sample recommendation information of the same sample object as a group of sample pairs, and obtaining the labeling information of the sample pairs.

Here, the labeling information characterizes a probability that the sample object information in the sample pair matches the sample recommendation information.

Step S404, inputting each sample pair corresponding to each sample object in the theme sample set and the labeling information of each sample pair into a preset theme network model for training and learning to obtain a trained theme network model.

And S405, constructing a dictionary tree according to the sensitive words in the sensitive word set.

Step S406, adding a query failure pointer to each node in the dictionary tree to obtain a dictionary tree-based search model.

Step S407, obtaining sample recommendation information of each sample object in the attraction sample set.

And step S408, extracting information of the sample recommendation information of each sample object to obtain a characteristic information set of each sample object.

And step S409, inputting the characteristic information set of each sample object into a preset attraction network model to obtain a trained attraction network model.

Step S410, obtaining sample recommendation information of each sample object in the smoothness sample set.

Step S411, performing word segmentation processing on the sample recommendation information of each sample object to obtain word segmentation of each sample recommendation information.

And step S412, inputting the word segmentation of each sample recommendation information into a preset smoothness network model to obtain a trained smoothness network model.

And step S413, constructing a trained case scoring model based on the trained theme network model, the trained attraction network model, the trained smoothness network model and the search model.

In some embodiments, the above steps S401 to S413 may also be performed after step S414.

Step S414, obtaining recommendation information of an object to be recommended and object information of the object to be recommended from a recommendation information delivery platform.

Step S415, inputting the recommendation information and the object information as a set of evaluation pairs to the trained topic network model, and obtaining a topic scoring result of the recommendation information.

Step S416, inputting the recommendation information into the search model to obtain a compliance grade result of the recommendation information.

And step S417, inputting the recommendation information into the trained attraction network model to obtain an attraction scoring result of the recommendation information.

And S418, inputting the recommendation information into a trained smoothness network model to obtain a smoothness scoring result of the recommendation information.

And step S419, determining the grading result of the recommendation information according to the thematic grading result, the compliance grading result, the attraction grading result and the compliance grading result.

Step S420, determining whether the scoring result of the recommendation information is greater than a first preset threshold.

When the scoring result of the recommendation information is greater than a first preset threshold value, the recommendation information is shown to meet the release requirement, and then the step S421 is executed; and when the scoring result of the recommendation information is smaller than or equal to a first preset threshold, indicating that the recommendation information does not meet the release requirement, and entering step S422.

In some embodiments, the above steps 419 to S420 may be replaced with steps S419 'to S421':

and step S419', calculating the variance of the theme scoring result, the compliance scoring result, the attraction scoring result and the naturalness scoring result.

Step S420', determine whether the variance is smaller than a second preset threshold.

When the variance is smaller than a second preset threshold value, the scoring results of all dimensions of the recommendation information are relatively average, and then the step S421' is executed; when the variance is greater than or equal to the second preset threshold, it indicates that the scoring result of the recommendation information in a certain dimension is too low or the scoring result in a certain dimension is too high, and at this time, it is determined that the recommendation information does not meet the release requirement, and the process proceeds to step S422.

Step S421', determining whether at least one of the theme rating result, the compliance rating result, the appeal rating result, and the compliance rating result is greater than a third preset threshold.

When at least one of the theme scoring result, the compliance scoring result, the attraction scoring result and the smoothness scoring result is greater than a third preset threshold value, indicating that the recommendation information meets the release result, then entering step S421; and when the theme grading result, the compliance grading result, the attraction grading result and the smoothness grading result are all smaller than or equal to a third preset threshold value, indicating that the recommendation information does not meet the release result, and entering step S422.

Step S421, determining that the evaluation result of the recommendation information is that the evaluation is passed.

And step S424 is entered, and the evaluated recommendation information is delivered.

Step S422, determining that the evaluation result of the recommendation information is that the evaluation fails.

Step S423, adjusting the recommendation information based on at least one of the theme score result, the compliance score result, the appeal score result, and the naturalness score result.

And adjusting the subject, the sensitive word, the information amount or the sentence of the recommendation information to enable the adjusted recommendation information to pass the evaluation result, so that the release condition is met, and entering the step S424.

Step S424, sending the evaluation result to the recommendation information delivery platform, where the recommendation information delivery platform delivers the evaluation result as recommendation information that the evaluation passes.

Next, an exemplary application of the embodiment of the present application in a practical application scenario will be described.

The advertising creativity is an important element of the advertisement and is related to the conversion rate of the commodity, the propagation of the brand and the like. The advertising creative pattern (corresponding to the recommendation information in the above) serves as a carrier of the advertising content, and the importance degree of the advertising creative pattern is visible. For an integrated online shopping mall selling over tens of thousands of brands and tens of millions of commodities, millions of advertisements need to be put in every day, how to evaluate millions of advertisement originality documents in the system, filter and reject inferior advertisement originality, reduce unnecessary advertisement cost and reduce enterprise operation cost is a very important problem. The existing method for evaluating the creative schemes of the advertisements mainly comprises the following steps:

1) the case evaluation method based on professionals comprises the following steps: the method mainly depends on professional knowledge background, personal experience and the like of evaluators to evaluate the creative patterns of the advertisements. For example, selecting an expert that can represent the consumer attitude to evaluate the advertising creative copy.

2) The file evaluation method based on questionnaire survey comprises the following steps: and (3) designing a questionnaire by combining the file content, screening suitable visited people according to the audience attributes of the advertisement products, evaluating the form, style, appeal points, understanding degree and the like of the file by the visited people, and selecting an advertisement creative idea with a possibly ideal effect for actual advertisement putting.

3) The effect pattern evaluation method based on actual delivery comprises the following steps: the method needs to put the advertisement actually, monitor relevant indexes such as click times, display times, cost and the like, and modify the document continuously and iteratively to optimize the creative document of the advertisement and improve the advertisement effect.

The prior art mainly has the following defects:

problems of the method 1): relying only on the preferences of the creative team, personal experience, and intuitive risks are too great and are also not responsible for the market. Moreover, for millions of advertising creative cases in the Jingdong system, it is impractical for experts to review the cases one by one.

Problems of the method 2): in conventional market research services, a minimum of one month is required from questionnaire design to final report. Methods based on questionnaires are therefore often inefficient and time consuming. In addition, this method is generally relatively expensive.

Problem of method 3): compared with the method 1) and the method 2), the method belongs to a method for pre-testing the effect of the creative documents before the advertisement is put in, and the method 3) belongs to posterior evaluation. Obviously, the evaluation method has higher evaluation cost and higher risk for evaluating and obtaining the creative scores of the paperwork.

In addition, methods 1), 2) and 3) are adopted, whether subjective evaluation is carried out or whether the quality of the creative schemes of the advertisements is indirectly evaluated according to the advertisement putting effect is achieved, and an objective quantitative method is not provided for the creatives of the advertisements by the methods, so that the evaluation methods are difficult to migrate and expand to the evaluation scene of the creative schemes of the e-commerce advertisements with large quantity of commodities and rich categories.

The embodiment of the application provides a quantitative evaluation model of an E-commerce advertisement creative document, which comprises the following steps: ICAN is used.

ICAN comprehensively considers four dimensional indexes of Compliance (C, company), attraction (A, application), theme (I, Integrated), naturalness (N, Natural) and the like of the advertisement creative case, and scores the full range and the multi-dimensional quantification of the case.

Fig. 5 is a schematic diagram illustrating an implementation principle of the evaluation method of the ad creative document provided in the embodiment of the present application, and as shown in fig. 5, the ica ad creative evaluation model provided in the embodiment of the present application mainly includes four scoring submodels: the system comprises a document compliance scoring model, a document attraction scoring model, a document theme scoring model and a document naturalness scoring model. The respective sections are described in detail below.

Section 1, literature theme scoring model:

the theme of the advertising copy is to evaluate whether it is consistent with the advertised goods. For example, an advertisement is written in a 'teapot' that dad likes, after a user clicks to enter, the user sees commodities such as electronic products and tea leaves, which not only fails to achieve the effect of popularizing the commodities, but also loses the user experience.

The embodiment of the application adopts sentence pair matching tasks of the BERT model to realize the theme scoring of the file. Fig. 6 is a schematic diagram of training an advertisement literature theme score model provided in the embodiment of the present application, and as shown in fig. 6, a training process of the literature theme score model is as follows:

1) manually marking partial positive samples and partial negative samples from the existing commodities and the advertising copy thereof to form a sample set S. The positive sample here refers to that the advertisement file is semantically related to the description of the goods popularized by the advertisement file; while negative examples are semantically unrelated.

2) And (3) taking the commodity description and the advertisement creative case of the sample set S as the input of the BERT sentence to the relation matching task, and taking whether the two are related as the labeling data of the model to perform transfer learning training.

3) After training S for K rounds (K10 in this example), a classifier based on BERT is obtained, and is set to F.

4) And (4) forming sentence pairs for all the existing advertisement documents and the description of each promoted commodity of the advertisement documents in pairs, and inputting the sentence pairs into the model F to generate the theme scores of the advertisement documents.

Suppose advertising copy D_iThe generalized commodity is described as A_i(i.e., the object information is A)_i) Sentence pair D_i、A_iAfter model F is input, F outputs the probability r of the correlation of the two_i，r_iCan be used as an advertisement file D_iSubject score of (1).

Section 2, the document compliance scoring model:

an advertising creative as a vehicle for advertising content should convey the advertising content in a healthy representation to guide consumers to establish correct value views. The creative literature of the advertisement should strictly comply with relevant laws and regulations such as the advertising law of the people's republic of China and the like, and firmly avoid the advertisement contents such as obscene, pornography, gambling, confusion, terrorism, violence and the like. Compliance is therefore also a very important and indispensable evaluation dimension when evaluating advertising creative documents.

The embodiment of the application can adopt an Aho-Corasick algorithm to score the compliance of the documents, and because the advertising creative documents containing sensitive words need to be strictly checked and killed, the score of each document is only 1 or 0.1 indicates that no sensitive words which do not conform to the laws and regulations are found in the document, namely the document conforms to the regulations; 0 indicates that there is a sensitive word in the document and thus the document is not compliant.

The Aho-Corasick is a classical multi-mode string matching algorithm, is widely applied to the pattern string matching scenes with large text strings and numerous target character strings, and is therefore suitable for compliance check of the advertising creative documents. The method for constructing the automatic machine for the sensitive words of the language case to detect the sensitive words in the language case comprises the following three steps: and constructing a sensitive word Trie tree (prefix) tree, adding a sensitive word query mismatch pointer to construct an AC automaton, performing pattern matching and returning a matched sensitive word.

The algorithm for constructing the trie tree comprises the following steps:

1) all text data is first acquired, divided into a line-by-line format.

2) Reading in each row of data, comparing the current comparison character value with the child node of the current node, and finding out the matched node

3) If the corresponding child node is found, the child node is taken as the current node, the character of the data is removed, and the step 2) is continued.

4) And if the corresponding child node is not found, inserting the newly-built node into the current node, taking the new node as the current node, and continuing to the step 2).

5) The termination condition of the operation is that all characters in the data are removed and compared.

(II) constructing an AC automaton according to the following algorithm flow:

1) the fail of all children of the root node is pointed to the root node, and then all children of the root node are listed in sequence.

2) If the queue is not empty:

2.1) dequeuing, marking the dequeued node as curr, and the failTo represents the node pointed to by the fail of curr, namely the failTo is curr

2.2) a. determining whether or not curr. child [ i ] ═ failt to. child [ i ] holds,

the following holds true: curl [ i ] fat ═ fat to [ i ] fat,

the method is not true: determining whether failTo ═ null holds

The following holds true: curl

The method is not true: execute failTo. faill, continue execution 2.2)

Child [ i ] enqueued, execute again step 2)

3) If the queue is empty: end up

(III) the mode matching operation process of the AC automaton is as follows:

1) the pointer representing the current node points to the root node of the AC automaton, i.e. curr root

2) Reading (next) character from text string

3) Finding a node matching the character from all child nodes of the current node,

if successful: and judging whether the current node and the node pointed by the fail of the current node indicate the end of one character string, if so, recording the index starting point of the text string in the corresponding character string storage result set (the index starting point is the length of the current index-character string and is + 1). curr points to the child node and proceeds to step 2).

If the failure: step 4) is performed.

4) If fail null (indicating that no character string in the target character string is the prefix of the input character string, which corresponds to a restart state machine) curr root, step 2 is executed,

otherwise, the pointer of the current node points to the fail node, and step 3) is executed.

Assuming that the existing sensitive word set black _ words is { high h, high imitation, high interest, simulation gun, real person game }, the two creatives are: creative 1? Poking here, the big card bag only needs one piece of money! "create 2 is a high-quality simulation potted landscape, and the volume is large and excellent. Ultraviolet ray resistance and wind pressure resistance. ", fig. 7 is a schematic training diagram of a compliance model of an ad document theme provided in an embodiment of the present application, and fig. 7 shows a whole process of scoring compliance of two ad creatives.

Section 3, case attraction scoring model:

the advertising copy influences the mind of the user by transmitting information to the user, and obviously, the copy with larger transmitted information can attract the user. For example, "good things all the elderly like", the amount of information is far less than "teapot dad would like".

In informatics, a quantization index that measures the amount of information is called "information entropy". After seeing the file information, the user receives the new information, and the information entropy of the cognition of the user is increased (namely the original ambiguous cognition becomes clear). For example, a user did not know that the teapot was at a reduced price before, seeing the advertisement "dad would like the teapot at 100 seventy-five folds" and learned the information that the teapot was at a reduced price. And the advertisement 'the old people like things and others do not tell him', the information provided for the user is little. The user always wants to see the advertisement with information content, which is embodied on the literature, and the concept is definite.

Therefore, the information entropy of the file is used in the embodiment of the application to evaluate the attraction. Specifically, the following "concept" is set:

product words, namely categories of commodities, such as concepts of 'mobile phones', 'fresh products', and the like;

points of interest, i.e., sales promotions for goods, such as concepts of "full reduction", "bonus", "discount", etc.;

attribute words such as "red", "log", "import", "qualified", "summer", and the like.

Assume N concepts (C)₁，C₂，C₃，…，C_N) Desk D_iThe probability belonging to each concept is a vector P_i＝[p_i,1,p_i,2,p_i,3,…,p_i,N]Wherein p is_i,jAs a case D_iBelong to concept C_jThe probability of (c). Then, case D_iIs entropy of

If the case D_iThere is no explicit concept, for example: d_i"good things everyone likes", then P_iWill be relatively even, E_iAnd will be relatively larger. When a user sees such a document, the user is likely to be unaware of the cloud and ignore it. On the contrary, if D_iClear concept, E_iWill be relatively smaller and it is more likely that users will be attracted to seeing such a document.

Fig. 8 is a schematic diagram of training an advertising copy theme appeal model provided in the embodiment of the present application, and as shown in fig. 8, the specific steps of training the copy theme appeal model are as follows:

1) extracting all commodities including their names, categories/brands, attribute words and related promotions from a commodity library of an e-commerce website;

2) using the name as a text, and using the category/brand, attribute words, promotional words and the like as labels to generate a training text set;

3) training a multi-label classifier by using the training text set, and setting the multi-label classifier as a model m; the Magpie model was used as μm in this example.

Finally, the m model can be used to predict the probability that a document belongs to each concept. Thus, there is a new document D_iThen, the probability distribution P is obtained by inputting the model_iCalculating its information entropy E_iI.e. the appeal score of the case.

Section 4, the document naturalness score model:

a good file must be smooth and natural, so that audiences can read the file smoothly and naturally, and the file is concise and clear. Therefore, when the ad creative document is quantitatively evaluated, the document compliance naturalness degree (corresponding to the compliance degree in the above) score is introduced as one dimension of document evaluation in the embodiment of the application.

In the embodiment of the application, the confusability (ppl) is used for quantifying the compliance naturalness of the file. The lower the confusion degree of the document, the more natural and smooth the document, on the contrary, the unsmooth the document exists. For sentence s ═ (w)₁，w₂，…，w_n) Wherein w is_iRepresenting the ith word in the sentence s, the number of wordsThe quantity n, and the calculation formula of the confusability thereof is shown in the following formula (3):

in the embodiment of the application, an N-Gram language model can be used, wherein the value of N is 2, 3 and 4. Calculating the confusion degree of the file by different N-Gram models, and obtaining the compliance naturalness f of the file by adopting weighted summation as shown in the following formula (4):

wherein alpha is_iAnd the weighted value corresponding to the confusion degree of N at different values.

Fig. 9 is a training schematic diagram of the subject naturalness model of the advertisement and literature provided in the embodiment of the present application, where a to-be-detected literature is input to the trained literature naturalness scoring model shown on the right side of fig. 9 to obtain a smoothness score of the to-be-detected literature, and then the smoothness score is obtained by performing normalization processing on the smoothness score.

Section 5, quantitative rating of advertising copy:

through the four sub-models, the file D is obtained_iScore for each dimension of (a): a theme point (r), a compliance point (c), an attraction force (a), and a compliance point (f). By means of the four scores, the quality of each case can be visually evaluated. Fig. 10 is a schematic diagram of the scores of the 3 documents in each dimension under the ICAN model, and corresponding description is given for the dimension with low score. If the unconventional document gives sensitive words, the unconventional document gives unconventional fragments, and the creative idea of article class mismatching gives the name of the mismatching article class.

In the embodiment of the application, each paper invents the scores of the dimensions under the ICAN model, and the radar map in FIG. 11 can be used for comparison. And the documents with lower scores can be improved or eliminated in subsequent decisions such as creative screening and the like.

The method and the device quantitatively evaluate the quality of the electronic commerce advertisement official documents by introducing four dimensions of themeness, compliance, attraction and naturalness, and can filter out the advertisement official documents with obviously too low single dimension or integrally low multiple dimensions through the ICAN model, so that the official documents can be adjusted and optimized later or directly removed. The attraction of the E-commerce advertisement copy is quantitatively evaluated by introducing the information entropy; by introducing BERT and sentence pair matching, the theme of the E-commerce advertisement case is quantitatively evaluated; by introducing four dimensions of themeness, compliance, attraction and naturalness, the comprehensive quantitative evaluation of the e-commerce advertisement file is realized.

Based on the foregoing embodiments, the present application provides an apparatus for evaluating recommendation information, where the apparatus includes modules and units included in the modules, and the modules may be implemented by a processor in a computer device; of course, the implementation can also be realized through a specific logic circuit; in the implementation process, the processor may be a Central Processing Unit (CPU), a Microprocessor Unit (MPU), a Digital Signal Processor (DSP), a Field Programmable Gate Array (FPGA), or the like.

Fig. 12 is a schematic structural diagram of the apparatus for evaluating recommendation information provided in an embodiment of the present application, and as shown in fig. 12, the apparatus 120 for evaluating recommendation information includes:

the first obtaining module 121 is configured to obtain recommendation information of an object to be recommended and object information of the object to be recommended from a recommendation information delivery platform;

the evaluation module 122 is configured to input the recommendation information and the object information into a trained case scoring model for evaluation, so as to obtain a scoring result of the recommendation information in each dimension, where the dimension includes a theme, a compliance, an attraction, and a compliance;

a determining module 123, configured to determine an evaluation result of the recommendation information based on a scoring result of the recommendation information in each dimension.

In some embodiments, the evaluation device 120 of the recommendation information may further include:

the second acquisition module is used for acquiring a theme sample set, a sensitive word set, an attraction sample set and a smoothness sample set;

the training module is used for respectively inputting the theme sample set, the attraction sample set and the smoothness sample set into a preset theme network model, a preset attraction network model and a preset smoothness network model to obtain a trained theme network model, a trained attraction network model and a trained smoothness network model;

the construction module is used for constructing a dictionary tree-based search model according to the sensitive word set;

and the construction module is used for constructing a trained case scoring model based on the trained theme network model, the trained attraction network model, the trained smoothness network model and the search model.

In some embodiments, the training module is further configured to:

In some embodiments, the construction module is further configured to:

In some embodiments, the training module is further configured to:

In some embodiments, the evaluation module is further configured to:

In some embodiments, the determining module is further configured to:

and the adjusting module is used for adjusting the recommendation information based on at least one of the thematic scoring result, the compliance scoring result, the attraction scoring result and the currency scoring result when the evaluation result is that the evaluation is failed.

and the sending module is used for sending the evaluation result to the recommendation information releasing platform so as to enable the recommendation information releasing platform to release the evaluation result as the recommendation information which passes the evaluation.

Here, it should be noted that: the above description of the items of the evaluation apparatus embodiment of the recommendation information is similar to the above description of the method, and has the same advantageous effects as the method embodiment. For technical details not disclosed in the embodiments of the apparatus for evaluating recommended information of the present application, those skilled in the art should understand with reference to the description of the embodiments of the method of the present application.

It should be noted that, in the embodiment of the present application, if the above evaluation method of the advertising copy is implemented in the form of a software functional module and is sold or used as a standalone product, it may also be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application may be essentially implemented or portions thereof contributing to the prior art may be embodied in the form of a software product stored in a storage medium, and including several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read Only Memory (ROM), a magnetic disk, or an optical disk. Thus, embodiments of the present application are not limited to any specific combination of hardware and software.

Accordingly, embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the steps in the evaluation method of recommendation information provided in the above embodiments.

Fig. 13 is a schematic diagram illustrating a component structure of the recommendation information evaluation device provided in the embodiment of the present application, and other exemplary structures of the recommendation information evaluation device 130 can be foreseen according to the exemplary structure of the recommendation information evaluation device 130 shown in fig. 13, so that the structures described herein should not be considered as limitations, for example, some components described below may be omitted, or components not described below may be added to adapt to special requirements of some applications.

The evaluation device 130 of recommendation information shown in fig. 13 includes: a processor 131, at least one communication bus 132, a user interface 133, at least one external communication interface 134, and memory 135. Wherein the communication bus 132 is configured to enable connected communication between these components. The user interface 133 may include a display screen, and the external communication interface 134 may include a standard wired interface and a wireless interface, among others. Wherein the processor 131 is configured to execute the program of the method for evaluating recommendation information stored in the memory to implement the steps in the method for evaluating recommendation information provided by the above-mentioned embodiments.

The above description of the evaluation apparatus and storage medium embodiment of recommendation information is similar to the description of the above method embodiment, with similar advantageous effects as the method embodiment. For technical details not disclosed in the embodiments of the evaluation device and the storage medium for the recommendation information of the present application, please refer to the description of the embodiments of the method of the present application for understanding.

It should be appreciated that reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that, in the various embodiments of the present application, the sequence numbers of the above-mentioned processes do not mean the execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are merely for description and do not represent the merits of the embodiments.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described device embodiments are merely illustrative, for example, the division of the unit is only a logical functional division, and there may be other division ways in actual implementation, such as: multiple units or components may be combined, or may be integrated into another system, or some features may be omitted, or not implemented. In addition, the coupling, direct coupling or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between the devices or units may be electrical, mechanical or other forms.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units; can be located in one place or distributed on a plurality of network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be separately regarded as one unit, or two or more units may be integrated into one unit; the integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit.

Those of ordinary skill in the art will understand that: all or part of the steps for realizing the method embodiments can be completed by hardware related to program instructions, the program can be stored in a computer readable storage medium, and the program executes the steps comprising the method embodiments when executed; and the aforementioned storage medium includes: a removable storage device, a ROM, a magnetic or optical disk, or other various media that can store program code.

Alternatively, the integrated units described above in the present application may be stored in a computer-readable storage medium if they are implemented in the form of software functional modules and sold or used as independent products. Based on such understanding, the technical solutions of the embodiments of the present application may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a device to perform all or part of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a removable storage device, a ROM, a magnetic or optical disk, or other various media that can store program code.

The above description is only for the embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for evaluating recommendation information, the method comprising:

2. The method of claim 1, further comprising:

3. The method of claim 2, wherein inputting the topic sample set to a preset topic network model to obtain a trained topic network model comprises:

4. The method of claim 2, wherein constructing a dictionary tree based lookup model from the set of sensitive words comprises:

5. The method of claim 2, wherein inputting the attraction sample set to a preset attraction network model to obtain a trained attraction network model comprises:

6. The method of claim 2, wherein inputting the compliance sample set to a preset compliance network model to obtain a trained compliance network model comprises:

7. The method of claim 2, wherein the inputting the recommendation information and the object information into a trained case scoring model for evaluation to obtain a scoring result of the recommendation information in each dimension comprises:

8. The method of claim 7, wherein the determining the evaluation result of the recommendation information based on the scoring result of the recommendation information in each dimension comprises:

9. The method of claim 7, wherein determining the evaluation result of the recommendation information based on the scoring results of the dimensions comprises:

10. The method according to claim 8 or 9, characterized in that the method further comprises:

11. The method according to any one of claims 1 to 9, further comprising:

12. An apparatus for evaluating recommendation information, the apparatus comprising:

13. An evaluation apparatus of recommendation information, characterized by comprising:

a processor; and

a memory for storing a computer program operable on the processor;

wherein the computer program realizes the steps of the method of any one of claims 1 to 11 when executed by a processor.

14. A computer-readable storage medium having stored thereon computer-executable instructions configured to perform the steps of the method of any one of claims 1 to 11.