CN110866776B

CN110866776B - Data calibration method for popularization resources, electronic equipment and readable storage medium

Info

Publication number: CN110866776B
Application number: CN201910970141.9A
Authority: CN
Inventors: 翟凯旋; 李东方
Original assignee: Shanghai Zhangmen Science and Technology Co Ltd
Current assignee: Shanghai Zhangmen Science and Technology Co Ltd
Priority date: 2019-10-12
Filing date: 2019-10-12
Publication date: 2023-11-24
Anticipated expiration: 2039-10-12
Also published as: CN110866776A

Abstract

The embodiment of the invention provides a data calibration method for popularization resources, electronic equipment and a readable storage medium. The data calibration method for popularization resources comprises the following steps: first estimated value information of a plurality of target popularization resources in a preset time period is obtained, a target segment function is determined according to each first estimated value information, fitting is carried out on the target segment function, a target fitting function is obtained, second estimated value information of the operation probability of the popularization resources to be displayed currently is obtained, and based on the target fitting function, the second estimated value information is calibrated to determine calibration information of the second estimated value information. Because the determined function is a target segment function according to each first estimated value information, and is different from a sequence preserving function adopted by a sequence preserving regression calibration scheme in the prior art, the calibration information is more approximate to the true value information because the degree of freedom of the target segment function is lower than that of the sequence preserving function and the target segment function is a continuous function, and the calibration effect is improved to a certain extent.

Description

Data calibration method for popularization resources, electronic equipment and readable storage medium

Technical Field

The present invention relates to the field of communications, and in particular, to a method for calibrating data of a popularization resource, an electronic device, and a readable storage medium.

Background

With the rise of the internet, web advertising has become a major profit model for large advertising platforms and advertisers. Before delivering the advertisement, the advertisement platform needs to order the advertisement to be delivered in consideration of the benefits of various aspects (including users, advertisers and advertisement platforms), and the ordering of the advertisement is mainly based on the estimated information (click rate estimated information and/or conversion rate estimated information) of the advertisement.

For targeted conversion bid (opc, optimized Cost Per Click) advertisements, the higher the conversion rate estimation information or the higher the product of the click rate estimation information and the conversion rate estimation information, the higher the ranking. For Cost Per Click (CPC) advertisements, the higher the Click rate estimate information, the higher the ranking. Therefore, the accuracy of the estimated information of the advertisement directly affects the ranking result, and the estimated information has an error with the true value information, so that the estimated information needs to be calibrated.

The existing calibration scheme of the order preserving regression (isotonic regression) is very easy to cause over fitting under the condition of sparse data, and the uneven generated by the step when the time interval is too large is easy to cause larger errors in local parts, so that the effect on the estimated value information after calibration is also poor.

Disclosure of Invention

The embodiment of the invention provides a data calibration method for popularization resources, electronic equipment and a readable storage medium, which are used for solving the problem that the effect of the existing calibration scheme on estimated value information after calibration is poor.

In a first aspect of an embodiment of the present invention, a data calibration method is provided, including:

obtaining first estimated value information of the operated probabilities of a plurality of popularization resources in a preset time period;

determining a target piecewise function according to each piece of first estimation information; the target piecewise function is used for describing the relation between first estimation information of the operated probability of the popularization resource and calibration information of the operated probability of the popularization resource;

fitting the target piecewise function to obtain a target fitting function;

obtaining second estimated value information of the operation probability of the popularization resource to be displayed currently;

and calibrating the second estimation information based on the target fitting function to determine calibration information of the second estimation information.

In a second aspect of the embodiments of the present invention, there is provided an electronic device including a processor, a memory, and a computer program stored on the memory and executable on the processor, the computer program implementing the steps of the data calibration method described above when executed by the processor.

In a third aspect of the embodiments of the present invention, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the data calibration method described above.

Aiming at the prior art, the invention has the following advantages:

according to the data calibration method for the popularization resources, first estimated value information of a plurality of target popularization resources in a preset time period is obtained, and a target piecewise function is determined according to each piece of first estimated value information, wherein the target piecewise function is used for describing the relation between the first estimated value information of the probability that the popularization resources are operated and calibration information of the probability that the popularization resources are operated; fitting the target piecewise function to obtain a target fitting function, obtaining second estimated value information of the current popularization resource to be displayed operated probability, and calibrating the second estimated value information based on the target fitting function to determine calibration information of the second estimated value information. Since the determined function is a target segment function according to each first estimation information, it is different from the order-preserving function (the order-preserving function is a monotonically increasing function) adopted in the order-preserving regression calibration scheme in the prior art. Since the order-preserving function belongs to a discontinuous function, the problem that the uneven generated by the step is caused to locally generate larger errors exists. The degree of freedom of the objective fitting function in the embodiment is lower than that of the order-preserving function, and the objective fitting function belongs to a continuous function, is relatively smooth and has no step problem, so that the calibration information is more approximate to the true value information, the calibration effect is improved to a certain extent, and the problem of poor fitting effect which is easily caused by adopting an order-preserving regression calibration scheme under the condition of sparse data can be avoided. The current popularization resource to be displayed can be an advertisement resource, the operated probability can be the clicked probability or the converted probability of the advertisement, namely the second estimated value information can be the clicked probability or the converted probability estimated value information of the advertisement. Since for an opc ad, the higher the estimated information of conversion rate or the higher the product of the estimated information of click rate and the estimated information of conversion rate, the higher the ranking of the opc ad. For CPC advertising, the higher the click-through rate estimate, the higher the ranking. Thus, the accuracy of the estimated information of the advertisement directly affects the ordering result of the advertisement, thereby affecting the profit of the advertiser. The embodiment of the invention improves the accuracy of the second estimated value information to a certain extent, so that the rationality of the sequencing result obtained after sequencing the advertisements can be ensured to a certain extent, and the income of advertisers is ensured.

The foregoing description is only an overview of the present invention, and is intended to be implemented in accordance with the teachings of the present invention in order that the same may be more clearly understood and to make the same and other objects, features and advantages of the present invention more readily apparent.

Drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to designate like parts throughout the figures. In the drawings:

FIG. 1 is a flow chart of steps of a method for calibrating data of a promotion resource according to an embodiment of the present invention;

FIG. 2 is a flowchart illustrating steps of another method for calibrating data of a promotional resource according to an embodiment of the present invention;

FIG. 3 is a flowchart illustrating steps of a method for determining coordinate information according to an embodiment of the present invention;

FIG. 4 is a flowchart illustrating steps of a data calibration method for promoting resources according to another embodiment of the present invention;

fig. 5 is a flowchart illustrating steps of another method for determining coordinate information according to an embodiment of the present invention.

Detailed Description

In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description.

It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.

Referring to fig. 1, fig. 1 is a step flowchart of a data calibration method for promotion resources according to an embodiment of the present invention, where the data calibration method of the present embodiment may be applied to a server. The method of the embodiment comprises the following steps:

step 101, obtaining first estimated value information of the operated probabilities of a plurality of popularization resources in a preset time period.

The plurality of promotion resources within the preset time period can be promotion resources within a week from the current time or promotion resources within a few days from the current time. The promotion resources within the preset time period can be obtained from the history log data, or the promotion resources within the preset time period can be obtained from other data sources. The promotion resource may be a promotion resource for a clicked operation or a promotion resource for a displayed operation, for example, a clicked advertisement in a displayed advertisement, and the promotion resource for a displayed advertisement. The first estimation information of the promotion resource can be click rate estimation information or conversion rate estimation information. Click rate estimation information can be obtained by inputting attribute information of popularization resources into a click rate estimation model, and conversion rate estimation information can be obtained by inputting attribute information of popularization resources into a conversion rate estimation model. The attribute information of the promotion resource of the clicked operation may include, for example, whether the advertisement is converted, advertiser characteristics, user portraits, and the like. The advertiser characteristics include characteristics of the advertisement itself, such as creative, type, etc., and the user images, such as geographic location information, age, gender, etc., where the user clicks on the advertisement. The attribute information of the promotion resource to be displayed is, for example, whether the advertisement to be displayed is clicked, the advertiser characteristics, the user portraits, etc. The conversion rate estimation model may be obtained by training a pre-built conversion rate model, the click rate estimation model may be obtained by training a pre-built click rate model, and a person skilled in the art may train the first model as well as train the second model according to the prior art.

Step 102, determining a target segment function according to each piece of first estimation information.

The target piecewise function is used for describing the relation between the first estimated value information of the operated probability of the popularization resource and the calibration information of the operated probability of the popularization resource.

In step 101, if the promotion resource is a promotion resource operated by clicking, the promotion resource operated probability is the promotion resource converted probability, and the first estimation information is the first conversion rate estimation information. Conversion Rate (CVR) is an indicator of how effective a Cost Per Action (CPA) advertisement is, simply the Conversion Rate of a user clicking on an advertisement to become a valid active or registered or even paid user. CVR is equal to the conversion times divided by the clicking times, and the first conversion rate estimated value information is estimated value information of the conversion rate output through a conversion rate estimated model. Wherein, according to each first estimated value information, the determining the target segment function can be realized by the steps of:

determining a plurality of coordinate information according to each first conversion rate estimation information;

drawing a target curve according to the plurality of coordinate information;

and determining a target piecewise function according to the shape of the target curve.

Wherein, according to each first conversion rate estimation information, determining a plurality of coordinate information can be realized by the following steps:

dividing all the first conversion rate estimated value information according to a first preset dividing rule to obtain a plurality of first estimated value information intervals;

calculating first ratio information of converted times and clicked times of popularization resources of clicked operation corresponding to each first estimated value information interval, and obtaining first conversion rate real value information corresponding to each first estimated value information interval;

calculating a first average value corresponding to each first estimated value information interval, wherein the first average value is an average value of first conversion rate estimated value information included in the first estimated value information interval;

and the coordinate information of the target curve is formed by taking a first average value corresponding to the first estimated value information interval as a first abscissa value and taking first conversion rate true value information corresponding to the first estimated value information interval as a first ordinate value.

Specifically, all the first conversion rate estimation information is divided according to a first preset division rule to obtain a plurality of first estimation information intervals, which can be exemplarily described with reference to the following tables 1 and 2: first conversion rate estimation information of each promotion resource, and whether each promotion resource is converted, 1 indicates that a click is converted, and 0 indicates that a click is not converted are shown in table 1. If the [0,1] section is divided into 10 section units, each section unit having a length of 0.1, the section unit 1 includes conversion rate estimation information of 0 or more and 0.1 or less, the section unit 2 includes conversion rate estimation information of 0.1 or more and 0.2 or less, and so on, the section unit 10 includes conversion rate estimation information of 0.9 or more and 1 or less, the first preset division rule is to divide the first conversion rate estimation information belonging to a section unit among all the first conversion rate estimation information into one first estimation information section, for example, referring to table 2, the PCVR (0.1) belonging to the promotion resource 1, the PCVR (0.11) of the promotion resource 2, and the PCVR (0.15) of the promotion resource 4 are divided into one first estimation information section (section 1) in the following table 2.

TABLE 1

TABLE 2

Taking the promotion resource of the clicked operation corresponding to the interval 1 as an example, it is known from the above table 1 that the promotion resource 1 is converted 1 time, the promotion resource 2 is not converted, and the promotion resource 4 is converted once, and it is known that the conversion number of the promotion resource of the clicked operation corresponding to the interval 1 is equal to 2, and the click number of the promotion resource of the clicked operation corresponding to the first estimation information interval is equal to 3, so that the first ratio information is 2 divided by 3 and is equal to 0.67, that is, the first conversion ratio actual value information corresponding to the interval 1 is 0.67. Since interval 1 includes the first conversion estimate information of 0.1, 0.11, and 0.15, the average value of the 3 first conversion estimate information is 0.13, and thus the first average value of interval 1 is 0.13. The first conversion rate true value information and the first average value corresponding to other intervals can be calculated, so that the coordinate information of the target curve can be determined. And then a target curve can be drawn according to the coordinate information, and a target piecewise function is determined according to the shape of the target curve. For example, the determined objective piecewise function is two linear functions and one logarithmic function. For example:

target piecewise function:

in this embodiment, only the data in table 1 and table 2 are used as an example, the number of popularization resources in the actual scene is relatively large, the number of section units dividing the [0,1] section into the same length is also relatively large, for example, the [0,1] section is divided into 100 section units with the same length, and the number of the divided section units is not limited in this embodiment, that is, the first preset dividing rule is not limited.

In step 101, if the promotion resource is a displayed promotion resource, the first estimation information is first click rate estimation information. The Click-Through-Rate (CTR) may be the actual number of clicks on the advertisement divided by the number of impressions of the advertisement. The first click rate estimation information is the click rate estimation information output by the click rate estimation model. Wherein, according to each first estimated value information, the determining the target segment function can be realized by the steps of:

determining a plurality of coordinate information according to each first click rate estimation information;

drawing a target curve according to the plurality of coordinate information;

Wherein, according to each first click rate estimation information, determining a plurality of coordinate information can be realized by the following steps:

dividing all the first click rate estimation information according to a first preset dividing rule to obtain a plurality of first estimation information intervals;

calculating first ratio information of the clicked times and the exhibited times of the exhibited popularization resources corresponding to each first estimated value information interval to obtain first click rate real value information corresponding to each first estimated value information interval;

Calculating a first average value corresponding to each first estimated value information interval, wherein the first average value is an average value of first click rate estimated value information included in the first estimated value information interval;

the coordinate information of the target curve is formed by taking a first average value corresponding to the first estimated value information interval as a first abscissa value and taking first click rate true value information corresponding to the first estimated value information interval as a first ordinate value.

The method for dividing all the first click rate estimation information described herein is similar to the method for dividing the first conversion rate estimation information described above, and will not be described here again. Other steps are similar to determining coordinate information of the target curve if the promotion resource is the promotion resource of the clicked operation, and are not described herein, and when the promotion resource is the promotion resource of the displayed promotion resource, the difference between the coordinate information of the target curve determined when the promotion resource is the promotion resource of the clicked operation is that: the first click rate real value information corresponding to the first estimated value information interval is equal to the first ratio information of the click times and the display times of the displayed popularization resources corresponding to the first estimated value information interval.

And 103, fitting the target piecewise function to obtain a target fitting function.

Fitting the target segment function to obtain a parameter value of the target segment function, thereby obtaining the target fitting function. For example, for a target piecewise function:the target fitting function obtained after fitting is +.>

Fitting the target segments to obtain parameter values of the parameters a, b, c, d and h, respectively, thereby obtaining a target fitting function. It should be noted that, the fitting method for the objective piecewise function may be performed by using other fitting methods such as a least square method, a newton interpolation method, etc. to obtain each parameter value of the piecewise function. Those skilled in the art can refer to a fitting method such as a least square method, a newton interpolation method, etc. in the prior art, and this embodiment will not be described in detail.

Step 104, obtaining second estimated value information of the operation probability of the popularization resource to be displayed currently.

If the first estimated value information obtained in the above step is the first conversion rate estimated value information, the attribute information of the popularization resource to be displayed currently may be input into the conversion rate estimation model in this step to obtain second estimated value information, that is, second conversion rate estimated value information. If the first estimated value information obtained in the step is the first click rate estimated value information, the attribute information of the popularization resource to be displayed currently can be input into a click rate estimated model in the step to obtain second estimated value information, namely the second click rate estimated value information. Or the first conversion rate estimated value information and the first click rate estimated value information are obtained through the steps, and the second conversion rate calculated value and the second click rate estimated value information can be obtained in the step respectively.

Step 105, calibrating the second estimation information based on the objective fitting function to determine calibration information of the second estimation information.

The second estimation information is calibrated based on the target fitting function, so as to determine calibration information of the second estimation information, which can be realized through the following steps:

determining a target variable interval in which the second conversion rate estimation information falls from a plurality of variable value intervals of the target fitting function;

and calculating the calibration information of the second conversion rate estimation information according to the second conversion rate estimation information and the subfunctions in the objective fitting function corresponding to the objective variable interval.

For example, if the second conversion estimate information obtained in step 104 is 0.1, then the value interval may be taken from the above-described illustrated plurality of variables: and determining a target variable interval in 0-0.2, 0.2< x-0.6 and x >0.6, wherein the target variable interval is 0-0.2. And substituting 0.1 into the sub-function of the objective fitting function corresponding to 0.2 and 0.2 to obtain the calibration information of the second conversion rate estimated value information as 0.12. If the second conversion rate estimated value information is 0.5, the target variable interval is 0.2< x less than or equal to 0.6, the sub-function of the target fitting function corresponding to the interval of 0.2< x less than or equal to 0.6 is f (x) =0.6x+0.05, and the calibration information of the second conversion rate estimated value information is obtained by substituting the second conversion rate estimated value information of 0.5 into the sub-function. Therefore, the problem that the calibration effect is poor because only one fixed fitting function is adopted to calibrate the estimated value information in the prior art is avoided.

It should be noted that, especially for the opcb advertisement, the conversion data is more sparse than the click data, and the calibration data is often selected to be more close to the on-line immediate data distribution, so that the problem of the sparse conversion rate calibration data is more serious, the effect of simply selecting the warranty regression calibration scheme is quite unsmooth, and the calibration effect of the on-line experiment is relatively poor. In this embodiment, a target piecewise function is adopted to fit the target piecewise function, so as to obtain a target fitting function, the obtained target fitting function is also a piecewise function, and a proper sub-function is selected from a plurality of sub-functions included in the target fitting function to calibrate the second estimated value information, so that the calibration accuracy is improved to a certain extent, and the calibrated second estimated value information is more approximate to the real value information.

Calculating calibration information of the second conversion rate estimation information according to the second conversion rate estimation information and the subfunctions in the objective fitting function corresponding to the objective variable interval

Alternatively, the second estimation information is calibrated based on the objective fitting function to determine calibration information of the second estimation information, which may be achieved by:

Determining a target variable interval in which the second click rate estimation information falls from a plurality of variable value intervals of the target fitting function;

and calculating the calibration information of the second click rate estimation information according to the second click rate estimation information and the subfunctions in the target fitting function corresponding to the target variable interval.

For example, if the second click rate estimation information is obtained in step 104, the second click rate estimation information may be substituted into the sub-function of the objective fitting function corresponding to the case where the promotion resource is the displayed promotion resource, and the specific calculation method is similar to the method for calculating the calibration information of the second conversion rate estimation information when the promotion resource is the promotion resource operated by clicking, so that details are not repeated herein.

According to the data calibration method provided by the embodiment, the first estimated value information of each popularization resource in a preset time period is obtained, the objective piecewise function is determined according to each first estimated value information, the objective piecewise function is fitted, the objective fitting function is obtained, the second estimated value information of the current popularization resource to be displayed is obtained, and the calibration information of the second estimated value information is determined according to the objective fitting function and the second estimated value information. Since the determined function is a target segment function according to each first estimation information, it is different from the order-preserving function (the order-preserving function is a monotonically increasing function) adopted in the order-preserving regression calibration scheme in the prior art. Since the order-preserving function belongs to a discontinuous function, the problem that the uneven generated by the step is caused to locally generate larger errors exists. The degree of freedom of the objective fitting function in the embodiment is lower than that of the order-preserving function, and the objective fitting function belongs to a continuous function, is relatively smooth and has no step problem, so that the calibration information is more approximate to the true value information, the calibration effect is improved to a certain extent, and the problem of poor fitting effect which is easily caused by adopting an order-preserving regression calibration scheme under the condition of sparse data can be avoided. The current popularization resource to be displayed can be an advertisement resource, the operated probability can be the clicked probability or the converted probability of the advertisement, namely the second estimated value information can be the clicked probability or the converted probability estimated value information of the advertisement. Since for an opc ad, the higher the estimated information of the converted probability or the product of the estimated information of the click rate probability and the estimated information of the converted probability, the higher the ranking of the opc ad. For CPC advertisements, the higher the estimate of the probability of being clicked, the higher the ranking. Thus, the accuracy of the estimated information of the advertisement directly affects the ranking result, and thus, the advertiser's revenue. The embodiment of the invention improves the accuracy of the second estimated value information to a certain extent, so that the rationality of the sequencing result obtained after sequencing the advertisements can be ensured to a certain extent, and the income of advertisers is ensured.

Referring to fig. 2, fig. 2 is a flowchart illustrating steps of another method for calibrating data of a promotion resource according to an embodiment of the present invention. The promotion resource in the method is the promotion resource of the clicked operation, and the method comprises the following steps:

step 201, obtaining first estimation information of the operated probabilities of a plurality of popularization resources in a preset time period.

This step may refer to the explanation of step 101, and will not be repeated here.

Step 202, dividing all promotion resources of the clicked operation into a plurality of first blocks according to main features corresponding to the conversion rate estimation model.

The main features corresponding to the conversion rate estimation model refer to main features, such as advertiser features and user portraits, on which the pre-constructed conversion rate model is trained. The following describes, in an exemplary manner, the division of the promotion resources of all clicked operations into a plurality of first partitions in conjunction with table 3, and here takes the promotion resources shown in table 1 as an example: for example, the promotion resource 1, promotion resource 2 and promotion resource 3 are divided into the partition 1, and the promotion resource 4 and promotion resource 5 are divided into the partition 2, which is not described here, please refer to the following table 3:

TABLE 3 Table 3

And 203, calculating first ratio information of converted times and clicked times of popularization resources of clicked operations included in each first partition, and obtaining conversion rate real value information of the first partition.

For example, the promotion resource 1 included in the block 1 is converted, the promotion resource 2 and the promotion resource 3 are not converted (taking the promotion resource 1 as the advertisement 1 as an example, the promotion resource 1 is converted, that is, the advertisement 1 is converted), so that the number of times of converted promotion resources of the clicked operation included in the block 1 is 1, the total number of times of clicking is 3, the first ratio information is the ratio information of 1 to 3, the conversion ratio actual value information of the block 1 is 0.33, the conversion ratio actual value information of other blocks can be calculated, and the calculated conversion ratio actual value information of each block can be shown in the following table 4.

TABLE 4 Table 4

Step 204, calculating a second average value of the first conversion rate estimation information of the clicked promotion resource included in each first partition.

For example, the first conversion rate estimation information of the promotion resource of the clicked operation included in the partition 1 is: 0.1, 0.11, and 0.22, so that the second average of the first conversion estimate information of the promoted resource of the clicked operation included in the partition 1 is 0.14. The second average of the first conversion estimate information of the promoted resource of the clicked operation included in the other blocks may also be calculated, and specifically may be described with reference to table 4.

Step 205, calculating ratio information of the conversion rate actual value information of the first block and the second average value corresponding to the first block, and obtaining second ratio information corresponding to the first block.

For example, the second ratio information corresponding to the partition 1 is equal to the ratio information of 0.33 to 0.14, that is, the second ratio information corresponding to the partition 1 is equal to 2.35, and the second ratio information corresponding to other first partitions can be obtained as well, which can be specifically shown in the above table 4.

And 206, clustering the plurality of first blocks according to the second ratio information corresponding to each first block to obtain at least one clustered second block.

For example, the second ratio information may be gathered into one type with a value greater than 1, and the second ratio information may be gathered into one type with a value less than or equal to 1. Then the segments 1 and 2 can be grouped into one type, the segments 3 and 4 are grouped into one type, and the clustered segments comprise two second segments, wherein the first second segment corresponds to the segments 1 and 2, and the second segment corresponds to the segments 3 and 4.

Step 207, determining a target segment function according to each piece of first estimation information.

Wherein, according to each first estimated value information, the determining the target segment function can be realized by the following steps:

drawing a target curve according to the plurality of coordinate information;

In the above embodiments, a manner of determining a plurality of coordinate information according to each first conversion rate estimation information is described, herein, another implementation manner of determining a plurality of coordinate information according to each first conversion rate estimation information is described, that is, how to determine a plurality of coordinate information after performing steps 202 to 206 (i.e., partitioning and clustering the clicked popularization resource), and referring to fig. 3, fig. 3 is a flowchart of a method for determining coordinate information according to an embodiment of the present invention. The implementation method comprises the following steps:

step 301, dividing the first conversion rate estimated value information included in all the first blocks corresponding to the second blocks according to a second preset dividing rule, so as to obtain a plurality of second estimated value information intervals.

It should be noted that, the second preset dividing rule divides the first conversion rate estimation information belonging to a certain interval unit in the first conversion rate estimation information included in all the first blocks corresponding to the second blocks into a second estimation information interval.

And dividing the first conversion rate estimated value information included in all the first blocks corresponding to the second blocks according to a second preset dividing rule to obtain a plurality of second estimated value information intervals. For example, the first conversion rate estimation information included in the block 1 and the block 2 corresponding to the first and second blocks illustrated in step 206 is divided to obtain a plurality of second estimation information intervals. The method for obtaining the plurality of second estimation information intervals in this step is similar to that illustrated in step 102, except that the popularization resources are divided differently.

For example, the first conversion rate estimation information included in all the first partitions corresponding to the first and second partitions includes: first conversion rate estimation information (0.1) of popularization resource 1, first conversion rate estimation information (0.11) of popularization resource 2, first conversion rate estimation information (0.22) of popularization resource 3, first conversion rate estimation information (0.15) of popularization resource 4, and first conversion rate estimation information (0.21) of popularization resource 5, 0.1, 0.11, and 0.15 may be divided into one second estimation information interval, and 0.21 and 0.22 may be divided into one second estimation information interval. The first conversion rate estimation information of the promotion resource 1 is, for example, the first conversion rate estimation information of the advertisement 1. Specific reference may be made to the following Table 5:

TABLE 5

And 302, calculating third ratio information of converted times and clicked times of the promotion resource of the clicked operation corresponding to each second estimated value information interval, and obtaining second conversion rate true value information corresponding to each second estimated value information interval.

As can be seen from the above table 4, the number of times of being converted by the promotion resources (promotion resource 1, promotion resource 2, and promotion resource 4) of the clicked operation corresponding to the interval a is 2, the number of times of being clicked is 3, and the third ratio information is equal to 2 divided by 3, that is, 0.67, so the second conversion rate real value information corresponding to the interval a is equal to 0.67. The second conversion actual value information corresponding to the interval B may also be calculated to be 0.165.

In this embodiment, only a small amount of data is used for the exemplary description, and the number of second conversion rate actual value information corresponding to the determined second estimated value information section is relatively large because the second estimated value information section in the actual scene is relatively large.

Step 303, calculating a second average value corresponding to each second estimated value information interval, where the second average value is an average value of the first conversion rate estimated value information included in the second estimated value information interval.

For example, the average value of the first conversion rate estimation information (0.1, 0.11, 0.15) included in the section a is equal to 0.12, and the average value of the first conversion rate estimation information (0.22, 0.21) included in the section B is equal to 0.165. Specifically, refer to table 5.

And 304, forming coordinate information of a target curve corresponding to the second block by taking a second average value corresponding to the second estimated value information interval as a first abscissa value and taking second conversion rate true value information corresponding to the second estimated value information interval as a first ordinate value.

The coordinate information of the first and second blocks corresponding to the target curve is referred to, for example, the coordinate information shown in the above table 5. After the coordinate information of the target curve corresponding to the second block is determined, the target curve can be drawn according to the determined coordinate information, and the drawn target curve is the target curve corresponding to the first block and the second block. The target curve corresponding to the second segment may also be plotted.

Step 208, determining a target piecewise function corresponding to each second piecewise block according to the shape of the target curve corresponding to each second piecewise block.

Step 209, fitting the target piecewise function corresponding to each second piecewise block to obtain a target fitting function corresponding to each target piecewise function.

Step 210, obtaining second estimated value information of the operation probability of the popularization resource to be displayed currently.

Step 211, calibrating the second estimation information based on the objective fitting function to determine calibration information of the second estimation information.

Note that the second estimation information in this embodiment is second conversion rate estimation information. The objective fitting function obtained in this embodiment is multiple, that is, multiple piecewise functions can be obtained, each piecewise function corresponds to multiple variable intervals, the sub-function of each piecewise function is more finely distinguished, each sub-function corresponds to a respective variable interval, thus the objective variable interval in which the estimated information of the second conversion rate falls can be more accurately determined, and further, the sub-function corresponding to the objective variable interval is used for calibrating the estimated information of the second conversion rate, so that the problem that overfitting is very easy to be caused by adopting an order preserving regression calibration scheme under the sparse data condition can be avoided, the calibration information is more approximate to the real value information, the calibration effect is improved to a certain extent, and in this embodiment, the accuracy of the calibration information of the estimated information of the second conversion rate is improved. In general, when advertisements are ranked according to the product of click rate estimation information and conversion rate estimation information, conversion rate estimation information has a larger weight, so that accuracy of conversion rate estimation information has a larger influence on the ranking result, and accuracy of advertisement conversion rate estimation information directly influences the ranking result, so that benefits of advertisers can be influenced. The embodiment of the invention improves the accuracy of the second conversion rate estimated value information to a certain extent, so that the rationality of the sequencing result obtained after sequencing the advertisements can be ensured to a certain extent, and the income of advertisers is ensured.

Referring to fig. 4, fig. 4 is a flowchart illustrating steps of a data calibration method for promoting resources according to another embodiment of the present invention. The popularization resource in the method is the displayed popularization resource, and the method comprises the following steps:

step 401, obtaining first estimation information of the operated probabilities of a plurality of popularization resources in a preset time period.

And step 402, dividing all the displayed popularization resources into a plurality of first blocks according to the main features corresponding to the click rate estimation model.

The main features corresponding to the click rate estimation model refer to main features based on which a pre-constructed click rate model is trained, such as advertisement main features and user portraits. The division of the plurality of first partitions is similar to step 202 and is not repeated here. It should be noted that 1 indicates that the displayed promotion resource is clicked, and 0 indicates that the displayed promotion resource is not clicked. The specific divided first plurality of partitions may be referred to in the following table 6:

TABLE 6

Step 403, calculating first ratio information of the clicked times and the exhibited times of the exhibited popularization resource included in each first partition, and obtaining click rate real value information of the first partition.

For example, the promotion resource 1 included in the partition 1 is clicked, the promotion resource 2 and the promotion resource 3 are not clicked, so that the number of times of clicked promotion resources displayed in the partition 1 is 1, the total number of times of clicked promotion resources is 3, the first ratio information is the ratio information of 1 to 3, the click rate authenticity information of the partition 1 is 0.33, the click rate authenticity information of other partitions can be calculated, and the calculated click rate authenticity information of each partition can be shown by referring to the following table 7.

TABLE 7

Step 404, calculating a second average value of the first click rate estimation information of the displayed popularization resource included in each first partition.

For example, the first click rate estimation information of the displayed popularization resource included in the partition 1 is: 0.1, 0.11, and 0.22, so that the second average of the first click rate estimate information for the presented promotional resource comprised by tile 1 is 0.14. The second average of the first click rate estimation information of the displayed promotion resource included in the other blocks can be calculated as well, and specifically, the table 7 can be referred to.

And step 405, calculating ratio information of the click rate reality value information of the first block and a second average value corresponding to the first block to obtain second ratio information corresponding to the first block.

For example, the second ratio information corresponding to the partition 1 is equal to the ratio information of 0.33 to 0.14, that is, the second ratio information corresponding to the partition 1 is equal to 2.35, and the second ratio information corresponding to other first partitions can be obtained as well, which can be specifically shown in the above table 7.

Step 406, clustering the plurality of first blocks according to the second ratio information corresponding to each first block to obtain at least one clustered second block.

Step 407, determining a target piecewise function according to each piece of first estimation information.

drawing a target curve according to the plurality of coordinate information;

In the above embodiments, a manner of determining a plurality of coordinate information according to each first click rate estimation information is described, and another implementation manner of determining a plurality of coordinate information according to each first click rate estimation information is described herein, namely how to determine a plurality of coordinate information after performing steps 402 to 406 (i.e. partitioning the displayed popularization resource and clustering the partitions) is described herein, and referring to fig. 5, fig. 5 is a step flowchart of another method for determining coordinate information provided by an embodiment of the present invention.

The implementation method comprises the following steps:

step 501, dividing the first click rate estimation information included in all the first blocks corresponding to the second blocks according to a second preset division rule, so as to obtain a plurality of second estimation information intervals.

It should be noted that, the second preset dividing rule is: and dividing the first click rate estimated value information belonging to a certain interval unit in the first click rate estimated value information included in all the first blocks corresponding to the second blocks into a second estimated value information interval.

The step is similar to the step 301, and will not be described again, specifically, the following second estimation information interval divided in table 8 may be referred to:

TABLE 8

Step 502, calculating third ratio information of the clicked times and the exhibited times of the exhibited popularization resource corresponding to each second estimated value information interval, and obtaining second click rate real value information corresponding to each second estimated value information interval.

As can be seen from the above table 4, the number of times the displayed promotion resources (promotion resource 1, promotion resource 2, and promotion resource 4) corresponding to the interval a are clicked is 2, the number of times the displayed promotion resources are 3, and the third ratio information is equal to 2 divided by 3, that is, 0.67, so the second click rate authenticity value information corresponding to the interval a is equal to 0.67. The second click rate fidelity value information corresponding to interval B may also be calculated to be 0.165.

In this embodiment, only a small amount of data is used for the exemplary description, and the number of second click rate real value information corresponding to the determined second estimated value information section is relatively large because the second estimated value information section in the actual scene is relatively large.

Step 503, calculating a second average value corresponding to each second estimated value information interval, where the second average value is an average value of the first click rate estimated value information included in the second estimated value information interval.

For example, the average value of the first click rate estimation information (0.1, 0.11, 0.15) included in the section a is equal to 0.12, and the average value of the first click rate estimation information (0.22, 0.21) included in the section B is equal to 0.165. Specifically, refer to table 8 above.

And 504, forming coordinate information of a target curve corresponding to the second block by taking a second average value corresponding to the second estimated value information interval as a first abscissa value and second click rate true value information corresponding to the second estimated value information interval as a first ordinate value.

Step 408, determining a target piecewise function corresponding to each second piece according to the shape of the target curve corresponding to each second piece.

And 409, fitting the target piecewise function corresponding to each second piecewise function to obtain a target fitting function corresponding to each target piecewise function.

Step 410, obtaining second estimated value information of the operation probability of the popularization resource to be displayed currently.

Step 411, calibrating the second estimation information based on the objective fitting function to determine calibration information of the second estimation information.

Note that, the second estimation information in this embodiment is second click rate estimation information. The objective fitting function obtained in the embodiment is multiple, that is, multiple piecewise functions can be obtained, each piecewise function corresponds to multiple variable intervals, sub-functions of each piecewise function are more finely distinguished, each sub-function corresponds to a respective variable interval, so that the objective variable interval in which the second click rate estimated value information falls can be more accurately determined, the sub-function corresponding to the objective variable interval is used for calibrating the second click rate estimated value information, the problem that overfitting is very easy to cause by adopting a sequence preserving regression calibration scheme under the condition of sparse data can be avoided, the calibration information is more close to real value information, and the calibration effect is improved to a certain extent. In this embodiment, in order to improve accuracy of the calibration information of the estimated value information of the second click rate, the higher the estimated value information of the click rate is, the higher the ranking is for CPC advertisement. Therefore, the accuracy of the estimated click rate information has a great influence on the sequencing result, and the accuracy of the estimated click rate information directly influences the sequencing result of advertisements, thereby influencing the benefits of advertisers. According to the embodiment of the invention, the accuracy of the second click rate estimation information is improved to a certain extent, so that the reasonability of the ordering result obtained after the advertisements are ordered can be ensured to a certain extent, and the benefits of advertisers are ensured.

In addition, the embodiment of the invention also provides an electronic device, which comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program realizes the processes of the data calibration method embodiment of the promotion resource of the embodiment when being executed by the processor, and can achieve the same technical effects, and the repetition is avoided, so that the description is omitted.

The embodiment of the invention also provides a computer readable storage medium, on which a computer program is stored, which when executed by a processor, implements the processes of the data calibration method embodiment of the popularization resource, and can achieve the same technical effects, and in order to avoid repetition, the description is omitted here. The computer readable storage medium may be a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, an optical disk, or the like.

The embodiment of the invention also provides a computer program which can be stored on a cloud or local storage medium. The computer program, when being executed by a computer or processor, is adapted to carry out the respective steps of the method for data calibration of a promotional resource according to an embodiment of the present invention and to carry out the respective modules in the device for data calibration of a promotional resource according to an embodiment of the present invention.

In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described by differences from other embodiments, and identical and similar parts between the embodiments are all enough to be referred to each other.

As will be readily appreciated by those skilled in the art: any combination of the above embodiments is possible, and thus is an embodiment of the present invention, but the present specification is not limited by the text.

The data calibration methods of the promotional resources provided herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general-purpose systems may also be used with the teachings herein. The required structure for a system constructed with aspects of the present invention will be apparent from the description above. In addition, the present invention is not directed to any particular programming language. It will be appreciated that the teachings of the present invention described herein may be implemented in a variety of programming languages, and the above description of specific languages is provided for disclosure of enablement and best mode of the present invention.

In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it should be appreciated that in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be construed as reflecting the intention that: i.e., the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art will appreciate that the modules in the apparatus of the embodiments may be adaptively changed and disposed in one or more apparatuses different from the embodiments. The modules or units or components of the embodiments may be combined into one module or unit or component and, furthermore, they may be divided into a plurality of sub-modules or sub-units or sub-components. Any combination of all features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or units of any method or apparatus so disclosed, may be used in combination, except insofar as at least some of such features and/or processes or units are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings), may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features but not others included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.

Various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that some or all of the functions of some or all of the components in a data calibration method according to an embodiment of the invention may be implemented in practice using a microprocessor or Digital Signal Processor (DSP). The present invention can also be implemented as an apparatus or device program (e.g., a computer program and a computer program product) for performing a portion or all of the methods described herein. Such a program embodying the present invention may be stored on a computer readable medium, or may have the form of one or more signals. Such signals may be downloaded from an internet website, provided on a carrier signal, or provided in any other form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, third, etc. do not denote any order. These words may be interpreted as names.

Claims

1. A method for data calibration of a promotional resource, comprising:

obtaining first estimated value information of the operated probabilities of a plurality of popularization resources in a preset time period, wherein the first estimated value information comprises first click rate estimated value information or first conversion rate estimated value information, and the popularization resources comprise advertisement resources;

Determining a target segment function according to each piece of first estimation information, wherein the target segment function comprises the following steps: determining a plurality of coordinate information according to each piece of first click rate estimation information or first conversion rate estimation information; drawing a target curve according to the plurality of coordinate information; determining a target piecewise function according to the shape of the target curve;

the target piecewise function is used for describing the relation between first estimation information of the operated probability of the popularization resource and calibration information of the operated probability of the popularization resource;

fitting the target piecewise function to obtain a target fitting function;

obtaining second estimated value information of the operation probability of the popularization resource to be displayed currently, wherein the second estimated value information comprises second conversion rate estimated value information or second click rate estimated value information, the second estimated value information is obtained by inputting attribute information of the popularization resource to be displayed currently into a conversion rate estimated model or a click rate estimated model, and the attribute information at least comprises: one of advertisement conversion state, advertiser characteristics and user portraits, wherein the advertiser characteristics at least comprise one of originality and type;

calibrating the second estimation information based on the target fitting function to determine calibration information of the second estimation information;

The determining a plurality of coordinate information according to each of the first click rate estimation information or the first conversion rate estimation information includes:

dividing all the first click rate estimated value information or all the first conversion rate estimated value information according to a first preset dividing rule to obtain a plurality of first estimated value information intervals;

calculating first ratio information of the clicked times and the exhibited times of the displayed popularization resources corresponding to each first estimated value information interval to obtain first click rate real value information corresponding to each first estimated value information interval, or calculating first ratio information of the converted times and the clicked times of the popularization resources of the clicked operation corresponding to each first estimated value information interval to obtain first conversion rate real value information corresponding to each first estimated value information interval;

calculating a first average value corresponding to each first estimated value information interval, wherein the first average value is an average value of first click rate estimated value information or an average value of first conversion rate estimated value information included in the first estimated value information interval;

and forming coordinate information of the target curve by taking an average value of first conversion rate estimated value information corresponding to the first estimated value information interval as a first abscissa value, taking first conversion rate true value information corresponding to the first estimated value information interval as a first ordinate value, or taking an average value of first click rate estimated value information corresponding to the first estimated value information interval as a first abscissa and taking a first click rate true value corresponding to the first estimated value information interval as a first ordinate.

2. The method of claim 1, wherein the promotion resource is a promotion resource operated by a click, the promotion resource operated probability is a promotion resource converted probability, and the first estimation information is first conversion rate estimation information.

3. The method of claim 2, wherein said determining a plurality of coordinate information from each of said first conversion estimate information comprises:

calculating first ratio information of converted times and clicked times of popularization resources of clicked operations corresponding to each first estimated value information interval, and obtaining first conversion rate true value information corresponding to each first estimated value information interval;

and forming coordinate information of the target curve by taking a first average value corresponding to the first estimated value information interval as a first abscissa value and taking first conversion rate true value information corresponding to the first estimated value information interval as a first ordinate value.

4. A method according to claim 2 or 3, further comprising, after said obtaining the first estimation information of the probability that a plurality of promotion resources are operated in the preset time period:

dividing all promotion resources subjected to clicking operation into a plurality of first blocks according to main features corresponding to the conversion rate estimation model;

calculating first ratio information of converted times and clicked times of promotion resources of clicked operations included in each first block to obtain conversion rate real value information of the first block;

calculating a second average value of first conversion rate estimation information of the promotion resource of the clicked operation included in each first partition;

calculating ratio information of conversion rate actual value information of the first block and a second average value corresponding to the first block to obtain second ratio information corresponding to the first block;

clustering the first blocks according to the second ratio information corresponding to each first block to obtain at least one clustered second block;

wherein the determining a plurality of coordinate information according to each of the first conversion rate estimation information includes:

and determining coordinate information of a target curve corresponding to the second block according to the first conversion rate estimated value information included in all the first blocks corresponding to the second block.

5. The method of claim 4, wherein said determining a plurality of coordinate information based on each of said first conversion estimate information comprises:

dividing first conversion rate estimated value information included in all first blocks corresponding to the second blocks according to a second preset dividing rule to obtain a plurality of second estimated value information intervals;

calculating third ratio information of the converted times and the clicked times of the promotion resource of the clicked operation corresponding to each second estimated value information interval to obtain second conversion rate true value information corresponding to each second estimated value information interval;

calculating a second average value corresponding to each second estimated value information interval, wherein the second average value is an average value of first conversion rate estimated value information included in the second estimated value information interval;

and forming coordinate information of a target curve corresponding to the second block by taking a second average value corresponding to the second estimated value information interval as a first abscissa value and taking second conversion rate true value information corresponding to the second estimated value information interval as a first ordinate value.

6. The method of claim 5, wherein determining a target piecewise function based on the shape of the target curve comprises:

Determining a target piecewise function corresponding to each second piecewise block according to the shape of a target curve corresponding to each second piecewise block;

the fitting the target segment function to determine a parameter value of the target segment function, and obtaining a target fitting function comprises the following steps:

fitting the target segment function corresponding to each second segment to obtain a target fitting function corresponding to each target segment function.

7. A method according to claim 2 or 3, wherein the second estimation information is second conversion estimation information;

the calibrating the second estimation information based on the objective fitting function to determine calibration information of the second estimation information includes:

determining a target variable interval in which the second conversion rate estimation information falls from a plurality of variable interval values of the target fitting function;

and calculating the calibration information of the second conversion rate estimation information according to the second conversion rate estimation information and the subfunctions in the objective fitting function corresponding to the objective variable interval, wherein the calibration information is the calibration information after calibrating the second conversion rate estimation information of the current popularization resource to be displayed with the operation probability.

8. The method of claim 3, wherein the conversion rate estimation model is obtained by inputting attribute information of popularization resources in a popularization resource training set into a pre-constructed conversion rate model for training.

9. The method of claim 1, wherein the promotional resource is a presented promotional resource, the promotional resource operated probability is the promotional resource clicked probability, and the first valuation information is first click rate valuation information;

the determining the target segment function according to each piece of the first estimation information comprises the following steps:

determining a plurality of coordinate information according to each piece of first click rate estimation information;

drawing a target curve according to the plurality of coordinate information;

10. The method of claim 9, wherein said determining a plurality of coordinate information based on each of said first click rate estimation information comprises:

and forming coordinate information of the target curve by taking a first average value corresponding to the first estimated value information interval as a first abscissa value and taking first click rate true value information corresponding to the first estimated value information interval as a first ordinate value.

11. The method according to claim 9 or 10, further comprising, after the obtaining the first estimation information of the probability that the plurality of promotion resources are operated in the preset time period:

dividing all the displayed popularization resources into a plurality of first blocks according to main features corresponding to the click rate estimation model;

calculating first ratio information of the clicked times and the exhibited times of the exhibited popularization resources included in each first block to obtain click rate real value information of the first block;

calculating a second average value of first click rate estimation information of the displayed popularization resource included in each first partition;

calculating ratio information of click rate reality value information of the first block and a second average value corresponding to the first block to obtain second ratio information corresponding to the first block;

wherein the determining a plurality of coordinate information according to each of the first click rate estimation information includes:

and determining coordinate information of a target curve corresponding to the second block according to first click rate estimation information included in all the first blocks corresponding to the second block.

12. The method of claim 11, wherein said determining a plurality of coordinate information based on each of said first click rate estimation information comprises:

dividing first click rate estimated value information included in all first blocks corresponding to the second blocks according to a second preset dividing rule to obtain a plurality of second estimated value information intervals;

calculating third ratio information of the clicked times and the exhibited times of the exhibited popularization resources corresponding to each second estimated value information interval to obtain second click rate true value information corresponding to each second estimated value information interval;

calculating a second average value corresponding to each second estimated value information interval, wherein the second average value is an average value of first click rate estimated value information included in the second estimated value information interval;

And forming coordinate information of a target curve corresponding to the second block by taking a second average value corresponding to the second estimated value information interval as a first abscissa value and taking second click rate true value information corresponding to the second estimated value information interval as a first ordinate value.

13. The method of claim 12, wherein said determining a target piecewise function from the shape of the target curve comprises:

14. The method according to claim 9 or 10, wherein the second estimation information is second click rate estimation information;

and calculating the calibration information of the second click rate estimation information according to the second click rate estimation information and the subfunction in the objective fitting function corresponding to the objective variable interval, wherein the calibration information is the calibration information after calibrating the second click rate estimation information of the current to-be-displayed popularization resource operated probability.

15. The method of claim 10, wherein the click rate estimation model is obtained by inputting attribute information of a promotion resource in a promotion resource training set into a pre-constructed click rate model for training.

16. An electronic device comprising a processor, a memory and a computer program stored on the memory and running on the processor, which when executed by the processor performs the steps of the method of data calibration of promotional resources according to any of claims 1-15.

17. A computer readable storage medium, wherein a computer program is stored on the computer readable storage medium, which computer program, when executed by a processor, implements the method of data calibration of promotional resources according to any of claims 1 to 15.