CN106484688B

CN106484688B - Data processing method and system

Info

Publication number: CN106484688B
Application number: CN201510522784.9A
Authority: CN
Inventors: 黄艳香; 向宇; 徐钊; 张文郁
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2015-08-24
Filing date: 2015-08-24
Publication date: 2020-07-24
Anticipated expiration: 2035-08-24
Also published as: CN106484688A

Abstract

The data processing method and system provided by the invention configure the one-to-one correspondence relationship between the user identifier of the first data and the second data and the third data, use the first data as cache data, obtain the cache data corresponding to the user identifier by querying in the relationship according to the user identifier, query the third data corresponding to the operated second data in the cache data according to the operated second data, and use the third data corresponding to the clicked second data as real data for background use when the third data corresponding to the operated second data meets a preset condition, that is, only the real data is counted, thereby improving the accuracy of background data processing.

Description

Data processing method and system

Technical Field

The present invention relates to the field of mobile terminals, and in particular, to a data processing method and system.

Background

With the development of intelligent mobile products such as mobile phones, mobile terminal applications are being developed unprecedentedly, such as WeChat, mobile phone Tencent news, and the like. In these mobile-end applications, there is typically some "pre-pull" process to improve the user experience. For convenience of description, a user interface of the news terminal of the mobile phone is shown in combination with fig. 1, so that a user can smoothly read news of different channels when sliding left and right, when the user clicks a "video" channel, the news of channels such as "guangdong", "finance", "entertainment" and the like are loaded into the user client in advance, and a set of exposure data for the user is generated. The exposure data generated by such pre-pull processing is called pseudo exposure data because the user has not actually seen the news of channels such as "guangdong", "finance", "entertainment", etc. in the example, when counting the exposure and clicking data of the news, the pseudo exposure data can cause wrong counting results if not distinguished.

In order to solve the aforementioned problems, the general data statistics utilizes the way of offline storage and periodic calculation, firstly, user behavior data in a period of time is intensively transmitted offline to a distributed file system, then the offline data is periodically counted, in this case, all behaviors of a user are visible, pseudo exposure data and real exposure data can be distinguished through the behaviors of the user after pre-pulling, the news under the 'video' and 'entertainment' channels actually clicked by the user generates real exposure, and pre-pulling exposure data under the 'guangdong' and 'financial' channels which are not clicked by the user are discarded without participating in the statistics.

Because the offline data statistical scheme adopts a batch processing calculation mode, data is stored in a disk firstly and then is processed periodically, the calculation mode cannot generate real-time statistical data, great time delay is brought, and the current real-time requirement cannot be met.

Disclosure of Invention

In view of this, embodiments of the present invention provide a data processing method and system.

An object of the present invention is to provide a data processing method, including:

the method comprises the steps that first data generated when a user operates a client are obtained, wherein the user has a user identifier for identifying an identity, the first data comprises the user identifier, a plurality of second data corresponding to the user identifier and a plurality of third data, and the second data and the third data are in one-to-one correspondence;

configuring a one-to-one corresponding relation between the user identification of the first data and the second data and the third data, and taking the first data as cache data;

acquiring object operation data generated by the user operation client, wherein the object operation data comprises the user identifier and operated second data, and the operated object is one of the second data;

inquiring in the relation according to the user identification to obtain the cache data corresponding to the user identification;

inquiring third data corresponding to the operated second data in the cache data according to the operated second data;

and when the third data corresponding to the operated second data meets the preset condition, taking the third data corresponding to the clicked second data as real data for the background to use.

Further, the first data is original exposure data, the second data is an exposure channel, the third data is an exposure channel news ID list, the object operation data is channel click data, and the first data generated by a user operating a client is acquired, where the user has a user identifier for identifying an identity, the first data includes a user identifier, a plurality of second data corresponding to the user identifier, and a plurality of third data, and the second data corresponds to the third data one to one, including:

acquiring original exposure data generated by a user operating a client, wherein the user has a user ID for identifying an identity, the original exposure data comprises a user ID, a plurality of exposure channels corresponding to the user ID and a plurality of exposure channel news ID lists, and the exposure channels correspond to the exposure channel news ID lists one by one;

the configuring a one-to-one correspondence relationship between the user identifier of the first data and the second data and the third data, and using the first data as cache data includes:

storing the original exposure data into a KV storage system based on a memory as an exposure cache by taking the user ID as a key and taking an exposure channel and an exposure channel news ID list as a value;

acquiring object operation data generated by the user operating client, wherein the object operation data comprises the user identifier and operated second data, and the operated object is one of the second data, and the method comprises the following steps:

acquiring channel click data generated by the user operating client, wherein the channel click data comprises the user ID and a clicked exposure channel, and the clicked exposure channel is one of the exposure channels;

the obtaining of the cache data corresponding to the user identifier by querying the relationship according to the user identifier includes:

inquiring the exposure cache corresponding to the user ID in the KV storage system according to the user ID;

inquiring third data corresponding to the operated second data in the cache data according to the operated second data, wherein the inquiring includes:

inquiring an exposure channel news ID list corresponding to the clicked exposure channel in the exposure cache according to the clicked exposure channel;

when the third data corresponding to the operated second data meets the preset condition, taking the third data corresponding to the clicked second data as real data for the background to use, wherein the method comprises the following steps:

and when the exposure channel news ID list corresponding to the clicked exposure channel meets the preset condition, taking the exposure channel news ID list corresponding to the clicked exposure channel as real exposure data for the background to use.

Further, the exposure channel news ID list includes an exposure time and a valid time,

before the step of using the exposure channel news ID list corresponding to the clicked exposure channel as real exposure data for the background to use when the exposure channel news ID list corresponding to the clicked exposure channel meets the preset condition, the method further comprises the following steps:

acquiring the click action occurrence time of the clicked exposure channel;

calculating a time difference value between the exposure time of the clicked exposure channel news ID list and the click behavior occurrence time;

the news ID list of the exposure channel corresponding to the clicked exposure channel meets the preset conditions, and the method comprises the following steps:

and if the time difference is not more than the effective time, the exposure channel news ID list corresponding to the clicked exposure channel meets the preset condition.

Further, after querying an exposure channel news ID list corresponding to the clicked exposure channel in the exposure cache according to the clicked exposure channel, the method further includes:

and when the exposure channel news ID list corresponding to the clicked exposure channel does not meet the preset condition, discarding the exposure channel news ID list corresponding to the clicked exposure channel as pseudo exposure data.

Further, before querying the exposure cache corresponding to the user ID in the KV storage system according to the user ID, the method further includes:

obtaining news click data generated by the user operation client, wherein the news click data comprises the user ID, an exposure channel where the news is located and a news ID, and the exposure channel where the news is located is one of the exposure channels;

after querying the exposure cache corresponding to the user ID in the KV storage system according to the user ID, the method further includes:

inquiring an exposure channel news ID list corresponding to the exposure channel of the news in the exposure cache according to the exposure channel of the news;

and acquiring the position information corresponding to the news ID in an exposure channel news ID list corresponding to the exposure channel where the news is located according to the news ID, so that the position information can be used by a background.

Further, the effective time is any time from 1 minute to 10 minutes.

It is another object of the present invention to provide a data processing system comprising:

the system comprises a first acquisition unit, a second acquisition unit and a third acquisition unit, wherein the first acquisition unit is used for acquiring first data generated by a user operating a client, the user has a user identifier for identifying an identity, the first data comprises the user identifier, a plurality of second data corresponding to the user identifier and a plurality of third data, and the second data and the third data are in one-to-one correspondence;

a storage unit, configured to configure a one-to-one correspondence relationship between the user identifier of the first data and the second data and the third data, and use the first data as cache data;

a second obtaining unit, configured to obtain object operation data generated by operating a client by the user, where the object operation data includes the user identifier and operated second data, and the operated object is one of the second data;

the first query unit is used for querying the relation according to the user identifier to obtain the cache data corresponding to the user identifier;

the second query unit is used for querying third data corresponding to the operated second data in the cache data according to the operated second data;

and the first processing unit is used for taking the third data corresponding to the clicked second data as real data for background use when the third data corresponding to the operated second data meets a preset condition.

The system comprises a first acquisition unit, a second acquisition unit and a third acquisition unit, wherein the first acquisition unit is used for acquiring original exposure data generated by a user operating a client, the user has a user ID for identifying an identity, the original exposure data comprises a user ID, a plurality of exposure channels corresponding to the user ID and a plurality of exposure channel news ID lists, and the exposure channels correspond to the exposure channel news ID lists one by one;

the storage unit is used for storing the original exposure data into a KV storage system based on a memory by taking the user ID as a key and taking an exposure channel and exposure channel news ID list as a value as an exposure cache;

a second obtaining unit, configured to obtain channel click data generated by operating a client by the user, where the channel click data includes the first user ID and a clicked exposure channel, and the clicked exposure channel is one of the multiple exposure channels;

the first query unit is used for querying the exposure cache corresponding to the user ID in the KV storage system according to the user ID;

the second query unit is used for querying an exposure channel news ID list corresponding to the clicked exposure channel in the exposure cache according to the clicked exposure channel;

and the first processing unit is used for taking the exposure channel news ID list corresponding to the clicked exposure channel as real exposure data for background use when the exposure channel news ID list corresponding to the clicked exposure channel meets the preset condition.

Further, the news ID list of the exposure channel includes an exposure time and a valid time, and the data processing system further includes:

the third acquisition unit is used for acquiring the click behavior occurrence time of the clicked exposure channel;

a calculating unit, configured to calculate a time difference between an exposure time of the clicked exposure channel news ID list and an occurrence time of the click behavior;

and the first processing unit is further used for enabling an exposure channel news ID list corresponding to the clicked exposure channel to meet a preset condition when the time difference is not larger than the effective time.

Further, the data processing system further comprises:

and the second processing unit is used for discarding the exposure channel news ID list corresponding to the clicked exposure channel as pseudo exposure data when the exposure channel news ID list corresponding to the clicked exposure channel does not meet the preset condition.

Further, the data processing system further comprises:

a fourth obtaining unit, configured to obtain news click data generated by the user operating client, where the news click data includes the user ID, an exposure channel where the news is located, and a news ID, where the exposure channel where the news is located is one of the multiple exposure channels;

the third query unit is used for querying an exposure channel news ID list corresponding to the exposure channel of the news in the exposure cache according to the exposure channel of the news;

and the fourth query unit is used for acquiring the position information corresponding to the news ID from the news ID list of the exposure channel corresponding to the exposure channel where the news is located according to the news ID, so that the position information can be used by a background.

According to the technical scheme, the embodiment of the invention has the following advantages:

Drawings

FIG. 1 is a schematic diagram of a news terminal of a prior art cell phone;

FIG. 2a is a flow chart of one embodiment of a data processing method provided by the present invention;

FIG. 2b is a flow chart of another embodiment of a data processing method provided by the present invention;

FIG. 3 is a flow chart of another embodiment of a data processing method provided by the present invention;

FIG. 4 is a flow chart of another embodiment of a data processing method provided by the present invention;

FIG. 5 is a block diagram of one embodiment of a data processing system provided by the present invention.

Detailed Description

In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Referring to fig. 2a, an embodiment of the data processing method provided by the present invention includes:

s1, first data generated by a user operating a client is obtained, wherein the user has a user identification for identifying identity, the first data comprises the user identification, a plurality of second data corresponding to the user identification and a plurality of third data, and the second data and the third data are in one-to-one correspondence.

In this embodiment, the first data may be original exposure data, the second data may be an exposure channel, the third data may be an exposure channel news ID list, the object operation data may be channel click data, when a user browses web news using a client, the client performs a pre-pull operation, a news ID list of an adjacent channel is prepared for the client in advance, the part of data serves as original exposure data, each user may have a user ID for identifying an identity of the user, each user ID may generate original exposure data, each user ID may correspond to several exposure channels, and each exposure channel corresponds to one exposure news ID list.

S2, configuring the one-to-one correspondence relationship between the user identification of the first data and the second data and the third data, and taking the first data as cache data.

The relationship mentioned here is to establish a relationship for querying by using a user identifier, and may be performed by using a KV storage system, and for KV storage, a storage system using key-value as a format may be queried by a key, and it should be understood by a person skilled in the art that, therefore, the meaning of KV storage is not specifically described, and a user ID is used as a key, that is, in a key value list, there may be a plurality of user IDs, not only limited to a first user ID, but also when constructing a KV storage system, the first user ID is used as a key, then an exposure channel and exposure channel news ID list corresponding to pre-pull is operated by the first user ID is used as a value, and the exposure channel and exposure channel news ID list correspond one-to-one, that is, the exposure channel and exposure channel news ID list corresponding to pre-pull may be queried by the first user ID.

S3, obtaining object operation data generated by the user operating the client, where the object operation data includes the user identifier and operated second data, and the operated object is one of the second data.

The operated second data may be a clicked exposure channel, that is, a channel actually operated by a user, and after the KV storage system is established, when channel click data generated by the first user ID operating client is received, that is, when the first user ID clicks a certain channel, channel click data may be generated, which indicates that the user of the first user ID is browsing news under the channel, so that a news ID list pre-pulled for the channel may be considered to be seen by the user and may be used as real exposure data, and the channel click data includes the first user ID and a clicked exposure channel, where the clicked channel is one of the multiple exposure channels, and the clicked exposure channel is a channel currently browsed by the user of the first user ID.

S4, according to the user identification, inquiring in the relationship to obtain the cache data corresponding to the user identification.

Because the user identifier and the cache data are in one-to-one correspondence in the relationship, the exposure cache corresponding to the first user ID in the database, namely the exposure channel and the exposure channel news ID list pre-pulled by the first user ID, can be obtained by querying the corresponding cache data according to the user identifier and by using the characteristic of the KV storage system and according to the key of the first user ID.

And S5, inquiring third data corresponding to the operated second data in the cache data according to the operated second data.

After the exposure channel and the exposure channel news ID list corresponding to the first user ID are obtained, the information in the exposure cache is inquired by using the clicked exposure channel, and the exposure channel news ID list corresponding to the clicked exposure channel can be obtained.

And S6, when the third data corresponding to the operated second data meet the preset condition, using the third data corresponding to the clicked second data as real data for the background to use.

The validity of the obtained third data is verified, the third data is used by a background when a preset condition is met, when the third data is an exposure channel news list, the generation time of the exposure channel news ID list can be recorded, the effective time is set, whether the time interval is not more than the effective time is calculated based on the time of receiving the clicking action of the user on the channel, if not, the obtained data are limited, the exposure channel news ID list corresponding to the clicked exposure channel can be used as real exposure data for the background, namely, the user really browses the information of the channel, the background can count the real exposure data and serve as a reference for news recommendation, and the news recommendation accuracy is improved.

The data processing method provided by the invention has the advantages that KV storage is carried out on original exposure data generated by pre-drawing, then the user ID in channel click data is utilized to query in a KV-stored database to obtain an exposure cache, a clicked channel is utilized to query an exposure channel news ID list in the exposure cache, the exposure channel news ID list obtained by querying the clicked channel is used as real exposure data for a background to use, so that only the exposure channel news ID list clicked by the channel in the original exposure data is counted when the background is used for counting, namely only the real exposure data is counted, and the original exposure data is screened by utilizing KV storage and preset conditions, so that the accuracy of background data processing is improved.

Referring to fig. 2b, the data processing method according to the present invention further provides an embodiment, where the first data is original exposure data, the second data is an exposure channel, the third data is an exposure channel news ID list, and the object operation data is channel click data, and the method includes:

s101, original exposure data generated when a user operates a client are obtained, wherein the user has a user ID for identifying identity, the original exposure data comprises the user ID, a plurality of exposure channels corresponding to the user ID and a plurality of exposure channel news ID lists, and the exposure channels correspond to the exposure channel news ID lists one by one.

When a user browses webpage news by using a client, the client performs pre-pull operation, a news ID list of adjacent channels is prepared for the client in advance, the data is used as original exposure data, each user can have a user ID for identifying the identity of the user, each user ID can generate original exposure data, each user ID can correspond to a plurality of exposure channels, each exposure channel corresponds to an exposure channel news ID list, each exposure channel news ID list is determined when the original exposure data are generated, the original data can comprise exposure channels and exposure channel news ID lists corresponding to a plurality of user IDs, different user IDs can generate corresponding original exposure data when the operation is performed, for example, a user of the user ID is browsing ' military channels ', the client can browse news ID lists of ' social channels ' adjacent to ' military channels ' and news IDs of ' political channels ' adjacent to ' military channels The list is pre-extracted as the original exposure data of the user ID, and when a user of another user ID browses a "social channel", the client pre-extracts a news ID list of a "military channel" adjacent to the "social channel" and a news ID list of an "entertainment channel" as the original exposure data corresponding to the user ID.

S102, storing the original exposure data into a KV memory system based on a memory by taking the user ID as a key and taking an exposure channel and exposure channel news ID list as a value as an exposure cache.

For the KV storage, a storage system using key-value as a format can query a corresponding value through a key, and it should be understood by those skilled in the art that, the meaning of KV storage is not specifically described, a user ID is used as a key, that is, in a key value list, there may be a plurality of user IDs, not only limited to a first user ID, when the KV storage system is constructed, the first user ID is used as a key, an exposure channel and exposure channel news ID list corresponding to pre-pull operation of the first user ID is used as a value, the exposure channel and exposure news ID lists correspond one-to-one, that is, the exposure channel and exposure channel news ID list corresponding to pre-pull can be queried by using the first user ID, and similarly, a second user ID can be used without repeated description, and the original data is stored in a database of the KV storage system in this way, and the original exposure data is referred to as an exposure cache after being stored in the KV system, and will not be described in detail hereinafter.

S103, obtaining channel click data generated by operating the client by using the first user ID, wherein the channel click data comprises the first user ID and a clicked exposure channel, and the clicked exposure channel is one of the exposure channels.

After the KV storage system is established, when channel click data generated by operating the client with the first user ID is received, that is, when the first user ID clicks a certain channel, the channel click data may be generated, which indicates that the user with the first user ID is browsing news under the channel, so that a news ID list pre-pulled for the channel may be considered as what the user sees and may be used as real exposure data, and the channel click data includes the first user ID and a clicked exposure channel, where the clicked channel is one of the multiple exposure channels, the clicked exposure channel is a channel currently browsed by the user with the first user ID, for example, when the user with the current first user ID browses a "military channel", the clicked exposure channel is a "military channel", and the clicked exposure channel is a "political channel" of the multiple exposure channels pre-pulled before, One of a "social channel" and a "military channel".

S104, inquiring the exposure cache corresponding to the first user ID in the KV storage system according to the first user ID.

By using the characteristics of the KV storage system, the exposure cache corresponding to the first user ID in the database, that is, the exposure channel and exposure channel news ID list pre-pulled by the first user ID, such as the news ID list under "military channel" and "military channel", the news ID list under "hour administration channel" and "hour administration channel", and the news ID list under "social channel" and "social channel" mentioned above, can be obtained by querying according to the key of the first user ID, and the channels that the user has browsed are determined from the pre-pulled exposure channel and exposure news ID list corresponding to the first user ID, that is, the real exposure data is determined.

S105, inquiring an exposure channel news ID list corresponding to the clicked exposure channel in the exposure cache according to the clicked exposure channel.

After the exposure channel and the exposure channel news ID list corresponding to the first user ID are obtained, the clicked exposure channel is used for inquiring information in the exposure cache, and the exposure channel news ID list corresponding to the clicked exposure channel can be obtained.

And S106, when the exposure channel news ID list corresponding to the clicked exposure channel meets the preset condition, taking the exposure channel news ID list corresponding to the clicked exposure channel as real exposure data for a background to use.

After the exposure channel news ID list corresponding to the clicked exposure channel is obtained, validity verification is needed, for example, time limitation is conducted on exposure cache data, exposure cache exceeding a certain time is discarded to save storage space, specifically, the generation time of the exposure channel news ID list can be recorded, the valid time is set, whether the time interval is not more than the valid time or not is calculated according to the time of receiving the clicking action of the user on the channel, if the time interval is not more than the valid time, the obtained data can be determined to be limited, the exposure channel news ID list corresponding to the clicked exposure channel can be used as real exposure data for a background, the user really browses information of the channel, the background use can count the real exposure data and serve as reference for news recommendation, and the accuracy of news recommendation is improved.

Referring to fig. 3, the data processing method according to another embodiment of the present invention includes:

s201, original exposure data generated by a user operating a client is obtained, wherein the user has a user ID for identifying an identity, the original exposure data comprises a user ID, a plurality of exposure channels corresponding to the user ID and a plurality of exposure channel news ID lists, the exposure channels correspond to the exposure channel news ID lists one by one, and the user ID at least comprises a first user ID.

Step 201 is similar to step S101 in the previous embodiment, and is not described herein.

S202, storing the original exposure data into a KV memory system based on a memory by taking the user ID as a key and taking an exposure channel and exposure channel news ID list as a value as an exposure cache.

The exposure channel news ID list comprises exposure time and effective time, the exposure time and the effective time are stored in the KV storage system, the exposure time of the exposure channel news ID list, namely the generation time of the exposure channel news ID list, can be obtained after the exposure channel news ID list is inquired, the effective time is used for verifying the inquired information, data exceeding the effective time are discarded, space is saved, and the accuracy of data statistics is improved, the effective time can be flexibly set, for example, any time from 1 minute to 10 minutes can be set, the effective time can be specifically selected according to needs, and the limitation is not performed.

S203, obtaining channel click data generated by operating the client with the first user ID, wherein the channel click data comprises the first user ID and a clicked exposure channel, and the clicked exposure channel is one of the exposure channels.

Step 203 is similar to step S103 in the previous embodiment, and is not described herein.

S204, inquiring the exposure cache corresponding to the first user ID in the KV storage system according to the first user ID.

Step 204 is similar to step S104 in the previous embodiment, and is not described herein.

S205, inquiring an exposure channel news ID list corresponding to the clicked exposure channel in the exposure cache according to the clicked exposure channel.

Step 205 is similar to step S105 in the previous embodiment, and is not described herein.

And S206, acquiring the click action occurrence time of the clicked exposure channel.

When a user clicks a channel, recording the current time, putting the click occurrence time into the channel click data, and analyzing the click occurrence time by receiving the channel click data to be used as a reference point for judging whether the exposure time exceeds the effective time.

S207, calculating a time difference value between the exposure time of the clicked exposure channel news ID list and the click behavior occurrence time.

The exposure time is generated when the exposure channel news ID list is pre-drawn and used for prompting the generation time of the exposure channel news ID list, the duration of the exposure channel news ID list can be calculated by utilizing the occurrence time of clicking behaviors, namely the time difference, and whether the exposure channel news ID list can be used as real exposure data or not can be determined by comparing the time difference with the effective time.

And S208, when the time difference is not greater than the effective time, performing an exposure channel news ID list corresponding to the clicked exposure channel, if so, performing S109, and if not, performing S110.

If the time difference is not greater than the effective time, the exposure channel news ID list corresponding to the clicked exposure channel meets the preset condition, and if the time difference is greater than the effective time, the exposure channel news ID list corresponding to the clicked exposure channel does not meet the preset condition.

S209, taking the exposure channel news ID list corresponding to the clicked exposure channel as real exposure data for background use.

Step 209 is similar to step S106 in the previous embodiment, and is not described herein.

S210, discarding the exposure channel news ID list corresponding to the clicked exposure channel as pseudo exposure data.

The background needs to perform operations such as statistics on the real exposure data, so that the dummy exposure data needs to be eliminated, and the dummy exposure data can be discarded in order to save the space of the KV storage system, that is, the dummy exposure data is deleted to save the space.

By providing real-time accurate effective exposure data, more accurate news data analysis is obtained, and statistical errors caused by false exposure data in the original exposure data are reduced.

In order to better provide accurate recommendation data, the data processing method of the present invention further provides an embodiment of the data processing method for the user to operate on specific news, which is described below with reference to fig. 4.

S301, original exposure data generated by a user operating a client is obtained, wherein the user has a user ID for identifying an identity, the original exposure data comprises a user ID, a plurality of exposure channels corresponding to the user ID and a plurality of exposure channel news ID lists, the exposure channels correspond to the exposure channel news ID lists one by one, and the user ID at least comprises a first user ID.

Step S301 is similar to step S201, and is not described in detail here.

S302, storing the original exposure data into a KV memory system based on a memory by taking the user ID as a key and taking an exposure channel and exposure channel news ID list as a value as an exposure cache.

Step S302 is similar to step S202, and is not described herein.

S303, obtaining channel click data generated by operating the client by using the first user ID, wherein the channel click data comprises the first user ID and a clicked exposure channel, and the clicked exposure channel is one of the exposure channels.

Step S303 is similar to step S203, and is not described in detail here.

S304, obtaining news click data generated by operating the client by using the first user ID, wherein the news click data comprises the first user ID, an exposure channel where the news is located and a news ID, and the exposure channel where the news is located is one of the exposure channels.

The method comprises the steps that clicking behaviors of users are greatly related to positions of news, the positions of the news are different in attraction to different users, therefore, the positions of the news when the users click the news are obtained, accurate recommendation or data analysis is very important, after news clicking data of the users with the first user ID are received, KV storage based on a memory is inquired according to the first user ID to obtain an exposure cache of the first user ID, an exposure channel news ID list of a channel where the news is newly found in the exposure cache is extracted according to the channel where the news is found, and because news IDs in the exposure channel news ID list are arranged in sequence, the positions of the clicked news IDs in the list can be inquired, namely, the recommended positions of the news when the news is exposed to the users.

S305, inquiring the exposure cache corresponding to the first user ID in the KV storage system according to the first user ID.

Step S305 is similar to step S204, and is not described in detail here.

S306, inquiring an exposure channel news ID list corresponding to the clicked exposure channel in the exposure cache according to the clicked exposure channel.

Step S306 is similar to step S205, and is not described in detail here.

S307 acquires the click behavior occurrence time of the clicked exposure channel.

Step S307 is similar to step S206 and will not be described herein.

S308 calculates a time difference between the exposure time of the clicked exposure channel news ID list and the click behavior occurrence time.

Step S308 is similar to step S207, and is not described in detail here.

S309, when the time difference is not larger than the effective time, performing an exposure channel news ID list corresponding to the clicked exposure channel, if so, performing S310, and if not, performing S313.

S310, taking the exposure channel news ID list corresponding to the clicked exposure channel as real exposure data for background use.

Step 301 is similar to step S209 in the previous embodiment, and is not described herein.

S311, inquiring an exposure channel news ID list corresponding to the exposure channel of the news in the exposure cache according to the exposure channel of the news.

Inquiring KV storage based on the memory according to the first user ID to obtain an exposure cache of the first user ID, and extracting an exposure channel news ID list of a channel where news is located in the exposure cache according to the channel where the news is located.

S312, according to the news ID, obtaining the position information corresponding to the news ID in the exposure channel news ID list corresponding to the exposure channel where the news is located, so that the position information can be used by a background.

The method comprises the steps of extracting an exposure channel news ID list of a channel where news is located in an exposure cache according to the channel where the news is located, wherein news IDs in the exposure channel news ID list are arranged in sequence, so that the position of the clicked news ID in the list, namely the recommended position of the news when the news is exposed to a user, can be inquired, the position of the clicked news is obtained Through processing news clicking behaviors of the user, and fine-grained CTR (Chinese: Click Through Rate, English: Click Through Rate) data analysis conditions at the recommended position level are provided.

S313, discarding the exposure channel news ID list corresponding to the clicked exposure channel as pseudo exposure data.

The actual news recommending service verifies that the position of the clicked news is obtained by processing the user news clicking behavior, the attractiveness of different recommending positions is distinguished, the user interest is more accurately understood, and the recommending accuracy of a news recommending algorithm is effectively improved.

In order to facilitate understanding of the data processing method of the present application, an application scenario is provided below to facilitate understanding.

The method includes that a user browses webpage news on a client, a channel currently browsed is a channel B, a channel A and a channel C are adjacent to the channel B, a user ID is a first user ID, in order to facilitate the user to quickly view a news ID list in the channel A or the channel C when the user slides left and right, the channel list of the channel A and the news ID list of the channel C need to be pre-pulled, and at the moment, the pre-pulling time is exposure time, and is determined as follows: when 10 hours 00 minutes, assuming that the news ID list of the a channel includes a1, a2, a3, the news ID list of the B channel is B1, B2, B3, the news ID list of the C channel is C1, C2, C3, the news ID list of the a channel, the news ID list of the B channel, and the news ID list of the C channel are used as raw exposure data, the raw exposure data is stored in KV storage system as exposure buffers with the first user ID being key, the news ID list of the a channel and the a channel, the news ID list of the B channel, the news ID list of the C channel and the exposure time being valid, when the a channel is operated by the first user ID, channel click data of the a channel is generated, the click channel data includes the first user ID, the clicked a channel, and the occurrence time (defined as 10 hours 05 minutes) and the effective time (defined as 10 minutes), the exposure cache corresponding to the first user ID is inquired in the KV storage system by using the first user ID, then, a clicked A channel is used for inquiring news ID lists a1, a2 and a3 of a channel corresponding to the exposure cache, the time difference is calculated to be 5 minutes according to the click occurrence time and the exposure time, and the time difference is not more than the effective time, so that the news ID lists of the channel A and the channel A in the original exposure data are real exposure data and can be used for background statistics, and the time when the user of the first user ID clicks the channel C is later than 10 minutes (the time difference is more than 10 minutes and exceeds the effective time) or the channel C is not clicked, the news ID list of the channel C can be deleted as pseudo exposure data to save space.

In the foregoing, a data processing method is introduced, and as shown in fig. 5, correspondingly, an embodiment of the data processing system is further provided, where the data processing method includes:

a first obtaining unit 401, configured to obtain first data generated by a user operating a client, where the user has a user identifier for identifying an identity, the first data includes the user identifier, a plurality of second data corresponding to the user identifier, and a plurality of third data, and the second data corresponds to the third data one to one;

a storage unit 402, configured to configure a one-to-one correspondence relationship between the user identifier of the first data and the second data and the third data, and use the first data as cache data;

a second obtaining unit 403, configured to obtain object operation data generated by operating a client by the user, where the object operation data includes the user identifier and operated second data, and the operated object is one of the second data;

a first querying unit 404, configured to query the relationship according to the user identifier to obtain the cache data corresponding to the user identifier;

a second querying unit 405, configured to query, according to the operated second data, third data corresponding to the operated second data in the cache data;

and the first data processing unit 406 is configured to use, when third data corresponding to the operated second data meets a preset condition, the third data corresponding to the clicked second data as real data for a background.

Optionally, the first obtaining unit 401 is configured to obtain original exposure data generated by a user operating a client, where the user has a user ID used for identifying an identity, the original exposure data includes a user ID, a plurality of exposure channels corresponding to the user ID, and a plurality of exposure channel news ID lists, the exposure channels correspond to the exposure channel news ID lists one by one, and the user ID at least includes a first user ID.

And a storage unit 402, configured to store the original exposure data in the memory-based KV storage system as an exposure cache with the user ID as a key and the exposure channel and exposure channel news ID list as a value.

A second obtaining unit 403, configured to obtain channel click data generated by operating a client with a first user ID, where the channel click data includes the first user ID and a clicked exposure channel, and the clicked channel is one of the exposure channels.

A first querying unit 404, configured to query, in the KV storage system according to the first user ID, the exposure cache corresponding to the first user ID.

A second querying unit 405, configured to query, according to the clicked exposure channel, an exposure channel news ID list corresponding to the clicked exposure channel in the exposure cache.

The first data processing unit 406 is configured to use the exposure channel news ID list corresponding to the clicked exposure channel as real exposure data for the background to use when the exposure channel news ID list corresponding to the clicked exposure channel meets a preset condition.

According to the data processing system, KV storage is carried out on original exposure data generated by pre-pulling, then an exposure cache is obtained by inquiring a user ID in channel click data in a KV-stored database, an exposure channel news ID list is inquired in the exposure cache by using a clicked channel, the exposure channel news ID list obtained by inquiring the clicked channel is used as real exposure data for a background to use, so that only the exposure channel news ID list clicked by the channel in the original exposure data is counted when the background is used for counting, namely only the real exposure data is counted, and the original exposure data is screened by using KV storage and preset conditions, and the accuracy of background data processing is improved.

a third obtaining unit 407, configured to obtain a click behavior occurrence time of a clicked exposure channel;

a calculating unit 408, configured to calculate a time difference between an exposure time of the clicked exposure channel news ID list and an occurrence time of the click behavior;

and the first data processing unit is also used for enabling an exposure channel news ID list corresponding to the clicked exposure channel to meet a preset condition when the time difference value is not greater than the effective time.

Further, the data processing system further comprises:

a second data processing unit 409, configured to discard the exposure channel news ID list corresponding to the clicked exposure channel as pseudo exposure data when the exposure channel news ID list corresponding to the clicked exposure channel does not meet a preset condition.

Further, the data processing system further comprises:

a fourth obtaining unit 410, configured to obtain news click data generated by operating a client with a first user ID, where the news click data includes the first user ID, an exposure channel where the news is located, and a news ID, and the exposure channel where the news is located is one of the multiple exposure channels;

a third querying unit 411, configured to query, according to the exposure channel where the news is located, an exposure channel news ID list corresponding to the exposure channel where the news is located in the exposure cache;

a fourth querying unit 412, configured to obtain, according to the news ID, location information corresponding to the news ID from an exposure channel news ID list corresponding to an exposure channel where the news is located, so that the location information can be used by a background.

The data processing system mentioned above, there is also provided a terminal as a carrier for the data processing system.

A terminal comprising a data processing system as described above, but of course the terminal also needs to comprise the necessary hardware structure, as will be described in more detail below.

The terminal may be any terminal device including a mobile phone, a tablet computer, a PDA (Personal Digital Assistant), a vehicle-mounted computer, etc., taking the terminal as the mobile phone as an example:

the mobile phone comprises: radio Frequency (RF) circuit, memory, input unit, touch display screen, sensor, audio circuit, wireless fidelity (WiFi) module, processor, and power supply.

The following describes the components of the mobile phone in detail:

in General, the RF circuit includes, but is not limited to, an antenna, at least one Amplifier, a transceiver, a coupler, a low Noise Amplifier (L ow Noise Amplifier, L NA), a duplexer, etc. in addition, the RF circuit can also communicate with a network and other devices through wireless communication.

The memory can be used for storing software programs and modules, and the processor executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory. The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the cellular phone, and the like. Further, the memory may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.

The input unit may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the cellular phone. Specifically, the input unit may include a touch panel and other input devices. The touch panel, also called a touch screen, may collect touch operations of a user (for example, operations of the user on or near the touch panel using any suitable object or accessory such as a finger, a stylus, etc.) and drive the corresponding connection device according to a preset program. Alternatively, the touch panel may include two parts, a touch detection device and a touch controller. The touch detection device detects the touch direction of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts the touch information into touch point coordinates, sends the touch point coordinates to the processor, and can receive and execute commands sent by the processor. In addition, the touch panel may be implemented in various types such as a resistive type, a capacitive type, an infrared ray, and a surface acoustic wave. The input unit may include other input devices in addition to the touch panel. In particular, other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

The touch screen may include a display panel, and optionally, the display panel may be configured in the form of a liquid crystal display (e.g., L essential crystalline display, abbreviated as L CD), an Organic light Emitting Diode (e.g., Organic L light-Emitting Diode, abbreviated as O L ED), etc. further, the touch panel may cover the display panel, and when the touch panel detects a touch operation on or near the touch panel, the touch panel may be transmitted to the processor to determine the type of the touch event, and then the processor may provide a corresponding visual output on the display panel according to the type of the touch event.

The handset may also include at least one sensor, such as a light sensor, motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor that adjusts the brightness of the display panel according to the brightness of ambient light, and a proximity sensor that turns off the display panel and/or the backlight when the mobile phone is moved to the ear. As one of the motion sensors, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally, three axes), can detect the magnitude and direction of gravity when stationary, and can be used for applications of recognizing the posture of a mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer and tapping), and the like; as for other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, and an infrared sensor, which can be configured on the mobile phone, further description is omitted here.

The audio circuit can transmit the electric signal converted from the received audio data to the loudspeaker, and the electric signal is converted into a sound signal by the loudspeaker to be output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit and converted into audio data, which is then output to the processor for processing, and then transmitted to, for example, another mobile phone via the RF circuit, or the audio data is output to the memory for further processing.

WiFi belongs to short-distance wireless transmission technology, and the mobile phone can help a user to receive and send e-mails, browse webpages, access streaming media and the like through a WiFi module, and provides wireless broadband internet access for the user.

The processor is a control center of the mobile phone, is connected with each part of the whole mobile phone by various interfaces and lines, and executes various functions and processes data of the mobile phone by running or executing software programs and/or modules stored in the memory and calling the data stored in the memory, thereby carrying out the integral monitoring on the mobile phone. Alternatively, the processor may include one or more processing units; preferably, the processor may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor.

The mobile phone further includes a power supply (such as a battery) for supplying power to each component, and preferably, the power supply may be logically connected to the processor through a power management system, so that functions of managing charging, discharging, and power consumption are implemented through the power management system.

Although not shown, the mobile phone may further include a camera, a bluetooth module, etc., which are not described herein.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by associated hardware instructed by a program, which may be stored in a computer-readable storage medium, and the storage medium may include: a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic or optical disk, or the like.

It will be understood by those skilled in the art that all or part of the steps in the method for implementing the above embodiments may be implemented by hardware that is instructed to implement by a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

While the data processing method and the related apparatus provided by the present invention have been described in detail, those skilled in the art will appreciate that the present invention is not limited to the above embodiments, and that the present invention is not limited to the above embodiments.

Claims

1. A data processing method, comprising:

the method comprises the steps that first data generated when a user operates a client are obtained, wherein the user has a user identifier for identifying an identity, the first data comprises the user identifier, a plurality of second data corresponding to the user identifier and a plurality of third data, and the second data and the third data are in one-to-one correspondence; the first data is original exposure data, the second data is an exposure channel, and the third data is an exposure channel news ID list;

when third data corresponding to the operated second data meet a preset condition, the third data corresponding to the operated second data are used as real data for a background to use; the preset conditions include: the duration of the third data is not greater than the validity time of the third data.

2. The method according to claim 1, wherein the object operation data is channel click data, and the obtaining of first data generated by a user operating a client, wherein the user has a user identifier for identifying an identity, the first data includes a user identifier, a plurality of second data corresponding to the user identifier, and a plurality of third data, and the second data corresponds to the third data one to one, and includes:

when the third data corresponding to the operated second data meets the preset condition, taking the third data corresponding to the operated second data as real data for the background to use, wherein the method comprises the following steps:

3. The data processing method of claim 2, wherein the exposure channel news ID list includes an exposure time and a valid time,

acquiring the click action occurrence time of the clicked exposure channel;

4. The data processing method according to claim 2, after querying an exposure channel news ID list corresponding to the clicked exposure channel in the exposure cache according to the clicked exposure channel, further comprising:

5. The data processing method according to claim 2, wherein before querying the exposure cache corresponding to the user ID in the KV storage system according to the user ID, the method further comprises:

after the user inquires the exposure cache corresponding to the user ID in the KV storage system, the method further includes:

6. The data processing method of claim 3, wherein the effective time is any time from 1 minute to 10 minutes.

7. A data processing system, comprising:

the system comprises a first acquisition unit, a second acquisition unit and a third acquisition unit, wherein the first acquisition unit is used for acquiring first data generated by a user operating a client, the user has a user identifier for identifying an identity, the first data comprises the user identifier, a plurality of second data corresponding to the user identifier and a plurality of third data, and the second data and the third data are in one-to-one correspondence; the first data is original exposure data, the second data is an exposure channel, and the third data is an exposure channel news ID list; a storage unit, configured to configure a one-to-one correspondence relationship between the user identifier of the first data and the second data and the third data, and use the first data as cache data;

the first data processing unit is used for taking third data corresponding to the operated second data as real data for a background to use when the third data corresponding to the operated second data meet a preset condition; the preset conditions include: the duration of the third data is not greater than the validity time of the third data.

8. The data processing system of claim 7,

the first acquisition unit is further used for acquiring original exposure data generated by a user operating a client, wherein the user has a user ID for identifying an identity, the original exposure data comprises the user ID, a plurality of exposure channels corresponding to the user ID and a plurality of exposure channel news ID lists, and the exposure channels correspond to the exposure channel news ID lists one by one;

the storage unit is also used for storing the original exposure data into a KV storage system based on the memory by taking the user ID as a key and taking an exposure channel and exposure channel news ID list as a value as an exposure cache;

the second obtaining unit is further configured to obtain channel click data generated by the user operating the client, where the channel click data includes the user ID and a clicked exposure channel, and the clicked exposure channel is one of the exposure channels;

the first query unit is further configured to query the exposure cache corresponding to the user ID in the KV storage system according to the user ID;

the second query unit is further configured to query, according to the clicked exposure channel, an exposure channel news ID list corresponding to the clicked exposure channel in the exposure cache;

the first data processing unit is further used for enabling the exposure channel news ID list corresponding to the clicked exposure channel to serve as real exposure data to be used by a background when the exposure channel news ID list corresponding to the clicked exposure channel meets a preset condition.

9. The data processing system of claim 8, wherein the exposure channel news ID list includes an exposure time and a valid time, the data processing system further comprising:

10. The data processing system of claim 8, further comprising:

and the second data processing unit is used for discarding the exposure channel news ID list corresponding to the clicked exposure channel as pseudo exposure data when the exposure channel news ID list corresponding to the clicked exposure channel does not meet the preset condition.

11. The data processing system of claim 8, further comprising:

a fourth obtaining unit, configured to obtain news click data generated by the user operating client, where the news click data includes the user ID, an exposure channel where the news is located, and a news ID, and the exposure channel where the news is located is one of the exposure channels;

12. A terminal, comprising: a memory and a processor;

the memory is used for storing a computer program;

the processor is configured to execute a computer program stored in the memory;

the computer program is for performing the data processing method of any one of claims 1 to 6.

13. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium; the computer program is for performing the data processing method of any one of claims 1 to 6.