CN110471945B - Active data processing method, system, computer equipment and storage medium - Google Patents
Active data processing method, system, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110471945B CN110471945B CN201910606335.0A CN201910606335A CN110471945B CN 110471945 B CN110471945 B CN 110471945B CN 201910606335 A CN201910606335 A CN 201910606335A CN 110471945 B CN110471945 B CN 110471945B
- Authority
- CN
- China
- Prior art keywords
- event
- data
- active
- user
- statistical
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title description 13
- 238000012545 processing Methods 0.000 claims abstract description 38
- 238000000034 method Methods 0.000 claims abstract description 34
- 238000004590 computer program Methods 0.000 claims description 13
- 230000002159 abnormal effect Effects 0.000 claims description 11
- 230000005856 abnormality Effects 0.000 claims description 8
- 238000012216 screening Methods 0.000 claims description 3
- 238000013075 data extraction Methods 0.000 abstract description 9
- 238000013515 script Methods 0.000 abstract description 8
- 238000010586 diagram Methods 0.000 description 7
- 230000006399 behavior Effects 0.000 description 4
- 238000001514 detection method Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 3
- 230000002349 favourable effect Effects 0.000 description 3
- 230000006978 adaptation Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004140 cleaning Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Fuzzy Systems (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention relates to the technical field of data processing, and provides a method and a system for processing active data, wherein the method comprises the following steps: classifying event information and user information in the buried active data; classifying and storing event information according to event types to obtain an active event list; combining the user information of the same user, and storing the user information in a classified manner according to the user attribute type to obtain a user list; and acquiring an input statistical caliber, and carrying out associated inquiry from the active event list and the user list according to the statistical caliber to obtain target active data. According to the method, the buried point active data are stored according to the classification, the buried point active data meeting the statistical standard can be searched correspondingly according to the statistical standard of the statistical caliber under each classification, the target active data meeting the statistical caliber can be screened after the buried point active data are associated, script writing according to the relation among various types of statistical standards in the statistical caliber can be avoided, the workload is reduced, and the efficiency of active data extraction is improved.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method for processing active data, a system for processing active data, a computer device, and a storage medium.
Background
With the advancement of more and more service platforms for informationizing service processing, the method is popular with vast users. When the service platform has more users and operation service types, the traffic of the service platform can be correspondingly increased. By carrying out data embedding on the service platform and counting active data, the flow of the service platform can be analyzed, and then the optimization requirement or the current user requirement of the service platform can be analyzed to a certain extent.
In the prior art, active data under a contracted statistical caliber can be generally obtained through a fixed interface, and corresponding scripts are required to be written for other statistical calibers to obtain the active data. However, the process of writing the script according to the statistical caliber is complex, and the efficiency of active data extraction is reduced.
Disclosure of Invention
The present invention aims to solve at least one of the above technical drawbacks, in particular the technical drawback of low active data extraction efficiency.
The invention provides a processing method of active data, which comprises the following steps:
acquiring buried point active data of a service platform, and classifying event information and user information in the buried point active data;
classifying and storing the event information according to the event type to obtain an active event list;
combining the user information of the same user, and storing the user information in a classified manner according to the user attribute type to obtain a user list;
and acquiring an input statistical caliber, and carrying out association query from the active event list and the user list according to the statistical caliber to obtain target active data.
In one embodiment, the step of obtaining buried active data of the service platform includes:
and acquiring original buried point data of the service platform, detecting a missing value and an abnormal value of the original buried point data, and removing the original buried point data with the missing value and the abnormal value and repeated original buried point data to obtain the buried point active data.
In one embodiment, before the step of performing the association query from the active event table and the user list according to the statistical caliber, the method further comprises:
receiving newly-added buried point active data, and separating newly-added event information and newly-added user information from the newly-added buried point active data; classifying and storing the newly added event information according to the event type to obtain an incremental event table, and adding the incremental event table to the active event table; and updating the user list according to the newly added user information.
In one embodiment, after the step of adding the incremental event table to the active event table, further comprising:
receiving the uploading data number of the newly added buried active data; recording a first event number and a second event number of the active event table, and taking a difference value of the first event number and the second event number as an input data number, wherein the first event number and the second event number are the number of event information before and after the increment event table is added in the active event table respectively; judging whether the number of the uploaded data is equal to the number of the input data; when the number of the uploaded data is equal to the number of the input data, judging that the data of the active event list is complete, and executing the step of carrying out association inquiry from the active event list and the user list according to the statistical caliber; and when the number of the uploaded data is not equal to the number of the input data, generating reminding information for representing data abnormality.
In one embodiment, the step of performing an association query from the active event table and the user list according to the statistical caliber includes:
separating an event statistical caliber and a user attribute statistical caliber from the statistical caliber; retrieving target event information belonging to the event statistics caliber from the active event table; invoking target user information of a user associated with the target event information in the user list; screening the target user information according to the user attribute statistical caliber to obtain active user information; and counting the active user information to obtain the target active data, and drawing a user map.
In one embodiment, the step of performing an association query from the active event table and the user list according to the statistical caliber includes:
separating the event statistical aperture and the user attribute statistical aperture from the statistical aperture; retrieving target user information belonging to the user attribute statistical caliber from the user list; retrieving target event information of a user associated with the target user information from the active event table; and counting the target active data belonging to the event counting caliber from the target event information.
In one embodiment, the step of obtaining the statistical aperture includes:
the caliber data acquired in a label generator is used as a parameter label, and the parameter label is stored and displayed, wherein the label generator is used for generating the parameter label related to the event type and/or the user attribute type; and obtaining parameter labels in a parameter pool, and generating the statistical caliber according to the parameter labels in the parameter pool, wherein the parameter pool is an input frame for identifying the statistical caliber corresponding to the parameter labels.
The invention also provides a processing system of the active data, which comprises:
the classification module is used for acquiring buried point active data of the service platform and classifying event information and user information in the buried point active data;
the storage module is used for classifying and storing the event information according to the event type to obtain an active event list;
the merging module is used for merging the user information of the same user, and classifying and storing the user information according to the user attribute type to obtain a user list;
and the query module is used for acquiring an input statistical caliber, and carrying out association query from the active event list and the user list according to the statistical caliber to obtain target active data.
The invention also provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the active data processing method according to any of the embodiments.
The present invention also provides a storage medium storing computer readable instructions, on which is stored a computer program which, when executed by a processor, implements the steps of the active data processing method according to any of the above embodiments.
According to the processing method, the system, the computer equipment and the storage medium of the active data, the buried point active data are stored according to the classification, the buried point active data meeting the statistical standard can be searched correspondingly according to the statistical standard of the statistical caliber under each classification, the buried point active data meeting the statistical standard under each classification are associated, the target active data meeting the statistical caliber can be screened out, script writing according to the relation among various types of statistical standards in the statistical caliber is avoided, the workload is reduced, and the active data extraction efficiency is improved.
Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Drawings
The foregoing and/or additional aspects and advantages of the invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:
FIG. 1 is an environmental diagram of an implementation of a method of processing active data provided in one embodiment;
FIG. 2 is a flow chart of a method of processing active data in one embodiment;
FIG. 3 is a flow chart of a method of processing active data in one embodiment;
FIG. 4 is a schematic diagram of a processing system for active data in one embodiment;
fig. 5 is a schematic diagram showing an internal structure of the computer device in one embodiment.
Detailed Description
Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are illustrative only and are not to be construed as limiting the invention.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless expressly stated otherwise, as understood by those skilled in the art. It will be understood by those skilled in the art that all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs unless defined otherwise. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
As shown in fig. 1, fig. 1 is an implementation environment diagram of a method for processing active data provided in one embodiment, where the implementation environment includes a computer device 110 and a service platform 120.
The computer device 110 is connected to a service platform 120. The service platform 120 may be provided with a data embedding point, and the data embedding point may be used for subsequent collection of active data in the service platform 120. The computer device 110 has computing and storage capabilities that can analyze the active data in the service platform 120. The computer device 110 may be, but is not limited to, a notebook computer, a desktop computer, etc.
In one embodiment, as shown in fig. 2, fig. 2 is a flowchart of a method for processing active data in one embodiment, and in this embodiment, a method for processing active data is provided, where the method for processing active data may be applied to the computer device 110, and may specifically include the following steps:
step S210: and acquiring buried point active data of the service platform, and classifying event information and user information in the buried point active data.
In this step, the buried active data may record the relationship between the user and the behavior event, so that event information and user information in the buried active data may be obtained by processing data related to the behavior event and related to the user in the buried active data, respectively.
Specifically, the step of acquiring the buried active data of the service platform in step S210 may include:
the method comprises the steps of obtaining original buried point data of a service platform, carrying out missing value detection and abnormal value detection on the original buried point data, and removing the original buried point data with missing values and abnormal values and repeated original buried point data to obtain buried point active data.
The original buried data in the service platform is generally collected according to the buried data, and the collected original buried data occasionally has defects such as deletion, abnormality, repetition and noise. According to the method for obtaining the buried point active data, the original buried point data with the missing value and the abnormal value and the repeated original buried point data can be respectively removed through cleaning the original buried point data, namely the original buried point data with the bad condition is removed, the buried point active data with complete data is obtained, errors caused by data abnormality during subsequent buried point active data processing can be avoided as much as possible, and the error rate is reduced.
Step S220: and classifying and storing the event information according to the event type to obtain an active event table.
In this step, the event information is classified according to the event type, the event information of the determined event type and the event type are stored, that is, the buried active data is stored according to the event type. The event types can be planned and designed according to services and functions provided by the service platform, and a plurality of event types can be predefined. In a service platform, for example, a bank insurance service, event types including a policy event, a user conversion event, a user registration event, a login and logout event, and the like may be divided, where the divided event types are often related to a service type and service data of interest related to a service department thereof.
Step S230: and combining the user information of the same user, and storing the user information in a classified manner according to the user attribute type to obtain a user list.
The user can operate a plurality of events in the service platform, so that part of the embedded point active data come from the same user, the user information of the same user in the embedded point active data is combined, the same user has one piece of user information, and repeated storage of the user information can be reduced. In the step, according to the user information of the non-repeated users, a user list containing user information corresponding to all users in the active data of the buried points can be generated; while also determining the user attribute type of the user information. The more the types of the user attribute types are, the more the types of the user attribute types are favorable for adapting to the fineness and the high statistical caliber or drawing the fine user patterns. The classifying process of the user information can determine the user attribute types corresponding to various information in the user information and store the various information and the corresponding user attribute types in an associated mode. The user information may include various information such as user gender, user age, user location area, user occupation, and user account amount, and the user attribute types may include gender, age, area, occupation, account amount, etc., the user gender in the user information is classified into gender types and stored in gender types, and so on, the information such as user age, user location area, user occupation, and user account amount in the user information is stored.
Step S240: and acquiring an input statistical caliber, and carrying out associated inquiry from the active event list and the user list according to the statistical caliber to obtain target active data.
In the step, target active data meeting the statistical caliber is inquired in an active event list and a user list.
The step of obtaining the statistical caliber in step S240 may include:
s241: and taking the caliber data acquired in the label generator as a parameter label, and storing and displaying the parameter label, wherein the label generator is used for generating the parameter label related to the event type and/or the user attribute type.
In this step, the user inputs caliber data into the tag generator through the visual operation page, and uses the caliber data as a parameter tag, and the parameter tag can represent certain caliber data for representing a statistical standard. The parameter tab may be displayed on a visual operations page and may be dragged. The parameter tag may be established by inputting the caliber data into the tag generator, and the tag generator may define the commonly used caliber data in advance as the parameter tag.
After the parameter tab is dragged to the parameter pool by the user, S242: the parameter labels in the parameter pool can be identified and obtained, and the statistical caliber is generated according to the parameter labels in the parameter pool, wherein the parameter pool is an input frame for identifying the statistical caliber corresponding to the parameter labels.
In the step, the parameter labels in the parameter pool are identified, the statistical caliber can be calculated according to caliber data corresponding to the identified parameter labels, the statistical caliber is prevented from being determined by writing scripts, and the workload is reduced.
In the mode of acquiring the input statistical aperture, the statistical aperture can be quickly and directly constructed under the visualization, the workload of determining the statistical aperture is reduced, and the efficiency of active data extraction is improved.
According to the processing method of the active data, the buried active data are stored according to the classification, the buried active data meeting the statistical standard can be searched correspondingly according to the statistical standard of the statistical caliber under each classification, the buried active data meeting the statistical standard under each classification are associated, the target active data meeting the statistical caliber can be screened out, script writing according to the relation among various types of statistical standards in the statistical caliber is avoided, the workload is reduced, and the active data extraction efficiency is improved.
Further, after obtaining the user list, the active event list and the user list may be stored in a database of the policy system, and the association operation may be performed by the policy system between the active event list and the user list, so as to facilitate a subsequent association query.
In one embodiment, before the step of performing the association query from the active event table and the user list according to the statistical aperture in step S240, the method may further include:
s251: and receiving the newly added buried point active data, and separating newly added event information and newly added user information from the newly added buried point active data.
In this step, incremental data of the active data of the buried point newly added in a certain preset time is received, namely the newly added active data of the buried point.
S252: and classifying and storing the newly added event information according to the event type to obtain an incremental event table, and adding the incremental event table into the active event table.
In this step, the new time information is added to the active event table in a manner of adding an increment, specifically, the new event information may be classified and stored as an increment event table, and the increment event table is merged into the active event table.
S253: and updating the user list according to the newly added user information.
In this step, since the user information of the same user in the user list is not repeated, the user list may be updated according to the user information of the same user combined after the incremental data is received; the user list can be updated by comparing the newly added user information with the user information of the user list, searching for the newly added user information which does not exist in the user list, and adding the newly added user information to the user list.
According to the processing method of the active data, the active event list and the user list corresponding to the buried active data are updated by receiving the newly added data within a certain preset time, so that massive buried active data do not need to be received each time, the waste of data transmission and data storage resources is avoided, and the cost can be reduced.
In one embodiment, after the step of adding the incremental event table to the active event table in S251, it may further include:
s261: and receiving the uploading data number of the newly added buried active data.
In this step, the newly added buried active data may be received while the increment of the received data is being received.
S262: and recording the first event number and the second event number of the active event table, and taking the difference value of the first event number and the second event number as the recorded data number, wherein the first event number and the second event number are the number of event information before and after adding the increment event table in the active event table respectively.
In this step, the increment of the data in the active event table after the event information is newly added is calculated.
S263: and judging whether the number of the uploaded data is equal to the number of the recorded data.
In this step, it is determined whether the received increment corresponds to the increment of the stored data, i.e., whether the number of uploaded data is equal to the number of recorded data.
S264: and when the number of the uploaded data is equal to the number of the recorded data, judging that the data of the active event list is complete, and executing the step of carrying out association inquiry from the active event list and the user list according to the statistical caliber.
In this step, when the received increment corresponds to the increment of the stored data, that is, the number of the uploaded data is equal to the number of the input data, it indicates that the received and stored data have consistency, and after the integrity of the data is ensured, the subsequent steps can be continuously executed.
S265: and when the number of the uploaded data is not equal to the number of the input data, generating reminding information for representing data abnormality.
In this step, when the received increment and the increment of the stored data are different, that is, the number of the uploaded data is not equal to the number of the input data, the data is missing or abnormal in the transmission or storage process, and the reminding information indicating the abnormal data can be generated and sent for reminding.
According to the processing method of the active data, whether the newly added data meets consistency or not is checked through the received increment and the increment of the stored data, and the integrity of the buried active data after the newly added data can be ensured.
In one embodiment, the step of performing the association query from the active event table and the user list according to the statistical caliber in step S240 may include:
a1: and separating the event statistical caliber and the user attribute statistical caliber from the statistical caliber.
A2: and calling target event information belonging to the event statistics caliber from the active event list.
A3: target user information of a user associated with the target event information is retrieved from the user list.
A4: and screening target user information according to the user attribute statistical caliber to obtain active user information.
A5: and counting active user information to obtain target active data, and drawing a user map.
According to the processing method of the active data, according to the target event information corresponding to the event statistical caliber in the statistical caliber, the correlation inquiry is carried out with the user attribute statistical caliber, the active user information is counted, the target active data is obtained, and the user map can be drawn according to the target active data.
In one embodiment, the step of performing the association query from the active event table and the user list according to the statistical caliber in step S240 may include:
b1: and separating the event statistical caliber and the user attribute statistical caliber from the statistical caliber.
B2: and calling the target user information belonging to the user attribute statistical caliber from the user list.
B3: and retrieving target event information of the user associated with the target user information in the active event table.
B4: and counting the target active data belonging to the event statistics caliber from the target event information.
According to the method for processing the active data, the target active data can be obtained after the target event information is counted according to the target user information corresponding to the user attribute statistic caliber in the statistic caliber and then the correlation inquiry is carried out under the event statistic caliber.
In another embodiment, as shown in fig. 3, fig. 3 is a flowchart of a method for processing active data in one embodiment, where the method for processing active data may specifically include the following steps:
s310: and acquiring original buried point data of the service platform, cleaning the original buried point data and acquiring buried point active data. And carrying out missing value detection and abnormal value detection on the original buried point data, and removing the original buried point data with missing values and abnormal values and repeated original buried point data to obtain buried point active data.
S320: and classifying event information and user information in the buried active data.
S330: and classifying and storing the event information according to the event type to obtain an active event table. The event information is classified according to event type, the event information of the determined event type and the event type are stored, namely, the buried point active data is stored according to the event type. The event types can be planned and designed according to services and functions provided by the service platform, and a plurality of event types can be predefined. In a service platform, for example, a bank insurance service, event types including a policy event, a user conversion event, a user registration event, a login and logout event, and the like may be divided, where the divided event types are often related to a service type and service data of interest related to a service department thereof.
S340: and combining the user information of the same user, and storing the user information in a classified manner according to the user attribute type to obtain a user list. The user can operate a plurality of events in the service platform, so that part of the buried active data come from the same user, the same user has one piece of user information, and the user information of the same user in the buried active data is combined for reducing the repeatedly stored user information. In the step, according to the user information of the non-repeated users, a user list containing user information corresponding to all users in the active data of the buried points can be generated; while also determining the user attribute type of the user information. The more the types of the user attribute types are, the more the types of the user attribute types are favorable for adapting to the fineness and the high statistical caliber or drawing the fine user patterns.
S350: and receiving the newly added buried point active data, and updating the active event list and the user list. And receiving the uploading data number of the newly added buried active data. And separating the newly added event information and the newly added user information from the newly added buried active data. And classifying and storing the newly added event information according to the event type to obtain an incremental event table, and adding the incremental event table into the active event table. And updating the user list according to the newly added user information.
S360: and checking the consistency of the data in the updated active event list and the user list. And recording the first event number and the second event number of the active event table, and taking the difference value of the first event number and the second event number as the recorded data number, wherein the first event number and the second event number are the number of event information before and after adding the increment event table in the active event table respectively. And judging whether the number of the uploaded data is equal to the number of the recorded data. And when the number of the uploaded data is equal to the number of the recorded data, judging that the data of the active event list is complete, and executing the step of carrying out association inquiry from the active event list and the user list according to the statistical caliber. And when the number of the uploaded data is not equal to the number of the input data, generating reminding information for representing data abnormality.
S370: and obtaining the input statistical caliber. Inputting caliber data in a label generator, generating parameter labels, and storing and displaying the parameter labels, wherein the label generator is used for generating the parameter labels related to event types and/or user attribute types. After the parameter labels are dragged to the parameter pool, the parameter labels in the parameter pool can be obtained, and the statistical caliber is generated according to the parameter labels in the parameter pool, wherein the parameter pool is an input frame for identifying the statistical caliber corresponding to the parameter labels.
S380: and carrying out association inquiry from the active event list and the user list according to the statistical caliber to obtain target active data.
Separating an event statistical caliber and a user attribute statistical caliber from the statistical caliber; retrieving target event information belonging to an event statistics caliber from an active event list; invoking target user information of a user associated with the target event information in a user list; filtering target user information according to the user attribute statistical caliber to obtain active user information; and counting active user information to obtain target active data, and drawing a user map.
Or separating the event statistical caliber and the user attribute statistical caliber from the statistical caliber; target user information belonging to the user attribute statistical caliber is called in the user list; retrieving target event information of a user associated with the target user information in an active event table; and counting the target active data belonging to the event statistics caliber from the target event information.
According to the processing method of the active data, the buried active data are stored according to the classification, the buried active data meeting the statistical standard can be searched correspondingly according to the statistical standard of the statistical caliber under each classification, the buried active data meeting the statistical standard under each classification are associated, the target active data meeting the statistical caliber can be screened out, script writing according to the relation among various types of statistical standards in the statistical caliber is avoided, the workload is reduced, and the active data extraction efficiency is improved. Meanwhile, the statistical caliber can be quickly and directly built under the visualization, the workload of determining the statistical caliber is reduced, and the efficiency of active data extraction is improved.
In one embodiment, as shown in fig. 4, fig. 4 is a schematic structural diagram of a processing system for active data in one embodiment, where the processing system for active data in this embodiment may include a classification module 410, a storage module 420, a merging module 430, and a query module 440, where:
the classification module 410 is configured to obtain the embedded active data of the service platform, and classify event information and user information in the embedded active data.
In the classification module 410, the buried active data may record the relationship between the user and the behavior event, so that the event information and the user information in the buried active data may be obtained by processing the data related to the behavior event and the data related to the user in the buried active data, respectively.
The storage module 420 is configured to store the event information in a classified manner according to the event type, so as to obtain an active event table.
The storage module 420 may classify event information by event type, store event information for a determined event type and the event type, i.e., store buried active data by event type. The event types can be planned and designed according to services and functions provided by the service platform, and a plurality of event types can be predefined. In a service platform, for example, a bank insurance service, event types including a policy event, a user conversion event, a user registration event, a login and logout event, and the like may be divided, where the divided event types are often related to a service type and service data of interest related to a service department thereof.
And the merging module 430 is configured to merge user information of the same user, and store the user information in a classified manner according to the user attribute type to obtain a user list.
The user can operate a plurality of events in the service platform, so that part of the embedded point active data come from the same user, the user information of the same user in the embedded point active data is combined, the same user has one piece of user information, and repeated storage of the user information can be reduced. The merging module 430 can generate a user list containing user information corresponding to all users in the active data of the buried point according to the user information of the non-repeated users; while also determining the user attribute type of the user information. The more the types of the user attribute types are, the more the types of the user attribute types are favorable for adapting to the fineness and the high statistical caliber or drawing the fine user patterns. The classifying process of the user information can determine the user attribute types corresponding to various information in the user information and store the various information and the corresponding user attribute types in an associated mode. The user information may include various information such as user gender, user age, user location area, user occupation, and user account amount, and the user attribute types may include gender, age, area, occupation, account amount, etc., the user gender in the user information is classified into gender types and stored in gender types, and so on, the information such as user age, user location area, user occupation, and user account amount in the user information is stored.
And the query module 440 is configured to obtain an input statistical caliber, and perform an associated query from the active event table and the user list according to the statistical caliber, so as to obtain target active data.
The query module 440 may query the active event table and the user list for target active data that meets the statistical aperture.
According to the processing system of the active data, the buried active data are stored according to the classification, the buried active data meeting the statistical standard can be searched correspondingly according to the statistical standard of the statistical caliber under each classification, the buried active data meeting the statistical standard under each classification are associated, the target active data meeting the statistical caliber can be screened out, script writing according to the relation among various types of statistical standards in the statistical caliber is avoided, the workload is reduced, and the active data extraction efficiency is improved.
For specific limitations of the processing system for active data, reference may be made to the above limitation of the processing method for active data, and no further description is given here. The various modules in the active data processing system described above may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
As shown in fig. 5, fig. 5 is a schematic diagram of an internal structure of the computer device in one embodiment. The computer device includes a processor, a non-volatile storage medium, a memory, and a network interface connected by a system bus. The non-volatile storage medium of the computer device stores an operating system and a computer program, and when the computer program is executed by the processor, the computer program can enable the processor to implement a method for processing active data. The processor of the computer device is used to provide computing and control capabilities, supporting the operation of the entire computer device. The memory of the computer device may have stored therein a computer program which, when executed by the processor, causes the processor to perform a method of processing active data. The network interface of the computer device is for communicating with a terminal connection. It will be appreciated by those skilled in the art that the structure shown in fig. 5 is merely a block diagram of some of the structures associated with the present application and is not limiting of the computer device to which the present application may be applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided that includes a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the method for processing active data in any of the embodiments described above when the computer program is executed by the processor.
In an embodiment, a computer readable storage medium is provided, on which a computer program is stored which, when executed by a processor, implements the steps of the active data processing method of any of the embodiments described above.
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited in order and may be performed in other orders, unless explicitly stated herein. Moreover, at least some of the steps in the flowcharts of the figures may include a plurality of sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, the order of their execution not necessarily being sequential, but may be performed in turn or alternately with other steps or at least a portion of the other steps or stages.
The foregoing is only a partial embodiment of the present invention, and it should be noted that it will be apparent to those skilled in the art that modifications and adaptations can be made without departing from the principles of the present invention, and such modifications and adaptations are intended to be comprehended within the scope of the present invention.
Claims (7)
1. A method for processing active data, comprising the steps of:
acquiring buried point active data of a service platform, and classifying event information and user information in the buried point active data;
classifying and storing the event information according to the event type to obtain an active event list;
combining the user information of the same user, and storing the user information in a classified manner according to the user attribute type to obtain a user list;
acquiring an input statistical caliber, and carrying out association query from the active event list and the user list according to the statistical caliber to obtain target active data;
the step of obtaining the statistical caliber specifically comprises the following steps:
the caliber data acquired in a label generator is used as a parameter label, and the parameter label is stored and displayed, wherein the label generator is used for generating the parameter label related to the event type and/or the user attribute type;
acquiring parameter labels in a parameter pool, and calculating the statistical caliber according to caliber data corresponding to the parameter labels in the parameter pool, wherein the parameter pool is an input frame for identifying the statistical caliber corresponding to the parameter labels;
wherein, before the step of performing the association query from the active event list and the user list according to the statistical caliber, the method further comprises:
receiving newly-added buried point active data, and separating newly-added event information and newly-added user information from the newly-added buried point active data;
classifying and storing the newly added event information according to the event type to obtain an incremental event table, and adding the incremental event table to the active event table;
updating the user list according to the newly added user information;
after the step of adding the incremental event table to the active event table, the method further comprises:
receiving the uploading data number of the newly added buried active data;
recording a first event number and a second event number of the active event table, and taking a difference value of the first event number and the second event number as an input data number, wherein the first event number and the second event number are the number of event information before and after the increment event table is added in the active event table respectively;
judging whether the number of the uploaded data is equal to the number of the input data;
when the number of the uploaded data is equal to the number of the input data, judging that the data of the active event list is complete, and executing the step of carrying out association inquiry from the active event list and the user list according to the statistical caliber;
and when the number of the uploaded data is not equal to the number of the input data, generating reminding information for representing data abnormality.
2. The method for processing active data according to claim 1, wherein the step of obtaining buried active data of the service platform comprises:
and acquiring original buried point data of the service platform, detecting a missing value and an abnormal value of the original buried point data, and removing the original buried point data with the missing value and the abnormal value and repeated original buried point data to obtain the buried point active data.
3. The method of processing active data according to claim 1, wherein the step of performing an associative query from the active event list and the user list according to the statistical aperture comprises:
separating an event statistical caliber and a user attribute statistical caliber from the statistical caliber;
retrieving target event information belonging to the event statistics caliber from the active event table;
invoking target user information of a user associated with the target event information in the user list;
screening the target user information according to the user attribute statistical caliber to obtain active user information;
and counting the active user information to obtain the target active data, and drawing a user map.
4. The method of processing active data according to claim 1, wherein the step of performing an associative query from the active event list and the user list according to the statistical aperture comprises:
separating the event statistical aperture and the user attribute statistical aperture from the statistical aperture;
retrieving target user information belonging to the user attribute statistical caliber from the user list;
retrieving target event information of a user associated with the target user information from the active event table;
and counting the target active data belonging to the event counting caliber from the target event information.
5. A system for processing active data, comprising:
the classification module is used for acquiring buried point active data of the service platform and classifying event information and user information in the buried point active data;
the storage module is used for classifying and storing the event information according to the event type to obtain an active event list;
the merging module is used for merging the user information of the same user, and classifying and storing the user information according to the user attribute type to obtain a user list;
the query module is used for acquiring an input statistical caliber, and carrying out association query from the active event list and the user list according to the statistical caliber to obtain target active data;
the query module is specifically configured to:
the caliber data acquired in a label generator is used as a parameter label, and the parameter label is stored and displayed, wherein the label generator is used for generating the parameter label related to the event type and/or the user attribute type;
acquiring parameter labels in a parameter pool, and calculating the statistical caliber according to caliber data corresponding to the parameter labels in the parameter pool, wherein the parameter pool is an input frame for identifying the statistical caliber corresponding to the parameter labels;
the processing system further comprises a user list updating module, which is specifically used for:
receiving newly-added buried point active data, and separating newly-added event information and newly-added user information from the newly-added buried point active data;
classifying and storing the newly added event information according to the event type to obtain an incremental event table, and adding the incremental event table to the active event table;
updating the user list according to the newly added user information;
the processing system further comprises an abnormality reminding information generation module which is specifically used for:
receiving the uploading data number of the newly added buried active data;
recording a first event number and a second event number of the active event table, and taking a difference value of the first event number and the second event number as an input data number, wherein the first event number and the second event number are the number of event information before and after the increment event table is added in the active event table respectively;
judging whether the number of the uploaded data is equal to the number of the input data;
when the number of the uploaded data is equal to the number of the input data, judging that the data of the active event list is complete, and executing the step of carrying out association inquiry from the active event list and the user list according to the statistical caliber;
and when the number of the uploaded data is not equal to the number of the input data, generating reminding information for representing data abnormality.
6. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method of processing active data according to any one of claims 1 to 4 when the computer program is executed by the processor.
7. A computer-readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of processing active data according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910606335.0A CN110471945B (en) | 2019-07-05 | 2019-07-05 | Active data processing method, system, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910606335.0A CN110471945B (en) | 2019-07-05 | 2019-07-05 | Active data processing method, system, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110471945A CN110471945A (en) | 2019-11-19 |
| CN110471945B true CN110471945B (en) | 2024-02-06 |
Family
ID=68506766
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910606335.0A Active CN110471945B (en) | 2019-07-05 | 2019-07-05 | Active data processing method, system, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110471945B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112966174A (en) * | 2019-12-13 | 2021-06-15 | 北京默契破冰科技有限公司 | Method, apparatus, and computer storage medium for processing data |
| CN111752799A (en) * | 2020-06-24 | 2020-10-09 | 中国建设银行股份有限公司 | Service link tracking method, device, equipment and storage medium |
| CN112579408A (en) * | 2020-10-29 | 2021-03-30 | 上海钱拓网络技术有限公司 | Classification method of embedded point information |
| CN112417281A (en) * | 2020-11-20 | 2021-02-26 | 京东方科技集团股份有限公司 | Data analysis method, device and equipment |
| CN114168430A (en) * | 2022-01-06 | 2022-03-11 | 携程旅游网络技术（上海）有限公司 | Front-end abnormal alarm configuration method, system, equipment and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105227751A (en) * | 2014-06-30 | 2016-01-06 | 腾讯科技（深圳）有限公司 | The method and system of realization event information digitalization |
| CN106557513A (en) * | 2015-09-29 | 2017-04-05 | 腾讯科技（深圳）有限公司 | Event information method for pushing and event information pusher |
| CN106980769A (en) * | 2017-04-05 | 2017-07-25 | 张莹 | A kind of destination client category system |
| CN107368488A (en) * | 2016-05-12 | 2017-11-21 | 阿里巴巴集团控股有限公司 | A kind of method for determining user behavior preference, the methods of exhibiting and device of recommendation information |
| CN107402910A (en) * | 2017-07-07 | 2017-11-28 | 北京今目标信息技术有限公司 | Data managing method, equipment and computer-readable recording medium based on list |
| CN109150641A (en) * | 2017-06-15 | 2019-01-04 | 北京国双科技有限公司 | A kind of data acquisition, querying method, device, storage medium and processor |
| CN109767259A (en) * | 2018-12-15 | 2019-05-17 | 深圳壹账通智能科技有限公司 | Based on operation event promotion method, apparatus, equipment and the medium for burying point data |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10055481B2 (en) * | 2016-07-20 | 2018-08-21 | LogsHero Ltd. | Method and system for automatic event classification |
-
2019
- 2019-07-05 CN CN201910606335.0A patent/CN110471945B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105227751A (en) * | 2014-06-30 | 2016-01-06 | 腾讯科技（深圳）有限公司 | The method and system of realization event information digitalization |
| CN106557513A (en) * | 2015-09-29 | 2017-04-05 | 腾讯科技（深圳）有限公司 | Event information method for pushing and event information pusher |
| CN107368488A (en) * | 2016-05-12 | 2017-11-21 | 阿里巴巴集团控股有限公司 | A kind of method for determining user behavior preference, the methods of exhibiting and device of recommendation information |
| CN106980769A (en) * | 2017-04-05 | 2017-07-25 | 张莹 | A kind of destination client category system |
| CN109150641A (en) * | 2017-06-15 | 2019-01-04 | 北京国双科技有限公司 | A kind of data acquisition, querying method, device, storage medium and processor |
| CN107402910A (en) * | 2017-07-07 | 2017-11-28 | 北京今目标信息技术有限公司 | Data managing method, equipment and computer-readable recording medium based on list |
| CN109767259A (en) * | 2018-12-15 | 2019-05-17 | 深圳壹账通智能科技有限公司 | Based on operation event promotion method, apparatus, equipment and the medium for burying point data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110471945A (en) | 2019-11-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110471945B (en) | Active data processing method, system, computer equipment and storage medium | |
| CN110399293B (en) | System test method, device, computer equipment and storage medium | |
| CN109598095B (en) | Method and device for establishing scoring card model, computer equipment and storage medium | |
| CN110428127B (en) | Automatic analysis method, user equipment, storage medium and device | |
| WO2020068868A1 (en) | Object detection machine learning | |
| CN111460312A (en) | Method and device for identifying empty-shell enterprise and computer equipment | |
| WO2020062702A1 (en) | Method and device for sending text messages, computer device and storage medium | |
| CN110956269A (en) | Data model generation method, device, equipment and computer storage medium | |
| WO2023056723A1 (en) | Fault diagnosis method and apparatus, and electronic device and storage medium | |
| CN114757468B (en) | Root cause analysis method for process execution abnormality in process mining | |
| CN112084180A (en) | Method, device, equipment and medium for monitoring vehicle-mounted application quality | |
| CN113538154A (en) | Risk object identification method and device, storage medium and electronic equipment | |
| CN114356257A (en) | Log printing method, apparatus, computer device, storage medium, and program product | |
| CN110704699A (en) | Data image construction method and device, computer equipment and storage medium | |
| CN111459796B (en) | Automated testing method, apparatus, computer device and storage medium | |
| CN110716866A (en) | Code quality scanning method and device, computer equipment and storage medium | |
| CN114116811B (en) | Log processing method, device, equipment and storage medium | |
| CN115576834A (en) | Software test multiplexing method, system, terminal and medium for supporting fault recovery | |
| CN115310011A (en) | Page display method and system and readable storage medium | |
| CN113723436A (en) | Data processing method and device, computer equipment and storage medium | |
| CN111966515A (en) | Business abnormal data processing method and device, computer equipment and storage medium | |
| CN114756401B (en) | Abnormal node detection method, device, equipment and medium based on log | |
| CN111045849A (en) | Method, device, server and storage medium for identifying reason of checking abnormality | |
| CN117556187B (en) | Cloud data restoration method and system based on deep learning and readable storage medium | |
| CN116738216B (en) | Association processing method and device for early warning big data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |