CN110928850A

CN110928850A - Traffic statistic method and device

Info

Publication number: CN110928850A
Application number: CN201810996059.9A
Authority: CN
Inventors: 孙铎春
Original assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Current assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Priority date: 2018-08-29
Filing date: 2018-08-29
Publication date: 2020-03-27

Abstract

The invention discloses a method and a device for flow statistics, and relates to the technical field of computers. One embodiment of the method comprises: receiving a browsing log to be analyzed to obtain a log format of the browsing log to be analyzed; matching the log format of the browsing log to be analyzed according to a preset interface list; and acquiring a corresponding browsing log processing implementation code in a preset interface list, and processing the browsing log to be analyzed to obtain flow data. The implementation mode can solve the problem that the non-invasive analysis capability is increased through dynamic and rapid iteration related to flow real-time monitoring in big data analysis.

Description

Traffic statistic method and device

Technical Field

The invention relates to the technical field of computers, in particular to a method and a device for flow statistics.

Background

In the field of flow real-time monitoring of big data analysis, unstructured browsing log data are converted into structured real-time flow data through URL matching analysis by aiming at browsing logs with multiple applications, multiple channels and multiple formats through a unified entry, and then the real-time flow data of the same application are subjected to accumulation analysis within a certain time interval and are stored, so that the real-time flow statistics is completed. The browsing log refers to a data record reported by a flow buried point formed by a user browsing a page in an application.

At present, after completing compatible analysis of a newly added log format, that is, completing URL matching analysis of a log, a new log format is added, and for matching analysis of the log source and the log format, the method mainly comprises the following steps:

mode 1: on the basis of the original project and the original code, processing logic aiming at the new log format is added in the analysis method, so that the compatible analysis of the new log format is completed.

Mode 2: and restarting a new project, completing a series of original processes such as analysis and superposition statistics of a newly added log format, and completing the same service processing by a plurality of applications.

In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art:

in the current pursuit of efficiency and code quality, the compatible analysis of the newly added log format is a problem to be solved urgently for faster and more elegant iteration development. On the basis of the original project and method, the mode 1 is the most direct and effective mode for adding the processing logic of a new log format, but the influence of the mode on the original processing logic and the invasiveness of the original code are obvious, the influence degree on the original service logic is avoided as much as possible in iterative development, and the existing processing scheme is upgraded and modified as far as possible by using the noninvasiveness, so that the project analyzes the newly added log format, and the required data format output is obtained. In addition, the mode 2 is non-invasive to the original project and the original method, but for the business processing and repeated project development of the same scene, the code redundancy is almost doubled, the development time is consumed, and the mode is only stopped in the planning and is an iterative development scheme which is almost not adopted.

Disclosure of Invention

In view of this, embodiments of the present invention provide a method and an apparatus for traffic statistics, which can solve the problem of dynamically and quickly iteratively increasing non-intrusive analytic capability related to traffic real-time monitoring in big data analysis.

To achieve the above object, according to an aspect of the embodiments of the present invention, a method for traffic statistics is provided, including receiving a browsing log to be analyzed to obtain a log format of the browsing log to be analyzed; matching the log format of the browsing log to be analyzed according to a preset interface list; and acquiring a corresponding browsing log processing implementation code in a preset interface list, and processing the browsing log to be analyzed to obtain flow data.

Optionally, the method further comprises:

defining a method of the requirement of analyzing the browsing log in each format into an interface;

defining an interface list of the analysis capability of the browsing log in the implementation class of each interface;

and connecting the interface lists analyzed by all the browsing logs in a file configuration mode to obtain a preset interface list.

Optionally, the method further comprises:

when the requirement of analysis of the browsing logs in the new format is added in the preset interface list, the realization of the analysis of the browsing logs in the new format is registered as Bean, then the Bean is configured into a list, and the list is associated with the definition of the preset interface list in an injection mode.

Optionally, the method further comprises:

the browsing log to be analyzed is an unstructured browsing log, and the flow data is structured real-time flow data.

In addition, according to an aspect of the embodiments of the present invention, there is provided a traffic statistic apparatus, including a receiving module, configured to receive a browsing log to be analyzed, so as to obtain a log format of the browsing log to be analyzed; the matching module is used for matching the log format of the browsing log to be analyzed according to a preset interface list; and the analysis module is used for acquiring the corresponding browsing log processing implementation code in the preset interface list and processing the browsing log to be analyzed to obtain the flow data.

Optionally, the matching module is further configured to:

Optionally, the browsing log to be analyzed is an unstructured browsing log, and the traffic data is structured real-time traffic data.

According to another aspect of the embodiments of the present invention, there is also provided an electronic device, including:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the method of any of the embodiments of traffic statistics described above.

According to another aspect of an embodiment of the present invention, there is also provided a computer readable medium, on which a computer program is stored, which when executed by a processor, implements the method according to any of the above embodiments based on flow statistics.

One embodiment of the above invention has the following advantages or benefits: the method comprises the steps of receiving a browsing log to be analyzed to obtain a log format of the browsing log to be analyzed; matching the log format of the browsing log to be analyzed according to a preset interface list; and acquiring a corresponding browsing log processing implementation code in a preset interface list, and processing the browsing log to be analyzed to obtain flow data. Therefore, the invention provides a noninvasive expansion solution for analyzing the multi-application, multi-channel and multi-format browsing logs, namely, the invention is suitable for the application-variable log sources and log formats, and provides a quick and noninvasive solution for dynamically increasing the analysis capability.

Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.

Drawings

The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:

FIG. 1 is a schematic diagram of a main flow of a method of traffic statistics according to an embodiment of the invention;

fig. 2 is a schematic view of a main flow of a method of traffic statistics according to a referential embodiment of the present invention;

FIG. 3 is a schematic diagram of the main blocks of an apparatus for traffic statistics according to an embodiment of the present invention;

FIG. 4 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;

fig. 5 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.

Detailed Description

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

Fig. 1 is a schematic diagram of a main flow of a method for traffic statistics according to an embodiment of the present invention, where the method for traffic statistics may include:

step S101, receiving the browsing log to be analyzed to obtain the log format of the browsing log to be analyzed.

And step S102, matching the log format of the browsing log to be analyzed according to a preset interface list.

As an embodiment, the interface list may be preset, specifically, the process is as follows:

firstly, defining a method of each format for the requirement of the analysis of the browsing log as an interface, and then defining an interface list of the analysis capability of the browsing log in the implementation class of each interface. And then, connecting the interface lists analyzed by all the browsing logs in a file configuration mode to obtain a preset interface list.

It should be noted that if a requirement for parsing the travel log in the new format is added to the preset interface list, the implementation of parsing the travel log in the new format is registered as Bean, and then the Bean is configured into a list, and the list is associated with the definition of the preset interface list in an injection manner. All objects in Spring are managed by a Spring core container, and all objects are collectively referred to as beans.

Step S103, acquiring a corresponding browsing log processing implementation code in a preset interface list, and processing the browsing log to be analyzed to obtain flow data.

In an embodiment, the browsing log to be analyzed is an unstructured browsing log, and the traffic data is structured real-time traffic data.

According to the various embodiments described above, it can be seen that the browse log analysis mode provided by the present invention can perform circular matching analysis on the interface list until the corresponding log format is matched, so as to obtain the corresponding browse log processing implementation code. Meanwhile, the requirement for analyzing the browsing logs is defined as an interface, and a connected interface list is formed, so that non-invasive flow statistics is realized for the browsing logs in various formats and newly added formats.

Fig. 2 is a schematic diagram of a main flow of a method of traffic statistics according to a referential embodiment of the present invention, which may include:

step S201, defining a method for analyzing the browsing log in each format as an interface.

Preferably, the parsing of the travel log has the same entry (i.e. all parameters included in the travel log to be parsed are unstructured, such as character strings) and the same exit (i.e. the parsed real-time traffic data is structured, such as the result return value is an object array or null), so the method required by the travel log parsing can be defined as an interface. Where an interface refers to a reference type that defines an agreement.

Step S202, an interface list of the analysis capability of the browsing log is defined in the implementation class of each interface.

In an embodiment, each interface list may implement different log parsing capabilities for different log formats.

Step S203, connecting all the interface lists analyzed by the browsing logs in a file configuration manner.

In the embodiment, the interface lists for parsing the browsing logs are connected in a configuration file manner, that is, the browsing log processing implementation codes for different log formats (for example, processing the browsing logs of a PC end, a mobile end, an application end, and the like) are added through the configuration file, so that the original browsing log parsing method is implemented without intrusion. The implementation refers to the description or behavior of a specific flow for performing logical or physical operations on an agreement.

Further, if a new type of format is added to the connected interface List, the new type of format may be implemented by Bean-structured scanning of Spring, that is, the implementation of parsing the browsing log in the new type of format is registered as a Bean, and then configured as a List (the List is a List), and the List is associated with the definition of the connected interface List by means of injection.

Step S204, receiving the browsing log to be analyzed to obtain the log format of the browsing log to be analyzed.

Step S205, according to the connected interface list, determining whether the log format of the browsing log to be analyzed is successfully matched, if so, performing step S206, otherwise, exiting the process.

In the embodiment, in the connected interface list, the log formats of the incoming browsing logs (e.g., character strings) to be parsed are matched, if the matching is successful, that is, a corresponding log format exists (which is not null), step S206 is performed, if the matching is unsuccessful, that is, a corresponding log format does not exist, that is, the processing result is null directly, and the process is exited.

Step S206, acquiring the corresponding browsing log processing implementation code in the connected interface list so as to process the browsing log to be analyzed.

Step S207, obtain non-empty structured traffic data.

Fig. 3 is a device for traffic statistics according to an embodiment of the present invention, and as shown in fig. 3, the device for traffic statistics includes a receiving module 301, a matching module 302, and an analyzing module 303. The receiving module 301 receives the browsing log to be analyzed to obtain a log format of the browsing log to be analyzed. The matching module 302 matches the log format of the browsing log to be analyzed according to a preset interface list. The parsing module 303 obtains a corresponding browsing log processing implementation code in a preset interface list, and processes a browsing log to be parsed to obtain flow data.

Preferably, the matching module 302 may preset an interface list, specifically as follows:

Preferably, if the requirement of parsing the browsing log in the new format is added to the preset interface list, the matching module 302 registers the implementation of parsing the browsing log in the new format as Bean, and then configures the Bean into a list, and associates the list with the definition of the preset interface list in an injection manner.

In addition, it is worth to be noted that the browsing log to be analyzed is an unstructured browsing log, and the traffic data is structured real-time traffic data.

It should be noted that, in the implementation of the apparatus for flow rate statistics of the present invention, the details of the method for flow rate statistics are already described in detail, and therefore, the repeated descriptions herein are not repeated.

Fig. 4 shows an exemplary system architecture 400 of a method or apparatus for traffic statistics to which embodiments of the present invention may be applied.

As shown in fig. 4, the system architecture 400 may include

terminal devices

401, 402, 403, a network 404, and a server 405. The network 404 serves as a medium for providing communication links between the

terminal devices

401, 402, 403 and the server 405. Network 404 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.

A user may use

terminal devices

401, 402, 403 to interact with a server 405 over a network 404 to receive or send messages or the like. The

terminal devices

401, 402, 403 may have installed thereon various communication client applications, such as shopping-like applications, web browser applications, search-like applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).

The

terminal devices

401, 402, 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 405 may be a server providing various services, such as a background management server (for example only) providing support for shopping websites browsed by users using the

terminal devices

401, 402, 403. The backend management server may analyze and perform other processing on the received data such as the product information query request, and feed back a processing result (for example, target push information, product information — just an example) to the terminal device.

It should be noted that the method for traffic statistics provided by the embodiment of the present invention is generally executed by the server 405, and accordingly, the device for traffic statistics is generally disposed in the server 405.

It should be understood that the number of terminal devices, networks, and servers in fig. 4 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Referring now to FIG. 5, shown is a block diagram of a computer system 500 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.

As shown in fig. 5, the computer system 500 includes a Central Processing Unit (CPU)501 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)502 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 503. In the RAM503, various programs and data necessary for the operation of the system 500 are also stored. The CPU501, ROM502, and RAM503 are connected to each other via a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.

The following components are connected to the I/O interface 505: an input portion 506 including a keyboard, a mouse, and the like; an output portion 507 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. The driver 510 is also connected to the I/O interface 505 as necessary. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as necessary, so that a computer program read out therefrom is mounted into the storage section 508 as necessary.

In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 509, and/or installed from the removable medium 511. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 501.

It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a receiving module, a matching module, and an analyzing module. Wherein the names of the modules do not in some cases constitute a limitation of the module itself.

As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: receiving a browsing log to be analyzed to obtain a log format of the browsing log to be analyzed; matching the log format of the browsing log to be analyzed according to a preset interface list; and acquiring a corresponding browsing log processing implementation code in a preset interface list, and processing the browsing log to be analyzed to obtain flow data.

According to the technical scheme of the embodiment of the invention, the problem of dynamically and quickly iteratively increasing the non-invasive analysis capability in real-time flow monitoring in big data analysis can be solved.

The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method of traffic statistics, comprising:

receiving a browsing log to be analyzed to obtain a log format of the browsing log to be analyzed;

matching the log format of the browsing log to be analyzed according to a preset interface list;

and acquiring a corresponding browsing log processing implementation code in a preset interface list, and processing the browsing log to be analyzed to obtain flow data.

2. The method of claim 1, further comprising:

3. The method of claim 2, further comprising:

4. The method of claim 1, further comprising:

5. An apparatus for flow statistics, comprising:

the receiving module is used for receiving the browsing logs to be analyzed so as to obtain the log format of the browsing logs to be analyzed;

the matching module is used for matching the log format of the browsing log to be analyzed according to a preset interface list;

and the analysis module is used for acquiring the corresponding browsing log processing implementation code in the preset interface list and processing the browsing log to be analyzed to obtain the flow data.

6. The apparatus of claim 5, wherein the matching module is further configured to:

7. The apparatus of claim 6, wherein the matching module is further configured to:

8. The apparatus of claim 5, wherein the travel log to be parsed is an unstructured travel log and the traffic data is structured real-time traffic data.

9. An electronic device, comprising:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-4.

10. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-4.