CN108616415B - Data association method and device - Google Patents
Data association method and device Download PDFInfo
- Publication number
- CN108616415B CN108616415B CN201810219031.4A CN201810219031A CN108616415B CN 108616415 B CN108616415 B CN 108616415B CN 201810219031 A CN201810219031 A CN 201810219031A CN 108616415 B CN108616415 B CN 108616415B
- Authority
- CN
- China
- Prior art keywords
- data
- user
- network behavior
- online
- online data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/02—Capturing of monitoring data
- H04L43/028—Capturing of monitoring data by filtering
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0805—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability
- H04L43/0817—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability by checking functioning
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/08—Network architectures or network communication protocols for network security for authentication of entities
- H04L63/0876—Network architectures or network communication protocols for network security for authentication of entities based on the identity of the terminal or configuration, e.g. MAC address, hardware or software configuration or device fingerprint
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/08—Network architectures or network communication protocols for network security for authentication of entities
- H04L63/0892—Network architectures or network communication protocols for network security for authentication of entities by using authentication-authorization-accounting [AAA] servers or protocols
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/52—Network services specially adapted for the location of the user terminal
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/535—Tracking the activity of the user
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/54—Presence management, e.g. monitoring or registration for receipt of user log-on information, or the connection status of the users
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L61/00—Network arrangements, protocols or services for addressing or naming
- H04L61/50—Address allocation
- H04L61/5007—Internet protocol [IP] addresses
- H04L61/5014—Internet protocol [IP] addresses using dynamic host configuration protocol [DHCP] or bootstrap protocol [BOOTP]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computing Systems (AREA)
- Business, Economics & Management (AREA)
- Accounting & Taxation (AREA)
- Environmental & Geological Engineering (AREA)
- Power Engineering (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The disclosure relates to a data association method and device. The method comprises the following steps: acquiring user authentication data and terminal online data according to a first frequency, integrating the user authentication data and the terminal online data comprising the same identification to obtain user online data, and persisting the user online data into a user online data table; acquiring a flow log in real time, performing streaming data processing on the flow log to obtain network behavior data, and persisting the network behavior data into a network behavior data table; according to the second frequency, the user online data, the network behavior data and the access equipment position data which comprise the same identification are correlated from the user online data table, the network behavior data table and the access equipment position data table, so that the positions of the user and the user in the wireless network and the network behavior can be correlated, data support is provided for analyzing the network behavior of the user, and higher-quality network service or security protection guarantee is provided for the user.
Description
Technical Field
The present disclosure relates to the field of communications technologies, and in particular, to a data association method and apparatus.
Background
At present, wireless network and network behavior auditing equipment are deployed in places such as schools, venues, cities and the like, and network behaviors of users under the coverage of the wireless network are monitored while internet access services are provided for the users. Through big data analysis, the network service or security protection guarantee with higher quality is provided for the user. The network behavior auditing device may refer to a device capable of monitoring a network behavior of a user under network coverage, such as a Deep Packet Inspection (DPI) device. For example, a network behavior auditing device may be installed at a primary network outlet for monitoring network behavior of users of the primary network outlet.
In the related art, a wireless network automatically allocates an IP (Internet Protocol) address to a user by using a DHCP (Dynamic Host Configuration Protocol), so that the user is allocated to a different IP address each time the user logs in the wireless network. Because the user does not have a long-term corresponding relationship with the IP address, the network behavior auditing device can only monitor the network behavior of each IP address, but cannot monitor the network behavior of each user.
At present, the network behavior auditing equipment can be connected with an authentication system to realize the association of a user and an IP address, thereby realizing the monitoring of the network behavior of the user. However, the data volume of the flow log is large, and the efficiency of offline association is low, so that the timeliness of offline association is poor, and the user cannot be associated with the IP address in real time. Furthermore, even if the association of the user with the IP address is implemented, the location of the user cannot be determined.
Disclosure of Invention
In view of this, the present disclosure provides a data association method and an apparatus, so as to solve the problem of low efficiency of associating a user, a location of the user in a wireless network, and a network behavior in the related art.
According to an aspect of the present disclosure, there is provided a data association method, the method including:
acquiring user authentication data and terminal online data according to a first frequency, integrating the user authentication data and the terminal online data which comprise the same identification to obtain user online data, and persisting the user online data into a user online data table;
acquiring a flow log in real time, performing streaming data processing on the flow log to obtain network behavior data, and persisting the network behavior data into a network behavior data table;
and according to a second frequency, associating the user online data, the network behavior data and the access equipment position data which comprise the same identification from the user online data table, the network behavior data table and the access equipment position data table.
According to another aspect of the present disclosure, there is provided a data association apparatus, the apparatus including:
the first processing module is used for acquiring user authentication data and terminal online data according to a first frequency, integrating the user authentication data and the terminal online data which comprise the same identification to obtain user online data, and persisting the user online data into a user online data table;
the second processing module is used for acquiring a flow log in real time, performing streaming data processing on the flow log to obtain network behavior data, and persisting the network behavior data into a network behavior data table;
and the data association module is used for associating the user online data, the network behavior data and the access equipment position data which comprise the same identification from the user online data table, the network behavior data table and the access equipment position data table according to a second frequency.
According to another aspect of the present disclosure, there is provided a data association apparatus including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to perform the above method.
According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the above-described method.
The disclosed data association method and device, according to the first frequency, obtains user authentication data and terminal on-line data, and integrates the user authentication data and terminal on-line data including the same identification, obtains user on-line data, persists the user on-line data into a user on-line data table, obtains a flow log in real time, and performs streaming data processing on the flow log to obtain network behavior data, persists the network behavior data into a network behavior data table, according to the second frequency, associates the user on-line data, the network behavior data and the access device position data including the same identification from the user on-line data table, the network behavior data table and an access device position data table, thereby quickly and timely associating the user, the user position under the wireless network and the network behavior, and providing data support for analyzing the network behavior of the user, therefore, better network service or security guarantee is provided for the user.
Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, which proceeds with reference to the accompanying drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.
FIG. 1 shows a schematic diagram of data association according to an embodiment of the invention.
FIG. 2 shows a flow diagram of a data association method according to an embodiment of the present disclosure.
FIG. 3 shows a schematic diagram of data association according to an embodiment of the invention.
FIG. 4 shows a block diagram of a data association apparatus according to an embodiment of the present disclosure.
Fig. 5 shows a block diagram of a data association apparatus according to an embodiment of the present disclosure.
FIG. 6 shows a block diagram of a data association apparatus according to an embodiment of the invention.
Detailed Description
Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. In the drawings, like reference numbers can indicate functionally identical or similar elements. While the various aspects of the embodiments are presented in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.
The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.
Furthermore, in the following detailed description, numerous specific details are set forth in order to provide a better understanding of the present disclosure. It will be understood by those skilled in the art that the present disclosure may be practiced without some of these specific details. In some instances, methods, means, elements and circuits that are well known to those skilled in the art have not been described in detail so as not to obscure the present disclosure.
In order to make those skilled in the art better understand the technical solutions in the embodiments of the present invention, a brief description will be given below of some technical terms involved in the embodiments of the present invention.
The flow log may refer to a log file generated by the network behavior auditing device monitoring the network behavior of the user under the network coverage. The network behavior auditing device may refer to a device capable of monitoring network behaviors of users under network coverage, such as a DPI device. Network behavior may refer to various operations performed over a network, including accessing websites, mailing, uploading/downloading, instant messaging, and network games, to which the present disclosure is not limited.
By performing streaming data processing on the flow log, network behavior data can be obtained. The network behavior data table may refer to a database for storing network behavior data. Network behavior data may refer to data that records a user's network behavior. Table 1 shows network behavior data according to an embodiment of the invention. As shown in table 1, the network behavior data includes, but is not limited to, the following: account Name (User Name), User IP address (User IP), application Name (App Name), upload traffic (UP), download traffic (Down), statistics start Time (Create Time), and statistics End Time (End Time).
It should be noted that, the wireless network automatically allocates an IP address to the user by using DHCP, and the user is allocated a different IP address each time he logs in the wireless network. Therefore, as shown in table 1, although the network behavior data includes the account name and the user IP address, the account name does not include the real account name, but is replaced with the user IP address. Whereby the network behaviour data can only correspond to the user IP address and not to a specific user.
TABLE 1
| Field(s) | Name of field | Description of the invention |
| User Name | Name of account | Defaults to the user IP address |
| User IP | User IP address | IP address acquired by user using network |
| App Name | Application name | Using application names |
| UP | Upload flow rate | Upload traffic for a user |
| Down | Download traffic | Download traffic for a user |
| Create Time | Counting the start time | Counting the start time |
| End Time | End time of statistics | End time of statistics |
| …… | …… | …… |
The user authentication data may refer to data that records authentication of the user. An AAA (Authentication Authorization Accounting) server may refer to a device that is capable of handling user access requests, providing Authentication Authorization, and Accounting services. The AAA server authenticates, authorizes, and accounts the user, and stores the user authentication data in a general relational database, such as Mysql or Oracle.
User authentication data may be obtained by the AAA server. Table 2 shows user authentication data according to an embodiment of the present invention. As shown in table 2, the user authentication data includes, but is not limited to, the following: account Name (User Name), User Name (Full Name), terminal MAC address (User MAC), User IP address (Framed IP), access Start Time (Online Time Start), and access End Time (Online Time End).
TABLE 2
| Field(s) | Name of field | Description of the invention |
| User Name | Name of account | Account for logging in network |
| Full Name | User name | Real name of account number |
| User Mac | Terminal MAC address | User terminal MAC address |
| Framed IP | User IP address | IP address acquired by user using network |
| Online Time Start | Access start time | Time of account login |
| Online Time End | End time of access | Time of account exit |
| …… | …… | …… |
The terminal presence data may refer to data recording the presence of the terminal. A network management device may refer to a device that is capable of planning, controlling, and monitoring a network to ensure that the network is operating properly. An Access device may refer to a device capable of providing a user with Access to a network service, such as an AP (Wireless Access Point). The network management device stores the terminal online data into a general relational database, such as Mysql or Oracle.
The terminal online data can be acquired through the network management equipment. Table 3 shows terminal presence data according to an embodiment of the present invention. As shown in table 3, the terminal online data includes, but is not limited to, the following: account Name (User Name), terminal MAC address (User MAC), User IP address (User IP), Service Set Identifier (SSID), Channel (Channel), AP Serial number (AP Serial ID), and AP IP address (AP IP).
TABLE 3
| Field(s) | Name of field | Description of the invention |
| User Name | Name of account | This data field is typically available only when online and authenticated in conjunction |
| User Mac | Terminal MAC address | User terminal MAC address |
| User IP | User IP address | User IP address |
| SSID | Service set identification | Service set identification |
| Channel | Channel with a plurality of channels | Channel with a plurality of channels |
| AP Serial ID | AP sequence number | AP sequence number |
| AP IP | IP address of AP | IP address of AP |
| …… | …… | …… |
Note that, as shown in table 3, although the terminal online data includes an account name, the account name includes a real account name only when the user authentication data and the terminal online data are linked. And if the user authentication data and the terminal online data are not linked, the account name is null, or the terminal online data does not comprise the account name.
The user online and offline are not fixed, the AAA server generates user authentication data according to the user online and offline, and the network management equipment generates terminal online data according to the user online and offline. FIG. 1 shows a schematic diagram of data association according to an embodiment of the invention. As shown in fig. 1, the user authentication data includes a user IP address, a terminal MAC address, and an account name (as shown in table 2), and the terminal online data includes a user IP address, a terminal MAC address, and a serial number of the AP (as shown in table 3).
Therefore, the user authentication data and the terminal online data which comprise the same user IP address or the same terminal MAC address are integrated to obtain the user online data, and the user online data are persisted into a user online data table. The user presence data table may refer to a database for storing user presence data. User presence data may refer to data that records the presence of a user.
And the user online data can be obtained through the user online data table. Table 4 shows user online data according to an embodiment of the invention. As shown in Table 4, the user online data includes, but is not limited to, the following: account Name (User Name), User IP address (User IP), and AP Serial number (AP Serial ID). Through the user online data, it can be determined through which AP the user accesses the network, and thus the user's location can be determined by determining the location of the AP.
TABLE 4
| Field(s) | Name of field | Description of the invention |
| User Name | Name of account | Account for logging in network |
| User IP | User IP address | IP address acquired by user using network |
| AP Serial ID | AP sequence number | AP sequence number |
| …… | …… | …… |
An access device location data table may refer to a database for storing access device location data. Access device location data may refer to data that records the location of the access device. The location of the access device may refer to information that can indicate the location of the access device. The access device location may be described by latitude and longitude, for example (22 ° 42 '09 "N, 112 ° 13' 19" E); and may be described by text, such as the city of Beijing, Fengchi district XX street XX Hospital, to which the disclosure is not limited.
Table 5 shows access device location data according to an embodiment of the invention. As shown in table 5, the access device location data includes, but is not limited to, the following: access device serial number and access device location. Where the access device is an AP, the AP location data includes, but is not limited to, the following: the AP sequence number (AP Serial ID) and the AP Location (AP Location). Through the AP position data, the position of each AP can be determined, and therefore the position of the user can be determined through the positions of the APs.
TABLE 5
| AP Serial ID | AP Location |
| 001 | (22°42′09″N,112°13′19″E) |
| 002 | (31°14′36″N,115°33′19″E) |
| 003 | (33°20′36″N,110°32′10″E) |
| 004 | (24°20′02″N,101°32′10″E) |
| …… | …… |
FIG. 2 shows a flow diagram of a data association method according to an embodiment of the present disclosure. As shown in fig. 2, the method includes the following steps.
In step S21, the user authentication data and the terminal online data are obtained according to the first frequency, and the user authentication data and the terminal online data including the same identifier are integrated to obtain the user online data, and the user online data is persisted in the user online data table.
The first frequency may refer to the number of times of acquiring the user authentication data and the terminal online data in unit time. For example, the user authentication data and the terminal online data are acquired every 5 minutes or 10 minutes. The value of the first frequency may be set empirically, and the disclosure is not limited thereto.
In one implementation, acquiring user authentication data and terminal online data at a first frequency includes: and according to the first frequency, respectively acquiring user authentication data and terminal online data through an Application Programming Interface (API).
In one implementation, integrating the user authentication data and the terminal online data including the same identifier to obtain the user online data includes: integrating user authentication data and terminal online data comprising the same user IP address to obtain user online data; the user online data comprises: user IP address, account name and access device serial number.
It should be noted that, although the integration processing of the user authentication data and the terminal online data is described above by taking the same user IP address as an example, those skilled in the art will understand that the present disclosure should not be limited thereto. The technical personnel in the field can flexibly set how to integrate the user authentication data and the terminal online data according to the actual application scene. For example, the user authentication data and the terminal online data including the same terminal MAC address are integrated to obtain the user online data.
In step S22, the traffic log is obtained in real time, and the traffic log is subjected to streaming data processing to obtain network behavior data, and the network behavior data is persisted in the network behavior data table.
The flow log has high real-time performance, and the data volume per second is generally 1-10 ten thousand or more. The flow log is broadcasted by a User Datagram Protocol (UDP) message to form flow data, and the flow data is processed by the flow data to obtain network behavior data.
The stream data can refer to data continuously generated by thousands of data sources, and has the characteristics of continuous arrival of data in real time, independent arrival sequence, numerous data sources, complex formats, large data size and the like. The streaming data processing can refer to the real-time processing process of streaming data, and generally comprises the processes of data real-time acquisition, data real-time processing, real-time query service and the like.
According to the data association method, the flow data processing is adopted for the flow logs, the consumption of computing performance can be reduced, the timeliness of data is improved, and the association of data of hundred million levels of data volume is realized.
In one implementation, performing streaming data processing on the traffic log to obtain network behavior data includes: and distributing the flow logs acquired in real time to a message queue, and analyzing the flow logs output by the message queue according to an analysis protocol to obtain network behavior data.
A message queue may refer to a container that holds messages during transmission of the messages. For example, the message queue may be Kafka, which is a high throughput distributed publish-subscribe messaging system.
In step S23, the user presence data, the network behavior data, and the access device location data including the same identifier are associated from the user presence data table, the network behavior data table, and the access device location data table according to the second frequency.
The second frequency may refer to the number of times of acquiring the user online data, the network behavior data, and the access device location data in unit time. For example, user online data, network behavior data, and access device location data are acquired every 10 minutes or 15 minutes. The value of the second frequency may be set empirically, and the disclosure is not limited thereto.
In one implementation, the second frequency is less than or equal to the first frequency. It will be appreciated that the frequency of generating the user presence data may be approximately equal to the first frequency, i.e. the frequency of acquiring the user authentication data and the terminal presence data. Therefore, the second frequency is set to be less than or equal to the first frequency, and the user online data which can be acquired can be ensured to exist when the user online data, the network behavior data and the access equipment position data are acquired every time, so that the matching rate of the user online data and the network behavior data is improved.
In one implementation, associating the user presence data, the network behavior data, and the access device location data that include the same identification includes: associating the user online data and the network behavior data comprising the same user IP address; and associating the user online data and the access equipment position data which comprise the same access equipment serial number.
FIG. 3 shows a schematic diagram of data association according to an embodiment of the invention. As shown in fig. 3:
an IMC (Intelligent Management Center) platform may have both functions of an AAA server and a network Management device. And acquiring user authentication data and terminal online data generated by the IMC platform by the API according to the first frequency, integrating the user authentication data and the terminal online data comprising the same user IP address to obtain user online data, and persisting the user online data into a user online data table. The persistent user online data can be read or called at any time.
And broadcasting the flow log generated by the DPI equipment in a UDP message to form flow data. And the Logstash collects the flow logs in real time, and distributes the flow logs collected in real time to the Kafka for caching according to the current load of each queue included by the Kafka. And the Spark Streaming analyzes the flow log output by the Kafka according to an analysis protocol to obtain network behavior data. And persisting the network behavior data into a network behavior data table. The persistent network behavior data can be read or called at any time.
The Spark Streaming associates the user online data and the network behavior data including the same user IP address from the user online data table, the network behavior data table and the access device location data table according to the second frequency, associates the user online data and the access device location data including the same access device serial number, and stores the associated user online data, network behavior data and access device location data into an ES (Elastic Search).
Wherein, the Logstash is open source log collection analysis software. The processing performance of the Logstash can reach 7000 pieces/second. When the data volume of the flow log exceeds 7000 pieces/second, a special acquisition program can be considered to be adopted to acquire the flow log. Kafka is open source message queue software. Kafka can carry out cluster deployment, and the application of the cluster can greatly improve the processable data volume. Kafka transmits the data in the queue to Spark Streaming for batch processing, and debugging the batch size is performed through a moving window of Spark Streaming so as to achieve optimal processing performance. Through the association of micro-batches of data, the throughput of data is greatly increased. When the Spark Streaming processing speed exceeds the receiving speed of the program by 10-15%, the performance is optimal, and even if the data transmission has a peak period, the Spark Streaming can process the data in the Kafka in time through the buffering effect of the Kafka. The ES is a search server based on Lucene, can provide a full-text search engine with distributed multi-user capability, and has the advantages of real-time search, stability, reliability, rapidness, convenience and the like.
According to the data association method disclosed by the invention, as the flow log has higher real-time performance, the data volume per second is generally 1-10 ten thousand or more. If user authentication data and terminal presence data are requested once per receipt of a traffic log, the network management device and AAA server cannot complete such high frequency requests. Therefore, the user authentication data and the terminal online data which comprise the same identification are integrated to obtain the user online data, and then the user online data, the network behavior data and the access equipment position data which comprise the same identification are associated. Therefore, the efficiency of matching the user online data and the network behavior data can be increased, and the method is a very efficient processing mode within an error allowable range.
As an example, User authentication data { User Name: zhangsan, Full Name: Zhang III, User Mac: Mac1, Framed IP:1.1.1, Online Time Start:10:02:00, Online Time End:11:12:00} are obtained at a first frequency, and terminal Online data { User Name: zhangsan, User Mac: Mac1, User IP: 1.1.1.1.1, SSID: SSID1, Channel: Channel1, AP Serial ID:002, AP IP:2.2.2.2 }. Therefore, the User authentication data and the terminal online data which comprise the same User IP address { User IP:1.1.1.1} are integrated to obtain the User online data { User IP:1.1.1.1, User Name: zhangsan, AP Serial ID:002 }.
Network behavior data { User Name:1.1.1.1, User IP:1.1.1, App Name: WeiBo, UP:50M, Down:100M, Create Time:10:12:00, End Time:10:22:00} in the network behavior data table. According to a second frequency, associating the User online data and the network behavior data comprising the same User IP address { User IP:1.1.1.1} with each other, and associating the User online data and the AP position data (shown in Table 5) comprising the same AP Serial number { AP Serial ID:002} with each other to obtain { User Name: zhangsan, User IP:1.1.1.1, App Name: WeiBo, UP:50M, Down:100M, Create Time:10:12:00, End Time:10:22:00, AP Serial ID:002, AP Location (31 degree 14 '36' N,115 degree 33 '19' E) }
The data association method can quickly and timely associate the user, the position of the user under the wireless network and the network behavior, and provide data support for analyzing the network behavior of the user, so that better network service or security protection guarantee and the like can be provided for the user.
FIG. 4 shows a block diagram of a data association apparatus according to an embodiment of the present disclosure. As shown in fig. 4, the apparatus includes:
the first processing module 41 is configured to obtain user authentication data and terminal online data according to a first frequency, integrate the user authentication data and the terminal online data including the same identifier to obtain user online data, and persist the user online data into a user online data table; the second processing module 42 is configured to obtain a traffic log in real time, perform streaming data processing on the traffic log to obtain network behavior data, and persist the network behavior data into a network behavior data table; and a data association module 43, configured to associate, according to the second frequency, the user online data, the network behavior data, and the access device location data that include the same identifier from the user online data table, the network behavior data, and the access device location data table.
Fig. 5 shows a block diagram of a data association apparatus according to an embodiment of the present disclosure. As shown in fig. 5:
in one implementation, the first processing module 41 includes: the collecting submodule 411 is configured to obtain the user authentication data and the terminal online data through an application programming interface API according to the first frequency.
In one implementation, the second processing module 42 includes: the parsing submodule 421 is configured to distribute the traffic log obtained in real time to a message queue, and parse the traffic log output by the message queue according to a parsing protocol to obtain the network behavior data.
In one implementation, the user authentication data includes a user IP address and an account name; the terminal online data comprises a user IP address and an access equipment serial number; the network behavior data comprises a user IP address, an application name, uploading flow, downloading flow, counting start time and counting end time; the access device location data includes an access device serial number and a corresponding access device location.
In one implementation, the first processing module 41 includes: an integrating sub-module 412, configured to integrate the user authentication data and the terminal online data that include the same user IP address to obtain the user online data; the user online data comprises: user IP address, account name and access device serial number.
In one implementation, the data association module 43 includes: a first associating submodule 431, configured to associate the user online data and the network behavior data that include the same user IP address; a second associating submodule 432, configured to associate the user online data and the access device location data that include the same access device serial number.
The data association device can quickly and timely associate the user, the position of the user under the wireless network and the network behavior, and provide data support for analyzing the network behavior of the user, so that better network service or security protection guarantee and the like can be provided for the user.
FIG. 6 shows a block diagram of a data association apparatus according to an embodiment of the invention. Referring to fig. 6, the apparatus 900 may include a processor 901, a machine-readable storage medium 902 having stored thereon machine-executable instructions. The processor 901 and the machine-readable storage medium 902 may communicate via a system bus 903. Also, the processor 901 performs the data association method described above by reading machine-executable instructions in the machine-readable storage medium 902 corresponding to the data association logic.
The machine-readable storage medium 902 referred to herein may be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, and the like. For example, the machine-readable storage medium may be: a RAM (random Access Memory), a volatile Memory, a non-volatile Memory, a flash Memory, a storage drive (e.g., a hard drive), a solid state drive, any type of storage disk (e.g., an optical disk, a dvd, etc.), or similar storage medium, or a combination thereof.
Having described embodiments of the present disclosure, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein were chosen in order to best explain the principles of the embodiments, the practical application, or technical improvements to the techniques in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
Claims (12)
1. A method for associating data, the method comprising:
acquiring user authentication data and terminal online data according to a first frequency, integrating the user authentication data and the terminal online data which comprise the same identification to obtain user online data, and persisting the user online data into a user online data table; the user authentication data comprises a user IP address and an account name; the terminal online data comprises a user IP address and an access equipment serial number;
acquiring a flow log in real time, performing streaming data processing on the flow log to obtain network behavior data, and persisting the network behavior data into a network behavior data table;
according to a second frequency, associating the user online data, the network behavior data and the access equipment position data which comprise the same identification from the user online data table, the network behavior data table and the access equipment position data table; the second frequency is less than or equal to the first frequency.
2. The method of claim 1, wherein obtaining user authentication data and terminal presence data at a first frequency comprises:
and respectively acquiring the user authentication data and the terminal online data through an Application Programming Interface (API) according to the first frequency.
3. The method of claim 1, wherein performing streaming data processing on the traffic log to obtain network behavior data comprises:
and distributing the flow logs acquired in real time to a message queue, and analyzing the flow logs output by the message queue according to an analysis protocol to obtain the network behavior data.
4. The method according to any one of claims 1 to 3,
the network behavior data comprises a user IP address, an application name, uploading flow, downloading flow, counting start time and counting end time;
the access device location data includes an access device serial number and a corresponding access device location.
5. The method of claim 4, wherein integrating the user authentication data and the terminal online data including the same identifier to obtain user online data comprises:
integrating the user authentication data and the terminal online data comprising the same user IP address to obtain the user online data;
the user online data comprises: user IP address, account name and access device serial number.
6. The method of claim 5, wherein associating the user presence data, the network behavior data, and access device location data that include the same identification comprises:
associating the user online data and the network behavior data comprising the same user IP address;
and associating the user online data and the access equipment position data which comprise the same access equipment serial number.
7. An apparatus for associating data, the apparatus comprising:
the first processing module is used for acquiring user authentication data and terminal online data according to a first frequency, integrating the user authentication data and the terminal online data which comprise the same identification to obtain user online data, and persisting the user online data into a user online data table; the user authentication data comprises a user IP address and an account name; the terminal online data comprises a user IP address and an access equipment serial number;
the second processing module is used for acquiring a flow log in real time, performing streaming data processing on the flow log to obtain network behavior data, and persisting the network behavior data into a network behavior data table;
the data association module is used for associating the user online data, the network behavior data and the access equipment position data which comprise the same identification from the user online data table, the network behavior data table and the access equipment position data table according to a second frequency; the second frequency is less than or equal to the first frequency.
8. The apparatus of claim 7, wherein the first processing module comprises:
and the acquisition submodule is used for respectively acquiring the user authentication data and the terminal online data through an Application Programming Interface (API) according to the first frequency.
9. The apparatus of claim 7, wherein the second processing module comprises:
and the analysis submodule is used for distributing the flow log acquired in real time to a message queue and analyzing the flow log output by the message queue according to an analysis protocol to obtain the network behavior data.
10. The apparatus according to any one of claims 7 to 9,
the network behavior data comprises a user IP address, an application name, uploading flow, downloading flow, counting start time and counting end time;
the access device location data includes an access device serial number and a corresponding access device location.
11. The apparatus of claim 10, wherein the first processing module comprises:
the integration submodule is used for integrating the user authentication data and the terminal online data which comprise the same user IP address to obtain the user online data;
the user online data comprises: user IP address, account name and access device serial number.
12. The apparatus of claim 11, wherein the data association module comprises:
the first association submodule is used for associating the user online data and the network behavior data which comprise the same user IP address;
and the second association submodule is used for associating the user online data and the access equipment position data which comprise the same access equipment serial number.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810219031.4A CN108616415B (en) | 2018-03-16 | 2018-03-16 | Data association method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810219031.4A CN108616415B (en) | 2018-03-16 | 2018-03-16 | Data association method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108616415A CN108616415A (en) | 2018-10-02 |
| CN108616415B true CN108616415B (en) | 2020-11-27 |
Family
ID=63659018
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810219031.4A Active CN108616415B (en) | 2018-03-16 | 2018-03-16 | Data association method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108616415B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112260965A (en) * | 2020-10-21 | 2021-01-22 | 阳光保险集团股份有限公司 | Message processing method, device, equipment and storage medium |
| CN112600948B (en) * | 2020-12-09 | 2022-05-31 | 中国电建集团华东勘测设计研究院有限公司 | Equipment and user positioning method under IPoE network access environment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101442449A (en) * | 2008-12-18 | 2009-05-27 | ***通信集团浙江有限公司 | Method for completely auditing user behaviors under centralization access mode |
| CN105451219A (en) * | 2015-12-30 | 2016-03-30 | 迈普通信技术股份有限公司 | Data integration method and apparatus |
| CN105939327A (en) * | 2016-01-19 | 2016-09-14 | 杭州迪普科技有限公司 | Auditing log generation method and device |
| CN106878092A (en) * | 2017-03-28 | 2017-06-20 | 上海以弈信息技术有限公司 | A kind of network O＆M monitor in real time of multi-source heterogeneous data fusion is presented platform with analysis |
| CN107783993A (en) * | 2016-08-25 | 2018-03-09 | 阿里巴巴集团控股有限公司 | The storage method and device of data |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070180101A1 (en) * | 2006-01-10 | 2007-08-02 | A10 Networks Inc. | System and method for storing data-network activity information |
| CN101483553B (en) * | 2009-02-24 | 2011-09-21 | 中兴通讯股份有限公司 | Audit apparatus and method for customer network behavior |
| CN105159926B (en) * | 2015-08-04 | 2019-04-16 | 百度在线网络技术（北京）有限公司 | The method and apparatus for establishing the user information correlation of user |
| CN106202253A (en) * | 2016-06-27 | 2016-12-07 | 安徽科成信息科技有限公司 | A kind of network log on-line monitoring update method |
| CN106658500A (en) * | 2016-12-12 | 2017-05-10 | 张驰 | WLAN user client flow analysis method and system based on Portal |
| CN107071900A (en) * | 2017-04-28 | 2017-08-18 | 杭州迪普科技股份有限公司 | A kind of user facility positioning method and device |
-
2018
- 2018-03-16 CN CN201810219031.4A patent/CN108616415B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101442449A (en) * | 2008-12-18 | 2009-05-27 | ***通信集团浙江有限公司 | Method for completely auditing user behaviors under centralization access mode |
| CN105451219A (en) * | 2015-12-30 | 2016-03-30 | 迈普通信技术股份有限公司 | Data integration method and apparatus |
| CN105939327A (en) * | 2016-01-19 | 2016-09-14 | 杭州迪普科技有限公司 | Auditing log generation method and device |
| CN107783993A (en) * | 2016-08-25 | 2018-03-09 | 阿里巴巴集团控股有限公司 | The storage method and device of data |
| CN106878092A (en) * | 2017-03-28 | 2017-06-20 | 上海以弈信息技术有限公司 | A kind of network O＆M monitor in real time of multi-source heterogeneous data fusion is presented platform with analysis |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108616415A (en) | 2018-10-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9143422B2 (en) | Determining network node performance data based on location and proximity of nodes | |
| US8938534B2 (en) | Automatic provisioning of new users of interest for capture on a communication network | |
| US8972612B2 (en) | Collecting asymmetric data and proxy data on a communication network | |
| US10068227B1 (en) | Methods and apparatus for authenticating identity of web access from a network element | |
| WO2012106861A1 (en) | Terminal distribution information acquisition method, data acquisition device and communication system | |
| US10447530B2 (en) | Device metering | |
| US20130232161A1 (en) | Method and Apparatus of User Recognition and Information Distribution | |
| CN104702629B (en) | The collection method and device of a kind of resource access information | |
| US20140059024A1 (en) | System and method of storage, recovery, and management of data intercepted on a communication network | |
| TWI592046B (en) | Network sharing device, system and method | |
| RU2654854C1 (en) | Method for collecting data on wireless communication device user and machine-readable medium for method implementation | |
| US20160105787A1 (en) | Method and System for Discovering User Equipment in a Network | |
| CN104599086A (en) | Online storage cabinet grid inventory management and grid resource reservation method | |
| EP2857992A1 (en) | Online data processing method, device and system | |
| CN108616415B (en) | Data association method and device | |
| US20210185507A1 (en) | Location verification and enforcement for content access devices | |
| CN113225339B (en) | Network security monitoring method and device, computer equipment and storage medium | |
| US20220408215A1 (en) | Information processing method, apparatus, and system | |
| US11902036B2 (en) | Policy and charging control (PCC) in information centric networking | |
| CN107483970B (en) | Method and equipment for determining hot live video | |
| CN109286506B (en) | Method, system and device for charging flow | |
| EP4096328A1 (en) | Network function registration method and apparatus, network function discovery method and apparatus, and device and medium | |
| US20100333178A1 (en) | System and Method for Unique User Identification via Correlation of Public and Private Data by a Third-Party | |
| CN101175038B (en) | Data stream information transmission method, communication system and equipment | |
| US20150234901A1 (en) | Data Collector and Associated Method, Computer Program and Computer Program Product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |