CN109067711B - Rapid backtracking analysis method for network data packet - Google Patents
Rapid backtracking analysis method for network data packet Download PDFInfo
- Publication number
- CN109067711B CN109067711B CN201810776879.7A CN201810776879A CN109067711B CN 109067711 B CN109067711 B CN 109067711B CN 201810776879 A CN201810776879 A CN 201810776879A CN 109067711 B CN109067711 B CN 109067711B
- Authority
- CN
- China
- Prior art keywords
- data packet
- file
- epb
- information
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L2463/00—Additional details relating to network architectures or network communication protocols for network security covered by H04L63/00
- H04L2463/121—Timestamp
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a fast backtracking analysis method of a network data packet, which comprises the following steps: step 1) creating a header block SHB containing file description extension options, an interface description block IDB and a plurality of enhanced data packet blocks EPB, filling a network original data packet in each EPB, and adding data packet information and session information extension options for each data packet; creating a self-defined time stamp index block TIB for increasing the index options of the data packets in the file; generating a pcapng file from the network data packet according to the SHB-IDB-EPB-EPB- … -EPB-EPB-TIB format; and 2) selecting a pcapng file through a backtracking time window, quickly positioning the pcapng file on an EPB through a TIB, and extracting metadata information of a data packet by using a session information expansion option of the EPB. The method can improve the backtracking analysis efficiency in the network flow backtracking analysis system.
Description
Technical Field
The invention relates to the technical field of network security, in particular to a fast backtracking analysis method of a network data packet.
Background
The continuous, efficient and safe operation of the network is the basis for the normal operation of the user services. Therefore, a network manager is required to master key indexes of service application operation at any time, find abnormality and early warn in time, and active operation, maintenance and active management are realized; when a fault occurs, the problem point can be quickly and effectively positioned, the responsibility can be clearly distinguished, and the reason can be analyzed, so that the fault time can be reduced; once the network receives an attack or a security event occurs, means and evidences are needed to realize effective positioning, analysis and evidence obtaining. The network backtracking analysis system has long-time and large-capacity data storage capacity, can store various captured statistical data such as original data packets, data streams, network sessions, application logs and the like in real time for a long time, has quick data retrieval capacity, and can conveniently perform backtracking analysis on the occurred network behaviors, application data and host data.
The existing storage format and method of the network data packet can not realize fast and efficient backtracking analysis of a large amount of data packets.
Disclosure of Invention
The invention aims to overcome the technical defects and provide a quick backtracking analysis method for a network data packet, which can improve the backtracking analysis efficiency in a network traffic backtracking analysis system.
In order to achieve the above object, the present invention provides a method for fast backtracking analysis of a network data packet, where the method includes:
step 1) creating a header block SHB containing file description extension options, an interface description block IDB and a plurality of enhanced data packet blocks EPB, filling a network original data packet in each EPB, and adding data packet information and session information extension options for each data packet; creating a self-defined time stamp index block TIB for increasing the index options of the data packets in the file; generating a pcapng file from the network data packet according to the SHB-IDB-EPB-EPB- … -EPB-EPB-TIB format;
and 2) selecting a pcapng file through a backtracking time window, quickly positioning the pcapng file on an EPB through a TIB, and extracting metadata information of a data packet by using a session information expansion option of the EPB.
As a modification of the above method, the step 1) includes:
step 1-1) creating a title block SHB, and adding a file description expansion option for the title block SHB;
step 1-2) creating an interface description block IDB;
step 1-3) creating Enhanced Packet Blocks (EPBs), wherein each EPB fills a network original packet and adds packet information and a session information extension option to each packet;
step 1-4) judging whether the EPB quantity reaches a set threshold value, if the judgment result is negative, turning to step 1-3), otherwise, entering step 1-5);
step 1-5), creating a self-defined time stamp index block TIB and adding an in-file data packet index option for the TIB;
step 1-6) judging whether the size of the written data reaches a pcap file generation threshold, if the judgment result is negative, turning to the step 1-3), otherwise, generating a pcap file according to an SHB-IDB-EPB-EPB- … -EPB-EPB-TIB format, and generating a file name by using the timestamp of a head and tail data packet recorded in the file.
As an improvement of the above method, in the file description extension option of the SHB, the information to be recorded includes, but is not limited to: magic number, file generation time, software version and collector identification;
the magic number is used for adding an identifier to the reserved network data packet file;
the file generation time is used for recording the generation time of the saved network data packet file;
the software version is used for identifying the software version for generating the file;
and the collector identifier is used for identifying the name of the collector which generates the file.
As an improvement of the above method, in the EPB session information extension option, the information to be recorded includes, but is not limited to: packet type, application layer protocol type, L3 offset, L4 offset, L4 payload length, session ID, and timestamp;
the data packet type is represented by using a 32-bit mask, the lower 28 bits identify the type of a 2-4 layer protocol stack, the 28 bits identify the direction of the data packet, 0 is a packet sent by a client, 1 is a packet sent by a server, and 29-31 bits identify the calculation mode of a TAG;
the application layer protocol type is used for identifying the application layer protocol type carried by the network data packet;
the L3 offset is used to identify the layer 3 offset of the network packet, and if not, 0 is filled;
the L4 offset is used to identify a 4-layer offset of a network packet, and if not, 0 is filled;
the L4 load length is used for identifying the 4-layer load length of the network data packet, and if not, 0 is filled;
the TAG value is used for identifying the data packet and is obtained by quintuple information of the data packet according to a certain calculation mode;
the session ID option is used for identifying the data stream corresponding to the data packet;
the timestamp option is used to identify the time interval relative to the start of the stream, taking the value of milliseconds and rounding up.
As an improvement of the above method, in the protocol custom timestamp index block TIB, the information to be recorded includes: timestamp information and offset information;
the timestamp information is used for recording the relative time relative to the starting time of the file;
the offset information is used for recording the number of bytes relative to the starting position of the file.
As a modification of the above method, the step 2) includes:
step 2-1) according to the selected backtracking analysis time window, positioning a corresponding pcapng file according to the timestamp information in the file name, and simultaneously verifying magic number options in a file header block SHB;
step 2-2) jumping to a search starting position by utilizing a timestamp and offset information recorded by TIB in the pcapng file;
and 2-3) sequentially extracting metadata information in each EPB until the end position of the search time, wherein the metadata information comprises a network original data packet and a session information expansion option for rapid analysis.
The invention has the advantages that:
1. the storage method of the invention adopts pcapng format to record files, adds expansion options in the header block SHB and the enhanced data packet block EPB, self-defines the time stamp index block TIB to support the index in the file, and simultaneously the file name contains the start and end time stamps of the data packet. The organization format of the pcapng file format is SHB-IDB-EPB-EPB- … -EPB-EPB-TIB, and due to the expandability of the pcapng format, the readability of the file cannot be influenced by adding the expansion option fields and merging/additional data;
2. after the network data packet is stored by using the method, when backtracking analysis is carried out, firstly, files are selected through a backtracking time period, then, the analysis position is quickly positioned through TIB, and then, metadata information of the data packet is extracted by using EPB (electronic packet management) extension items, so that the efficiency of backtracking analysis is improved;
3. by the method, the network flow in the selected time period can be efficiently and quickly analyzed.
Drawings
Fig. 1 is a flowchart of a network packet storage method according to embodiment 1 of the present invention;
fig. 2 is a schematic structural diagram of a session extension option according to embodiment 1 of the present invention;
FIG. 3 is a schematic structural diagram of a timestamp index block TIB according to embodiment 1 of the present invention;
fig. 4 is a flowchart of a fast trace-back analysis method for network packets according to embodiment 2 of the present invention.
Detailed Description
The invention will now be further described with reference to the accompanying drawings and specific embodiments.
The invention discloses a fast backtracking analysis method of a network data packet, which realizes the high-efficiency storage of the network data packet by expanding a pcapng Format (PCAP Next Generation Dual File Format). A file description option is added in a Header block shb (section Header block) in the pcapng file, a Packet description information extension option is added in an enhanced Packet block epb (enhanced Packet block) for each Packet, and a custom timestamp Index block tib (timestamp Index block) supports intra-file indexing, while the file name contains the start and end timestamps of the Packet. The pcapng file format is organized in the format of SHB-IDB-EPB-EPB- … -EPB-EPB-TIB. And when backtracking analysis is carried out, firstly selecting files through a backtracking time period, then quickly positioning and analyzing positions through TIB (time information base), and then extracting metadata information of the data packet by utilizing EPB (extended packet buffer) items.
Example 1
As shown in fig. 1, in a network data packet storage method provided in embodiment 1 of the present invention, a network data packet stored by using the method can implement fast trace-back analysis on network traffic, and the method includes:
step 1), creating a title block SHB (section Header Block), and adding a file description expansion option;
in the protocol SHB extension option, the information to be recorded includes, but is not limited to: magic numbers, file generation time, software version, collector identification and the like.
The magic number is used for adding an identifier to the reserved network data packet file, and the identifier is used for verifying the file in backtracking analysis;
the file generation time is used for recording the generation time of the saved network data packet file;
the software version is used for identifying the software version for generating the file;
and the collector identifier is used for identifying the name of the collector which generates the file.
Step 2), an interface description block IDB is created;
step 3), creating an enhanced Packet block epb (enhanced Packet block) for filling the original packets and adding Packet information and session information extension options to each Packet;
as shown in fig. 2, in the protocol EPB extension option, the information to be recorded includes, but is not limited to: packet type, application layer protocol type, L3 offset, L4 offset, L4 payload length, session ID, and timestamp, among others.
The data packet type is represented by a 32-bit mask, the lower 28 bits (0-27) identify the type of a 2-4-layer protocol stack, the 28 bits identify the direction of the data packet, 0 is a packet sent by a client, 1 is a packet sent by a server, and 29-31 bits identify the calculation mode of a TAG;
the application layer protocol type is used for identifying the application layer protocol type carried by the network data packet;
the L3 offset is used to identify the layer 3 offset of the network packet, and if not, 0 is filled;
the L4 offset is used to identify a 4-layer offset of a network packet, and if not, 0 is filled;
the L4 load length is used for identifying the 4-layer load length of the network data packet, and if not, 0 is filled;
the TAG value is used for identifying the data packet and is obtained by quintuple information of the data packet according to a certain calculation mode;
the session ID option is used for identifying the data stream corresponding to the data packet;
the timestamp option is used to identify the time interval relative to the start of the stream, taking the value of milliseconds and rounding up.
Step 4) judging whether the EPB quantity reaches a set threshold value, if not, continuing to execute the step 3), and if so, executing the step 5);
step 5), creating a self-defined timestamp Index block TIB (timestamp Index Block), and adding data packet Index options in the file, wherein the steps comprise: a timestamp and a packet offset;
as shown in fig. 3, in the protocol custom timestamp index block TIB, the information to be recorded includes: a timestamp, and an offset.
The timestamp information is used for recording the relative time (microsecond) relative to the starting time of the file;
the offset information is used for recording the number of bytes relative to the starting position of the file.
And 6) judging whether the size of the written data reaches a pcap file generation threshold, if not, turning to the step 3), if so, generating a pcapng file according to an SHB-IDB-EPB-EPB- … -EPB-EPB-TIB format, and generating a file name by using the time stamps of the head and tail data packets recorded in the file.
Example 2
As shown in fig. 4, embodiment 2 of the present invention provides a method for fast backtracking and analyzing a network data packet, which is used to retrieve historical abnormal traffic, and the method first executes the data storage method provided in embodiment 1 to leave the network data packet in a system, and then executes the following steps:
step S1), according to the selected backtracking analysis time window, positioning the corresponding pcapng file according to the timestamp information in the file name;
step S2) jumping to the search start position by using the timestamp and offset information recorded by TIB in pcapng (PCAP Next Generation Dump File Format) File;
step S3), according to the given filtering rule (according to the protocol, the quadruple, the retention reason and the like), the option fields (the protocol, the TAG, the session ID and the like) in the EPBs are compared with the filtering rule in sequence until the end position of the searching time, and the data stream meeting the requirements is extracted from the historical abnormal traffic.
Finally, it should be noted that the above embodiments are only used for illustrating the technical solutions of the present invention and are not limited. Although the present invention has been described in detail with reference to the embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the spirit and scope of the invention as defined in the appended claims.
Claims (5)
1. A method for fast backtracking analysis of network packets, the method comprising:
step 1) creating a header block SHB containing file description extension options, an interface description block IDB and a plurality of enhanced data packet blocks EPB, filling a network original data packet in each EPB, and adding data packet information and session information extension options for each data packet; creating a self-defined time stamp index block TIB for increasing the index options of the data packets in the file; generating a pcapng file from the network data packet according to the SHB-IDB-EPB-EPB- … -EPB-EPB-TIB format;
step 2) selecting a pcapng file through a backtracking time window, quickly positioning the pcapng file on an EPB through a TIB, and extracting metadata information of a data packet by using a session information expansion option of the EPB;
the step 2) comprises the following steps:
step 2-1) according to the selected backtracking analysis time window, positioning a corresponding pcapng file according to the timestamp information in the file name, and simultaneously verifying magic number options in the file header block SHB;
step 2-2) jumping to a search starting position by utilizing a timestamp and offset information recorded by TIB in the pcapng file;
and 2-3) sequentially extracting metadata information in each EPB until the end position of the search time, wherein the metadata information comprises a network original data packet and a session information expansion option for rapid analysis.
2. The method for fast backtracking analysis of network data packets according to claim 1, wherein the step 1) comprises:
step 1-1) creating a title block SHB, and adding a file description expansion option for the SHB;
step 1-2) creating an interface description block IDB;
step 1-3) creating an enhanced data packet block EPB, filling network original data packets in the EPB, and adding data packet information and session information expansion options for each data packet;
step 1-4) judging whether the EPB quantity reaches a set threshold value, if the judgment result is negative, turning to step 1-3), otherwise, entering step 1-5);
step 1-5), creating a self-defined time stamp index block TIB and adding an in-file data packet index option for the TIB;
step 1-6) judging whether the size of the written data reaches a pcap file generation threshold, if the judgment result is negative, turning to the step 1-3), otherwise, generating a pcap file according to an SHB-IDB-EPB-EPB- … -EPB-EPB-TIB format, and generating a file name by using the timestamp of a head and tail data packet recorded in the file.
3. The method of claim 2, wherein in the file description extension option of the SHB, the information to be recorded includes but is not limited to: magic number, file generation time, software version and collector identification;
the magic number is used for adding an identifier to the reserved network data packet file;
the file generation time is used for recording the generation time of the saved network data packet file;
the software version is used for identifying the software version for generating the file;
and the collector identifier is used for identifying the name of the collector which generates the file.
4. The method according to claim 3, wherein in the EPB session information extension option, the information to be recorded includes but is not limited to: packet type, application layer protocol type, L3 offset, L4 offset, L4 payload length, session ID, and timestamp;
the data packet type is represented by using a 32-bit mask, the lower 28 bits identify the type of a 2-4 layer protocol stack, the 28 bits identify the direction of the data packet, 0 is a packet sent by a client, 1 is a packet sent by a server, and 29-31 bits identify the calculation mode of a TAG;
the application layer protocol type is used for identifying the application layer protocol type carried by the network data packet;
the L3 offset is used to identify the layer 3 offset of the network packet, and if not, 0 is filled;
the L4 offset is used to identify a 4-layer offset of a network packet, and if not, 0 is filled;
the L4 load length is used for identifying the 4-layer load length of the network data packet, and if not, 0 is filled;
the TAG value is used for identifying the data packet and is calculated by quintuple information of the data packet;
the session ID option is used for identifying the data stream corresponding to the data packet;
the timestamp option is used to identify the time interval relative to the start of the stream, taking the value of milliseconds and rounding up.
5. The method of claim 4, wherein in the TIB, the information to be recorded comprises: timestamp information and offset information;
the timestamp information is used for recording the relative time relative to the starting time of the file;
the offset information is used for recording the number of bytes relative to the starting position of the file.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810776879.7A CN109067711B (en) | 2018-07-16 | 2018-07-16 | Rapid backtracking analysis method for network data packet |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810776879.7A CN109067711B (en) | 2018-07-16 | 2018-07-16 | Rapid backtracking analysis method for network data packet |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109067711A CN109067711A (en) | 2018-12-21 |
| CN109067711B true CN109067711B (en) | 2020-04-14 |
Family
ID=64816661
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810776879.7A Active CN109067711B (en) | 2018-07-16 | 2018-07-16 | Rapid backtracking analysis method for network data packet |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109067711B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110704714A (en) * | 2019-09-27 | 2020-01-17 | 杭州九略智能科技有限公司 | Method and device for quickly indexing data of pcap file |
| CN113242151A (en) * | 2021-06-04 | 2021-08-10 | 上海天旦网络科技发展有限公司 | Specific data extraction method and system based on massive network data |
| CN113672629B (en) * | 2021-10-25 | 2021-12-28 | 北京金睛云华科技有限公司 | Distributed network traffic retrieval method and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102299828A (en) * | 2011-08-30 | 2011-12-28 | 中国科学院计算技术研究所 | Test traffic synthesis method and device for test on performance of network security equipment |
| CN102594625A (en) * | 2012-03-07 | 2012-07-18 | 北京启明星辰信息技术股份有限公司 | White data filter method and system in APT (Advanced Persistent Threat) intelligent detection and analysis platform |
| CN102801714A (en) * | 2012-07-26 | 2012-11-28 | 杭州电子科技大学 | Method for analyzing and reducing SQL (Structured Query Language) command in TNS (Transparent Network Substrate) protocol in by-pass manner |
| WO2016137884A1 (en) * | 2015-02-25 | 2016-09-01 | Quantea Corporation | Network traffic system and method of operation thereof |
| CN108040069A (en) * | 2017-12-28 | 2018-05-15 | 成都数成科技有限公司 | A kind of quick method for opening network data APMB package |
-
2018
- 2018-07-16 CN CN201810776879.7A patent/CN109067711B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102299828A (en) * | 2011-08-30 | 2011-12-28 | 中国科学院计算技术研究所 | Test traffic synthesis method and device for test on performance of network security equipment |
| CN102594625A (en) * | 2012-03-07 | 2012-07-18 | 北京启明星辰信息技术股份有限公司 | White data filter method and system in APT (Advanced Persistent Threat) intelligent detection and analysis platform |
| CN102801714A (en) * | 2012-07-26 | 2012-11-28 | 杭州电子科技大学 | Method for analyzing and reducing SQL (Structured Query Language) command in TNS (Transparent Network Substrate) protocol in by-pass manner |
| WO2016137884A1 (en) * | 2015-02-25 | 2016-09-01 | Quantea Corporation | Network traffic system and method of operation thereof |
| CN108040069A (en) * | 2017-12-28 | 2018-05-15 | 成都数成科技有限公司 | A kind of quick method for opening network data APMB package |
Non-Patent Citations (1)
| Title |
|---|
| pcap及pcapng格式解析中文版;L.Degioanni等;《Network Working Group》;20040902;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109067711A (en) | 2018-12-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7742414B1 (en) | Lightweight indexing for fast retrieval of data from a flow-level compressed packet trace | |
| CN109067711B (en) | Rapid backtracking analysis method for network data packet | |
| US10652265B2 (en) | Method and apparatus for network forensics compression and storage | |
| CN107634848B (en) | System and method for collecting and analyzing network equipment information | |
| US9305055B2 (en) | Method and apparatus for analysing data packets | |
| CN108400909B (en) | Traffic statistical method, device, terminal equipment and storage medium | |
| US7561569B2 (en) | Packet flow monitoring tool and method | |
| JP3655486B2 (en) | Event log method | |
| US8578024B1 (en) | Network application signatures for binary protocols | |
| US9717011B2 (en) | Event management in telecommunications networks | |
| CN114584401B (en) | Tracing system and method for large-scale network attack | |
| CN110457190A (en) | A kind of full link monitoring method, apparatus and system based on block chain | |
| EP3211834B1 (en) | Fast packet retrieval based on flow id and metadata | |
| CN112491652B (en) | Network flow sample processing method and device for testing | |
| EP3364627B1 (en) | Adaptive session intelligence extender | |
| CN103259737B (en) | A kind of method for rapidly positioning of flow of parallel storage high-speed network | |
| CN112486914B (en) | Data packet storage and quick-checking method and system | |
| CN106027414A (en) | HDFS-oriented parallel network message reading method | |
| CN111240599A (en) | Data stream storage method and device | |
| CN101267335B (en) | A method for guaranteeing successful alarm receiving/transmission in simple network management protocol | |
| US20160205118A1 (en) | Cyber black box system and method thereof | |
| CN111478806B (en) | Link tracking sampling method and system | |
| CN116545701A (en) | HTTP message rule matching method, system, equipment and medium | |
| CN114598622B (en) | Data monitoring method and device, storage medium and computer equipment | |
| CN113037551B (en) | Quick identification and positioning method for sensitive-related services based on traffic slice |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |