CN105515919A - Network flow monitoring method based on Hash compression algorithm - Google Patents
Network flow monitoring method based on Hash compression algorithm Download PDFInfo
- Publication number
- CN105515919A CN105515919A CN201610038717.4A CN201610038717A CN105515919A CN 105515919 A CN105515919 A CN 105515919A CN 201610038717 A CN201610038717 A CN 201610038717A CN 105515919 A CN105515919 A CN 105515919A
- Authority
- CN
- China
- Prior art keywords
- hash
- address
- occurrence
- traffic statistics
- bucket
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
Landscapes
- Engineering & Computer Science (AREA)
- Environmental & Geological Engineering (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention relates to a Hash compression realization method based on an FPGA (Field Programmable Gate Array) and belongs to the technical field of network communication. The invention provides the Hash compression realization method based on the FPGA; the method comprises the following steps: taking a quintuple receiving an IP (Internet Protocol) message as a flow mark; performing a Hash compression algorithm on the flow mark to reduce bit length of the flow mark, so as to determine entry addresses of a Hash lookup table; meanwhile, configuring Hash buckets with the corresponding quantity at each entry address, so as to solve a Hash conflict problem; after Hash lookup is matched accurately, obtaining a flow statistical table address corresponding to the flow mark, and performing updating operation on flow statistics. According to the method provided by the invention, the width of a key word bit of the flow mark to be matched is greatly reduced, so that resources of a storage device are saved; meanwhile, the Hash conflict problem is solved by adopting the Hash buckets, so that the complexity of solving the conflict problem is reduced and the accurate table lookup time is shortened.
Description
Technical field
The present invention relates to a kind of network flow monitoring method based on Hash compression algorithm, belong to network communication technology field.Being specially adapted to realize identifying and matched and searched being no more than 2K bar IP traffic in FPGA sheet, completing traffic statistics monitoring function simultaneously.
Background technology
Along with the deep development of network technology and the continuous expansion of network size, flow proliferation issues also becomes increasingly conspicuous.In order to can monitor network run, the abnormal flow problem existed in Timeliness coverage network, need a kind of stable, in time, network flow monitoring method reliably.At present, the network flow monitoring method based on IP traffic usually adopts external TCAM or DDR to carry out stream identification and traffic statistics on hardware implementing.Although the method meets the requirement of quick identification to IP traffic in network and traffic statistics, bring the problems such as power consumption increasing, cost increase.Especially, in the private network application that some network sizes are little, the method causes the significant wastage of hardware resource and cost.Therefore, need one to be applicable to realize in FPGA sheet in the private network application that some network sizes are little, too many memory resource need not be taken simultaneously, the network flow monitoring method of IP traffic identification and traffic statistics can be realized fast.
Summary of the invention
The object of this invention is to provide in a kind of private network little in network size, be applicable to the network flow monitoring method realized in FPGA sheet, the method has and takies the features such as resource is few, hardware cost is low, can identify fast and traffic statistics IP traffic.
Implementation of the present invention is to the quick identification of IP traffic and coupling based on the realization of Hash compression algorithm; The present invention to receive the five-tuple of IP message as traffic identifier, and by this traffic identifier by Hash compression algorithm, reduces its bit length, thus determines the entry address of Hash lookup table, greatly save the storage space shared by Hash table; Solving hash-collision problem owing to have employed Hash bucket simultaneously, because this reducing the complexity of Conflict solving, decreasing the hit time of tabling look-up.
In order to realize described goal of the invention, the present invention is realized by following technical scheme:
Based on a network flow monitoring method for Hash compression algorithm, it is characterized in that comprising the steps:
(1) step: extract as traffic identifier using the object IP address in IP message, source IP address, destination slogan, source port number and protocol type field, sends into Hash compression algorithm module and processes;
(2) step: Hash compression algorithm module, using traffic identifier as keyword, obtains Hash table entrance index address by HASH function, carries out Hash table matched and searched;
(3) step: the corresponding bucket in each Hash table entry address is dark is the Hash bucket of 4 list items, and after obtaining Hash table entry address, entering in the Hash bucket of its correspondence is that match objects carries out matched and searched with traffic identifier; If there is occurrence in Hash bucket, enter (4) step; If there is not occurrence in Hash bucket, enter (5) step;
(4) step: when there is occurrence in Hash bucket, extracts the traffic statistics table address in occurrence, enters traffic statistics table and carries out traffic statistics renewal, proceed to (6) step;
(5) step: when there is not occurrence in Hash bucket and in Hash bucket, list item is not occupied full, according to priority orders newly-built occurrence in unappropriated list item of list item 0,1,2,3, and in newly-built occurrence, distribute new traffic statistics table address, newly-built traffic statistics in the traffic statistics table that this address is corresponding, proceeds to (6) step simultaneously;
(6) step: in order to the generation preventing Hash bucket from taking spilling situation, sets up aging deleting mechanism to the list item of Hash table; Carried out a poll every 2 seconds to flow statistical form, when the byte statistical number had in traffic statistics table in list item does not increase in 2 seconds, the list item in its corresponding Hash table will carry out deletion action; Discharge the traffic statistics table address shared by this traffic identifier simultaneously;
Complete the network flow monitoring method based on Hash compression algorithm.
Wherein, the HASH function in (2) step in Hash compression algorithm module selects CRC-12 algorithm realization, and the evaluator of CRC-12 algorithm is x12+x11+x3+x+1; Wherein, x is factor of a polynomial.
Compared with prior art, it is few that the network flow monitoring method based on Hash compression algorithm provided by the invention takies memory resource, and hardware cost is low; The method to receive the five-tuple of IP message as traffic identifier, and by this traffic identifier by Hash compression algorithm, reduces its bit length, greatly saves the storage space shared by Hash table; Adopt Hash bucket to solve matching conflict problem simultaneously, improve IP traffic matched and searched performance, decrease the hit time of tabling look-up; The method is suitable for the private network communication application high to low-power consumption, low cost and stability requirement.
Accompanying drawing explanation
Fig. 1 is principle schematic of the present invention;
Fig. 2 is the extraction schematic diagram of traffic identifier;
Fig. 3 is the structural representation of Hash table in the present invention;
Fig. 4 is the structural representation of traffic statistics table in the present invention.
Embodiment
Below in conjunction with accompanying drawing 1-4, the present invention is elaborated.
As shown in Figure 1, a kind of network flow monitoring method based on Hash compression algorithm, is characterized in that comprising the steps:
(1) step: extract as traffic identifier (104bit) using the object IP address (32bit) in IP message, source IP address (32bit), destination slogan (16bit), source port number (16bit) and protocol type field (8bit), sends into Hash compression algorithm module and processes;
As shown in Figure 2, for an IPV4TCP data number message, its traffic identifier is: { 192,168,1,25,192,160,1,30,80,69,6};
(2) step: Hash compression algorithm module, using traffic identifier (104bit) as keyword, obtains Hash table entrance index address (12bit) by HASH function, carries out Hash table matched and searched; HASH function in Hash compression algorithm module selects CRC-12 algorithm realization; The evaluator of this CRC-12 algorithm is x12+x11+x3+x+1; Wherein, x is factor of a polynomial;
(3) step: the corresponding bucket in each Hash table entry address is dark is the Hash bucket of 4 list items, after obtaining Hash table entry address, entering in the Hash bucket of its correspondence is that match objects carries out matched and searched with traffic identifier, and the structure of this Hash table as shown in Figure 3; If there is occurrence in Hash bucket, enter (4) step; If there is not occurrence in Hash bucket, enter (5) step;
(4) step: when there is occurrence in Hash bucket, extracts the traffic statistics table address in occurrence, enters traffic statistics table and carries out traffic statistics renewal, proceed to (6) step; The structure of this traffic statistics table as shown in Figure 4;
(5) step: when there is not occurrence in Hash bucket and in Hash bucket, list item is not occupied full, according to priority orders newly-built occurrence in unappropriated list item of list item 0,1,2,3, and in newly-built occurrence, distribute new traffic statistics table address, newly-built traffic statistics in the traffic statistics table that this address is corresponding, proceeds to (6) step simultaneously;
(6) step: in order to the generation preventing Hash bucket from taking spilling situation, sets up aging deleting mechanism to the list item of Hash table; Carried out a poll every 2 seconds to flow statistical form, when the byte statistical number had in traffic statistics table in list item does not increase in 2 seconds, the list item in its corresponding Hash table will carry out deletion action; Discharge the traffic statistics table address shared by this traffic identifier simultaneously;
Complete the network flow monitoring method based on Hash compression algorithm.
More than be described with reference to the accompanying drawings the present invention; but those of ordinary skill in the art it should be understood that specification is for explaining claim, and protection scope of the present invention is as the criterion with claim; on basis of the present invention, any done amendment all should in required protection range.
Claims (2)
1., based on a Hash compression implementation method of FPGA, it is characterized in that comprising the steps:
(1) step: extract as traffic identifier using the object IP address in IP message, source IP address, destination slogan, source port number and protocol type field, sends into Hash compression algorithm module and processes;
(2) step: Hash compression algorithm module, using traffic identifier as keyword, obtains Hash table entrance index address by HASH function, carries out Hash table matched and searched;
(3) step: the corresponding bucket in each Hash table entry address is dark is the Hash bucket of 4 list items, and after obtaining Hash table entry address, entering in the Hash bucket of its correspondence is that match objects carries out matched and searched with traffic identifier; If there is occurrence in Hash bucket, enter (4) step; If there is not occurrence in Hash bucket, enter (5) step;
(4) step: when there is occurrence in Hash bucket, extracts the traffic statistics table address in occurrence, enters traffic statistics table and carries out traffic statistics renewal, proceed to (6) step;
(5) step: when there is not occurrence in Hash bucket and in Hash bucket, list item is not occupied full, according to priority orders newly-built occurrence in unappropriated list item of list item 0,1,2,3, and in newly-built occurrence, distribute new traffic statistics table address, newly-built traffic statistics in the traffic statistics table that this address is corresponding, proceeds to (6) step simultaneously;
(6) step: in order to the generation preventing Hash bucket from taking spilling situation, sets up aging deleting mechanism to the list item of Hash table; Carried out a poll every 2 seconds to flow statistical form, when the byte statistical number had in traffic statistics table in list item does not increase in 2 seconds, the list item in its corresponding Hash table will carry out deletion action; Discharge the traffic statistics table address shared by this traffic identifier simultaneously;
Complete the network flow monitoring method based on Hash compression algorithm.
2. described a kind of network flow monitoring method based on Hash compression algorithm according to claim 1, it is characterized in that: the HASH function in (2) step in Hash compression algorithm module selects CRC-12 algorithm realization, the evaluator of CRC-12 algorithm is x12+x11+x3+x+1; Wherein, x is factor of a polynomial.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610038717.4A CN105515919A (en) | 2016-01-20 | 2016-01-20 | Network flow monitoring method based on Hash compression algorithm |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610038717.4A CN105515919A (en) | 2016-01-20 | 2016-01-20 | Network flow monitoring method based on Hash compression algorithm |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105515919A true CN105515919A (en) | 2016-04-20 |
Family
ID=55723578
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610038717.4A Pending CN105515919A (en) | 2016-01-20 | 2016-01-20 | Network flow monitoring method based on Hash compression algorithm |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105515919A (en) |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106357479A (en) * | 2016-11-15 | 2017-01-25 | 中国人民解放军防空兵学院 | Full network flow monitoring method |
| CN106411622A (en) * | 2016-12-16 | 2017-02-15 | 成都西加云杉科技有限公司 | Table entry processing method and device |
| CN107248939A (en) * | 2017-05-26 | 2017-10-13 | 中国人民解放军理工大学 | Network flow high-speed associative method based on hash memories |
| WO2017185912A1 (en) * | 2016-04-26 | 2017-11-02 | 上海斐讯数据通信技术有限公司 | Method and apparatus for collecting statistics about terminal device information based on hash node |
| CN108574587A (en) * | 2017-03-09 | 2018-09-25 | 华为技术有限公司 | The capacity update method and device of distributed apparatus |
| CN110019250A (en) * | 2019-03-06 | 2019-07-16 | 清华大学 | Network measure method and computer readable storage medium based on hash function |
| CN111107068A (en) * | 2019-12-09 | 2020-05-05 | 江苏云涌电子科技股份有限公司 | Efficient rule matching method for FPGA and terminal |
| CN112286966A (en) * | 2020-09-23 | 2021-01-29 | 清华大学 | Data stream processing method, data stream recovery method, data stream processing device, data stream recovery device and storage medium |
| CN112632079A (en) * | 2020-12-30 | 2021-04-09 | 联想未来通信科技(重庆)有限公司 | Data stream identification query method and device |
| CN114244752A (en) * | 2021-12-16 | 2022-03-25 | 锐捷网络股份有限公司 | Flow statistical method, device and equipment |
| CN114710451A (en) * | 2022-04-08 | 2022-07-05 | 中山大学 | Block chain application flow identification method and system based on expanded hash and KNN |
| CN114785396A (en) * | 2022-03-09 | 2022-07-22 | 西安电子科技大学 | Method, system and terminal for configuring, searching mapping and managing flow of logical port |
| CN115580579A (en) * | 2022-09-28 | 2023-01-06 | 杭州迪普科技股份有限公司 | Message forwarding method and device, electronic equipment and storage medium |
| CN116055397A (en) * | 2023-03-27 | 2023-05-02 | 井芯微电子技术(天津)有限公司 | Queue entry maintenance method and device |
| CN116132319A (en) * | 2021-11-15 | 2023-05-16 | 华为技术有限公司 | Method and device for identifying stream |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101741743A (en) * | 2009-12-11 | 2010-06-16 | 东南大学 | Network address sort-based bidirectional stream combining method |
| CN102316173A (en) * | 2010-07-05 | 2012-01-11 | 国讯新创软件技术有限公司 | Method and device for aggregating network address |
| CN103581044A (en) * | 2013-11-04 | 2014-02-12 | 汉柏科技有限公司 | Flow statistic method and device |
-
2016
- 2016-01-20 CN CN201610038717.4A patent/CN105515919A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101741743A (en) * | 2009-12-11 | 2010-06-16 | 东南大学 | Network address sort-based bidirectional stream combining method |
| CN102316173A (en) * | 2010-07-05 | 2012-01-11 | 国讯新创软件技术有限公司 | Method and device for aggregating network address |
| CN103581044A (en) * | 2013-11-04 | 2014-02-12 | 汉柏科技有限公司 | Flow statistic method and device |
Non-Patent Citations (1)
| Title |
|---|
| 李渊，许鑫: "基于哈希压缩算法的网络流量监控方法研究", 《计算机与网络》 * |
Cited By (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017185912A1 (en) * | 2016-04-26 | 2017-11-02 | 上海斐讯数据通信技术有限公司 | Method and apparatus for collecting statistics about terminal device information based on hash node |
| CN106357479B (en) * | 2016-11-15 | 2019-08-23 | 中国人民解放军防空兵学院 | A kind of whole network flow monitoring method |
| CN106357479A (en) * | 2016-11-15 | 2017-01-25 | 中国人民解放军防空兵学院 | Full network flow monitoring method |
| CN106411622A (en) * | 2016-12-16 | 2017-02-15 | 成都西加云杉科技有限公司 | Table entry processing method and device |
| CN106411622B (en) * | 2016-12-16 | 2019-06-07 | 成都西加云杉科技有限公司 | A kind of table entry processing method and device |
| CN108574587A (en) * | 2017-03-09 | 2018-09-25 | 华为技术有限公司 | The capacity update method and device of distributed apparatus |
| CN108574587B (en) * | 2017-03-09 | 2020-07-24 | 华为技术有限公司 | Capacity updating method and device for distributed equipment |
| CN107248939B (en) * | 2017-05-26 | 2020-07-31 | 中国人民解放军理工大学 | Network flow high-speed correlation method based on hash memory |
| CN107248939A (en) * | 2017-05-26 | 2017-10-13 | 中国人民解放军理工大学 | Network flow high-speed associative method based on hash memories |
| CN110019250A (en) * | 2019-03-06 | 2019-07-16 | 清华大学 | Network measure method and computer readable storage medium based on hash function |
| CN111107068B (en) * | 2019-12-09 | 2021-03-09 | 江苏云涌电子科技股份有限公司 | Efficient rule matching method for FPGA and terminal |
| CN111107068A (en) * | 2019-12-09 | 2020-05-05 | 江苏云涌电子科技股份有限公司 | Efficient rule matching method for FPGA and terminal |
| CN112286966A (en) * | 2020-09-23 | 2021-01-29 | 清华大学 | Data stream processing method, data stream recovery method, data stream processing device, data stream recovery device and storage medium |
| CN112632079A (en) * | 2020-12-30 | 2021-04-09 | 联想未来通信科技(重庆)有限公司 | Data stream identification query method and device |
| CN116132319A (en) * | 2021-11-15 | 2023-05-16 | 华为技术有限公司 | Method and device for identifying stream |
| CN114244752A (en) * | 2021-12-16 | 2022-03-25 | 锐捷网络股份有限公司 | Flow statistical method, device and equipment |
| CN114785396B (en) * | 2022-03-09 | 2024-04-12 | 西安电子科技大学 | Logic port configuration, lookup mapping and traffic management method, system and terminal |
| CN114785396A (en) * | 2022-03-09 | 2022-07-22 | 西安电子科技大学 | Method, system and terminal for configuring, searching mapping and managing flow of logical port |
| CN114710451B (en) * | 2022-04-08 | 2023-06-23 | 中山大学 | Block chain application flow identification method and system based on extended hash and KNN |
| CN114710451A (en) * | 2022-04-08 | 2022-07-05 | 中山大学 | Block chain application flow identification method and system based on expanded hash and KNN |
| CN115580579A (en) * | 2022-09-28 | 2023-01-06 | 杭州迪普科技股份有限公司 | Message forwarding method and device, electronic equipment and storage medium |
| CN115580579B (en) * | 2022-09-28 | 2024-06-04 | 杭州迪普科技股份有限公司 | Message forwarding method and device, electronic equipment and storage medium |
| CN116055397A (en) * | 2023-03-27 | 2023-05-02 | 井芯微电子技术(天津)有限公司 | Queue entry maintenance method and device |
| CN116055397B (en) * | 2023-03-27 | 2023-08-18 | 井芯微电子技术(天津)有限公司 | Queue entry maintenance method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105515919A (en) | Network flow monitoring method based on Hash compression algorithm | |
| WO2016029684A1 (en) | Packet processing and forwarding device and method, and computer storage medium | |
| US11863439B2 (en) | Method, apparatus and storage medium for application identification | |
| WO2020258982A1 (en) | Method and system for analyzing security log of base station, and computer-readable storage medium | |
| US9307555B2 (en) | Method and system for mobile terminal to access the network through cell phone | |
| US11316804B2 (en) | Forwarding entry update method and apparatus in a memory | |
| WO2023115978A1 (en) | Packet processing method and apparatus, and electronic device | |
| CN104899204A (en) | Data storage method and device | |
| CN106465238A (en) | Service discovery method and device in neighbor awareness network | |
| CN104579970A (en) | Strategy matching method and device of IPv6 message | |
| WO2020248690A1 (en) | Message matching table lookup method, system, storage medium, and terminal | |
| CN113645140A (en) | Message statistical method, device, storage medium and network equipment | |
| CN104702508A (en) | Method and system for dynamically updating table items | |
| US20150200812A1 (en) | Determining when to relearn a relationship between a network address and a port | |
| US11444877B2 (en) | Packet flow identification with reduced decode operations | |
| CN111010362B (en) | Monitoring method and device for abnormal host | |
| CN114338529B (en) | Five-tuple rule matching method and device | |
| CN105634999A (en) | Aging method and device for medium access control address | |
| US11303566B2 (en) | Method, electronic device, and computer program product for data processing | |
| US9967178B1 (en) | Flow record size reduction | |
| US20190005533A1 (en) | Signal Matching for Entity Resolution | |
| CN114401243A (en) | Data packet transmission method, device, equipment and medium | |
| CN114448886A (en) | Flow table processing method and device | |
| US8825884B2 (en) | Method and device for protocol identification | |
| CN114697160A (en) | Tunnel message processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160420 |
|
| RJ01 | Rejection of invention patent application after publication |