CN112688837B - Network measurement method and device based on time sliding window - Google Patents
Network measurement method and device based on time sliding window Download PDFInfo
- Publication number
- CN112688837B CN112688837B CN202110283706.3A CN202110283706A CN112688837B CN 112688837 B CN112688837 B CN 112688837B CN 202110283706 A CN202110283706 A CN 202110283706A CN 112688837 B CN112688837 B CN 112688837B
- Authority
- CN
- China
- Prior art keywords
- message
- sliding window
- data plane
- structure body
- large flow
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention provides a network measurement method and a device based on a time sliding window, wherein the method comprises the following steps: establishing a sliding window consisting of queues, establishing a sketch structure body to receive messages, writing the received messages into the tail of the sliding window, and deleting overtime messages at the front end of the sliding window; the data plane updates the message count of the sketch structure body according to the statistical result of the sliding window; if the received message is judged to be a large flow, reporting to a control plane; the control plane receives the big flow reported by the data plane and acquires the key value and the statistical data of the big flow; the control plane maintains a large flow dictionary. According to the scheme of the invention, the defects of low precision and poor timeliness in the traditional network measurement based on the fixed time window are effectively overcome, the memory occupation is effectively compressed, and the network measurement with precision, real time and memory saving is realized.
Description
Technical Field
The invention relates to the field of network data analysis in the Internet, in particular to a network measurement method and device based on a time sliding window.
Background
Network measurement is a basic means of describing network behavior, quantifying various indexes in the network, and fully understanding and knowing the internet. It provides a network-wide view by monitoring data flow in the network, thereby helping users better understand network conditions. It relates to the content of measurement algorithm, network behavior analysis and control method, etc. Network measurements, including fault diagnosis, traffic measurement, protocol troubleshooting, and performance evaluation, are the basis for many other network applications, such as congestion control, anomaly detection, capacity planning, and quality of service.
In the prior art, network measurement (such as Netflow) is realized on a CPU, and in the face of huge number of messages in a network, the CPU is difficult to accurately count each message, and can only collect a small number of network messages by a sampling method to analyze to estimate the condition of the whole network, so that the measurement result has deviation.
The traditional network measurement adopts a method based on a fixed time window, the starting time and the ending time of each time window are determined, after the current time window is measured, all measurement data need to be cleared, and then the next measurement window is entered, so that the time is artificially divided into a series of time windows, continuity is lacked between two adjacent time windows, and the measurement accuracy is reduced. Because the measurement result is reported only at the end of each time window in the traditional network measurement, the delay is larger when the time window is larger, and the measurement result lacks timeliness; in the face of the message flow of Tbit per second, recording all messages consumes a large amount of storage space, which exceeds the storage capacity of the existing network equipment.
Disclosure of Invention
In order to solve the technical problems, the invention provides a network measurement method and device based on a time sliding window, and the method and device are used for solving the technical problems that continuity is lacked between fixed time windows, measurement accuracy is low, timeliness of measurement results is poor, and network equipment is inconvenient to store in the prior art.
According to a first aspect of the present invention, there is provided a network measurement method based on a time sliding window, the method comprising the steps of:
a network measurement method based on a time sliding window comprises the following steps:
step S101: establishing a sliding window consisting of queues, and establishing a sketch structure body, wherein the sketch structure body is provided with m levels of storage arrays, each level of storage array corresponds to one hash value, and each level of storage array is provided with n storage units, wherein m is greater than 0, and n is greater than 0;
step S102: receiving a message, writing the received message into the tail part of the sliding window, and deleting the overtime message at the front end of the sliding window;
step S103: the data plane updates the message count of the sketch structure body according to the statistical result of the sliding window; if the received message is judged to be a large flow, reporting to a control plane; otherwise, go to step S102;
step S104: the control plane receives the big flow reported by the data plane and acquires the key value and the statistical data of the big flow;
step S105: the control plane maintains a large flow dictionary, and if the large flow reported by the data plane exists in the dictionary, the data of the large flow is updated in the large flow dictionary; if not, adding the large stream into the large stream dictionary.
Further, the step S102: receiving a message, writing the received message into the tail part of the sliding window, and deleting the overtime message at the front end of the sliding window, wherein:
receiving a message by a data plane, decapsulating the received message, and extracting a key value of the message as a basis for distinguishing different message streams; the key value of the message is determined by a five-tuple, the five-tuple comprises a source IP address, a destination IP address, a source port number, a destination port number and a protocol type number of the message, and the key value of the message can be determined by all elements of the five-tuple or the combination of any one or more elements in the five-tuple.
Further, the data plane calculates m hash values according to the key values of the messages, and the number of the hash values is the same as the number of the stages of the sketch structure body; and carrying out hash calculation on key values of the same message by m mutually independent hash functions to obtain m hash values, wherein the value range of each hash value is 0 to n-1, and the hash values are used for indexing each level of storage arrays in the sketch structure body.
Further, the data plane writes the current timestamp and m hash values of the message into the tail of the time sliding window together as a record of the current message; and the data plane reads the message record at the front end of the time sliding window, acquires a timestamp and m hash values corresponding to the message record at the front end, and if the difference between the timestamp of the message at the front end and the timestamp of the message at the tail end exceeds a set threshold, the message at the front end is taken out of the time sliding window until the difference between the timestamp of the message at the front end and the timestamp of the message at the tail end is less than or equal to the set threshold.
Further, the step S103: the data plane updates the message count of the sketch structure body according to the statistical result of the sliding window; if the received message is judged to be a large flow, reporting to a control plane; otherwise, go to step S102, where:
and the data plane updates the message count of the sketch structure body according to a newly added message at the tail and a front-end overtime message given by the time sliding window: and for the newly added tail message, the count of the tail message needs to be added into the sketch structure, m hash values are used as indexes to access m-level storage arrays in the sketch structure respectively, and corresponding storage units are modified.
According to a second aspect of the present invention, there is provided a network measurement apparatus based on a time sliding window, the apparatus comprising:
an initialization module: the method comprises the steps that a sliding window consisting of queues is established, a sketch structure body is established, the sketch structure body is provided with m levels of storage arrays, each level of storage array corresponds to one Hash value, n storage units are arranged in each level of storage array, and m is greater than 0, and n is greater than 0;
a message receiving module: configuring to receive messages, writing the received messages into the tail part of the sliding window, and deleting overtime messages at the front end of the sliding window;
an update module: configuring a data plane to update the message count of the sketch structure body according to the statistical result of the sliding window; if the received message is judged to be a large flow, reporting to a control plane; otherwise, triggering a message receiving module;
a data acquisition module: configuring the control plane to receive the big flow reported by the data plane and obtain the key value and the statistical data of the big flow;
the large flow dictionary maintenance module: configuring to maintain a large flow dictionary for the control plane, and if a large flow reported by the data plane exists in the dictionary, updating data of the large flow in the large flow dictionary; if not, adding the large stream into the large stream dictionary.
Further, the message receiving module receives a message from the data plane, decapsulates the received message, and extracts a key value of the message as a basis for distinguishing different message streams; the key value of the message is determined by a five-tuple, the five-tuple comprises a source IP address, a destination IP address, a source port number, a destination port number and a protocol type number of the message, and the key value of the message can be determined by all elements of the five-tuple or the combination of any one or more elements in the five-tuple.
Further, the data plane calculates m hash values according to the key values of the messages, and the number of the hash values is the same as the number of the stages of the sketch structure body; and carrying out hash calculation on key values of the same message by m mutually independent hash functions to obtain m hash values, wherein the value range of each hash value is 0 to n-1, and the hash values are used for indexing each level of storage arrays in the sketch structure body.
According to a third aspect of the present invention, there is provided a network measurement system based on a time sliding window, comprising:
a processor for executing a plurality of instructions;
a memory to store a plurality of instructions;
wherein the plurality of instructions are configured to be stored by the memory and loaded and executed by the processor to perform the network measurement method based on the time sliding window.
According to a fourth aspect of the present invention, there is provided a computer readable storage medium having a plurality of instructions stored therein; the plurality of instructions are configured to be loaded by a processor and to perform the network measurement method based on the time sliding window as described above.
According to the scheme of the invention, in a data plane of an SDN, the number of each message flow in a recent period of time is recorded through a sliding window consisting of a first-in first-out queue and a sketch structure consisting of a multi-stage storage unit; the sliding window records the hash value and the timestamp of the newly arrived message and gives the hash value and the timestamp of the overtime message for the sketch structure to update the message count; the data plane reports the measurement result to the control plane in real time; the control plane records all measurement data acquired by the data plane. The invention effectively overcomes the defects of low precision and poor timeliness in the traditional network measurement based on the fixed time window, effectively compresses the memory occupation and realizes the network measurement with precision, real time and memory saving. The SDN network equipment is adopted to realize a network measurement algorithm on a programmable data plane, and the programmable data plane of the SDN network equipment is realized on the basis of hardware, so that the SDN network equipment has higher throughput rate, solves the problem that the traditional network measurement based on CPU software is limited by the throughput rate of the software and can only carry out network measurement in a sampling mode, thereby causing larger measurement error, and realizes more accurate network measurement; by adopting the network measurement based on the sliding time window, the message record can be updated in real time along with the time of the time window, the problem that the junction of the time window is lack of continuity in the traditional network measurement based on the fixed time window is solved, and more accurate network measurement is realized; by adopting network measurement based on the sliding window, the message records in the latest period of time are always recorded in the sliding window, and the message records in the latest period of time are only recorded in the window when the tail time of each window is reached in the fixed window, so that the sliding window can report the large stream in real time, the fixed window can only inquire the large stream at the tail time of each window, and the timeliness of the sliding window is higher; the sketch algorithm is adopted to compress the storage space of the message flow counting, and the hash value is stored in the sliding window instead of the message original key value, so that the storage cost of the sliding window is compressed, the storage cost caused by directly recording all messages is avoided, and the storage space is saved.
The foregoing description is only an overview of the technical solutions of the present invention, and in order to make the technical solutions of the present invention more clearly understood and to implement them in accordance with the contents of the description, the following detailed description is given with reference to the preferred embodiments of the present invention and the accompanying drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention. In the drawings:
FIG. 1 is a flow chart of a network measurement method based on a time sliding window according to an embodiment of the present invention;
FIG. 2 is a diagram of a network measurement system architecture based on a time sliding window according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating an example of network measurement application based on a time sliding window according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a time sliding window according to an embodiment of the present invention;
FIG. 5 is a schematic view of a fixed window;
fig. 6 is a block diagram of an apparatus for network measurement based on a time sliding window according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the specific embodiments of the present invention and the accompanying drawings. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Defining:
SDN: software Defined Networking (SDN) Defined is a novel Network innovation architecture proposed by the research group of clean-slate project of stanford university, usa, and is an implementation mode of Network virtualization. The core technology OpenFlow separates the control plane and the data plane of the network equipment, thereby realizing the flexible control of network flow, enabling the network to be more intelligent as a pipeline, and providing a good platform for the innovation of a core network and application.
Sketch: the sketch is a hash-based data structure, can store flow characteristic information in real time in a high-speed network environment, only occupies smaller space resources, and has the balance characteristic of theoretically provable estimation precision and memory. The sketch algorithm is usually implemented based on hardware, the sketch does not store key values of messages, records flow information in a data structure with a fixed size, and allocates storage units for flows by using a hash algorithm, so that messages with different key values are allowed to share the same storage unit. By allowing hash collisions to exist, the sketch algorithm can achieve the effect of occupying as little storage space as possible with a certain precision. Common sketch algorithms include Count Min (CM), CU, Count, Counter Tree, Counter Braids, OpenSketch, FlowRadar, SketchVisor, Elastic sketch, Nitrosketch, sketchlen, and the like.
Large flow: refers to a stream that occurs more than a certain threshold number of times in a data stream. This can be given in two ways: 1) the absolute size of the large flow threshold, i.e. the number of times a flow occurs; 2) the relative size of the large flow threshold, i.e., the frequency of occurrence of a flow in the totality of flows.
First, a flow chart of a network measurement method based on a time sliding window according to an embodiment of the present invention is described with reference to fig. 1. As shown in fig. 1-2, the method comprises the steps of:
step S101: establishing a sliding window consisting of queues, and establishing a sketch structure body, wherein the sketch structure body is provided with m levels of storage arrays, each level of storage array corresponds to one hash value, and each level of storage array is provided with n storage units, wherein m is greater than 0, and n is greater than 0;
step S102: receiving a message, writing the received message into the tail part of the sliding window, and deleting the overtime message at the front end of the sliding window;
step S103: the data plane updates the message count of the sketch structure body according to the statistical result of the sliding window; if the received message is judged to be a large flow, reporting to a control plane; otherwise, go to step S102;
step S104: the control plane receives the big flow reported by the data plane and acquires the key value and the statistical data of the big flow;
step S105: the control plane maintains a large flow dictionary, and if the large flow reported by the data plane exists in the dictionary, the data of the large flow is updated in the large flow dictionary; if not, adding the large stream into the large stream dictionary.
In this embodiment, the network measurement system implements network measurement of the time sliding window, and the network measurement system uses an SDN architecture. The network measurement system has a data plane 10 and a control plane 20, and utilizes the feature that the SDN separates the data plane from the control plane so that the data plane can be conveniently programmed by means of software. The network measurement system defines a time sliding window 101 and a sketch structure body 102 in a data plane 10, wherein the time sliding window 101 is a first-in first-out queue and stores messages in the latest period of time, the time sliding window 101 records newly arrived messages, deletes overtime messages and provides the messages to the sketch structure body 102 to count the messages in the latest period of time. The sketch structure 102 obtains the statistics of the newly arrived packet, and reports the statistics to the control plane 20 if the packet is a large flow. The control plane program 201 is responsible for receiving the large streams counted by the sketch structure 102, the large stream dictionary 202 records all the counted large streams, and the control plane 20 updates the large stream dictionary 202.
The step S102: receiving a message, writing the received message into the tail part of the sliding window, and deleting the overtime message at the front end of the sliding window, wherein:
receiving a message by a data plane, decapsulating the received message, and extracting a key value of the message as a basis for distinguishing different message streams; the key value of the message is determined by a five-tuple, the five-tuple comprises a source IP address, a destination IP address, a source port number, a destination port number and a protocol type number of the message, and the key value of the message can be determined by all elements of the five-tuple or the combination of any one or more elements in the five-tuple.
And the data plane calculates m hash values according to the key values of the messages, wherein the number of the hash values is the same as the number of the stages of the sketch structure body. And carrying out hash calculation on key values of the same message by m mutually independent hash functions to obtain m hash values, wherein the value range of each hash value is 0 to n-1, and the hash values are used for indexing each level of storage arrays in the sketch structure body.
In this embodiment, the values of m and n are determined according to the measurement accuracy. The data plane writes the current timestamp and m hash values of the message into the tail of the time sliding window together to be used as a record of the current message; and the data plane reads the message record at the front end of the time sliding window, acquires a timestamp and m hash values corresponding to the message record at the front end, and if the difference between the timestamp of the message at the front end and the timestamp of the message at the tail end exceeds a set threshold, the message at the front end is taken out of the time sliding window until the difference between the timestamp of the message at the front end and the timestamp of the message at the tail end is less than or equal to the set threshold. Therefore, the time sliding window stores the time stamps and hash values of all the messages in the latest period of time, and the depth of the time sliding window is the number of the messages in the latest period of time and is determined by the network load in the latest period of time.
The step S103: the data plane updates the message count of the sketch structure body according to the statistical result of the sliding window; if the received message is judged to be a large flow, reporting to a control plane; otherwise, go to step S102, where:
and the data plane updates the message count of the sketch structure body according to a newly added message at the tail and a front-end overtime message given by the time sliding window: and for the newly added tail message, the count of the tail message needs to be added into the sketch structure, m hash values are used as indexes to access m-level storage arrays in the sketch structure respectively, and corresponding storage units are modified.
In this embodiment, specific update operations are slightly different according to different sketch algorithms, and generally, m hash values are used as indexes to respectively access m-level storage arrays in a sketch structure, and modify corresponding storage units, for example, a CM algorithm is used to perform an increment operation on each level of indexed storage units, and a CU algorithm only needs to select the smallest one of m levels to perform the increment operation; for the timeout message, the count of the timeout message needs to be deleted from the sketch, and the deletion process is opposite to the addition process, and the reduction operation needs to be performed on the memory location indexed by the hash value.
After deleting all overtime messages, inquiring the count of newly arrived messages in the sketch structure, if the size of the newly arrived messages exceeds a given threshold value, determining that the current message belongs to a large flow, reporting key values and statistical data of the large flow to a control plane, and performing next processing; otherwise, the step S102 is returned to process the next new message.
The step S104: the control plane receives the big flow reported by the data plane, acquires the key value and the statistical data of the big flow, wherein,
and the control plane receives the data plane report and acquires the key values and the statistical sizes of the newly detected large flows.
The step S105: the control plane maintains a large flow dictionary, and if the large flow reported by the data plane exists in the dictionary, the data of the large flow is updated in the large flow dictionary; if not, the large stream is added to a large stream dictionary, wherein,
the control plane maintains a large flow dictionary, wherein dictionary key values are message key values, and the dictionary values are message flow sizes; if a record exists in the large flow dictionary, the dictionary key value of the record is the same as the large flow key value of the new report, and the value of the record is larger than the size of the current new message flow, the original record in the dictionary is reserved; if the large flow dictionary has a record that the dictionary key value is the same as the large flow of the new report and the value is smaller than the size of the current new message flow, updating the value in the large flow dictionary to the size of the current new message flow; and if the large flow dictionary does not have records with the same dictionary key values as the large flow key values of the new message, respectively taking the newly detected large flow key values and the statistical values thereof as the key values and the values of the large flow dictionary and inserting the key values and the statistical values into the large flow dictionary.
The network measurement method based on the time sliding window of the present invention is described below with reference to fig. 3.
As shown in fig. 3, for simplicity, assume that the time threshold for the sliding window is 4 seconds; the sketch structure adopts CM sketch, and the size m =3 and n = 5; there are two message flows s0 and s 1; using the source IP address of the message flow as a key value; three hash values for s0 are 2, 4 and 2, and three hash values for s1 are 0, 2 and 1; the high flow threshold is 3.
Step S301: the sliding window records 4 messages from the 1 st second to the 5 th second (not included), wherein s0 and s1 occur twice each;
step S302: updating the sliding window, after 5 seconds, the data plane newly receives a message s0, adding s0 to the end of the sliding window, and simultaneously deleting s1 which arrives after 1 second and exceeds the time threshold of the sliding window for 4 seconds from the sliding window;
step S303: adding the newly received message statistic into the sketch structure body: the CM sketch is adopted, three hash values of s0 are 2, 4 and 2, and the three hash values are respectively indexed to the storage units 2, 4 and 2 in the first stage to the third stage of the CM sketch; adding 1 to the statistic values of the memory units according to the counting mode of CM sketch;
step S304: deleting the overtime message statistic from the sketch structure body: the three hash values of the timeout message s1 are 0, 2 and 1, the three hash values are respectively indexed to the storage units 0, 2 and 1 in the first stage to the third stage of the CM sketch, and the statistical values of the storage units are respectively subtracted by 1;
step S305: and judging whether the newly arrived message is a large stream, namely respectively indexing the storage units 2, 4 and 2 in the first stage to the third stage of the CM sketch again by using the three hash values 2, 4 and 2 of s 0. According to the CM sketch statistical method, taking the memory cell with the minimum value as the statistical value of the message stream s0, namely 3; s0 has reached the big flow threshold, so s0 is considered a big flow and needs to be reported to the control plane;
step S306: reporting a new big flow, comprising: assuming that the source IP address of s0 is 192.168.1.1, since the source IP address is selected as the key value of the message flow, 192.168.1.1 is used as the key value, and 3 is used as the flow size;
step S307: updating the big stream dictionary, comprising: the large stream dictionary is empty at this time, so the key value 192.168.1.1 and stream size 3 are inserted into the dictionary as a new record.
The reason why the time sliding window is more accurate than the fixed window is explained below in conjunction with fig. 4-5.
Suppose there is a message from 0 to 2tS 0,S 1,S 1,S 0,S 1,S 0,S 1,S 0,S 0,S 0,S 1,S 1Arrive successively, for simplicity, onlyS 0AndS 1two different message flows, assuming a large flow threshold of 4.
As shown in fig. 4, in the sliding window based measurement method, the current time is in the sliding windowS 0Appear 4 times, canS 0Identified as a large flow.
In the fixed window-based measurement method shown in fig. 5, time 0 to 2t is divided into two segments, i.e., time 0 to t and time t to 2t, and after the measurement at time 0 to t is completed, the measurement data needs to be cleared and measurement at time t to 2t is performed. The large stream that occurs 4 times at both the time 0 to t and the time t to 2t of fig. 5 does not occur, and therefore the large streamS 0Is reported as a False Negative (False Negative).
Therefore, the time sliding window has continuity compared with the fixed window, and measurement errors at time intervals are eliminated, so that the measurement result is more accurate.
An embodiment of the present invention further provides a device for network measurement based on a time sliding window, as shown in fig. 6, the device includes:
an initialization module: the method comprises the steps that a sliding window consisting of queues is established, a sketch structure body is established, the sketch structure body is provided with m levels of storage arrays, each level of storage array corresponds to one Hash value, n storage units are arranged in each level of storage array, and m is greater than 0, and n is greater than 0;
a message receiving module: configuring to receive messages, writing the received messages into the tail part of the sliding window, and deleting overtime messages at the front end of the sliding window;
an update module: configuring a data plane to update the message count of the sketch structure body according to the statistical result of the sliding window; if the received message is judged to be a large flow, reporting to a control plane; otherwise, triggering a message receiving module;
a data acquisition module: configuring the control plane to receive the big flow reported by the data plane and obtain the key value and the statistical data of the big flow;
the large flow dictionary maintenance module: configuring to maintain a large flow dictionary for the control plane, and if a large flow reported by the data plane exists in the dictionary, updating data of the large flow in the large flow dictionary; if not, adding the large stream into the large stream dictionary.
The embodiment of the invention further provides a system for network measurement based on a time sliding window, which comprises:
a processor for executing a plurality of instructions;
a memory to store a plurality of instructions;
wherein the plurality of instructions are for being stored by the memory and loaded and executed by the processor to perform the method for network measurement based on a time sliding window as described above.
The embodiment of the invention further provides a computer readable storage medium, wherein a plurality of instructions are stored in the storage medium; the plurality of instructions are for loading and executing, by a processor, the method for network measurement based on a time sliding window as described above.
It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict.
In the embodiments provided in the present invention, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions in actual implementation, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit.
The integrated unit implemented in the form of a software functional unit may be stored in a computer readable storage medium. The software functional unit is stored in a storage medium and includes several instructions to enable a computer device (which may be a personal computer, a physical machine Server, or a network cloud Server, etc., and needs to install a Windows or Windows Server operating system) to perform some steps of the method according to various embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the present invention in any way, and any simple modification, equivalent change and modification made to the above embodiment according to the technical spirit of the present invention are still within the scope of the technical solution of the present invention.
Claims (5)
1. A network measurement method based on a time sliding window is characterized by comprising the following steps:
step S101: establishing a sliding window consisting of queues, and establishing a sketch structure body, wherein the sketch structure body is provided with m levels of storage arrays, each level of storage array corresponds to one hash value, and each level of storage array is provided with n storage units, wherein m is greater than 0, and n is greater than 0;
step S102: receiving a message, writing the received message into the tail part of the sliding window, and deleting the overtime message at the front end of the sliding window;
step S103: the data plane updates the message count of the sketch structure body according to the statistical result of the sliding window; if the received message is judged to be a large flow, reporting to a control plane; otherwise, go to step S102;
step S104: the control plane receives the big flow reported by the data plane and acquires the key value and the statistical data of the big flow;
step S105: the control plane maintains a large flow dictionary, and if the large flow reported by the data plane exists in the dictionary, the data of the large flow is updated in the large flow dictionary; if not, adding the large stream into a large stream dictionary;
the step S102: receiving a message, writing the received message into the tail part of the sliding window, and deleting the overtime message at the front end of the sliding window, wherein:
receiving a message by a data plane, decapsulating the received message, and extracting a key value of the message as a basis for distinguishing different message streams; the key value of the message is determined by a quintuple, wherein the quintuple comprises a source IP address, a destination IP address, a source port number, a destination port number and a protocol type number of the message, and the key value of the message can be determined by all elements of the quintuple or the combination of any one or more elements in the quintuple;
the data plane calculates m hash values according to the key values of the messages, and the number of the hash values is the same as the number of the stages of the sketch structure body; carrying out hash calculation on key values of the same message by m mutually independent hash functions to obtain m hash values, wherein the value range of each hash value is 0 to n-1 and the hash values are used for indexing each level of storage arrays in the sketch structure body;
the data plane writes the current timestamp and m hash values of the message into the tail of the time sliding window together to be used as a record of the current message; and the data plane reads the message record at the front end of the time sliding window, acquires a timestamp and m hash values corresponding to the message record at the front end, and if the difference between the timestamp of the message at the front end and the timestamp of the message at the tail end exceeds a set threshold, the message at the front end is taken out of the time sliding window until the difference between the timestamp of the message at the front end and the timestamp of the message at the tail end is less than or equal to the set threshold.
2. The method for network measurement based on time sliding window according to claim 1, wherein said step S103: the data plane updates the message count of the sketch structure body according to the statistical result of the sliding window; if the received message is judged to be a large flow, reporting to a control plane; otherwise, go to step S102, where:
and the data plane updates the message count of the sketch structure body according to a newly added message at the tail and a front-end overtime message given by the time sliding window: and for the newly added tail message, the count of the tail message needs to be added into the sketch structure, m hash values are used as indexes to access m-level storage arrays in the sketch structure respectively, and corresponding storage units are modified.
3. A network measurement device based on a time sliding window, the device comprising:
an initialization module: the method comprises the steps that a sliding window consisting of queues is established, a sketch structure body is established, the sketch structure body is provided with m levels of storage arrays, each level of storage array corresponds to one Hash value, n storage units are arranged in each level of storage array, and m is greater than 0, and n is greater than 0;
a message receiving module: configuring to receive messages, writing the received messages into the tail part of the sliding window, and deleting overtime messages at the front end of the sliding window;
an update module: configuring a data plane to update the message count of the sketch structure body according to the statistical result of the sliding window; if the received message is judged to be a large flow, reporting to a control plane; otherwise, triggering a message receiving module;
a data acquisition module: configuring the control plane to receive the big flow reported by the data plane and obtain the key value and the statistical data of the big flow;
the large flow dictionary maintenance module: configuring to maintain a large flow dictionary for the control plane, and if a large flow reported by the data plane exists in the dictionary, updating data of the large flow in the large flow dictionary; if not, adding the large stream into a large stream dictionary;
the message receiving module receives the message by the data plane, decapsulates the received message, and extracts a key value of the message as a basis for distinguishing different message streams; the key value of the message is determined by a quintuple, wherein the quintuple comprises a source IP address, a destination IP address, a source port number, a destination port number and a protocol type number of the message, and the key value of the message can be determined by all elements of the quintuple or the combination of any one or more elements in the quintuple;
the data plane calculates m hash values according to the key values of the messages, and the number of the hash values is the same as the number of the stages of the sketch structure body; carrying out hash calculation on key values of the same message by m mutually independent hash functions to obtain m hash values, wherein the value range of each hash value is 0 to n-1 and the hash values are used for indexing each level of storage arrays in the sketch structure body;
the data plane writes the current timestamp and m hash values of the message into the tail of the time sliding window together to be used as a record of the current message; and the data plane reads the message record at the front end of the time sliding window, acquires a timestamp and m hash values corresponding to the message record at the front end, and if the difference between the timestamp of the message at the front end and the timestamp of the message at the tail end exceeds a set threshold, the message at the front end is taken out of the time sliding window until the difference between the timestamp of the message at the front end and the timestamp of the message at the tail end is less than or equal to the set threshold.
4. A network measurement system based on a time sliding window, comprising:
a processor for executing a plurality of instructions;
a memory to store a plurality of instructions;
wherein the plurality of instructions are for being stored by the memory and loaded and executed by the processor to perform the time sliding window based network measurement method according to any one of claims 1-2.
5. A computer-readable storage medium having stored therein a plurality of instructions; the plurality of instructions for loading and executing by a processor the time sliding window based network measurement method according to any one of claims 1-2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110283706.3A CN112688837B (en) | 2021-03-17 | 2021-03-17 | Network measurement method and device based on time sliding window |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110283706.3A CN112688837B (en) | 2021-03-17 | 2021-03-17 | Network measurement method and device based on time sliding window |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112688837A CN112688837A (en) | 2021-04-20 |
| CN112688837B true CN112688837B (en) | 2021-06-08 |
Family
ID=75455605
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110283706.3A Active CN112688837B (en) | 2021-03-17 | 2021-03-17 | Network measurement method and device based on time sliding window |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112688837B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113407363B (en) * | 2021-06-23 | 2024-05-17 | 京东科技控股股份有限公司 | Sliding window counting method and device based on remote dictionary service |
| CN114285515B (en) * | 2021-12-14 | 2023-12-19 | 昆高新芯微电子(江苏)有限公司 | Method and device for realizing arbitrary TSN time window period |
| CN113965584B (en) * | 2021-12-21 | 2022-05-13 | 北京达佳互联信息技术有限公司 | Message processing method, device, apparatus and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105989061A (en) * | 2015-02-09 | 2016-10-05 | 中国科学院信息工程研究所 | Rapid indexing method for repeated detection of multi-dimensional data under sliding window |
| CN106502625A (en) * | 2016-10-27 | 2017-03-15 | 成都知道创宇信息技术有限公司 | The evaluation method of top n cardinal datas in a kind of high-speed data-flow |
| WO2019032914A1 (en) * | 2017-08-09 | 2019-02-14 | Verdigris Technologies, Inc. | System and methods for providing waveforms over wireless systems from power monitors |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9124515B2 (en) * | 2010-11-22 | 2015-09-01 | Hewlett-Packard Development Company, L.P. | Elephant flow detection in a computing device |
| CN103647670B (en) * | 2013-12-20 | 2017-12-26 | 北京理工大学 | A kind of data center network flow analysis method based on sketch |
| CN109861881B (en) * | 2019-01-24 | 2021-11-19 | 大连理工大学 | Elephant flow detection method based on three-layer Sketch framework |
| CN110011876B (en) * | 2019-04-19 | 2022-05-03 | 福州大学 | Sketch network measurement method based on reinforcement learning |
-
2021
- 2021-03-17 CN CN202110283706.3A patent/CN112688837B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105989061A (en) * | 2015-02-09 | 2016-10-05 | 中国科学院信息工程研究所 | Rapid indexing method for repeated detection of multi-dimensional data under sliding window |
| CN106502625A (en) * | 2016-10-27 | 2017-03-15 | 成都知道创宇信息技术有限公司 | The evaluation method of top n cardinal datas in a kind of high-speed data-flow |
| WO2019032914A1 (en) * | 2017-08-09 | 2019-02-14 | Verdigris Technologies, Inc. | System and methods for providing waveforms over wireless systems from power monitors |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112688837A (en) | 2021-04-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112688837B (en) | Network measurement method and device based on time sliding window | |
| JP7039685B2 (en) | Traffic measurement methods, devices, and systems | |
| CN109861881B (en) | Elephant flow detection method based on three-layer Sketch framework | |
| US10097464B1 (en) | Sampling based on large flow detection for network visibility monitoring | |
| US9979624B1 (en) | Large flow detection for network visibility monitoring | |
| US10536360B1 (en) | Counters for large flow detection | |
| US11171869B2 (en) | Microburst detection and management | |
| JPH04263536A (en) | Apparatus and system for monitoring of network | |
| US10003515B1 (en) | Network visibility monitoring | |
| US11050649B2 (en) | Delay measurement method of network node device, apparatus, and network node device | |
| EP3823217A1 (en) | Network flow measurement method, network measurement device and control plane device | |
| JPWO2015182629A1 (en) | Monitoring system, monitoring device and monitoring program | |
| EP4075749A1 (en) | Detection method and detection device for heavy flow data stream | |
| CN112822077A (en) | Method and system for measuring total network flow in data center network and packet loss detection method | |
| CN114189480B (en) | Flow sampling method, device, electronic equipment and medium | |
| CN115776449A (en) | Train Ethernet communication state monitoring method and system | |
| CN112583658B (en) | Available bandwidth measuring method, storage medium and equipment | |
| CN111935769B (en) | Poor cell identification method, device and equipment | |
| US8645593B2 (en) | Signal processor, transmission apparatus, and method for processing signal | |
| US9900207B2 (en) | Network control protocol | |
| CN113965492A (en) | Data flow statistical method and device | |
| CN114785396B (en) | Logic port configuration, lookup mapping and traffic management method, system and terminal | |
| CN112671616B (en) | Flow monitoring method and device, programmable chip, storage medium and electronic device | |
| CN115701049A (en) | Communication delay estimation method and related device | |
| CN115695254A (en) | Network remote measuring method, system, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |