CN114785707A - Hierarchical massive stream cooperative monitoring method - Google Patents

Hierarchical massive stream cooperative monitoring method Download PDF

Info

Publication number
CN114785707A
CN114785707A CN202210526869.4A CN202210526869A CN114785707A CN 114785707 A CN114785707 A CN 114785707A CN 202210526869 A CN202210526869 A CN 202210526869A CN 114785707 A CN114785707 A CN 114785707A
Authority
CN
China
Prior art keywords
flow
task
network
hierarchical
switch
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202210526869.4A
Other languages
Chinese (zh)
Other versions
CN114785707B (en
Inventor
宋超
吴金秋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
University of Electronic Science and Technology of China
Original Assignee
University of Electronic Science and Technology of China
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by University of Electronic Science and Technology of China filed Critical University of Electronic Science and Technology of China
Priority to CN202210526869.4A priority Critical patent/CN114785707B/en
Publication of CN114785707A publication Critical patent/CN114785707A/en
Application granted granted Critical
Publication of CN114785707B publication Critical patent/CN114785707B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/06—Generation of reports
    • H04L43/062—Generation of reports related to network traffic
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/15—Correlation function computation including computation of convolution operations
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/10—Flow control; Congestion control
    • H04L47/12—Avoiding congestion; Recovering from congestion
    • H04L47/125—Avoiding congestion; Recovering from congestion by balancing the load, e.g. traffic engineering
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D30/00—Reducing energy consumption in communication networks
    • Y02D30/50—Reducing energy consumption in communication networks in wire-line communication networks, e.g. low power modes or reduced link rate

Abstract

The invention belongs to the technical field of network measurement, and particularly provides a hierarchical big stream cooperative monitoring method which is used for solving the problems that the existing centralized method is migrated to distributed cooperative monitoring and sub-streams of distributed lower hierarchical big stream cooperative monitoring are lost; the invention solves the problem of cooperative monitoring by modeling and solving the problem of joint optimization of task deployment and network flow selection, and realizes better load balancing effect on the whole network switch while acquiring a cooperative monitoring strategy; the invention provides a distributed hierarchical big flow measurement framework for solving the problem of sub-flow loss, wherein a bloom filter is used for supporting a whole network switch to cooperatively monitor hierarchical big flows, a big flow component is used for storing identifiers and count values of candidate big flows, a small flow component is used for storing the count values of network flows thrown by the big flow component, lost sub-flows are recovered through the small flow component, and the measurement precision of the hierarchical big flows is improved. Finally, the invention obviously improves the measurement precision of the hierarchical mass flow.

Description

Hierarchical massive stream cooperative monitoring method
Technical Field
The invention belongs to the technical field of network measurement, and particularly provides a hierarchical large flow cooperative monitoring method.
Background
Network measurement provides necessary information for network management application, and plays a significant role in ensuring the stability and safety of a network; hierarchical Heavy flows (Hierarchical heaves), as one of network measurement tasks, are used to identify Heavy flows (heaves) based on public IP prefix aggregation, with network applications such as anomaly detection, DDoS detection, and the like. Existing research uses a centralized algorithm to identify hierarchically large flows among all network flows passing through it on a single switch, but the continuous expansion of network traffic scale makes it difficult for the centralized algorithm to reach the desired measurement accuracy again; in the measurement of the hierarchical large flow, the measurement error of a lower hierarchy affects the precision of a higher hierarchy, which is called a dependency problem; due to the resource limitations and dependency issues described above, the accuracy of performing hierarchical large flow measurements on a single switch is very low.
With the proposal of the software-defined network and the software-defined measurement, the method supports the efficient deployment of measurement tasks on a plurality of switches in a distributed mode for the cooperative monitoring of network flows; however, in a distributed environment, multiple sub-streams (with common IP prefix) from the same large stream are measured scattered across different switches, and part of the sub-streams will be ignored because they are too small, resulting in aggregation at a higher level, which is called the sub-stream loss problem by the present invention. Current research work mainly focuses on balancing the load of the switch, neglecting the characteristics of specific tasks, especially the sub-flow loss problem in hierarchical large-flow cooperative monitoring; based on the above, the invention provides a hierarchical concurrent monitoring method for the mass flow.
Disclosure of Invention
The invention aims to provide a hierarchical big flow cooperative monitoring method aiming at the problems in the prior art; in order to solve the dependency problem of hierarchical large flow measurement in a centralized environment, the invention provides cooperative monitoring of hierarchical large flows in a distributed manner, firstly, a problem formulated by a hierarchical large flow cooperative monitoring strategy is expressed as a task deployment and network flow selection joint optimization problem, and then, the task deployment and network flow selection strategy is obtained by approximate solving, so that the purpose is to utilize resources of a whole network switch to perform hierarchical large flow cooperative monitoring according to the strategy, and the problem of low precision caused by limited resources and dependency problems is solved; in order to solve the sub-flow loss problem of distributed lower-level large-flow cooperative monitoring, the invention provides a distributed level large-flow measurement framework, each measurement task deployed on each switch under the framework has a compact data structure, and the framework consists of a Bloom filter (Bloom filter), a large-flow component (Heavy part) and a small-flow component (Light part), wherein the Bloom filter is used for enabling a plurality of switches to cooperatively monitor the level large-flow according to a strategy of a joint optimization problem, the large-flow component is used for storing an identifier and a count value of a candidate large-flow, and the small-flow component is used for storing the count value of a network flow thrown by the large-flow component and realizing data recovery.
In order to achieve the purpose, the invention adopts the technical scheme that:
a hierarchical big flow cooperative monitoring method is characterized by comprising the following steps:
step 1, a control plane makes a task deployment strategy and a network flow selection strategy and sends the task deployment strategy and the network flow selection strategy to a data plane;
step 2, each switch in the data plane deploys tasks according to the task deployment strategy, meanwhile, task measurement of network flows is carried out according to the network flow selection strategy, and measurement data are periodically uploaded to the control plane;
and 3, carrying out data combination and data recovery on the measurement data uploaded by each switch in the data plane and the control plane, and inquiring the hierarchical large flow identification result to obtain a measurement report.
Further, in step 1, the task deployment strategy and the network flow selection strategy satisfy the following joint optimization constraints:
Figure BDA0003644730050000021
x＝(xjl∈{0,1}:vj∈V,tl∈T)，
y＝(yijl∈{0,1}:fi∈F,vj∈V,tl∈T)，
Figure BDA0003644730050000022
yijl＜xjl,fi∈F,vj∈V,tl∈T，
Figure BDA0003644730050000023
wherein V represents a switch set, F represents a network flow set, and T represents a task set; f. ofiRepresenting the ith network flow, v, in the set F of network flowsjDenotes the jth switch, F, in the set of switches VjTransit switch v represented in network flow set FjOf network flows, ViRepresenting a network flow fiSet of switches passed, tlRepresents the l-th task in the task set T, rlRepresenting a task tlSize of occupied memory space, RjRepresenting exchanges vjA storage space constraint of; x and y are both indication vectors, respectively indicate task deployment and network flow selection strategies, xjlAs task tlAt the exchange vjIndication variable of whether deployment is in: x is the number ofjl1 denotes task tlAt exchange vjIn deployment, x jl0 denotes task tlAt the exchange vjMiddle undeployed, yijlFor network flow fiTask t oflAt the exchange vjIs measured or not: y isijl1 denotes a network flow fiTask t of (1)lAt exchange vjMiddle measurement, y ijl0 denotes the network flow fiTask t oflAt the exchange vjNo measurement in (1).
Furthermore, the solution process of the joint optimization constraint is as follows: performing linear relaxation on the joint optimization constraint and solving to obtain an indication vector
Figure BDA0003644730050000024
And
Figure BDA0003644730050000025
for indicating vectors
Figure BDA0003644730050000031
For the purpose of
Figure BDA0003644730050000032
In (1)
Figure BDA0003644730050000033
By probability
Figure BDA0003644730050000034
X is to bejlIs set to 1; setting an entrance switch to deploy all measurement tasks;
for indicating a vector
Figure BDA0003644730050000035
Adopting off-line solution or on-line solution;
and (3) offline solution: for each network flow fiEach task t oflAccording to probability
Figure BDA0003644730050000036
With task t deployed on its forwarding pathlSelects one of the internal switches (2) and corresponds to the indication variable yijlIs set to 1; if no internal switch is selected, selecting the corresponding inlet switch and corresponding to the indication variable yijlIs set to 1;
and (3) online solution: using greedy thinking in network flow fiFor each task t of the first arrivallSelecting in network flow fiThe task t is deployed on the forwarding pathlThe switch with the minimum current measurement load is measured, and the corresponding indicator variable y is measuredijlIs set to 1.
Further, in step 2, the data structure of each measurement task deployed on each switch in the data plane includes three parts: a bloom filter, a large flow component, and a small flow component; wherein, the bloom filter is used for realizing the cooperative monitoring of the network flow by the whole network switch, and comprises: implementation indicating variable yijlInquiring and generalizing prefix inquiry of network flow corresponding to the task measured by the current switch; the big stream part is used for storing the candidatesAn identifier of a large flow and a count value, and a small flow component for storing the count value of the network flow thrown by the large flow component.
Further, in step 2, the specific process of task measurement is as follows: for each network flow fiEach measurement task t oflIn network flow fiJudging whether a task t should be carried out on the current switch or not by a bloom filter in each switch on a forwarding pathlThe measurement of (2); if yes, the current task t is split according to the granularity of the task splitlCorresponding network flow fiThe generalized prefix of the flow component is used as an identifier to be inserted into the large flow component and the bloom filter, and the count value corresponding to the large flow component is updated; and, if the network flow f 'is thrown out of the big flow component'iThen the network flow f is senti' insert into the small flow element and update the corresponding calculated value of the small flow element.
Further, in step 3, the specific process of the data synthesis is as follows: a candidate big flow list is obtained from a big flow section of each switch in the data plane, and count values of network flows having the same identifier are merged as an estimate value of the network flow size.
Further, in step 3, the specific process of data recovery is as follows: according to the candidate big flow list, inquiring the bloom filter of each switch and judging whether the current switch measures the network flow or not for the identifier of each network flow; when the query result is true, querying the major flow component of the network flow, judging whether the identifier of the network flow exists, if so, returning to 0, and if not, querying the minor flow component of the network flow and returning to a count value; and accumulating the count value of the returned small flow component and the estimated value of the current network flow to be used as a new estimated value of the network flow, thereby completing the recovery process of the network flow measurement data.
Further, in step 3, the specific process of result query is as follows: the result query starts from the bottom layer, and whether the estimation value of each network flow is larger than a preset threshold value is judged; if so, reporting the current hierarchical big flow as the current hierarchical big flow, and deducting the count values of all the parent prefixes of the current hierarchical big flow by using the estimated value of the current hierarchical big flow; the operation is carried out from bottom to top, and the large flows identified in each hierarchy are gathered together to become the hierarchical large flows identified in the whole network.
In conclusion, the beneficial effects of the invention are as follows: the provided hierarchical large flow cooperative monitoring method mainly comprises the following steps:
1. task deployment and network flow selection joint optimization problem:
1) deployment of hierarchical large flow measurement tasks in a distributed network environment
In the hierarchical large flow cooperative monitoring, each network flow needs to complete corresponding measurement at h levels, the measurement of each level l from bottom to top is regarded as a measurement task, and the h measurement tasks are deployed in a switch of the whole network under certain constraint conditions, which is defined as a task deployment problem;
2) selective measurement of a flow through a network by a network measurement switch
In the hierarchical large flow cooperative monitoring, each measurement task of each network flow selects one switch from the switches with the task in the process of passing through the switch for measurement, and the problem is defined as the network flow selection problem;
3) joint optimization problem of hierarchical large flow measurement task deployment and network flow selection
The task deployment and network flow selection strategies are obtained by modeling task deployment and network flow selection joint optimization problems and approximately solving the problems; the approximate solution scheme has strong adaptability, so that the load of the whole network switch is more balanced, the resource is more fully and reasonably utilized, and a better load balancing effect is realized; meanwhile, the precision of the level heavy current measurement is improved under the better load balancing effect.
2. Distributed hierarchical big flow measurement framework:
1) distributed collaborative monitoring using bloom filters
The bloom filter is used for realizing cooperative monitoring of the network flow by the switch of the whole network, and mainly has two functions: for dependent on an indicating variable yijlImplementing the function of selecting the monitoring network flow on the exchange, i.e. querying the current network flow fiTask t of (1)lWhether or not to switch vjPerforming the measurement; the method is used for realizing generalized prefix query of network flows corresponding to tasks measured by a current switch, and inserting the generalized prefixes of the corresponding network flows into a bloom filter in the task measurement process so as to query whether the network flows identified by the prefixes are subjected to task measurement at the current switch or not according to the generalized prefixes in the data recovery process;
2) monitoring structure design of large flow component and small flow component combination
The large flow component is used for storing the identifier and the count value of the candidate large flow, and can filter most of the flow by setting the large flow component, so that the over-estimation problem of the small flow component is relieved; the big stream component may specifically be a counter-based algorithm (e.g. Space Saving); the small flow part is used for storing the count value of the network flow thrown by the large flow part, namely the count value used for keeping the small flow is very important for recovering the lost count value; the streamlet component may specifically be a Sketch-based (Sketch) algorithm (e.g., Count-Min Sketch);
the distributed hierarchical big flow measurement framework provided by the invention can support the whole network switch to cooperatively monitor the hierarchical big flow through the bloom filter, and can improve the measurement precision of the hierarchical big flow after the small flow component recovers the lost sub-flow by selecting a proper data structure in the big flow component and the small flow component to carry out measurement.
Drawings
FIG. 1 is a diagram illustrating an architecture of a software-defined measurement system according to an embodiment of the present invention.
Fig. 2 is a schematic diagram of task deployment and network flow selection policy in an embodiment of the present invention.
Fig. 3 is a schematic diagram of a distributed hierarchical mainstream measurement framework according to an embodiment of the present invention.
Detailed Description
To make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the following embodiments and accompanying drawings.
The invention aims to carry out hierarchical large-flow cooperative monitoring in the whole network and ensure that the measurement load of each switch is equivalent; in short, each network flow can complete all measurement tasks through the cooperation of a plurality of switches on the forwarding path of the network flow, and each switch only needs to deploy part of the measurement tasks, so that the resource utilization rate is improved in a finer-grained manner; it is important to ensure that each network flow can complete all measurement tasks through cooperation of multiple switches on its forwarding path. In cooperative monitoring, on one hand, a measurement task of a hierarchical large flow is expected to be deployed to a switch of a whole network under resource constraint, which is called as task deployment; on the other hand, for each task of each network flow, a switch with the task deployed therein is selected from switches on a forwarding path thereof to perform task measurement, which is referred to as network flow selection in the present invention.
Based on this, the invention provides a hierarchical big flow cooperative monitoring method, which comprises the following steps: task deployment, network flow selection and distributed hierarchical large flow measurement; the task deployment and network flow selection are used for acquiring task deployment and network flow selection strategies of hierarchical massive flow cooperative monitoring; the distributed hierarchical big flow measurement is used for carrying out cooperative monitoring on the hierarchical big flow according to a strategy and periodically carrying out data merging, recovery and query; the following examples are given to further illustrate the present invention.
Example 1
The embodiment provides a hierarchical concurrent monitoring method for a massive flow, which is implemented based on a software defined measurement system, and the software defined measurement system is shown in fig. 1, and specifically includes: the system comprises a control plane and a data plane, wherein the data plane is composed of a plurality of programmable switches; the control plane is responsible for formulating task deployment and network flow selection strategies, issuing the strategies to the programmable switch of the data plane, and combining, recovering and reporting the periodically uploaded information of the data plane; and the programmable switch of the data plane is responsible for measuring network flow according to the strategy of the control plane, periodically collecting data and uploading the data to the control plane.
In this embodiment, the hierarchical concurrent massive flow monitoring method based on the software defined measurement system includes the following steps:
step 1, a control plane formulates a task deployment strategy and a network flow selection strategy and sends the task deployment strategy and the network flow selection strategy to a data plane;
the task deployment strategy is as follows: which measurement tasks are deployed on each switch, wherein the network flow selection strategy refers to which switch each measurement task of each network flow is selected to be performed on; the key of cooperative monitoring is to formulate a task deployment strategy and a network flow selection strategy, so that the invention establishes the joint optimization problem of task deployment and network flow selection as follows:
Figure BDA0003644730050000061
the constraint conditions are as follows:
x＝(xjl∈{0,1}:vj∈V,tl∈T) (2)
y＝(yijl∈{0,1}:fi∈F,vj∈V,tl∈T) (3)
Figure BDA0003644730050000062
yijl＜xjl,fi∈F,vj∈V,tl∈T (5)
Figure BDA0003644730050000063
v represents a switch set, F represents a network flow set, and T represents a task set; f. ofiRepresents the ith network flow, v, in the set F of network flowsjDenotes the jth switch, F, in the set of switches VjTransit switch v represented in network flow set FjOf network flows, ViRepresenting a network flow fiSet of switches passed, tlRepresenting the ith task in the task set T, rlRepresenting a task tlSize of occupied memory space, RjRepresenting exchanges vjOf a storage spaceConstraining;
in the task deployment and network flow selection joint optimization problem, the joint optimization target shown in the formula (1) is to minimize the maximum measurement load of the whole network switch, and a task deployment and network flow selection strategy is found, namely an x and y vector formula is obtained; (2) and (3) in the formula (3), x and y are both indicating vectors and respectively indicate task deployment and network flow selection strategies, wherein xjlAs task tlAt the exchange vjIndication variable of whether deployment is in: x is the number ofjl1 denotes task tlAt the exchange vjIn deployment, xjl0 denotes task tlAt the exchange vjNot deployed in, yijlFor network flow fiTask t of (1)lAt exchange vjOf (2) is measured or not: y isijlNetwork flow f is denoted by 1iTask t oflAt the exchange vjMiddle measurement, yijl0 denotes the network flow fiTask t of (1)lAt exchange vjNo measurement in; equation (4) for ensuring per network flow fiEach task t oflSwitch V capable of being on its forwarding pathiSelecting one switch for measurement; equation (5) for ensuring for network flow fiSelected ongoing task tlV. of a switchjThe task must be deployed; equation (6) for ensuring deployment at switch vjThe total amount of memory space occupied by all measurement tasks in (a) should not exceed its memory capacity limit.
Further, since the joint optimization problem is an integer programming problem, the embodiment performs linear relaxation and solves the problem, that is, the constraint equations (2) and (3) are replaced by the following equations:
x＝(xjl∈[0,1]:vj∈V,tl∈T) (7)
y＝(yijl∈[0,1]:fi∈F,vj∈V,tl∈T) (8)
solutions due to linear relaxation (respectively in
Figure BDA0003644730050000071
And
Figure BDA0003644730050000072
expressed) is of floating-point type, therefore, the present embodiment obtains an integer solution by rounding according to probability, specifically:
1) task deployment strategy: to is directed at
Figure BDA0003644730050000073
In (1)
Figure BDA0003644730050000074
By probability
Figure BDA0003644730050000075
X is to bejlIs set to 1;
particularly, in order to ensure that each task of each network flow can be executed by finding a switch, all measurement tasks are deployed by setting an entrance switch according to the result of the approximate solution;
2) network flow selection policy: an off-line or on-line mode is adopted;
off-line: for each network flow fiEach task t oflAccording to probability
Figure BDA0003644730050000076
With task t deployed on its forwarding pathlSelects one of the internal switches (2) and corresponds to the indication variable yijlIs set to 1; if no internal switch is selected, selecting the corresponding entry switch and corresponding to the indication variable yijlIs set to 1;
online: using greedy thinking in network flow fiFor each task t of the first arrivallSelecting a network flow fiThe task t is deployed on the forwarding pathlOf the switches of (1) that are currently measuring the least loaded switch, i.e. the corresponding indicator variable yijlIs set to 1.
Step 2, each switch in the data plane deploys tasks according to the task deployment strategy, meanwhile, task measurement of network flows is carried out according to the network flow selection strategy, and measurement data are periodically uploaded to the control plane;
further, in the data plane, the data structure of each measurement task deployed on each switch includes three parts: a bloom filter, a large flow component and a small flow component; wherein the content of the first and second substances,
the bloom filter is used for realizing cooperative monitoring of the network flow by the switch of the whole network, and mainly has two functions: for dependent on an indicator variable yijlImplementing the function of selecting the monitoring network flow on the exchange, i.e. querying the current network flow fiTask t of (1)lWhether or not to switch vjPerforming the measurement; the method is used for realizing generalized prefix query of network flows corresponding to tasks measured by a current switch, and inserting the generalized prefixes of the corresponding network flows into a bloom filter in the task measurement process so as to query whether the network flows identified by the prefixes are subjected to task measurement at the current switch or not according to the generalized prefixes in the data recovery process;
the large flow component is used for storing the identifier and the count value of the candidate large flow, most of flow can be filtered out by setting the large flow component, and the problem of overhigh estimation of the small flow component is relieved; the big stream component may specifically be a counter-based algorithm (e.g. Space Saving);
the small flow part is used for storing the count value of the network flow thrown by the large flow part, namely the count value of the small flow is reserved, and the method is very important for recovering the lost count value; the streamlet component may specifically be a Sketch (Sketch) based algorithm (e.g. Count-Min Sketch);
further, the specific process of task measurement is as follows: for each network flow fiEach measurement task t oflIn network flow fiJudging whether a task t should be carried out on the current switch or not by a bloom filter in each switch on a forwarding pathlMeasuring (2); if yes, the current task t is split according to the granularity of the task splitlCorresponding network flow fiThe generalized prefix of (2) is used as an identifier to be inserted into the big flow component and the bloom filter, and the count value corresponding to the big flow component is updated; and, if a network flow f is thrown in the big flow componenti′，The network flow f is senti' insert in the small flow element and update the corresponding calculated value of the small flow element.
Step 3, for the measurement data uploaded by each switch in the data plane, the control plane performs data merging and data recovery, and queries the hierarchical large flow identification result to obtain a measurement report;
further, the specific process of the data synthesis is as follows: acquiring a candidate big flow list from a big flow component of each switch in a data plane, and combining count values of network flows with the same identifier to serve as an estimated value of the size of the network flows;
the specific process of the data recovery is as follows: according to the candidate big flow list, inquiring the bloom filter of each switch and judging whether the current switch measures the network flow or not for the identifier of each network flow; when the query result is true, querying a large flow component of the network flow, judging whether an identifier of the network flow exists or not (if the query result is not true, no operation is needed), if so, returning 0 (indicating that no count value possibly lost exists on the current switch), and if not, querying a small flow component of the network flow and returning the count value; accumulating the count value of the returned small flow component and the estimated value of the existing network flow to be used as a new estimated value of the network flow, thereby completing the recovery process of the network flow measurement data;
the specific process of the result query is as follows: the result query starts from the bottom layer, and whether the estimation value of each network flow is larger than a preset threshold value is judged; if yes, reporting the current level as a big flow of the current level, and deducting the count values of all the parent prefixes of the current level by using the estimated value of the current level (if no, not performing any operation); the operation is carried out from bottom to top, and the large flows identified in each hierarchical report are gathered together to become the hierarchical large flows identified in the whole network.
Fig. 2 is a schematic diagram of a task deployment policy and a network flow selection policy in this embodiment, where a network topology of a data plane is composed of 5 switches: v. of1、v2、v3、v4And v5Total 6 network flows: f. of1、f2、f3、f4、f5And f6Each network flow needs to complete 5 levels of measurement tasks t1、t2、t3、t4And t5Each switch can only deploy 3 measurement tasks under the resource constraint; therefore, it is not possible for any one network flow to complete all measurement tasks on only one switch; as can be seen from FIG. 2, the data plane has deployed part of the measurement tasks, e.g., task t, on each switch according to the task deployment policy from the control plane2、t4And t5Deployed in switch v1Upper, task t1、t3And t5Deployed in switch v2C, removing; there are two possible forwarding paths a-v between hosts a and B1-v2-v3-B and A-v1-v4-v3B, according to the task deployment strategy given in fig. 2, all measurement tasks can be completed by network flows on any forwarding path, since they can cooperate to provide measurements for all 5 tasks through multiple switches; similarly, between hosts A and C, all measurement tasks may also be on paths A-v1-v4-v5Is completed on-C. In order to enable each network flow to complete all measurement tasks under the cooperation of a plurality of switches, a corresponding switch for completing each measurement task is selected for each network flow according to a network flow selection strategy, the strategy can ensure that each network flow completes all measurement tasks under the cooperation of a plurality of switches, and the measurement load of each switch is 6; for example, the network flow t4Task t of4At the exchange v1Upper measurement, task t1、t3And t5At exchange v2Upper measurement, task t2At the exchange v3Performing upper measurement; for network flow f2Task t thereof2And t5Selecting at switch v1Upper measurement, task t1And t4Selecting at switch v3Upper measurement, and task t3Then choose to be in switch v4And (4) measuring. It follows that the strategy of completing all measurement tasks under the same switch as the network flow (assumingAll measurement tasks can be deployed on one switch, at least one switch needs to measure all tasks of two network flows, namely the measurement load of at least one switch is 10).
FIG. 3 is a diagram of a distributed hierarchical big stream measurement framework, in which the big stream component selects Space Saving and the small stream component selects CM Sketch (Count-Min Sketch); the data plane contains three switches, and the current data flow has been updated according to the update algorithm of the corresponding structure and the update process of the above-mentioned framework. The control plane obtains candidate macroflows { (10.1.1. fill, 40), (10.1.2. fill, 9), (10.2.1. fill, 19) } from the macroflow components of the three switches in the data plane; at this time, it can be known from the candidate large flow list that the identifiers of the two network flows with the largest count value are (10.1.1 °) and (10.2.1 °), respectively, which are different from (10.1.1 °) and (10.1.2 °) given by the accurate result; further, according to the candidate big stream list, the missing values are recovered from the small stream component by the following steps: the network flow (10.1.1.) is at switch v due to the query result according to the bloom filter2Measured in, but at v2There is no corresponding record in the large stream component, so the count value that may be lost is 8 by looking up from the small stream component; similarly, for network flow (10.1.2.), it exists in switch v1The count value that may be lost is thus obtained as 20; however, for network flows (10.2.1.) it is recorded in switch v3The count value that may be lost is therefore 0; after recovering the missing count value, a new candidate big flow can be obtained at this time as { (10.1.1. times., 48), (10.1.2. times., 29), (10.2.1. times., 19) }, and the identifiers of the two network flows with the largest count value at this time are (10.1.1.) and (10.1.2.), respectively, consistent with an accurate result.
Where mentioned above are merely embodiments of the invention, any feature disclosed in this specification may, unless stated otherwise, be replaced by alternative features serving equivalent or similar purposes; all of the disclosed features, or all of the method or process steps, may be combined in any combination, except mutually exclusive features and/or steps.

Claims (8)

1. A hierarchical big flow cooperative monitoring method is characterized by comprising the following steps:
step 1, a control plane makes a task deployment strategy and a network flow selection strategy and sends the task deployment strategy and the network flow selection strategy to a data plane;
step 2, each switch in the data plane deploys tasks according to the task deployment strategy, meanwhile, task measurement of network flows is carried out according to the network flow selection strategy, and measurement data are periodically uploaded to the control plane;
and 3, carrying out data combination and data recovery on the measurement data uploaded by each switch in the data plane and the control plane, and inquiring the hierarchical large flow identification result to obtain a measurement report.
2. The method for cooperatively monitoring the hierarchical big flow according to claim 1, wherein in the step 1, the task deployment strategy and the network flow selection strategy satisfy the following joint optimization constraints:
Figure FDA0003644730040000011
x＝(xjl∈{0,1}:vj∈V,tl∈T)，
y＝(yijl∈{0,1}:fi∈F,vj∈V,tl∈T)，
Figure FDA0003644730040000012
yijl＜xjl,fi∈F,vj∈V,tl∈T，
Figure FDA0003644730040000013
v represents a switch set, F represents a network flow set, and T represents a task set; f. ofiRepresents the ith network flow, v, in the set F of network flowsjDenotes the jth switch, F, in the set of switches VjTransit switch v represented in network flow set FjOf network flows, ViRepresenting a network flow fiSet of switches passed, tlRepresenting the ith task in the task set T, rlRepresenting a task tlSize of occupied memory space, RjRepresenting exchanges vjA storage space constraint of; x and y are both indication vectors, respectively indicate task deployment and network flow selection strategies, xjlAs task tlAt exchange vjIndication variable of whether deployment is in: x is the number ofjl1 denotes task tlAt exchange vjIn deployment, xjl0 denotes task tlAt the exchange vjNot deployed in, yijlFor network flow fiTask t oflAt the exchange vjIs measured or not: y isijlNetwork flow f is denoted by 1iTask t oflAt the exchange vjMiddle measurement, yijl0 denotes the network flow fiTask t oflAt the exchange vjNo measurement in (1).
3. The method for cooperative monitoring of hierarchical high flows according to claim 1, wherein in step 2, the data structure of each measurement task deployed on each switch in the data plane comprises three parts: a bloom filter, a large flow component and a small flow component; wherein, the bloom filter is used for realizing the cooperative monitoring of the network flow by the whole network switch, and comprises: implementation indicating variable yijlInquiring and generalizing prefix inquiry of network flow corresponding to the task measured by the current switch; the large flow component is used for storing the identifier and the count value of the candidate large flow, and the small flow component is used for storing the count value of the network flow thrown by the large flow component.
4. The method for cooperatively monitoring the hierarchical high flow according to claim 1, wherein in the step 2, the specific process of task measurement is as follows: for each network flow fiEach measurement task t oflIn network flow fiJudging whether a task t should be carried out on the current switch or not by a bloom filter in each switch on a forwarding pathlMeasuring (2); if yes, the current task t is split according to the granularity of the task splittinglCorresponding network flow fiThe generalized prefix of (2) is used as an identifier to be inserted into the big flow component and the bloom filter, and the count value corresponding to the big flow component is updated; and, if the network flow f is thrown out of the big flow componenti', then the network flow f is senti' insert into the small flow element and update the corresponding calculated value of the small flow element.
5. The hierarchical mainstream cooperative monitoring method according to claim 1, wherein in the step 3, the specific process of data combination comprises: a candidate big flow list is obtained from the big flow section of each switch in the data plane, and count values of network flows having the same identifier are merged as an estimate value of the network flows.
6. The hierarchical concurrent massive flow monitoring method as claimed in claim 1, wherein in step 3, the specific process of data recovery is: according to the candidate big flow list, inquiring the bloom filter of each switch and judging whether the current switch measures the network flow or not for the identifier of each network flow; when the query result is true, querying a major flow component of the network flow, judging whether the identifier of the network flow exists, if so, returning to 0, and if not, querying a minor flow component of the network flow and returning a count value; and accumulating the count value of the returned small flow component and the estimation value of the current network flow to be used as a new network flow estimation value, namely finishing the data recovery of the network flow measurement data.
7. The hierarchical mainstream cooperative monitoring method according to claim 1, wherein in the step 3, the specific process of the result query is as follows: the result query starts from the bottommost layer, and whether the estimation value of each network flow is larger than a preset threshold value is judged; if so, reporting the current hierarchical big flow as the current hierarchical big flow, and deducting the count values of all the parent prefixes of the current hierarchical big flow by using the estimated value of the current hierarchical big flow; and performing the operation from bottom to top, and summarizing the large flows identified in each hierarchy into the hierarchical large flows identified in the whole network.
8. The hierarchical high-flow cooperative monitoring method according to claim 2, wherein the solution process of the joint optimization constraint is as follows: performing linear relaxation on the joint optimization constraint and solving to obtain an indication vector
Figure FDA0003644730040000021
And
Figure FDA0003644730040000022
for indicating a vector
Figure FDA0003644730040000023
For the purpose of
Figure FDA0003644730040000024
In (1)
Figure FDA0003644730040000025
By probability
Figure FDA0003644730040000026
X is to bejlIs set to 1; setting an entrance switch to deploy all measurement tasks;
for indicating vectors
Figure FDA0003644730040000027
Adopting off-line solution or on-line solution;
and (3) offline solution: for each network flow fiEach task t oflAccording to probability
Figure FDA0003644730040000028
With task t deployed on its forwarding pathlSelects one of the internal switches (2) and corresponds to the indication variable yijlIs set to 1; if no internal switch is selected, selecting the corresponding entry switch and corresponding to the indication variable yijlIs set to 1;
and (3) online solution: using greedy thought in network flows fiFor each task t of the first arrivallSelecting in network flow fiThe task t is deployed on the forwarding pathlThe switch with the minimum current measurement load is measured, and the corresponding indicator variable y is measuredijlIs set to 1.
CN202210526869.4A 2022-05-16 2022-05-16 Hierarchical large-flow collaborative monitoring method Active CN114785707B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210526869.4A CN114785707B (en) 2022-05-16 2022-05-16 Hierarchical large-flow collaborative monitoring method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210526869.4A CN114785707B (en) 2022-05-16 2022-05-16 Hierarchical large-flow collaborative monitoring method

Publications (2)

Publication Number Publication Date
CN114785707A true CN114785707A (en) 2022-07-22
CN114785707B CN114785707B (en) 2023-06-20

Family

ID=82437999

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210526869.4A Active CN114785707B (en) 2022-05-16 2022-05-16 Hierarchical large-flow collaborative monitoring method

Country Status (1)

Country Link
CN (1) CN114785707B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116032829A (en) * 2023-03-24 2023-04-28 广东省电信规划设计院有限公司 SDN network data stream transmission control method and device

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102770845A (en) * 2009-12-04 2012-11-07 国际商业机器公司 Optimization of archive management scheduling
CN104301246A (en) * 2014-10-27 2015-01-21 盛科网络（苏州）有限公司 Large-flow load balanced forwarding method and device based on SDN
US9252972B1 (en) * 2012-12-20 2016-02-02 Juniper Networks, Inc. Policy control using software defined network (SDN) protocol
CN106357547A (en) * 2016-09-08 2017-01-25 重庆邮电大学 Software-defined network congestion control algorithm based on stream segmentation
CN107395693A (en) * 2017-07-04 2017-11-24 大连工业大学 The hospital clinical operation data selection equipment for the size stream classification applied in cloud data center system
CN112367217A (en) * 2020-10-20 2021-02-12 武汉大学 Cooperative type large flow detection method and system oriented to software defined network
CN113839835A (en) * 2021-09-27 2021-12-24 长沙理工大学 Top-k flow accurate monitoring framework based on small flow filtering

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102770845A (en) * 2009-12-04 2012-11-07 国际商业机器公司 Optimization of archive management scheduling
US9252972B1 (en) * 2012-12-20 2016-02-02 Juniper Networks, Inc. Policy control using software defined network (SDN) protocol
CN104301246A (en) * 2014-10-27 2015-01-21 盛科网络（苏州）有限公司 Large-flow load balanced forwarding method and device based on SDN
CN106357547A (en) * 2016-09-08 2017-01-25 重庆邮电大学 Software-defined network congestion control algorithm based on stream segmentation
CN107395693A (en) * 2017-07-04 2017-11-24 大连工业大学 The hospital clinical operation data selection equipment for the size stream classification applied in cloud data center system
CN112367217A (en) * 2020-10-20 2021-02-12 武汉大学 Cooperative type large flow detection method and system oriented to software defined network
CN113839835A (en) * 2021-09-27 2021-12-24 长沙理工大学 Top-k flow accurate monitoring framework based on small flow filtering

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
DA TONG，: ""High Throughput Hierarchical Heavy Hitter Detection in Data Streams"", 《HIGH THROUGHPUT HIERARCHICAL HEAVY HITTER DETECTION IN DATA STREAMS》 *
WENTAO WANG，ET AL.，: ""A Distributed Hierarchical Heavy Hitter Detection Method in Software-Defined Networking"", 《IEEE ACCESS》 *
谢坤，: ""SDN环境下数据中心网络能源优化机制的研究"", 《中国博士学位论文全文数据库 (信息科技辑)》 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116032829A (en) * 2023-03-24 2023-04-28 广东省电信规划设计院有限公司 SDN network data stream transmission control method and device
CN116032829B (en) * 2023-03-24 2023-07-14 广东省电信规划设计院有限公司 SDN network data stream transmission control method and device

Also Published As

Publication number Publication date
CN114785707B (en) 2023-06-20

Similar Documents

Publication Publication Date Title
CN105379196B (en) Method, system and computer storage medium for the routing of fault-tolerant and load balance
Yu et al. CountMax: A lightweight and cooperative sketch measurement for software-defined networks
CN107276794B (en) Switch migration algorithm in software defined network
CN104935476B (en) A kind of network traffics matrix measuring method based on SDN
US20060168168A1 (en) Assisted determination of data flows in communication/data networks
CN107911242A (en) A kind of cognitive radio based on industry wireless network and edge calculations method
CN111130928B (en) Network measurement method based on in-band detection in wide area network
CN109034562A (en) A kind of social networks node importance appraisal procedure and system
CN114785707A (en) Hierarchical massive stream cooperative monitoring method
CN101536428A (en) Method for tracking network parameters
Shen et al. Geographic location-based network-aware qos prediction for service composition
CN109952743B (en) System and method for low memory and low flow overhead high flow object detection
CN114814420B (en) Low-voltage distribution network topology identification method and system based on frozen data
CN109587000A (en) High latency method for detecting abnormality and system based on collective intelligence network measurement data
CN113507396B (en) Network state analysis method, device, equipment and machine-readable storage medium
CN110351166A (en) A kind of network level fine granularity flow measuring method based on traffic statistics characteristic
CN114401516A (en) 5G slice network anomaly detection method based on virtual network traffic analysis
CN106209404B (en) Analyzing abnormal network flow method and system
CN106649070A (en) Rapid optimization method and system for operating performance of Linux system
CN112994970B (en) In-band network telemetry INT method and system based on capture and arrangement
CN106254403A (en) The moving method of data and device
CN108400907A (en) A kind of link packet drop rate inference method under uncertain network environment
CN110430133A (en) A kind of inter-domain path mark prefix acquisition methods based on confidence interval
Qin et al. MCRA: multicost rerouting algorithm in SDN
Yalagandula et al. Correlations in end-to-end network metrics: impact on large scale network monitoring

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant