CN105471639A - Median-based network flow entropy evaluation method and apparatus - Google Patents

Abstract

The invention provides a median-based network flow entropy evaluation method and apparatus. The method includes the following steps: obtaining a data packet of network data flow and sending the data packet to slave nodes in a Storm cluster; receiving estimations value of an intermediate item S returned by the slave nodes, wherein the estimation values of the S are obtained through a preset entropy estimation algorithm by the slave nodes in the Storm cluster according to the received data packet; ordering the received estimation values of the S according to the magnitude; obtaining a median of the ordered estimation values of the S, taking the median as a final estimation value of the S; and obtaining an estimation value of an entropy of the network data flow according to the final estimation value of the S. In the method, the Storm cluster includes at least one salve node. The method increases the accuracy of entropy estimation while the storage space and the calculating complexity are not increased.

Description

Based on network traffics entropy evaluation method and the device of median

Technical field

The present invention relates to network traffic analysis technical field, particularly relate to a kind of network traffics entropy evaluation method based on median and device.

Background technology

In network traffic analysis field, the analysis based on flow point cloth can obtain than based on the abundanter result of simple stream quantitative analysis usually.The feature of flow distribution can be summarized and refine to entropy well, and be the extremely important one tolerance of network traffics, such as, the algorithm of a lot of Network anomaly detection all depends on the entropy of network flow.How quick on the net in high-speed backbone, the entropy of computing network flow is for abnormality detection in real time, traffic engineering, and the field tools such as flow analysis are of great significance.Calculate entropy and need the size recording each network flow, but the flow of backbone network is very large, the number of interior network flow per minute is tens of thousands easily, wants the state recording every bar stream to need to consume a large amount of memory spaces, increase the complexity of calculating widely, be difficult to accomplish real-time calculating.

And the entropy evaluation method of prior art, often memory space and computation complexity lower but entropy estimation accuracy rate not high, or the higher but memory space of entropy estimation accuracy rate and computation complexity higher.

Given this, the accuracy rate how improving entropy estimation when not increasing memory space and computation complexity becomes the current technical issues that need to address.

Summary of the invention

For solving above-mentioned technical problem, the invention provides a kind of network traffics entropy evaluation method based on median and device, the accuracy rate of entropy estimation can be improved when not increasing memory space and computation complexity.

First aspect, the invention provides a kind of network traffics entropy evaluation method based on median, comprising:

Obtain the packet of network data flow, and be sent to each from node of Storm cluster;

The estimated value of each middle entry S returned from node receiving described Storm cluster, the estimated value of described S be described Storm cluster each from node according to the packet received, obtained by default entropy estimating algorithm;

The estimated value of the S of reception is sorted according to size;

Obtain the median of estimated value of all S after sequence, using described median as the final estimated value of S

According to described obtain the estimated value of the entropy of described network data flow network traffics

Wherein, described Storm cluster comprises at least one from node.

Alternatively, described middle entry S is obtained by the first formulae discovery, and described first formula is:

$S=\frac{1}{m}{\Σ}_i{m}_i{\mathrm{logm}}_i;$

Wherein, the packet in described network data flow represents with i, m _ifor the frequency that packet i occurs in network data flow, i ∈ 1,2 ..., 2 ^4b, b is the byte length of individual data bag in described network data flow.

Alternatively, described in described basis obtain the estimated value of the entropy of network traffics comprise:

According to described by the second formula, obtain the estimated value of the entropy of network traffics

Described second formula is:

$\tilde{H}=log\left(m\right)-\tilde{S},$

Wherein, m is the total length of network data flow, and m is obtained by the 3rd formulae discovery;

Described 3rd formula is:

$m={\Σ}_{i=1}^n{m}_i,$

N is positive integer.

Alternatively, described default entropy estimating algorithm is entropy estimation approximate data, comprising:

Random selecting g × z position in described network data flow, generates random site set;

For the packet a of jth in described network data flow _j=i, judges whether j belongs to described random site set, if j belongs to described random site set, then for i increases a counter, and the counter set be associated with i is all added 1; If j does not belong to described random site set, and there is the counter be associated with i, then the counter be associated with i is added 1;

After having judged whether the j of all packets in described network data flow belongs to described random site set, obtain the counter matrices C=(c of g × z _pq);

According to described counter matrices C, build matrix X=(x _pq);

Obtain the mean value of described each row element of matrix X;

Obtain the median of all mean value, and using the estimated value of this median as S;

Wherein, x _pq=m × (c _pqlogc _pq-(c _pq-1) log (c _pq-1)), ε is relative evaluated error, and 1-δ is for estimating accuracy rate.

Alternatively, the mean value of each row element of the described matrix X of described acquisition, comprising:

By the 4th formula, obtain the mean value avg [p] of described each row element of matrix X;

Wherein, described 4th formula is:

$avg\[p\]=\frac{1}{z}{\Σ}_{q=1}^z{x}_{pq},p=1,2,...,g;q=1,2,...,z.$

Alternatively, described default entropy estimating algorithm is entropy estimation filtering algorithm, comprising:

According to default sample rate, the packet in the network data flow obtained is sampled;

Judge a jth packet a in described network data flow _jwhether=i is selected and whether there is the counter be associated with i;

If a jth packet a in network data flow _j=i is selected, and there is not the counter be associated with i, then create a counter for i and i is labeled as rill;

If a jth packet a in network data flow _j=i is selected, and there is the counter be associated with i, then i is labeled as large stream;

If a jth packet a in network data flow _j=i is not selected, and there is the counter be associated with i, then the counter be associated with i is added 1;

Judging a jth packet a in network data flow _jwhether=i is selected and after whether there is the counter that is associated with i, obtains being labeled as the counter matrices E of large stream and being labeled as the counter matrices M of rill;

According to the described counter matrices E being labeled as large stream, obtain the contribution margin S of large stream _e;

According to the described counter matrices M being labeled as rill, build matrix Y=(y _pq);

Obtain the mean value of described each row element of matrix Y;

Obtain the median of all mean value, and using the contribution margin S of this median as rill _m;

According to described S _eand S _m, obtain the estimated value of S;

Wherein, E=(E _t), t=1,2 ..., e; E is positive integer;

M＝(M _pq)，p＝1,2,…,g；q＝1,2,…,z；

y _pq＝m×(M _pqlogM _pq-(M _pq-1)log(M _pq-1))，p＝1,2,…,g；q＝1,2,…,z。

Alternatively, described in described basis, be labeled as the counter matrices E of large stream, obtain the contribution margin S of large stream _e, comprising:

According to the described counter matrices E being labeled as large stream, by the 5th formula, obtain the contribution margin S of large stream _e;

Wherein, described 5th formula is:

$S_e={\Σ}_{t=1}^e{E}_t{\mathrm{logE}}_t,t=1,2,...,e;$

And/or,

Described according to described S _eand S _m, obtain the estimated value of S, comprising:

According to described S _eand S _m, by the 6th formula, obtain the estimated value of S;

Wherein, described 6th formula is:

$\tilde{S}=S_e+S_m;$

And/or,

The mean value of each row element of the described matrix Y of described acquisition, comprising:

By the 7th formula, obtain the mean value avg [p] of described each row element of matrix Y;

Wherein, described 7th formula is:

$avg\[p\]=\frac{1}{z}{\Σ}_{q=1}^z{y}_{pq},p=1,2,...,g;q=1,2,...,z.$

Second aspect, the invention provides a kind of network traffics entropy estimating device based on median, comprising:

First acquisition module, for obtaining the packet of network data flow, and is sent to each from node of Storm cluster;

Receiver module, for the estimated value of each middle entry S returned from node of receiving described Storm cluster, the estimated value of described S be described Storm cluster each from node according to the packet received, obtained by default entropy estimating algorithm;

Order module, the estimated value for the S by reception sorts according to size;

Second acquisition module, for obtaining the median of the estimated value of all S after sequence, using described median as the final estimated value of S

3rd acquisition module, described in basis obtain the estimated value of the entropy of described network data flow network traffics

Wherein, described Storm cluster comprises at least one from node.

Alternatively, described middle entry S is obtained by the first formulae discovery, and described first formula is:

$S=\frac{1}{m}{\Σ}_i{m}_i{\mathrm{logm}}_i;$

Wherein, the packet in described network data flow represents with i, m _ifor the frequency that packet i occurs in network data flow, i ∈ 1,2 ..., 2 ^4b, b is the byte length of individual data bag in described network data flow.

Alternatively, described 4th acquisition module, specifically for

According to described by the second formula, obtain the estimated value of the entropy of network traffics

Described second formula is:

$\tilde{H}=log\left(m\right)-\tilde{S},$

Wherein, m is the total length of network data flow, and m is obtained by the 3rd formulae discovery;

Described 3rd formula is:

$m={\Σ}_{i=1}^n{m}_i,$

N is positive integer.

As shown from the above technical solution, the network traffics entropy evaluation method based on median of the present invention and device, calculating in real time by Storm platform is parallel, can improve the accuracy rate of entropy estimation when not increasing memory space and computation complexity.

Accompanying drawing explanation

The schematic flow sheet of a kind of network traffics entropy evaluation method based on median that Fig. 1 provides for one embodiment of the invention;

Fig. 2 is the Storm cluster topology schematic diagram of the network traffics entropy evaluation method based on median provided embodiment illustrated in fig. 1 when the default entropy estimating algorithm in embodiment illustrated in fig. 1 is entropy estimation approximate data;

Fig. 3 is the Storm cluster topology schematic diagram of the network traffics entropy evaluation method based on median provided embodiment illustrated in fig. 1 when the default entropy estimating algorithm in embodiment illustrated in fig. 1 is entropy estimation filtering algorithm;

Fig. 4 is that a part for Storm cluster embodiment illustrated in fig. 1 uses entropy estimation approximate data from node, the Storm cluster topology schematic diagram of the network traffics entropy evaluation method based on median provided embodiment illustrated in fig. 1 when residue another part uses entropy estimation filtering algorithm from node;

The structural representation of a kind of network traffics entropy estimating device based on median that Fig. 5 provides for one embodiment of the invention.

Embodiment

For making the object of the embodiment of the present invention, technical scheme and advantage clearly, below in conjunction with the accompanying drawing in the embodiment of the present invention, clear, complete description is carried out to the technical scheme in the embodiment of the present invention, obviously, described embodiment is only the present invention's part embodiment, instead of whole embodiments.Based on embodiments of the invention, those of ordinary skill in the art, not making the every other embodiment obtained under creative work prerequisite, belong to the scope of protection of the invention.

Fig. 1 shows the schematic flow sheet of the network traffics entropy evaluation method based on median that one embodiment of the invention provides, and as shown in Figure 1, the network traffics entropy evaluation method based on median of the present embodiment is as described below.

101, obtain the packet of network data flow, and (by message source spout) is sent to each from node of Storm cluster.

Wherein, described Storm cluster comprises at least one from node.

In a particular application, network flow data bag described in the present embodiment can carry the five-tuple information be made up of the agreement interconnected between source network (InternetProtocol is called for short IP) address, object IP address, source port, destination interface and protocol number.

Will be understood that, this step 101 continuously grabs the packet of network data flow by network interface card, can suppose all packets all from certain feature space 1,2,3 ..., n}.

The estimated value of each middle entry S returned from node 102, receiving described Storm cluster, the estimated value of described S be described Storm cluster each from node according to the packet received, obtained by default entropy estimating algorithm.

In a particular application, described middle entry S is obtained by the first formulae discovery, and described first formula is:

$S=\frac{1}{m}{\Σ}_i{m}_i{\mathrm{logm}}_i;$

Wherein, the packet in described network data flow represents with i, m _ifor the frequency that packet i occurs in network data flow, i ∈ 1,2 ..., 2 ^4b, b is the byte length of individual data bag in described network data flow.

For example, source IP address can be expressed as the binary coding of 4 bytes, and by that analogy, five-tuple can be expressed as the binary coding of 13 bytes.That is, all packets can with one 1 ~ 2 ⁵²an integer i represent, different i is corresponding different network flow.

103, the estimated value of the S of reception is sorted according to size.

104, the median of estimated value of all S after sequence is obtained, using described median as the final estimated value of S

105, described in basis obtain the estimated value of the entropy of described network data flow network traffics

In a particular application, described step 105, can comprise:

According to described by the second formula, obtain the estimated value of the entropy of network traffics

Described second formula is:

$\tilde{H}=log\left(m\right)-\tilde{S},$

Wherein, m is the total length of network data flow, and m is obtained by the 3rd formulae discovery;

Described 3rd formula is:

$m={\Σ}_{i=1}^n{m}_i,$

N is positive integer.

The network traffics entropy evaluation method based on median of the present embodiment, calculating in real time by Storm platform is parallel, can improve the accuracy rate of entropy estimation when not increasing memory space and computation complexity.

In an embody rule, presetting entropy estimating algorithm described in the present embodiment can be entropy estimation approximate data, comprises not shown step S1-S6:

S1, in described network data flow random selecting g × z position, generate random site set.

Wherein, ε is relative evaluated error, and 1-δ is for estimating accuracy rate.

S2, for the packet a of jth in described network data flow _j=i, judges whether j belongs to described random site set, if j belongs to described random site set, then for i increases a counter, and the counter set be associated with i is all added 1; If j does not belong to described random site set, and there is the counter be associated with i, then the counter be associated with i is added 1.

S3, after having judged whether the j of all packets in described network data flow belongs to described random site set, obtain the counter matrices C=(c of g × z _pq).

S4, according to described counter matrices C, build matrix X=(x _pq).

Wherein, x _pq=m × (c _pqlogc _pq-(c _pq-1) log (c _pq-1)), p=1,2 ..., g; Q=1,2 ..., z.

S5, obtain the mean value of described each row element of matrix X.

In a particular application, described step S5, by the 4th formula, obtains the mean value avg [p] of described each row element of matrix X;

Wherein, described 4th formula is:

$avg\[p\]=\frac{1}{z}{\Σ}_{q=1}^z{x}_{pq},p=1,2,...,g;q=1,2,...,z.$

S6, obtain the median of all mean value, and using the estimated value of this median as S.

This entropy estimation approximate data is a kind of (ε, δ) approximate data, and its implication to be more than or equal to the probability of (1-δ), with the relative error of maximum ε estimation entropy, can have following formula:

$\mathrm{Pr}\left(\right|S-\tilde{S}|\≤S\mathrm{\ϵ})\≥1-\mathrm{\δ}.$

Will be understood that, this entropy estimation approximate data, by the selection to z, can realize the control to relative evaluated error ε, by the selection to g, can realize the control to estimating accuracy rate (1-δ).

The advantage of this entropy estimation approximate data only needs reading data flow, do not need to store each packet, only need to safeguard g × z counter, greatly reduce demand and the computation complexity of memory space.Therefore, this algorithm is well suited for the entropy of estimation on line network traffics.But if evaluated error ε to be reduced and increase the accuracy rate (1-δ) estimated, certainly will to increase the value of z and g, thus need the counter number safeguarded greatly to increase.

Fig. 2 shows the Storm cluster topology schematic diagram of the network traffics entropy evaluation method based on median that described default entropy estimating algorithm provides for embodiment of the present invention during entropy estimation approximate data, one of them random site generates bolt, a counting machine generates bolt and and generates the example that S estimated value bolt just constitutes algorithm one, bolt is Message Processing person, the example of n part above-mentioned entropy estimation approximate data is contained in this storm topology, the bolt that the result of each example has delivered to the most right side sorts, get median as final entropy estimated value.

See Fig. 2, the present embodiment based on median network traffics entropy evaluation method Storm platform make Storm cluster each use above-mentioned entropy to estimate from node approximate data walks abreast calculates the estimated value of S in real time, then by the median of the estimated value using all S after sequence as the final estimated value of S and then according to described obtain the estimated value of the entropy of network data flow network traffics the accuracy rate of entropy estimation can be improved when not increasing memory space and computation complexity.

In another embody rule, presetting entropy estimating algorithm described in the present embodiment can be entropy estimation filtering algorithm, comprises not shown step T1-T8:

T1, according to default sample rate r, to obtain network data flow in packet sample.

T2, judge a jth packet a in described network data flow _jwhether=i is selected and whether there is the counter be associated with i; If a jth packet a in network data flow _j=i is selected, and there is not the counter be associated with i, then create a counter for i and i is labeled as rill; If a jth packet a in network data flow _j=i is selected, and there is the counter be associated with i, then i is labeled as large stream; If a jth packet a in network data flow _j=i is not selected, and there is the counter be associated with i, then the counter be associated with i is added 1.

T3, judging a jth packet a in network data flow _jwhether=i is selected and after whether there is the counter that is associated with i, obtains being labeled as the counter matrices E of large stream and being labeled as the counter matrices M of rill.

Wherein, E=(E _t), t=1,2 ..., e; E is positive integer; M=(M _pq), p=1,2 ..., g; Q=1,2 ..., z.

Be labeled as the counter matrices E of large stream described in T4, basis, obtain the large contribution margin S flowed _e.

In a particular application, described step T4 according to the described counter matrices E being labeled as large stream, by the 5th formula, can obtain the contribution margin S of large stream _e;

Wherein, described 5th formula is:

$S_e={\Σ}_{t=1}^e{E}_t{\mathrm{logE}}_t,t=1,2,...,e.$

Be labeled as the counter matrices M of rill described in T5, basis, build matrix Y=(y _pq).

Wherein, y _pq=m × (M _pqlogM _pq-(M _pq-1) log (M _pq-1)), p=1,2 ..., g; Q=1,2 ..., z.

T6, obtain the mean value of described each row element of matrix Y.

In a particular application, described step T6, by the 7th formula, obtains the mean value avg [p] of described each row element of matrix Y;

Wherein, described 7th formula is:

$avg\[p\]=\frac{1}{z}{\Σ}_{q=1}^z{y}_{pq},p=1,2,...,g;q=1,2,...,z.$

T7, obtain the median of all mean value, and using the contribution margin S of this median as rill _m.

T8, according to described S _eand S _m, obtain the estimated value of S.

In a particular application, described step T8 can according to described S _eand S _m, by the 6th formula, obtain the estimated value of S;

Wherein, described 6th formula is:

$\tilde{S}=S_e+S_m.$

This entropy estimation filtering algorithm is a kind of filtering algorithm, and its core concept distinguishes large stream (elephants) and rill (mice), calculates their contributions for final entropy respectively.

Owing to having distinguished size stream, can be more accurate with this entropy estimation filtering algorithm estimation network traffics entropy, but memory space and computation complexity also can slightly increase.Obviously, if we will increase estimation precision and the accuracy rate of filtering algorithm further, just need to increase sample rate r, this inherently increases the burden of process.

Similar Fig. 2, Fig. 3 show the Storm cluster topology schematic diagram of the network traffics entropy evaluation method based on median that described default entropy estimating algorithm provides for embodiment of the present invention during entropy estimation filtering algorithm.See Fig. 3, the present embodiment based on median network traffics entropy evaluation method Storm platform make Storm cluster each use above-mentioned entropy to estimate from node filtering algorithm walks abreast calculates the estimated value of S in real time, then by the median of the estimated value using all S after sequence as the final estimated value of S and then according to described obtain the estimated value of the entropy of network data flow network traffics the accuracy rate of entropy estimation can be improved when not increasing memory space and computation complexity.

In another embody rule, see described in the present embodiment shown in Fig. 4 based on the Storm cluster topology schematic diagram of the network traffics entropy evaluation method of median, the network traffics entropy evaluation method based on median of the present embodiment also can make a part for Storm cluster use above-mentioned entropy to estimate approximate data from node at Storm platform, residue another part uses above-mentioned entropy to estimate filtering algorithm from node, the parallel estimated value calculating S in real time, then by the median of the estimated value using all S after sequence as the final estimated value of S and then according to described obtain the estimated value of the entropy of network data flow network traffics the accuracy rate of entropy estimation can be improved when not increasing memory space and computation complexity.

Will be understood that, the Qie Nuofu of famous multiplication form can be utilized theoretically to define reason (TheoremformultiplicativeformofChernoffbound) prove to use after above-mentioned default entropy estimating algorithm, the network traffics entropy evaluation method based on median of the present embodiment can improve the accuracy rate of entropy estimation when not increasing memory space and computation complexity:

Qie Nuofu defines reason: assuming that X ₁, X ₂..., X _nbe separate with distribution, value is one group of stochastic variable of 0 or 1.Make X represent these stochastic variables and then the mathematic expectaion that μ=E [X] is X is made.Then for arbitrary arithmetic number 1> δ >0, have

$\mathrm{Pr}(X\≤(1-\mathrm{\δ})\mathrm{\μ})\≤e^{-\frac{{\mathrm{\δ}}^2\mathrm{\μ}}{2}}.$

Utilize Qie Nuofu to define reason, the correctness of the innovatory algorithm based on median can be proved.

First, suppose that described Storm cluster comprises n from node, we can be one group of independent identically distributed Bernoulli Jacob's variable X depending on n time operation result of this entropy estimation approximate data ₁, X ₂..., X _n, work as X _i=1, represent that the estimation of entropy is very accurate, work as X _i=0, represent that the estimation of entropy is unacceptable.If Pr is (X _i=1)=p, i=1,2 ..., n, then these Bernoulli Jacob's variablees and expectation be qie Nuofu according to multiplication form defines reason

$\mathrm{Pr}\left({\Σ}_{i=1}^n{X}_i\≤\frac{n}{2}\right)=\mathrm{Pr}\left({\Σ}_{i=1}^n{X}_i\≤\left(1-\left(1-\frac{1}{2p}\right)\right)\mathrm{\μ}\right)\≤e^{-\frac{\mathrm{\μ}}{2}{(1-1/2p)}^2}=$ $e^{-\frac{n}{2p}{(p-1/2)}^2};$

Utilize the complementary theorem in probability theory, have following formula

$\mathrm{Pr}\left({\Σ}_{i=1}^n{X}_i>\frac{n}{2}\right)\≥1-e^{-\frac{1}{2p}n{(p-\frac{1}{2})}^2};$

By the adjustment to random site number g × z and sample rate r, p>1/2 can be made.When n levels off to infinity, trend towards 1.That is, along with the increase running pass n, having more than n/2 result is that the probabilistic approximation of accurately estimation is in 1.

Then, our following proposition that will prove:

Run n and obtain n estimated value all over data flow algorithm, the median of this n estimated value is that the probability of accurate estimated value trends towards 1 along with the increase of n.

Prove: set the exact value of network traffics entropy as H, n estimated value is H ₁, H ₂..., H _n, might as well H be established ₁<H ₂< ... <H _n.According to H ₁, H ₂..., H _nwith the relative position of H, three kinds of situation discussion can be divided.

Situation 1:H ₁<H ₂< ... <H _i<H<H _i+1< ... <H _n;

Obviously, distance H is nearest individual estimated value is the most accurately individual estimated value.Due to the verified increase along with running pass n, have that to exceed half result be that the probabilistic approximation accurately estimated is in 1.Therefore, this individual estimated value is that the probability accurately estimated is tending towards 1.

Below, the median of n estimated value need only be proved whether belong to distance H nearest individual estimated value.Obviously, distance H is nearest individual estimated value is adjacent, might as well be set to H _s<H _s+1< ... <H _t, wherein utilize reduction to absurdity, might as well median be established do not belong to distance H nearest individual estimated value.Obviously, or or if the then number of estimated value produce contradiction.If the then number of estimated value produce contradiction.Therefore, must have i.e. median distance H must be belonged to nearest individual estimated value.According to formula (7), along with the increase of n, have more than n/2 result be the probabilistic approximation accurately estimated in 1, this is equivalent to median that the probability accurately estimated levels off to 1.Proposition must be demonstrate,proved.

Situation 2:H<H ₁<H ₂< ... <H _n

Obviously, now to estimate the most accurate (nearest from H) individual value is due to then median belong to estimation the most accurate individual value.According to formula (7), along with the increase of n, have more than n/2 result be the probabilistic approximation accurately estimated in 1, this is equivalent to median that the probability accurately estimated levels off to 1.Proposition must be demonstrate,proved.

Situation 3:H ₁<H ₂< ... <H _n<H

Proof procedure is similar to situation 2.

Now, proposition card is finished.

According to above-mentioned proposition, we only need run above-mentioned default entropy estimating algorithm n time and get the median of this n estimated value, just greatly can increase and estimate probability accurately.But running n secondary data flow algorithm just needs to read n pass certificate, and namely data must be stored on hard disk, namely can not lose with crossing.The advantage of data flow algorithm will not exist.Further, the time needed for entropy estimation correspondingly increases n doubly, is difficult to accomplish real-time estimation.Method described in the present embodiment utilizes storm cluster, runs the above-mentioned default entropy estimating algorithm of n part concurrently, processing load is distributed on multiple machine, can improves the accuracy of estimation, can meet again the requirement of real-time estimation.

In addition, for above-mentioned entropy estimation approximate data, following some emulation experiments of doing are to prove that the network traffics entropy evaluation method based on median of the present embodiment can improve the accuracy rate of entropy estimation when not increasing memory space and computation complexity.

First, gather the flow of about 2 hours from Chinese education and scientific research computer network CERNET, be stored as the file of about 300Gb.Reset this file flow in Storm cluster by TCPReplay software, the scene of On-line Estimation entropy can be simulated.

Then, selected relative error ε=0.22, according to document, the z value of needs is approximately when getting m=13403402 (flows corresponding to about 5 minutes), the value of z is approximately 10000.In actual applications, relative error can be reduced further by the value increasing z.In addition, we get g=1, then the probability that estimated value meets relative error is about 68.4%.G=1 why is selected to have two reasons.The first, have a mind to force down and estimate probability (but ensure that p>1/2) accurately, obtain clearer displaying to make method described in the present embodiment relative to the improvements of above-mentioned entropy estimation approximate data.The second, g increases by 1 at every turn, just need increase by 10000 counters, and z value controls the error size of estimation, must be guaranteed.So in actual applications, should the size of control g as far as possible.Simulated experiment illustrates the thought utilizing this patent, even if get g=1, also can ensure the accuracy rate estimated.

According to formula, the accuracy rate of estimation be made to be greater than 0.95, n and to be at least 53.So we are according to the topology of Fig. 2, run the example of 70 parts of above-mentioned entropy estimation approximate datas concurrently, wherein the Output rusults of single instance is as shown in table 1.The exact value of the entropy that our hand computation goes out is 4.5, and therefore, for the relative evaluated error of 0.22, the limit of estimated value is 5.5, and the estimation being namely less than 5.5 can be thought accurately, acceptable, and the estimation being greater than 5.5 is considered to unacceptable.Table 1 is the entropy estimated value of single above-mentioned entropy estimation approximate data example, table 1 shows in the Output rusults of 70 bolts has 23 results to be unacceptable, if that is we take single pass to run the way of above-mentioned entropy estimation approximate data, the probability of making mistakes is 32.9%.And if we get the median of these 70 entropy estimated values, this value is 5.10, is less than 5.5, can be considered as estimating accurately.

Table 1

Sequence number	Estimated value	Can accept	Sequence number	Estimated value	Can accept
						1	4.56	Be	2	4.68	Be
3	4.68	Be	4	4.74	Be
						5	4.75	Be	6	4.75	Be
7	4.77	Be	8	4.78	Be
						9	4.78	Be	10	4.80	Be
11	4.80	Be	12	4.81	Be
						13	4.82	Be	14	4.86	Be
15	4.86	Be	16	4.86	Be
						17	4.88	Be	18	4.88	Be
19	4.90	Be	20	4.90	Be
						21	4.90	Be	22	4.91	Be
23	4.91	Be	24	4.91	Be
						25	4.95	Be	26	4.95	Be
27	4.97	Be	28	4.97	Be
						29	5.00	Be	30	5.00	Be
31	5.00	Be	32	5.04	Be
						33	5.07	Be	34	5.07	Be
35	5.10	Be	36	5.10	Be
						37	5.15	Be	38	5.20	Be
39	5.20	Be	40	5.24	Be
						41	5.26	Be	42	5.26	Be
43	5.29	Be	44	5.33	Be
						45	5.33	Be	46	5.35	Be
47	5.38	Be	48	5.41	Be
						49	5.44	Be	50	5.46	Be
51	5.50	No	52	5.52	No
						53	5.56	No	54	5.58	No 10-->
55	5.60	No	56	5.60	No
						57	5.61	No	58	5.62	No
59	5.62	No	60	5.64	No
						61	5.65	No	62	5.66	No
63	5.69	No	64	5.69	No
						65	5.71	No	66	5.71	No
67	5.73	No	68	5.75	No

69

5.75

No

70

5.80

No

Next, we repeat 60 above-mentioned experiments, and obtain 60 entropy based on median and estimate, wherein only have 2 estimated results to be unacceptable, namely accuracy rate has brought up to 96.7%.As can be seen here, the innovatory algorithm based on median that this patent proposes can improve the accuracy rate of estimation really greatly.In addition, the estimated value result of entropy all can provide within 5 minutes, illustrated that the data flow algorithm based on median can realize by means of storm cluster really in real time.

Be directed to the situation that algorithm two and algorithm one are combined with algorithm two, our experiment obtains similar result, repeats no more herein.

Fig. 5 shows the structural representation of a kind of network traffics entropy estimating device based on median that one embodiment of the invention provides, as shown in Figure 5, the network traffics entropy estimating device based on median of the present embodiment, comprising: the first acquisition module 51, receiver module 52, order module 53, second acquisition module 54 and the 3rd acquisition module 55;

First acquisition module 51, for obtaining the packet of network data flow, and is sent to each from node of Storm cluster;

Receiver module 52, for the estimated value of each middle entry S returned from node of receiving described Storm cluster, the estimated value of described S be described Storm cluster each from node according to the packet received, obtained by default entropy estimating algorithm;

Order module 53, the estimated value for the S by reception sorts according to size;

Second acquisition module 54, for obtaining the median of the estimated value of all S after sequence, using described median as the final estimated value of S

3rd acquisition module 55, described in basis obtain the estimated value of the entropy of described network data flow network traffics

Wherein, described Storm cluster comprises at least one from node.

In a particular application, described middle entry S is obtained by the first formulae discovery, and described first formula is:

$S=\frac{1}{m}{\mathrm{\&Sigma;}}_i{m}_i{\mathrm{logm}}_i;$

Wherein, the packet in described network data flow represents with i, m _ifor the frequency that packet i occurs in network data flow, i ∈ 1,2 ..., 2 ^4b, b is the byte length of individual data bag in described network data flow.

In a particular application, described 4th acquisition module, can be specifically for

According to described by the second formula, obtain the estimated value of the entropy of network traffics

Described second formula is:

$\tilde{H}=log\left(m\right)-\tilde{S},$

Wherein, m is the total length of network data flow, and m is obtained by the 3rd formulae discovery;

Described 3rd formula is:

$m={\&Sigma;}_{i=1}^n{m}_i,$

N is positive integer.

The network traffics entropy estimating device based on median of the present embodiment, calculating in real time by Storm platform is parallel, can improve the accuracy rate of entropy estimation when not increasing memory space and computation complexity.

The network traffics entropy estimating device based on median of the present embodiment, may be used for the technical scheme performing embodiment of the method shown in earlier figures 1, it realizes principle and technique effect is similar, repeats no more herein.

" first " and " second " etc. are not make regulation to sequencing in embodiments of the present invention, just make difference to title, in embodiments of the present invention, do not make any restriction.

One of ordinary skill in the art will appreciate that: all or part of step realizing above-mentioned each embodiment of the method can have been come by the hardware that program command is relevant.Aforesaid program can be stored in a computer read/write memory medium.This program, when performing, performs the step comprising above-mentioned each embodiment of the method; And aforesaid storage medium comprises: ROM, RAM, magnetic disc or CD etc. various can be program code stored medium.

Last it is noted that above each embodiment is only in order to illustrate technical scheme of the present invention, be not intended to limit; Although with reference to foregoing embodiments to invention has been detailed description, those of ordinary skill in the art is to be understood that: it still can be modified to the technical scheme described in foregoing embodiments, or carries out equivalent replacement to wherein some or all of technical characteristic; And these amendments or replacement, do not make the essence of appropriate technical solution depart from the scope of various embodiments of the present invention technical scheme.

Claims (10)

1., based on a network traffics entropy evaluation method for median, it is characterized in that, comprising:

Obtain the packet of network data flow, and be sent to each from node of Storm cluster;

The estimated value of each middle entry S returned from node receiving described Storm cluster, the estimated value of described S be described Storm cluster each from node according to the packet received, obtained by default entropy estimating algorithm;

The estimated value of the S of reception is sorted according to size;

Obtain the median of estimated value of all S after sequence, using described median as the final estimated value of S

According to described obtain the estimated value of the entropy of described network data flow network traffics

Wherein, described Storm cluster comprises at least one from node.

2. method according to claim 1, is characterized in that, described middle entry S is obtained by the first formulae discovery, and described first formula is:

$S=\frac{1}{m}{\mathrm{\&Sigma;}}_i{m}_i{\mathrm{logm}}_i;$

Wherein, the packet in described network data flow represents with i, m _ifor the frequency that packet i occurs in network data flow, i ∈ 1,2 ..., 2 ^4b, b is the byte length of individual data bag in described network data flow.

3. method according to claim 2, is characterized in that, described in described basis obtain the estimated value of the entropy of network traffics comprise:

According to described by the second formula, obtain the estimated value of the entropy of network traffics

Described second formula is:

$\tilde{H}=\log \left(m\right)-\tilde{S},$

Wherein, m is the total length of network data flow, and m is obtained by the 3rd formulae discovery;

Described 3rd formula is:

$m={\mathrm{\&Sigma;}}_{i=1}^n{m}_i,$

N is positive integer.

4. method according to claim 3, is characterized in that, described default entropy estimating algorithm is entropy estimation approximate data, comprising:

Random selecting g × z position in described network data flow, generates random site set;

For the packet a of jth in described network data flow _j=i, judges whether j belongs to described random site set, if j belongs to described random site set, then for i increases a counter, and the counter set be associated with i is all added 1; If j does not belong to described random site set, and there is the counter be associated with i, then the counter be associated with i is added 1;

After having judged whether the j of all packets in described network data flow belongs to described random site set, obtain the counter matrices C=(c of g × z _pq);

According to described counter matrices C, build matrix X=(x _pq);

Obtain the mean value of described each row element of matrix X;

Obtain the median of all mean value, and using the estimated value of this median as S;

Wherein, x _pq=m × (c _pqlogc _pq-(c _pq-1) log (c _pq-1)), $g=2\log \left(\frac{1}{\mathrm{\&delta;}}\right),$ ε is relative evaluated error, and 1-δ is for estimating accuracy rate.

5. method according to claim 4, is characterized in that, the mean value of each row element of the described matrix X of described acquisition, comprising:

By the 4th formula, obtain the mean value avg [p] of described each row element of matrix X;

Wherein, described 4th formula is:

$avg\&lsqb;p\&rsqb;=\frac{1}{z}{\mathrm{\&Sigma;}}_{q=1}^z{x}_{pq},p=1,2,\mathrm{...},g;q=1,2,\mathrm{...},z.$

6. method according to claim 3, is characterized in that, described default entropy estimating algorithm is entropy estimation filtering algorithm, comprising:

According to default sample rate, the packet in the network data flow obtained is sampled;

Judge a jth packet a in described network data flow _jwhether=i is selected and whether there is the counter be associated with i;

If a jth packet a in network data flow _j=i is selected, and there is not the counter be associated with i, then create a counter for i and i is labeled as rill;

If a jth packet a in network data flow _j=i is selected, and there is the counter be associated with i, then i is labeled as large stream;

If a jth packet a in network data flow _j=i is not selected, and there is the counter be associated with i, then the counter be associated with i is added 1;

Judging a jth packet a in network data flow _jwhether=i is selected and after whether there is the counter that is associated with i, obtains being labeled as the counter matrices E of large stream and being labeled as the counter matrices M of rill;

According to the described counter matrices E being labeled as large stream, obtain the contribution margin S of large stream _e;

According to the described counter matrices M being labeled as rill, build matrix Y=(y _pq);

Obtain the mean value of described each row element of matrix Y;

Obtain the median of all mean value, and using the contribution margin S of this median as rill _m;

According to described S _eand S _m, obtain the estimated value of S;

Wherein, E=(E _t), t=1,2 ..., e; E is positive integer;

M＝(M _pq)，p＝1,2,…,g；q＝1,2,…,z；

y _pq＝m×(M _pqlogM _pq-(M _pq-1)log(M _pq-1))，p＝1,2,…,g；q＝1,2,…,z。

7. method according to claim 6, is characterized in that, is labeled as the counter matrices E of large stream described in described basis, obtains the contribution margin S of large stream _e, comprising:

According to the described counter matrices E being labeled as large stream, by the 5th formula, obtain the contribution margin S of large stream _e;

Wherein, described 5th formula is:

$S_e={\mathrm{\&Sigma;}}_{t=1}^e{E}_t{\mathrm{logE}}_t,t=1,2,\mathrm{...},e;$

And/or,

Described according to described S _eand S _m, obtain the estimated value of S, comprising:

According to described S _eand S _m, by the 6th formula, obtain the estimated value of S;

Wherein, described 6th formula is:

$\tilde{S}=S_e+S_m;$

And/or,

The mean value of each row element of the described matrix Y of described acquisition, comprising:

By the 7th formula, obtain the mean value avg [p] of described each row element of matrix Y;

Wherein, described 7th formula is:

$avg\&lsqb;p\&rsqb;=\frac{1}{z}{\mathrm{\&Sigma;}}_{q=1}^z{y}_{pq},p=1,2,\mathrm{...},g;q=1,2,\mathrm{...},z.$

8., based on a network traffics entropy estimating device for median, it is characterized in that, comprising:

First acquisition module, for obtaining the packet of network data flow, and is sent to each from node of Storm cluster;

Receiver module, for the estimated value of each middle entry S returned from node of receiving described Storm cluster, the estimated value of described S be described Storm cluster each from node according to the packet received, obtained by default entropy estimating algorithm;

Order module, the estimated value for the S by reception sorts according to size;

Second acquisition module, for obtaining the median of the estimated value of all S after sequence, using described median as the final estimated value of S

3rd acquisition module, described in basis obtain the estimated value of the entropy of described network data flow network traffics

Wherein, described Storm cluster comprises at least one from node.

9. device according to claim 8, is characterized in that, described middle entry S is obtained by the first formulae discovery, and described first formula is:

$S=\frac{1}{m}{\mathrm{\&Sigma;}}_i{m}_i{\mathrm{logm}}_i;$

Wherein, the packet in described network data flow represents with i, m _ifor the frequency that packet i occurs in network data flow, i ∈ 1,2 ..., 2 ^4b, b is the byte length of individual data bag in described network data flow.

10. device according to claim 9, is characterized in that, described 4th acquisition module, specifically for

According to described by the second formula, obtain the estimated value of the entropy of network traffics

Described second formula is:

$\tilde{H}=\log \left(m\right)-\tilde{S},$

Wherein, m is the total length of network data flow, and m is obtained by the 3rd formulae discovery;

Described 3rd formula is:

$m={\mathrm{\&Sigma;}}_{i=1}^n{m}_i,$

N is positive integer.