CN104168188B - Internet AS deduction and router division method based on SVI - Google Patents

Abstract

The invention provides an internet AS deduction and router division method based on SVI. Firstly, a mathematical model is established, and the relation between nodes, community indicator vectors and subordinate vectors is determined; secondly, a prior probability model about the subordinate vectors and observation values is established; thirdly, the prior probability model is solved through the SVI algorithm, the degree of membership of the nodes is worked out, and the AS division result of the nodes is obtained. The invention further provides a method for checking the division result.

Description

Internet AS inference and router division method based on SVI

Technical Field

The invention belongs to the technical field of Internet, and particularly relates to an AS inference and router division method.

Background

With the continuous expansion of the Internet scale, the structure becomes increasingly complex, and it becomes more difficult to accurately acquire the network device interconnection structure. The network topology information has important application in the aspects of network monitoring and diagnosis, network simulation, routing protocol behavior research, network performance optimization, network security implementation and the like. The Internet may be divided into a plurality of independent administrative domains, called Autonomous Systems (AS), each AS consisting of a set of routers and other network devices and interconnections with a single routing policy, and managed by an administrator. Different ASs exchange routing information via an External Gateway Protocol (EGP), and a currently widely used Protocol is Border Gateway Protocol (BGP).

Existing internet topology models are mainly classified into a Router (RL) level topology model and an autonomous domain (AS) level topology model, but due to the complexity of the internet, the topology models cannot fully show the topological characteristics of the internet: the AS-level topology model cannot simulate a large-scale complex network range and cannot well reflect various actual network topology characteristics; because the existing router-level topology discovery algorithm can not ensure the accuracy and completeness of a measurement set, a certain difficulty is generated in RL-level topology modeling, and the theoretical analysis at the present stage is not mature enough.

The internet is one of complex networks, and a large number of existing complex network topology models can be used for analyzing the complex networks, but the complex network based topology models are directed at the basic rules of the complex networks generally existing in nature, and the generalized complex network models or generation models are pursued, so that the models can only be applied to the internet to a certain extent, and therefore, the adaptability of a complex network (particularly, a large-scale complex network) modeling method to the current internet topology structure needs to be discussed, and a model specially directed at the internet needs to be established.

Mmsb (mixed member social storage block model) is a good community division method appearing in recent years, and is suitable for analyzing and processing data with complex association relationship, and the data can be expressed as a complex graph (graph) or network (network) structure in mathematics. The MMSB provides an important concept of mixed membership and has higher dividing accuracy.

In the MMSB, each individual is represented by a node, the relationship among the nodes is represented by edges, the edges can be divided into directed edges and undirected edges, certain similarity exists among some nodes or the nodes are more closely related, and the structure of a module or a community is presented. Each node is associated with an attribute value (membership degree) for expressing the membership of the node and each other community, and the MMSB calculates the membership degree of each node through an observation value, so that the module division condition of the node can be analyzed.

The traditional method can work better when the network scale is small, and can not work or takes too long time due to high algorithm complexity when the network nodes are huge. SVI (Stochastic variational Inference),

random variation inference can work well when the network scale is large. SVI adopts variation and optimization method to solve the model, and can be realized in parallel.

The identification of the AS and the division of the routers into the AS in the Internet topology comprises at least two parts: firstly, collecting information related to topology in the Internet; and deducing the internet topological structure according to the collected information. The existing internet topology identification method has two defects:

(1) most of the existing topology identification methods are based on one or a small amount of information related to network topology to recover the network topology. Accurate internet topology recovery puts high requirements on the completeness and accuracy of single information, and is generally difficult to achieve. The actual situation is that: various information related to network topology can be obtained, but each information is not enough to completely and accurately recover the internet topology, all the information which can be obtained by people needs to be integrated, a large-scale optimization problem of multi-source information fusion is constructed by taking the aim that the finally obtained topology can meet all the information related to the topology to the maximum extent, and the internet topology is recovered through the solution of the optimization problem.

(2) The existing topology recovery method starts from a node or an edge of a network, and gradually adds the node and the edge according to a certain rule by deeply analyzing the acquired topology related information, and only adds one edge or one node each time. For a small-scale network, the method is suitable, but for a complex and huge system such as internet topology, the calculation complexity of the existing method is too high to be realized, and the internet topology needs to be restored through a small number of steps and with less calculation workload according to all topology related information obtained by us.

The above problems arise because: aiming at the complex huge system of the internet topology, a simple and effective mathematical method is lacked for accurate and complete description and expression, which is convenient for the related research of the internet topology and the explanation of the common structure of the ubiquitous complex network of the internet.

Disclosure of Invention

Aiming at the problems in the prior art, the invention provides an internet AS inference and router division method based on SVI. The scheme of the invention specifically comprises the following steps:

step S1: establishing a mathematical model, specifically comprising the following steps:

step S11: for each node, a dependent vector θ_iDirichlet (α), where i denotes a node and the symbol "-" denotes compliance, i.e., p θ_i＝Dirichlet(θ_iα), the right side of the equation is the Dirichlet distribution with the parameter of vector α;

step S12: for each pair of nodes i and j, where i<j, community indication vector z_i→jAnd a dependent vector theta_iAssociation, community indication vector z_i←jAnd a dependent vector theta_jAssociating; wherein the community indication vector z_i→jAnd a community indication vector z_i←jTaking a value from 1 to K, wherein K represents a community;

step S2: establishing a prior probability model from the dependent vector theta to the observed value y:

wherein,is shown asThe internal nodes of the community are closely connectedDensity, ∈ constant, ∈ minimum value of 1.0 × 10^-9Y is the connection condition between the nodes in the actual observation data, when the two nodes are connected, the value of y is 1, otherwise, 0 is taken;

step S3: solving a prior probability model, and calculating the membership degree theta of the node by solving the following formula to obtain an AS division result of the node;

L(γ,φ,λ)＝E(log(pθ,z,β|y)))-E(log(qθ,z,β)))；

where γ is a parameter of the distribution of the dependent vector θ, Φ is a parameter of the community indicating the distribution of the vector z, λ is a parameter of the distribution of β, respectively, the symbol E (-) indicates a mathematical expectation, and on the right side of the equality of the above equation, the antecedent and consequent terms are E (log (p θ, z, β | y))) and represent the expectation of log (q θ, z, β)) under the distribution q (θ, z, β), respectively;

step S4: solving the formula in step S3: l (γ, Φ, λ) ═ E (log (p θ, z, β | y))) -E (log (q (θ, z, β)));

step S4 specifically includes the following substeps:

step S41: initializing global variables

Step S42: extracting a certain amount of node pairs from the nodes and recording the node pairs as a set S;

step S43: for all node pairs in the set S, an optimal local variable phi is calculated according to the current global variable value_i→jAnd an optimum local variable phi_i←j；

Step S44: for each node a, updating γ a, and for each community k, updating λ k;

step S45: repeating steps S2, S3, S4 until convergence;

further, a method for checking results is provided based on the above algorithmMethod by solving the equationIf it is notThe result is consistent with the result of the division, so that the division is correct, otherwise, the parameters are adjusted, and the calculation is carried out again;

wherein,indicates the AS, y with the largest number of connections_iIndicating the number of connections between node i and each AS interior node.

The invention has the beneficial effects that: the invention relates to an SVI-based Internet AS inference and router division method

For the routing equipment, referring to an MMSB model, and adopting a random variation inference method (SVI) to infer the inclusion relation between the routing equipment and the AS, thereby obtaining the AS division condition of the routing equipment in a certain range.

Drawings

Fig. 1 is a particular network.

Fig. 2 illustrates verification of the AS partitioning result.

Detailed Description

To facilitate understanding of the technical solution, the following terms are first defined:

and (3) node: a routing device located in the Internet and operating at the IP layer;

no side direction: reversible network links or connections existing between nodes;

observing the network: obtaining nodes and connection relations among the nodes through a certain technology;

k: the number of AS;

n: the number of nodes;

latent variables: variables that cannot be directly observed;

global variables: the calculation process involves variables of all nodes;

local variables: the calculation process involves variables of a single node;

the expression plus black font in this text represents a vector.

The present invention will be further described with reference to the accompanying drawings.

A bayesian model of the routing device network AS discovery and a model solution algorithm that can be applied to large-scale networks are set forth below.

Assuming that a total of K groups or communities exist, each node i has a membership vector theta representing the membership relationship with each community_iIn association, the sum of all component values of the vector is 1, the greater the component value, the greater the probability that the node belongs to the respective AS.

For each pair of nodes { i, j }, a community indication vector z is defined_i→jAnd z_i←jThe former and the dependent variable theta_iCorrelation of the latter with theta_jAnd associating, wherein the value range is 1 to K, or a vector of K dimensions, only one component in the vector is 1, other components are 0, and the indicating variable indicates which community each node is located in. If the values of the two indicating variables are the same, a connection exists between the two nodes with high probability, and otherwise, the probability of the existence of the connection is small. The above description is based on the assumption that if the membership vectors of two nodes are identical or at least have large simultaneous component values on the same component, the two nodes are more likely to be located in the same community. The above description implies another assumption that the connection between nodes within the same community is tighter, i.e. there is a probability that one connection exists between nodes in the same communityThe probability of a connection between nodes in different communities is higher than that of a connection between nodes in different communities, and under the condition of an existing observed value, community division is more prone to consider that the nodes with connections are located in the same community more frequently. This assumption holds in most cases for the routing device AS partitioning, and in general, there are less cases where routers inside the AS are directly connected to routers outside the AS. Based on the foregoing description, the following mathematical model was established:

(1) for each node, a dependent vector θ_iDirichlet (α), the i representing a node, Dirichlet (α) representing a distribution function;

(2) for each pair of nodes i and j, where i < j;

community indication vector z_i→j～θ_i

Community indication vector z_i←j～θ_j

Probability of connection between nodes

In the above formulaIs shown asThe closer the connection degree between internal nodes of each community is, the closer the connection degree between the internal nodes of the communities is, the higher the probability of connection between two nodes belonging to the same community is, when the model is actually solved, β is considered to be a random variable obeying the Beta distribution with the parameter η, ∈ is a constant and takes a smaller value, y is the connection condition between the nodes in actual observed data, and when the two nodes are connected, the value of y is 1, so that a prior probability model from a community membership vector theta to an observed value y is established.

According to the Bayesian view, solving the model requires calculating the posterior probability of the form:

p(θ,z,β|y)＝p(θ,z,β,y)/p(y) (1)

(1) in the formula, the denominator part is the joint distribution of variables in the model, and the solution is easy. The calculation of the denominator part is very complex.

p(y)＝∫_θ∫_β∑zp(θ,z,β,y) (2)

(2) Wherein the number of summation terms of the summation part exceedsThe computational complexity is more than exponentially growing, and the summation part is not computable when N is large. Therefore, an approximation method needs to be sought.

The family of variogram functions q (θ, z, β) is defined below and, taking into account the mean field theory, can be decomposed into the following form:

(3) wherein q (θ)_n|γ_n) Is a family of Dirichlet distributions, q (Z | φ) is a corresponding family of discrete distributions, q (β)_k|λ_k) For the family of Beta distributions, when an approximate family of distributions is found that meets the requirements, γ, Φ, λ are the parameters of the distributions of θ, z, β, respectively.

Measuring the distance (approximation degree) between the formula (3) and the formula (1) by adopting a KL (Kullback-Leibler) distance, and converting the solution of the model into a process for calculating q (theta, z, beta):

q^*(θ,z,β)＝argmin_γ，Φ，λKL(q(θ,z,β)||p(θ,z,β|y)) (4)

(4) the formula has high computational complexity and is still difficult to solve. It can be demonstrated that solving the minimum KL distance in equation (4) is equivalent to solving the maximum value in equation (5).

L(γ,φ,λ)＝E(log(p(θ,z,β|y)))-E(log(q(θ,z,β))) (5)

(5) The mathematical expectation in the equation is relative to the distribution family q with γ, φ, λ as variables.

In the conventional model, a coordinate ascending algorithm is generally adopted, and the method can work well when the network size is not large. For large scale networks, O (N)²) Is still high. For this purpose, equation (5) may be calculated using the SVI algorithm. In each iteration, the coordinate ascending algorithm firstly calculates and updates all local variables, and then updates the global variable by the local variables until convergence. The coordinate ascending algorithm has to process each node data in each iteration, and the SVI algorithm allows only part of uniformly sampled node data to be processed in each iteration, so that the data amount required to be processed in each iteration is relatively small, and the iterative updating process of the parameters is accelerated.

SVI solves the pseudocode of equation (5) as follows:

(1) initializing global variables

(2) A subset of node pairs is extracted from the nodes and designated as set S.

(3) For all node pairs in the set S, an optimal local variable phi is calculated according to the current global variable value_i→jAnd phi_i←j。

(4) For each node a, update γ_aFor each community k, λ is updated_k。

And (5) repeating the steps (2), (3) and (4) until convergence.

And obtaining the AS division result of the equipment node according to the calculated node membership degree theta.

The updating formula in the step (4) is as follows:

wherein, y_ab,0＝y_ab，y_ab,1＝1-y_ab，ρ_t＝(t₀+t)^-k，k∈(0.5,1]，The value of (a) is the node pair y in the case of a multiple graph_abMultiple of y_abRepresenting the connection between nodes a and b, the gradient in the above equation is a natural gradient.

The updating formula in the step (3) is as follows:

φ_a→b,k|y_ab＝0∝expE_q[logθ_a,k]+φ_a←b,kE_q[log(1-β_k)]+(1-φ_a←b,k)log(1-∈)) (8)

φ_a→b,k|y_ab＝1∝exp(E_qlogθ_a,k]+φ_a←b,kE_q[log1-β_k)]+(1-φ_a←b,k)log∈) (9)

in step (2), the simplest sampling method is a sampling method with uniform random node pairs, in which case

It should be noted that the above algorithm only infers the AS partitioning of the network from the perspective of probability under the condition of known information, and the result is obtained from one side, and auxiliary information or verification needs to be provided in cooperation with other methods. Under some topological connection conditions, the method cannot effectively deduce the division condition of the AS, such AS the following topology:

this scheme does not accurately demarcate the boundaries between AS200 and AS300, AS in router R2 in fig. 1, and the components of the membership degree associated with AS200 and AS300 will be very close with a large probability, which will generate a large uncertainty in inferring the AS to which R2 belongs. To accurately partition R2, additional side information is needed.

Based on the above situation, the present document proposes an AS partitioning method for processing large-scale internet routing devices, and discovers an AS topology structure of an unknown network. The method can infer the AS division of the network and the interconnection condition between the ASs under the condition that only the routing equipment and the routing equipment are connected with each other, so that the topological structure of the network can be understood at the AS level. The probability statistical model of the network and the model solving and verifying method are provided aiming at the topological characteristics of the AS.

An algorithm for checking the calculation results is proposed below, for judging the rationality of the result division.

AS shown in fig. 2, assuming that the network to be analyzed has N nodes, which are divided into K ASs, all the nodes are divided into an AS by model calculation. For each router node R_iSuppose a node R is divided_iAll nodes except the node R are divided correctly, and the node R is counted_iAnd the connection number yi in each AS internal node, finding out the AS with the maximum connection number,if it is notThe partition conforms to the model if it is consistent with the result of the partitionIf the partition is correct, otherwise, the parameters should be adjusted and the calculation should be performed again.

The invention has the beneficial effects that: the Internet AS inference and router division method based on the SVI is used for inferring the inclusion relationship between the routing equipment and the AS by referring to the MMSB model and adopting a random variation inference method (SVI) aiming at the routing equipment, so that the AS division condition of the routing equipment in a certain range is obtained. The invention has the following advantages:

(1) the concept of mixed membership is introduced, the probability that the equipment belongs to a certain AS is deduced, and topological structure information contained in observation data is reflected more accurately from the perspective of the probability;

(2) under the condition of insufficient information quantity, the structure of the network equipment is excavated to the greatest extent, so that not only can the internal structure of the AS be found, but also the interconnection relationship between the AS can be found, and the network topology structure can be displayed more clearly and comprehensively;

(3) by adopting the SVI algorithm, the large-scale routing equipment network can be processed quickly and efficiently.

It will be appreciated by those of ordinary skill in the art that the embodiments described herein are intended to assist the reader in understanding the principles of the invention and are to be construed as being without limitation to such specifically recited embodiments and examples. Various modifications and alterations to this invention will become apparent to those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims (3)

1. The internet AS inference and router division method based on the SVI is characterized by comprising the following steps:

step S1: establishing a mathematical model, specifically comprising the following steps:

step S11: for each node, a dependent vector θ_iDirichlet (α), where i denotes a node and the symbol "-" denotes the meaning of compliance, i.e., p (θ)_i)＝Dirichlet(θ_iα), the right side of the equation is the Dirichlet distribution with the parameter of vector α;

step S12: for each pair of sectionsPoints i and j, where i<j, community indication vector z_i→jAnd a dependent vector theta_iAssociation, community indication vector z_i→jAnd a dependent vector theta_jAssociating; wherein the community indication vector z_i→jAnd a community indication vector z_i←jVectors of K dimensions, K representing the total number of communities, community indication vector z_i→jAnd a community indication vector z_i←jIndicating which community each node is located in;

step S2: establishing a prior probability model from the dependent vector theta to the observed value y:

$p(y_{ij}=1|Z_{i\&RightArrow;j},Z_{i\&LeftArrow;j},\mathrm{\&beta;})=\left\{\begin{array}{cc}{\mathrm{\&beta;}}_{z_{i\&RightArrow;j}}& {\mathrm{ifZ}}_{i\&RightArrow;j}=Z_{i\&LeftArrow;j}\\ \&Element;& {\mathrm{ifZ}}_{i\&RightArrow;j}\&NotEqual;Z_{i\&LeftArrow;j}\end{array}\right.;$

wherein,denotes the Z th_i→jThe close degree of the connection among the internal nodes of the community, ∈ is a constant, ∈ takes a minimum value of 1.0 × 10^-9Y is the connection condition between the nodes in the actual observation data, when the two nodes are connected, the value of y is 1, otherwise, 0 is taken;

step S3: solving a prior probability model, and calculating the membership degree theta of the node by solving the following formula to obtain an AS division result of the node;

L(γ，φ，λ)＝E(log(p(θ，z，β|y)))-E(log(q(θ，z，β)))；

where γ is a parameter of the distribution of the dependent vector θ, Φ is a parameter of the community indicating the distribution of the vector z, λ is a parameter of the distribution of β, respectively, the symbol E (·) represents a mathematical expectation, on the right side of the equality of the above equation, the antecedent and consequent terms are E (log (p (θ, z, β | y))) respectively representing an expectation of finding log (p (θ, z, β | y)) under the distribution p (θ, z, β | y), and E (log (q (θ, z, β))) representing an expectation of finding log (q (θ, z, β)) under the distribution q (θ, z, β);

step S4: solving the formula in step S3: l (γ, Φ, λ) ═ E (log (p (θ, z, β | y))) -E (log (q (θ, z, β)));

step S4 specifically includes the following substeps:

step S41: initializing global variables

Step S42: extracting a certain amount of node pairs from the nodes and recording the node pairs as a set S;

step S43: for all node pairs in the set S, an optimal local variable phi is calculated according to the current global variable value_i→jAnd an optimum local variable phi_i←j；

The update formula of step S43 is as follows:

φ_a→b，k|y_ab＝0∝

exp(E_q[logθ_a，k]+φ_a←b，kE_q[log(1-β_k)]+(1-φ_a←b，k)log(1-∈))；

φ_a→b，k|y_ab＝1∝

exp(E_q[logθ_a，k]+φ_a←b，kE_q[log(1-β_k)]+(1-φ_a←b，k)log∈)；

wherein phi is_a→b，kAnd phi_a←b，kAn optimal local variable representing the kth community, E_q[logθ_a，k]In (A), the subscript q represents the log θ found under the distribution q (θ, z, β)_a，kMathematical expectation of (1), y_abRepresenting the connection between nodes a and b, exp (-) represents an exponential function based on e, β_kRepresenting the degree of closeness of connection among the internal nodes of the kth community;

step S44: for each node a, updating γ a, for each community K, updating λ K, K · 1,2,3, · K;

the update formula of step S44 is as follows:

$\&part;{\mathrm{\&gamma;}}_{a,k}={\mathrm{\&alpha;}}_{k+}\frac{1}{g\left(a,b\right)}{\mathrm{\&phi;}}_{a\&RightArrow;b,k}-{\mathrm{\&gamma;}}_{a,k};$

$\&part;{\mathrm{\&lambda;}}_{k,i}={\mathrm{\&eta;}}_i+\frac{1}{g\left(a,b\right)}{\mathrm{\&phi;}}_{a\&RightArrow;b,k}\&CenterDot;{\mathrm{\&phi;}}_{a\&LeftArrow;b,k}\&CenterDot;y_{ab,i}-{\mathrm{\&lambda;}}_{k,i};$

$\mathrm{\&gamma;}=\mathrm{\&gamma;}+{\mathrm{\&rho;}}_t\&part;\mathrm{\&gamma;};$

$\mathrm{\&lambda;}=\mathrm{\&lambda;}+{\mathrm{\&rho;}}_t\&part;\mathrm{\&lambda;};$

wherein, y_ab，0＝y_ab，y_ab，1＝1-y_ab，ρ_t＝(t₀+t)^-kT denotes the number of iterations, t₀Denotes the iteration initial value, k ∈ (0.5, 1)]，The value of (a) is the node pair y in the case of a multiple graph_abMultiple of y_abRepresenting the connection condition between the nodes a and b, wherein the gradient in the formula is a natural gradient;

step S45: step S2, step S3, step S4 are repeated until convergence.

2. The SVI-based internet AS inference and router partitioning method AS in claim 1, wherein β is a random variable that obeys a Beta distribution with parameter η.

3. A method for validating results of SVI-based internet AS inference and router partitioning AS claimed in claim 1 or 2, by solving the equationTo judge the divisionThe accuracy of the result, which means that a label i is obtained, and that the label i makes y_iTaking the maximum value, ifThe result is consistent with the result of the division, so that the division is correct, otherwise, the parameters are adjusted, and the calculation is carried out again;

wherein,indicates the AS, y with the largest number of connections_iIndicating the number of connections between node i and each AS interior node.