CN105956187A - Selection method of majority-class user network access characteristics - Google Patents

Abstract

The invention discloses a selection method of majority-class user network access characteristics, and relates to the field of big data analysis. The selection method comprises the following steps: firstly, downloading a log from a gateway server, carrying out statistics on the network access data of a real user, and forming an initial data set; secondly, creating N majority-class virtual users, and constructing a virtual data set; thirdly, carrying out correlation analysis on the virtual data set, and selecting access characteristics remarkably associated with the majority-class user; and finally, carrying out factor analysis to further lower a characteristic dimensionality. The virtual user is created to automatically balance the ratio of the majority-class user, and therefore, the network access characteristics still can be extracted when the number of target users occupies a small ratio.

Description

A kind of choosing method of minority class subscriber network access feature

Technical field

The present invention relates to big data analysis field, particularly relate to the choosing of a kind of minority class subscriber network access feature Access method.

Background technology

Popular along with Internet technology, the quantity of Internet user is more and more huger.Network has oozed Having entered the every aspect of life, therefore the network behavior to user is analyzed being the most significant, and it can To contribute to understanding user, provide the foundation of science for carrying out decision-making.Such as, in electricity business field, analyze and use The purchasing behavior at family can realize the accurate input of Commdity advertisement；In education sector, analyze teen-age online Behavior, the bad network that can correct them in time accesses；In information security field, the online of detection user Behavior, can stop the access of disabled user in time.Choose and can express the network of user network behavior and access Feature, is requisite during user behavior analysis, is also most important link.

In current research modal subscriber network access feature be access dissimilar network address the frequency and Time.The number of these features is all the most, it is necessary to therefrom chooses important feature and reduces The purpose of dimension.The method of existing research selected characteristic has a lot, and such as correlation analysis is exactly a kind of letter Single Feature Selection Algorithms the most easily performed.But current feature selecting algorithm does not the most all account for target User belongs to the situation of minority class user, say, that the ratio accounting for total user is considerably less, as browsed certain In a large number of users of Commdity advertisement, only small part user is ready to buy.In this case, because number ratio Example unbalanced, can cause the feature chosen the most comprehensive, it is impossible to express the access information of targeted customer completely. The most how minority class user account for all ratios the least in the case of, access from substantial amounts of network adaptively Feature selects important feature, just seems that there is Research Significance and using value very much.

Summary of the invention

Because the drawbacks described above of prior art, the technical problem to be solved is to provide a kind of minority The choosing method of class subscriber network access feature, it is possible to minority class user account for all ratios the least in the case of, Access feature from substantial amounts of network adaptively and select important feature.

For achieving the above object, the invention provides the choosing method of a kind of minority class subscriber network access feature, It is characterized in that, comprise the steps:

Step one, from gateway server download log, the network access data of statistics real user, and forming Initial data set；Wherein, the dimension of initial data set is m × d, and m is total number of persons, and d is number of features, just The packet of beginning data set contains the data of index of coincidence distribution；

Step 2, the Virtual User of establishment N name minority class, build dummy data set；Wherein, virtual data The dimension of collection is (m+N) × d, and dummy data set meets probability with distribution with the data of initial data set；

Step 3, dummy data set is carried out correlation analysis, select and minority class user's significant correlation Access feature.

Furthermore, the choosing method of a kind of minority class subscriber network access feature also comprises step 4: right The dummy data set having carried out correlation analysis carries out factorial analysis, reduces intrinsic dimensionality further.

Furthermore, step 2 specifically includes:

A0, labelling minority class user S_i, wherein i ∈ [1, p], p is the total number of persons of minority class user；

A1, calculate the mean μ of each feature_j,j∈[1,d]；

The total number of persons of the virtual minority class user that A2, calculating needs create: N=m-p；

A3, judge that p, whether more than 1, if it is, continue executing with A4, the most directly replicates N name S₁ And perform described step 3；

A4, calculate every true minority class user and need corresponding virtual number Represent downwards Rounding operation；

A5, according to exponential to S_iIncrease n name Virtual User.

Furthermore, in the A5 step of step 2, specifically include as follows:

B0, judge whether i exceedes true minority class user number, if it is, terminate, otherwise continue to hold OK；

B1, execution i=i+1；

B2, judge S_iWhether the Virtual User number created is more than n, without then continuing B3, otherwise Redirect B0 to continue executing with；

B3, find min (p-1,5) name and the minimum real minority class user of its Euclidean distance；

B4, randomly selected in min (p-1,5) name user, be denoted ask∈[1,min(p-1,5)]；

B5, one random number R of generation, and R～U (0,1)；

B6, establishment Virtual User S '_i, jth feature S' of described Virtual User_iJ (), is expressed as:

$S_i^{\′}\left(j\right)=-{\mathrm{\μ}}_{j}ln\left(\right(1-R)e^{-\frac{S_i\left(j\right)}{{\mathrm{\μ}}_j}}+{\mathrm{Re}}^{-\frac{S_i^k\left(j\right)}{{\mathrm{\μ}}_j}});$

Wherein j ∈ [1, d]；

B7, combine these features, it is thus achieved that the characteristic set of newly created Virtual User is:

S'_i=[S'_i(1),S'_i(2),...,S'_i(j),...,S'_i(d)]。

Furthermore, in step 3, described correlation analysis is simple correlation analysis, its correlation coefficient For Pearson correlation coefficients, significant level is 0.05；The marked feature collection of output is combined into [Sig₁,Sig₂,...,Sig_s], Wherein s is the number of features of significant correlation, Sig_sIt it is the feature of significant correlation.

Furthermore, step 4 specifically includes:

C1, calculating Bartlett statistic and KMO statistic；

C2, judge Bartlett statistic whether less than 0.05 and KMO statistic whether less than 0.5, as The most no, then continue executing with, otherwise terminate；

C3, the selection characteristic root common factor more than 1, described common factor includes some minority class user networks The homogenous characteristics accessed；

C4, use varimax carry out the rotation of factor axle, highlight minority class subscriber network access feature.

The invention has the beneficial effects as follows: the present invention can be by creating Virtual User, autobalance minority class user Ratio, it is ensured that after newly-increased Virtual User, each feature still obeys original exponential, reach from Adapt to extract the purpose of the feature of minority class subscriber network access behavior.

Accompanying drawing explanation

Fig. 1 is the schematic flow sheet of the embodiment of the present invention one；

Fig. 2 is the schematic flow sheet creating school work difficulty university students's Virtual User；

Fig. 3 is the schematic flow sheet creating Virtual User based on exponential；

Fig. 4 is the schematic flow sheet of factor analysis.

Detailed description of the invention

The invention will be further described with embodiment below in conjunction with the accompanying drawings:

The present embodiment, with school work difficulty university students as minority class, utilizes the present invention to analyze its network and accesses feature.

As shown in Figures 1 to 4, the present embodiment provides a kind of school work difficulty College Students with Internet to access the choosing of feature Access method, comprises the steps:

Step one, from gateway server download log, the network access data of statistics school, and forming Initial data set；In data set, include non-school work difficulty university students and minority school work difficulty university students. URL, student number etc. is included from gateway server download log.Main Analysis school work difficulty university students accesses network Type, network visitation frequency, network access duration etc..The dimension of initial data set is m × d, and m is total learning Raw number, d is number of features, and feature mainly includes the frequency, accesses duration etc..It is noted that the frequency is special Levying, accessing duration characteristics is index of coincidence distribution.Additionally, in order to reduce data analysis workload, generally By dissimilar network address, the frequency and temporal characteristics are added up.General website can be divided into large-scale portal website, Industrial sustainability, transaction class website, classification information site, forum, government website, functional character website, joy Happy type website, enterprise web site etc..

Step 2, establishment N name school work difficulty university students's Virtual User (minority class), build dummy data set； M name real user and the virtual data that N name school work difficulty university students's Virtual User dimension is (m+N) × d Collection.By creating school work difficulty university students's Virtual User, minority class quantity is by increase and adds data set, has Help its network is accessed the extraction of feature.It is noted that the school work difficulty university students created is virtual User, its feature should be to meet probability with distribution with the data of real user.Exemplary, network accesses The frequency and to access duration be without memory, meets exponential, the most newly created virtual school work difficulty university Also true school work difficulty university students's exponential should be met.

Specifically, as shown in Figure 2 to analyze school work difficulty College Students with Internet access feature, step 2 includes:

A0, labelling real school work difficulty student users S_i, wherein i ∈ [1, p], p is that real school work is stranded The total number of persons of difficult student users；

A1, calculate the mean μ of each feature_j,j∈[1,d]；

The total number of persons of school work difficulty university students's Virtual User that A2, calculating needs create: N=m-p；

A3, judge that p, whether more than 1, if it is, continue executing with A4, the most directly replicates N name S₁ And perform described step 3；

A4, calculate every true school work difficulty university students and need the number of corresponding Virtual User Generation The downward rounding operation of table；

A5, according to exponential to S_iIncrease n name school work difficulty university students's Virtual User；

Furthermore, as it is shown on figure 3, described in A5 according to exponential to S_iIncrease n name school work difficulty Specifically comprising the following steps that of university students's Virtual User

B0, judge whether i exceedes true school work difficulty student users number, if it is, terminate, no Then continue executing with B1；

B1, execution i=i+1；

B2, judge S_iWhether the school work virtual number of difficulty university students created is more than n, without then continuing B3, otherwise redirects B0 and continues executing with；

B3, find min (p-1,5) name and the minimum real school work difficulty university students of its Euclidean distance；

B4, randomly selected in min (p-1,5) name user, be denoted ask∈[1,min(p-1,5)]；

B5, one random number R of generation, and R～U (0,1)；

B6, to ensure create school work difficulty university students Virtual User S '_iAfter, all school work difficulty university students are still Obey identical exponential, then for each feature j, it should meet:

$\frac{P\{S_i<x<S_i^{\&prime;}\}}{P\{S_i<x<S_i^k\}}=R---\left(1\right)$

The distribution function using exponential solves (1) formula, and available newly created school work difficulty university students is empty Intend jth feature S' of user_iJ (), can be expressed as follows:

$S_i^{\&prime;}\left(j\right)=-{\mathrm{\&mu;}}_{j}ln\left(\right(1-R)e^{-\frac{S_i\left(j\right)}{{\mathrm{\&mu;}}_j}}+{\mathrm{Re}}^{-\frac{S_i^k\left(j\right)}{{\mathrm{\&mu;}}_j}})---\left(2\right)$

Wherein j ∈ [1, d]

B7, combine these features, it is thus achieved that the characteristic set of newly created school work difficulty university students's Virtual User is:

S'_i=[S'_i(1),S'_i(2),...,S'_i(j),...,S'_i(d)] (3)

Step 3, dummy data set is carried out correlation analysis, select and the school work notable phase of difficulty university students The access feature closed.

In the present embodiment, using simple correlation analysis, its correlation coefficient is Pearson correlation coefficients, significantly Level is 0.05；The marked feature collection of output is combined into [Sig₁,Sig₂,...,Sig_s], wherein s is the feature of significant correlation Number, Sig_sIt it is the feature of significant correlation.

Step 4, carry out factorial analysis, reduce school work difficulty university students's intrinsic dimensionality further, specifically include:

C1, calculating Bartlett statistic and KMO statistic；

C2, judge Bartlett statistic whether less than 0.05 and KMO statistic whether less than 0.5, as The most no, then continue executing with, otherwise terminate；

C3, the selection characteristic root common factor more than 1, each common factor includes minority class subscriber network access Homogenous characteristics, these common factors are the classification to feature, and each common factor represents affects student's school work A category feature；

C4, use varimax carry out the rotation of factor axle, highlight minority class subscriber network access feature, The common factor interpretability making acquisition becomes apparent from, and helps to understand the principal element affecting college students ' academic.

To sum up, the present embodiment is by creating school work difficulty university students's Virtual User, autobalance minority class user Ratio, after simultaneously also ensureing the Virtual User increased newly, each of all school work difficulty university students's Virtual User Feature still obeys original exponential, reaches the spy of extracted in self-adaptive minority class subscriber network access behavior The purpose levied.Although the present embodiment is with school work difficulty university students as minority class and analyzes network access behavior, It is equally applicable to other minority class network and accesses behavior research case, repeat no more here.

The preferred embodiment of the present invention described in detail above.Should be appreciated that the ordinary skill of this area Personnel just can make many modifications and variations according to the design of the present invention without creative work.Therefore, all Technical staff passes through logical analysis the most on the basis of existing technology, pushes away in the art Reason or the limited available technical scheme of experiment, all should be at the protection model being defined in the patent claims In enclosing.

Claims (6)

1. the choosing method of a minority class subscriber network access feature, it is characterised in that comprise the steps:

Step one, from gateway server download log, the network access data of statistics real user, and forming Initial data set；Wherein, the dimension of described initial data set is m × d, and m is total number of persons, and d is number of features, The packet of described initial data set contains the data of index of coincidence distribution；

Step 2, the Virtual User of establishment N name minority class, build dummy data set；Wherein, described virtual The dimension of data set is (m+N) × d, and described dummy data set meets probability with dividing with the data of initial data set Cloth；

Step 3, dummy data set is carried out correlation analysis, select and minority class user's significant correlation Access feature.

The choosing method of a kind of minority class subscriber network access feature the most as claimed in claim 1, its feature Being, described choosing method also comprises step 4: the dummy data set carrying out correlation analysis is carried out because of Son is analyzed, and reduces intrinsic dimensionality further.

The choosing method of a kind of minority class subscriber network access feature the most as claimed in claim 1, its feature Being, described step 2 specifically includes:

Minority class user S described in A0, labelling_i, wherein i ∈ [1, p], p is the total number of persons of described minority class user；

A1, calculate the mean μ of each feature_j,j∈[1,d]；

The total number of persons of the described virtual minority class user that A2, calculating needs create: N=m-p；

A3, judge that p, whether more than 1, if it is, continue executing with A4, the most directly replicates N name S₁ And perform described step 3；

A4, calculate every true minority class user and need corresponding virtual number

A5, according to exponential to S_iIncrease the described Virtual User of n name.

The choosing method of a kind of minority class subscriber network access feature the most as claimed in claim 3, its feature Be: described in A5 according to exponential to S_iIncrease specifically comprising the following steps that of n name Virtual User

B0, judge whether i exceedes described true minority class user number, if it is, terminate, otherwise continue Continuous execution；

B1, execution i=i+1；

B2, judge S_iWhether the Virtual User number created is more than n, without then continuing B3, otherwise Redirect B0 to continue executing with；

B3, find min (p-1,5) name and the minimum real minority class user of its Euclidean distance；

B4, randomly selected in min (p-1,5) name user, be denoted ask∈[1,min(p-1,5)]；

B5, one random number R of generation, and R～U (0,1)；

B6, establishment Virtual User S '_i, jth feature S' of described Virtual User_iJ (), is expressed as:

$S_i^{\&prime;}\left(j\right)=-{\mathrm{\&mu;}}_{j}ln\left(\right(1-R)e^{-\frac{S_i\left(j\right)}{{\mathrm{\&mu;}}_j}}+{\mathrm{Re}}^{-\frac{S_i^k\left(j\right)}{{\mathrm{\&mu;}}_j}})$

Wherein j ∈ [1, d]；

B7, combine these features, it is thus achieved that the characteristic set of newly created Virtual User is:

S'_i=[S'_i(1),S'_i(2),...,S'_i(j),...,S'_i(d)]。

The choosing method of a kind of minority class subscriber network access feature the most as claimed in claim 1, its feature Being: in described step 3, described correlation analysis is simple correlation analysis, and its correlation coefficient is Pierre Gloomy correlation coefficient, significant level is 0.05；The marked feature collection of output is combined into [Sig₁,Sig₂,...,Sig_s], wherein s is The number of features of significant correlation, Sig_sIt it is the feature of significant correlation.

The choosing method of a kind of minority class subscriber network access feature the most as claimed in claim 2, its feature Being, described step 4 sequentially includes the following steps:

C1, calculating Bartlett statistic and KMO statistic；

C2, judge Bartlett statistic whether less than 0.05 and KMO statistic whether less than 0.5, as The most no, then continue executing with step C3, otherwise terminate；

C3, the selection characteristic root common factor more than 1, described common factor includes some minority class user networks The homogenous characteristics accessed；

C4, use varimax carry out the rotation of factor axle, highlight minority class subscriber network access feature.