CN109617915B - Abnormal user mining method based on page access topology - Google Patents

Abstract

The invention discloses an abnormal user mining method based on page access topology, which comprises the following steps: step 1: according to the information extracted from the log, taking IP as a unit and time as a standard, sequencing URLs in an ascending order, and constructing an access topology database; step 2: extracting a training sample; and step 3: indexing the IP by using the URL for the training sample; and 4, step 4: taking the IP in the training sample as an initial search subset, removing the IP which does not access the URL in the initial search subset by using a reverse index, and updating the search subset; repeating the steps until the search subset is empty or the search subset cannot be updated; calculating a score according to the cycle times; and 5: sorting the IPs in ascending order according to scores, setting a threshold value, and taking the IPs smaller than the set threshold value as abnormal users; the invention describes the behavior of a normal user through the topological structure, judges the abnormal degree of the user, and has better self-adaptive capability and lower omission factor.

Description

Abnormal user mining method based on page access topology

Technical Field

The invention relates to an abnormal user mining method, in particular to an abnormal user mining method based on page access topology.

Background

The existing network security products generally adopt a plurality of rules or strategies to describe the behavior boundary of a user, and if a certain characteristic of the user breaks through the threshold of a normal user, a processing action is triggered; for example, the conventional abnormal user detection method mostly adopts a static feature matching means, for example, regular expression matching SQL, XSS injection attack, and the like.

The conventional method has the following disadvantages: firstly, static characteristics are easy to bypass, so that the missing rate is high; secondly, the static features can only match the abnormal conditions existing in the feature library, and the unknown abnormality cannot be detected.

Disclosure of Invention

The invention provides an abnormal user mining method based on a page access topology, which starts from users and the access topology, utilizes the access topology to depict user behaviors, improves the detection accuracy and reduces the missing report rate.

The technical scheme adopted by the invention is as follows: an abnormal user mining method based on page access topology comprises the following steps:

step 1: according to the information extracted from the log, taking IP as a unit and time as a standard, sequencing URLs in an ascending order, and constructing an access topology database;

step 2: extracting training samples from the database obtained in the step 1;

and step 3: indexing the IP by using the URL for the training sample in the step 2;

and 4, step 4: taking the IP in the training sample as an initial search subset, removing the IP which does not access the URL in the initial search subset by using a reverse index, and updating the search subset; repeating the steps until the search subset is empty or the search subset cannot be updated; calculating a score according to the cycle times;

and 5: and sorting the IPs in ascending order according to the scores, setting a threshold value, and setting the IPs smaller than the set threshold value as abnormal users.

Further, the information extracted from the log in step 1 includes a target URL, a request source IP and a request time, and removes request parameters of the URL.

Further, the score calculation process in step 4 is as follows:

in the formula: s is the score, N is the number of training samples in step 2, c (N) is,

in the formula: h (N-1) is the harmonic progression, which is a function of N-1.

The invention has the beneficial effects that:

(1) starting from a user and an access topology, the method uses the access topology to depict the user behavior and detects the user with abnormal access behavior;

(2) the invention can effectively utilize the information of the repeatability, the page type, the sequence of page access and the like when the user accesses the page, improves the accuracy of detection, reduces the rate of missing report and has certain adaptivity;

(3) the invention uses the topological structure formed by the access web pages to depict the behaviors of normal users and judge the abnormal degree of the users, and has better self-adaptive capacity and lower omission ratio.

Drawings

FIG. 1 is a schematic flow chart of the present invention.

FIG. 2 is a schematic view of the score calculation process of the present invention.

Detailed Description

The invention is further described with reference to the following figures and specific embodiments.

As shown in fig. 1 and fig. 2, an abnormal user mining method based on a page access topology includes the following steps:

step 1: and according to the information extracted from the log, sequencing the URLs in an ascending order by taking the IP as a unit and taking time as a standard, and constructing an access topology database.

Extracting a target URL, a request source IP and request time in the log, and removing parameters of the URL;

the URL in the access log often contains the requested parameters, which are irrelevant to the method of the invention and need to be removed; for example:

http://www.target.com？sreach_words＝a,b,c,d

after removing the parameters, the following steps are changed:

http://www.target.com

and (4) sequencing URLs in an ascending order by taking a single IP as a unit and using a time standard to construct an access topology database.

For example, the information corresponding to an IP in the log is as follows:

ip	url	time of access
			xxx	url1	2018-06-11:15:35:28
xxx	url2	2018-06-11:15:36:05
			xxx	url3	2018-06-11:15:38:25
xxx	url4	2018-06-11:15:40:50

The request topology to obtain the IP is: 1 → 2 → 3 → 4.

Step 2: extracting training samples from the database obtained in the step 1;

and step 3: indexing the IP by using the URL for the training sample in the step 2;

and 4, step 4: taking the IP in the training sample as an initial search subset, removing the IP which does not access the URL in the initial search subset by using a reverse index, and updating the search subset; repeating the steps until the search subset is empty or the search subset cannot be updated; calculating a score according to the cycle times;

and 5: and sorting the IPs in ascending order according to the scores, setting a threshold value, and setting the IPs smaller than the set threshold value as abnormal users.

The processes from step 2 to step 4 can be used as a normal score calculation module, and can calculate the normal score of the specified IP in the access topology database; for convenience of explanation, assume that the IP and its access topology that needs to be detected are:

IP3：1→2→3→4→7→8→10

firstly, selecting a certain number of samples from a constructed access topology database without sampling back as training samples; a certain number of the constructed model parameters can be adjusted according to actual needs to achieve the best detection effect, and 256 samples are selected in the invention.

Then, carrying out reverse indexing on the selected training samples, wherein the reverse indexing takes the content as an index and takes the previous index as the content; corresponding to the invention, the IP is indexed by using the URL, for example, the data before indexing is as follows:

IP0：URL1→URL2→URL3→URL4→URL7→URL8→URL10

IP1：URL1→URL2→URL3→URL4→URL8→URL10

IP2：URL1→URL2→URL3→URL7→URL8→URL10

the reverse index is followed by:

URL1：IP0，IP1，IP2

URL2：IP0，IP1，IP2

URL3：IP0，IP1，IP2

URL4：IP0，IP1，IP2

URL5：IP1

URL7：IP2

URL8：IP0，IP1，IP2

URL10：IP0，IP1，IP2

the calculation speed can be improved by constructing the reverse index.

And then randomly selecting a URL from the corresponding access topology in the IP to be detected, and taking the IP in the training sample as an initial search subset.

Finally, the IP which does not access the URL in the search subset is removed by using the reverse index, the search subset is updated by using the result, and the process is repeated until the search subset is empty or the search subset cannot be updated; and recording the cycle times, and obtaining a normal score according to the cycle times.

Assume that the selected training samples are:

IP0：1→6→9→11→12→13→14

IP1：1→2→3→4→5→8→10

IP2：1→2→3→4→8→10

IP4：1→2→3→4→7→10

IP5：1→2→3→4→5→8→10

IP6：1→2→3→7→8→10

IP7：1→2→3→7→10

IP8：1→2→3→4→5→8→7→4→5→8→7→4→5→8→10

the initial search subset is { IP0, IP1, IP2, IP4, …, IP8 }.

Selecting URL3, removing IP of un-visited URL3 in un-searched subset, and obtaining new searched subset as follows: { IP1, IP2, IP4, …, IP8 }.

And selecting URL7, removing the IP of the URL7 which is not visited in the search subset, and obtaining a new search subset of { IP4, IP6, IP7 and IP8 }.

And selecting the URL8, removing the IP of the URL8 which is not visited in the search subset, and obtaining a new search subset of { IP6, IP8 }.

At this time, since the IP6 and IP7 in the search subset have accessed each URL that the sample IP3 that needs to be detected has accessed, the search subset cannot be updated, and the loop stops.

Calculating a normal score according to the cycle times:

let t be the cycle number, score s be:

in the formula: s is the score, N is the number of training samples in step 2, c (N) is,

in the formula: h (N-1) is a harmonic series, which is a function of N-1;

wherein H (N-1) ═ log (N-1) + 0.5772156649.

And calculating the normal score of each IP in the access topology database through a normal score module, sequencing the IPs in an ascending order, setting a threshold value, and determining the IP smaller than the threshold value as an abnormal user.

Because the abnormal user often has behavior characteristics different from those of the normal user, the access topology is the most intuitive expression form of the access behavior; therefore, abnormal users can be detected by accessing the topology; the access topology of an anomalous user tends to have one or more of the following characteristics:

some URLs in the access topology are rare among normal users;

a large number of duplicate URLs in the access topology;

the length of the access topology is much longer than normal users.

According to these features, it is therefore possible to select from the log the behaviour with the above access features and the corresponding user using the method of the invention.

The traditional abnormal user detection based on behaviors usually depicts user behaviors through the characteristics of unit time frequency, total access times and the like of user access; the access frequency, the total access number and other parameters do not utilize the sequence of the user accessing the pages, and the information of the type, the repeatability and the like of the user accessing the pages is also ignored. These parameters therefore often do not characterize the user's behaviour very well and the effect of the detection is therefore limited.

The invention starts from the user and the access topology, describes the user behavior by using the access topology, effectively utilizes the information of the repetition rate, the page type, the sequence of page access and the like when the user accesses the page, improves the detection accuracy, reduces the missing report rate and has certain self-adaptability.

The abnormal user in the invention refers to a user whose request is greatly different from the normal user behavior, such as a web crawler, a user with an attack request, and the like; the access topology refers to a topology structure formed by URLs corresponding to a certain IP.

Claims (2)

1. An abnormal user mining method based on page access topology is characterized by comprising the following steps:

step 1: according to the information extracted from the log, taking IP as a unit and time as a standard, sequencing URLs in an ascending order, and constructing an access topology database;

step 2: extracting training samples from the database obtained in the step 1;

and step 3: indexing the IP by using the URL for the training sample in the step 2;

and 4, step 4: taking the IP in the training sample as an initial search subset, removing the IP which does not access the URL in the initial search subset by using a reverse index, and updating the search subset; repeating the steps until the search subset is empty or the search subset cannot be updated; calculating a score according to the cycle times; the calculation process is as follows:

in the formula: s is the score, t is the number of cycles, N is the number of training samples in step 2, c (N) is the average number of steps in which the binary tree search fails,

in the formula: h (N-1) is a harmonic series, which is a function with respect to N-1;

and 5: and sorting the IPs in ascending order according to the scores, setting a threshold value, and setting the IPs smaller than the set threshold value as abnormal users.

2. The method for mining abnormal users based on page access topology as claimed in claim 1, wherein the information extracted from the log in step 1 includes target URL, request source IP and request time, and removes the request parameters of URL.

Images