CN111556016A - Network flow abnormal behavior identification method based on automatic encoder - Google Patents
Network flow abnormal behavior identification method based on automatic encoder Download PDFInfo
- Publication number
- CN111556016A CN111556016A CN202010217930.8A CN202010217930A CN111556016A CN 111556016 A CN111556016 A CN 111556016A CN 202010217930 A CN202010217930 A CN 202010217930A CN 111556016 A CN111556016 A CN 111556016A
- Authority
- CN
- China
- Prior art keywords
- data
- model
- abnormal
- saids
- layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 91
- 206010000117 Abnormal behaviour Diseases 0.000 title claims abstract description 13
- 230000002159 abnormal effect Effects 0.000 claims abstract description 18
- 238000001514 detection method Methods 0.000 claims description 33
- 238000012549 training Methods 0.000 claims description 24
- 238000005070 sampling Methods 0.000 claims description 14
- 238000010606 normalization Methods 0.000 claims description 13
- 238000007781 pre-processing Methods 0.000 claims description 10
- 230000008569 process Effects 0.000 claims description 8
- 239000013598 vector Substances 0.000 claims description 4
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 3
- 230000004913 activation Effects 0.000 claims description 3
- 238000010801 machine learning Methods 0.000 abstract description 11
- 230000006399 behavior Effects 0.000 description 10
- 238000013135 deep learning Methods 0.000 description 10
- 230000009467 reduction Effects 0.000 description 10
- 238000012545 processing Methods 0.000 description 9
- 238000013528 artificial neural network Methods 0.000 description 7
- 238000004422 calculation algorithm Methods 0.000 description 7
- 238000002474 experimental method Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 4
- 238000000513 principal component analysis Methods 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 238000003066 decision tree Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000011156 evaluation Methods 0.000 description 3
- 238000007637 random forest analysis Methods 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 238000012512 characterization method Methods 0.000 description 2
- 230000007123 defense Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000007636 ensemble learning method Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 230000000306 recurrent effect Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000007635 classification algorithm Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 230000008092 positive effect Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1416—Event detection, e.g. attack signature detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/24323—Tree-organised classifiers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N7/00—Computing arrangements based on specific mathematical models
- G06N7/01—Probabilistic graphical models, e.g. probabilistic networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Mathematical Physics (AREA)
- Computer Security & Cryptography (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computational Mathematics (AREA)
- Mathematical Analysis (AREA)
- Mathematical Optimization (AREA)
- Pure & Applied Mathematics (AREA)
- Probability & Statistics with Applications (AREA)
- Algebra (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Medical Informatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computer Hardware Design (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention provides a network flow abnormal behavior identification method based on an automatic encoder, which belongs to the cross technical field of combination of machine learning and information safety, and aims to balance the category distribution of normal flow data and abnormal flow data in flow data by using a comprehensive few oversampling methods and combine the automatic encoder, so that nonlinear structure information can be effectively extracted from mass data, and abnormal behaviors in network flow can be identified.
Description
Technical Field
The invention provides an effective network flow abnormal behavior identification method. The method combines a comprehensive few oversampling methods and an automatic encoder classification algorithm, and belongs to the cross technical field of combination of machine learning and information safety.
Background
With the rapid development of the information age, the internet has become an indispensable part of people's lives. However, the frequency of the attack behaviors in the network and the scale of the attack events are also increasing, and these attack behaviors not only cause huge economic loss, but also pose serious threats to social stability and national security, and maintaining the security of the network space has become a problem to be solved urgently. In order to better maintain the security of network space, ensure the availability of various network resources and prevent various attack behaviors, the intrusion detection technology as an active defense method becomes a hot problem of current research. The intrusion detection system is an active safety protection technology, can monitor the transmission behavior of data in a network, and sends out an alarm or interrupts an abnormal transmission behavior after finding suspicious transmission.
The concept of intrusion detection was first proposed by James Anderson in 1980 to monitor attack behavior. There are a lot of studies on the detection of network intrusion behaviors, and these works can be classified into misuse-based intrusion detection systems (MIDS) and anomaly-based intrusion detection systems (AIDS). The MIDS is also called an intrusion detection system based on signature, and detects attack behaviors according to the existing knowledge. Although the MIDS has higher accuracy and lower false alarm rate, it cannot detect unknown attacks that are not in the signature database. Unlike MIDS, AIDS can detect unknown attacks by comparing normal and abnormal behavior. Thus, AIDS is drawing increasing attention, the most important of which is the use of traditional feature-based machine learning methods, such as decision trees, random forests, na iotave bayes, etc. However, intrusion detection based on the conventional machine learning method usually emphasizes feature engineering, and is a shallow learning method. With the increase of massive high-dimensional data in a network and the increase of network bandwidth, the complexity of the data and the diversity of characteristics are continuously improved, and the purposes of analysis and prediction are difficult to achieve through shallow learning.
In recent years, deep neural network technology has enjoyed great success in the fields of image recognition, natural language processing, speech recognition, and the like. The deep neural network is a method for performing characterization learning on data, can learn the intrinsic rule of the data, and adapts to the requirements of high-dimensional learning and prediction by constructing a nonlinear network structure formed by a plurality of hidden layers. The current intrusion detection method based on deep learning also has a development prospect, comprises an automatic encoder, a deep belief network, a recurrent neural network, a convolutional neural network, a gated recurrent unit and the like, and achieves certain success. However, these deep learning methods for intrusion detection still have some problems.
For example, due to the category imbalance problem, many studies do not consider the overall distribution of traffic data, the decision function is biased towards most samples, low frequency attack samples are considered as noise and are ignored, so that the model is difficult to capture effective features, and low frequency attacks are difficult to detect. On the other hand, some studies do not process high-dimensional data when converting symbolic data into numerical data, which results in low training efficiency, memory space consumption and poor detection performance. Therefore, the efficiency and the accuracy of intrusion detection can be better improved by performing dimension reduction processing on the traffic data.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a novel deep neural network intrusion detection method, namely a network flow abnormal behavior identification method based on an automatic encoder, which uses a comprehensive few oversampling methods to balance the category distribution of normal flow data and abnormal flow data in flow data and combines the automatic encoder, thereby effectively extracting nonlinear structure information from mass data.
In order to achieve the purpose, the invention adopts the specific technical scheme that:
a network flow abnormal behavior identification method based on an automatic encoder comprises the following steps:
1) constructing a sparse abnormal intrusion detection model SAIDS by using an automatic encoder;
2) training the SAIDS model, comprising the following steps:
the SAIDS model preprocesses original training data, and balances the category distribution of normal flow and abnormal flow in flow data by adopting a comprehensive few oversampling method (SMOTE) to the preprocessed training data to obtain balanced data;
classifying normal flow and abnormal flow according to the balance data, calculating a loss value, and finding out a model parameter corresponding to the minimum loss value to obtain a trained SAIDS model;
3) detecting the network traffic to be identified by using the trained SAIDS model, wherein the steps comprise:
the SAIDS model preprocesses the network traffic to be identified, classifies the preprocessed network traffic into normal traffic and abnormal traffic, and identifies abnormal behaviors.
Further, the original training data carries class labels for normal traffic and abnormal traffic.
Further, the normalized data is obtained through preprocessing, wherein the preprocessing comprises converting the symbolic data into numerical data by using one-hot coding, and normalizing the numerical data.
Further, the normalization processing refers to the reduction of numerical data to the range of [0,1] by adopting a Min-Max normalization method.
Further, a linear interpolation is adopted in the comprehensive minority oversampling method, and new data are generated by multiplying the difference between the data samples in the minority class and a randomly selected nearest neighbor sample by a random number between 0 and 1 and then summing the difference and the data samples in the minority class.
Furthermore, the SAIDS model mainly comprises a discarding layer and an automatic encoder except a network structure which is responsible for preprocessing original data and obtaining balanced data by adopting an SMOTE method; the discarding layer preprocesses the balance data to prevent overfitting; the automatic encoder comprises an input layer, an encoding layer and a decoding layer, wherein the input layer receives preprocessed balance data, the encoding layer maps the balance data into low-dimensional features, the decoding layer reconstructs the low-dimensional features into input data, and the input data is classified into normal flow and abnormal flow.
Further, the discarding layer is to perform element product processing on the input balance data and the vector with the probability obeying the Bernoulli distribution.
Further, the SAIDS model selects the Relu activation function and the Adam optimizer when training, and calculates the loss value using the mean square error.
The reason why the invention chooses to integrate a few oversampling methods for balancing the class distribution of the traffic data is that it has the following advantages: (1) the under-sampling method obtains a balanced data set by deleting most types of data, and important data information may be lost, so that the over-sampling method generally has better processing effect and higher use frequency than the under-sampling method. (2) By integrating a few oversampling methods and adopting the theory of linear interpolation, the overfitting phenomenon is effectively reduced, and the limitation in the sampling process is reduced.
Due to the fact that the dimensionality of the data is too high, training efficiency is low, the needed storage space can be reduced by reducing the dimensionality of the data, the calculation speed is increased, redundant features are removed, and the data are better expressed. The traditional linear dimensionality reduction methods such as principal component analysis and the like are difficult to capture nonlinear information in data, and the kernel function-based nonlinear dimensionality reduction methods such as principal component analysis and the like are high in calculation complexity and difficult to apply to large-scale data sets. The automatic encoder is used as a dimension reduction method in deep learning, nonlinear structure information can be effectively extracted from mass data sets, and higher-level features can be obtained. Therefore, the invention adopts the automatic encoder algorithm to construct the intrusion detection system, thereby improving the detection capability of massive high-dimensional data.
Compared with the prior art, the invention has the following positive effects:
the invention performs experiments on a plurality of real network traffic data sets, and evaluates the performance of the model by using the overall accuracy, precision, recall rate and F1 value. Comprehensive experiment results show that the model provided by the invention is superior to the existing baseline recognition methods such as decision trees, random forests, gated neural networks and the like in performance.
Drawings
Fig. 1 is a flowchart of the entire method for identifying abnormal network traffic behavior according to this embodiment.
FIGS. 2A-2B are graphs of the NSL-KDD data set used in the present example; where fig. 2A is the original training data set and fig. 2B is the data set processed by the SMOTE method.
Fig. 3A-3B are distribution diagrams of UNSW-NB15 data sets used in the present embodiment, where fig. 3A is the original training data set and fig. 3B is the data set processed by the SMOTE method.
Fig. 4A-4B are cubic graphs of performance comparisons for the deep learning method, where fig. 4A is an evaluation case for the NSL-KDD dataset and fig. 4B is an evaluation case for the UNSW-NB15 dataset.
Detailed Description
In order to make the technical solutions in the embodiments of the present invention better understood and make the objects, features, and advantages of the present invention more comprehensible, the technical core of the present invention is described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The embodiment provides an effective network traffic abnormal behavior identification method. The method has the general idea that network traffic data is preprocessed firstly, the preprocessing comprises two parts of sign data digitization and numerical data normalization, then the distribution condition of the network traffic data is changed by using a comprehensive few oversampling methods, and a model is established by combining an automatic encoder method, so that the attack behavior in the network traffic data can be detected.
The overall flow chart of the method is shown in fig. 1, and the details of the steps of the method are described as follows:
(1) data pre-processing
The validation dataset used in the method is the NSL-KDD dataset and the UNSW-NB15 dataset. Specifically, the NSL-KDD dataset is a subset of the KDDCup1999 dataset, which is provided by the United states defense advanced research planning agency and contains weeks of attack data that can be used to evaluate intrusion detection performance. However, the KDDCup1999 data set has the problems of redundant records, more preference of a classifier for frequent records and the like. To solve these problems. The NSL-KDD data set effectively solves the problems of redundant features and repeated recording of a KDDCup1999 data set, and the training data set and the testing data set are reasonable in quantity. The NSL-KDD data contains TCP/IP connection records, each record containing 41 features, such as basic features, content features, and traffic features, as well as a category label and a difficulty label. The UNSW-NB15 data set was created in the network wide laboratory of the australian network security center, collecting 16 hours of data on 22 days 1 month 2015 and 15 hours of data on 17 days 2 month 2015, with contemporary actual network traffic, containing more comprehensive attack activity. The UNSW-NB15 dataset contains 49 features, including traffic features, basic features, content features, temporal features, etc.
In order to remove redundant data, the detection efficiency is improved, and the time consumption is reduced. The method carries out data preprocessing on the network flow data and comprises two parts of sign characteristic numeralization and numerical data normalization.
And (3) symbol characteristic numeralization: the symbol feature data is usually contained in the intrusion detection data set, and the symbol data is difficult to be directly processed by the model, so that the symbol data is converted into digital data by using a one-hot encoder in the step. For example, the protocol _ type feature in the NSL-KDD dataset contains three characters, TCP, UDP and ICMP respectively. Mapping the three characters into 3 binary vectors through one-hot coding, wherein the mapping results are as follows: [1,0,0], [0,1,0], and [0,0,1 ]. In this way, all symbol characteristics are mapped by one-hot encoding. For the category label, the normal traffic data in the dataset is labeled as 0 and the abnormal traffic data is labeled as 1.
Normalization of numerical data: the data normalization can solve the problem that the dimensionality of different characteristic data is greatly different, and therefore the data normalization method is widely used in a data preprocessing step. In order to ensure the reliability of the detection result, the normalization processing needs to be performed on the numerical data in the two data sets, wherein the normalization refers to reducing all the characteristic data to [0,1]]Within the range. The method aims to adopt a Min-Max normalization method to process data, and the conversion formula is as follows:where x represents the attribute value of a feature, xmaxMaximum value, x, representing such characteristic propertyminRepresents the minimum value of such characteristic attribute, and x' represents the result of normalizing x.
(2) Changing data class distribution
In network traffic data, the abnormal traffic data is usually much smaller than the normal traffic data, resulting in the decision function being biased towards most samples, and the low frequency attack samples are considered as noise and ignored. Therefore, in order to improve the detection performance of the model, data with a small amount of data needs to be processed. There are generally two processing methods, namely, a solution at an algorithm level and a solution at a data level, wherein the solution at the algorithm level is usually to modify a classifier algorithm or optimize the performance of a learning algorithm, and the distribution of data of different categories is adjusted by adjusting the importance of the categories in the learning or decision making process. Common data-level solutions include under-sampling and over-sampling methods, which balance the distribution of data classes by sampling. Undersampling balances the class distribution by reducing the amount of data for the majority of classes and oversampling balances the class distribution by increasing the amount of data for the minority of classes.
Because the under-sampling method obtains the balanced data set by deleting most types of data, important data information may be lost, and therefore, the over-sampling method generally has a better processing effect and a higher use frequency than the under-sampling method, and the method adopts a famous over-sampling method, namely a comprehensive few over-Sampling Method (SMOTE), to process data with a small data volume, thereby improving the detection performance. SMOTE is a method for randomly generating new samples between a few class samples and their neighbors, and improves class distribution by adding few class data. The SMOTE method adopts the theory of linear interpolation, effectively reduces the overfitting phenomenon and reduces the limitation in the sampling process. The formula for generating synthetic data by the SMOTE method is as follows:
ynew=yi+(yi-yj) × formula (1)
Wherein, ynewRepresenting newly generated synthetic data, yiRepresenting data samples in a few categories, yjRepresents from yiRepresents a random number between 0 and 1, of the k nearest neighbors that are randomly chosen. The distribution of the NSL-KDD raw training data set and the distribution of the training data set after processing using the SMOTE method are shown in fig. 2A-2B. The distribution of UNSW-NB15 raw training data sets and the distribution of training data sets processed using the SMOTE method are shown in fig. 3A-3B.
(3) Model training
Due to the fact that the dimensionality of the data is too high, training efficiency is low, the needed storage space can be reduced by reducing the dimensionality of the data, the calculation speed is increased, redundant features are removed, and the data are better expressed. The traditional linear dimensionality reduction methods such as principal component analysis and the like are difficult to capture nonlinear information in data, and the kernel function-based nonlinear dimensionality reduction methods such as principal component analysis and the like are high in calculation complexity and difficult to apply to large-scale data sets. The automatic encoder is used as a dimension reduction method in deep learning, nonlinear structure information can be effectively extracted from mass data sets, and higher-level features can be obtained. Therefore, the automatic encoder is very suitable for the tasks of dimension reduction and feature learning, and the intrusion detection system is constructed by adopting the automatic encoder algorithm, so that the detection capability of massive high-dimensional data is improved.
The automatic encoder is a three-layer neural network comprising an input layer, an encoding layer and a decoding layer, and is an unsupervised learning structure consisting of an encoder and a decoder. After data preprocessing and data category distribution processing, an automatic encoder is used for carrying out dimension reduction on training data and training a model. Specifically, in order to avoid overfitting, the discarding layer is added to preprocess the balance data, so that the overfitting phenomenon is prevented, and the balance data preprocessed by the discarding layer is used as the input of the automatic encoder. For an auto-encoder, the encoder maps the input data to low-dimensional features, and the decoder reconstructs the input data using the mapped low-dimensional features. Through reconstruction input, the hidden layer can learn the characterization information of the input data, the characteristic dimensionality of the data set can be effectively reduced, and the integrity of the characteristic information is guaranteed. The formula for the discarded layer is as follows:
r to Bernoulli (p) formula (2)
λ (n) ═ r ═ α (n) formula (3)
Where r is an independent and uniformly distributed vector, obeying a bernoulli distribution with probability p, whose shape is the same as α (n), representing the product of the elements, α (n) representing the output of the current layer, and λ (n) representing the output of the discarded layer.
(4) Abnormal behavior detection
Finally, the Relu activation function and the Adam optimizer are selected and the loss value is calculated using the mean square error. The formula for the mean square error is as follows:
wherein,is the true value of the ith data,is the predicted value of the ith data, and m is the total number of samples of the test data. After the models are trained, the best training model with the minimum loss value is selected to classify the test data, and the detection result is evaluated by combining with the evaluation index.
(5) Comparison of results
The invention performs experiments on a plurality of real network traffic data sets, and evaluates the performance of the model by using the overall accuracy, precision, recall rate and F1 value. In order to verify the effectiveness of the proposed method (SMOTE + AE), the invention carries out comparison experiments on both the machine learning method and the deep learning method. Seven commonly used machine learning methods are used for comparison, which are respectively as follows: decision tree, random forest, gaussian naive bayes, polynomial naive bayes, bernoulli naive bayes, Adaboost, extreme gradient boost algorithms. Baseline comparison experiments for the deep learning method were gated-round network (GRU), SMOTE + gated-round network, sparse auto-encoder (SAE), SMOTE + sparse auto-encoder and auto-encoder.
1) Performance comparison with machine learning methods
The results of this experiment comparing the performance of the NSL-KDD dataset with the UNSW-NB15 dataset are shown in Table 1. As can be seen from table 1, for the machine learning method, the detection result based on the tree method is generally better than the detection result based on the probability method because the naive bayes method is difficult to process the features having correlation. The detection capability of the ensemble learning method is superior to that of a single classifier because it is difficult for a single classifier to fully summarize the features of a particular data set, and the ensemble learning method can capture more information. As can also be seen from table 1, the accuracy of the SAIDS method on the two data sets was 91.08% and 97.58%, respectively, which is superior to the traditional machine learning method, demonstrating the effectiveness of the method herein. This is due to the fact that the shallow learning method does not handle unbalanced flow data, resulting in a model biased towards the normal category with a large number of samples. As the complexity of data increases, the learning ability of the shallow learning method is limited.
TABLE 1 comparison of Performance in NSL-KDD and UNSW-NB15 data set machine learning methods
2) Performance comparison with deep learning methods
The deep learning method can learn the intrinsic rules of the data, so that the method is more suitable for fitting and predicting the network traffic data. The results of comparing the SAIDS model proposed by the present invention with the most advanced deep learning model are shown in FIGS. 4A-4B. It can be seen from the figure that the SAIDS performance proposed by the present invention is superior to other five methods, and can effectively detect network intrusion data. The detection performance of the model processed using the SMOTE method is generally better than that of the model processed without the SMOTE method, which demonstrates the importance of balancing the data class distribution. The invention considers the processing of unbalanced data and redundant characteristics at the same time, and the detection result is superior to that of a single gated neural network and a sparse automatic encoder.
The performance comparison result with the machine learning and deep learning method shows that the SAIDS model provided by the invention has better prediction accuracy rate for the detection of network traffic and has the potential of practical application.
The above-mentioned embodiments only express the embodiments of the present invention, and the description thereof is specific, but not construed as limiting the scope of the present invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent should be subject to the appended claims.
Claims (10)
1. A network flow abnormal behavior identification method based on an automatic encoder comprises the following steps:
1) constructing a sparse abnormal intrusion detection model SAIDS by using an automatic encoder;
2) training the SAIDS model, comprising the following steps:
the SAIDS model preprocesses original training data, and balances the category distribution of normal flow and abnormal flow in the flow data by adopting a comprehensive few oversampling methods for the preprocessed training data to obtain balanced data;
classifying normal flow and abnormal flow according to the balance data, calculating a loss value, and finding out a model parameter corresponding to the minimum loss value to obtain a trained SAIDS model;
3) detecting the network traffic to be identified by using the trained SAIDS model, wherein the steps comprise:
the SAIDS model preprocesses the network traffic to be identified, classifies the preprocessed network traffic into normal traffic and abnormal traffic, and identifies abnormal behaviors.
2. The method of claim 1, wherein raw training data comprises an NSL-KDD dataset and an UNSW-NB15 dataset.
3. The method of claim 1, wherein the raw training data carries class labels for normal traffic and abnormal traffic.
4. The method of claim 1, wherein the normalized data is obtained by preprocessing, the preprocessing comprising converting the symbolic data into numerical data using one-hot encoding, and normalizing the numerical data.
5. The method of claim 4, wherein the normalization process reduces the numerical data to a range of [0,1 ].
6. The method of claim 5, wherein the normalization is performed using a Min-Max normalization method.
7. The method of claim 1, wherein the integrated minority over-sampling method uses linear interpolation, and new data is generated by multiplying a difference between data samples in the minority class and a randomly selected nearest neighbor sample by a random number between 0 and 1, and summing the difference and the data samples in the minority class.
8. The method of claim 1, wherein the SAIDS model includes a drop layer and an auto-encoder; the discarding layer preprocesses the balance data to prevent overfitting; the automatic encoder comprises an input layer, an encoding layer and a decoding layer, wherein the input layer receives preprocessed balance data, the encoding layer maps the balance data into low-dimensional features, the decoding layer reconstructs the low-dimensional features into input data, and the input data is classified into normal flow and abnormal flow.
9. The method of claim 8, wherein the discarding layer is an elemental product of the input balance data and a vector having a probability that obeys a bernoulli distribution.
10. The method of claim 1, 8 or 9, wherein the SAIDS model is trained by selecting a Relu activation function and an Adam optimizer and using a mean square error to calculate the loss value.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010217930.8A CN111556016B (en) | 2020-03-25 | 2020-03-25 | Network flow abnormal behavior identification method based on automatic encoder |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010217930.8A CN111556016B (en) | 2020-03-25 | 2020-03-25 | Network flow abnormal behavior identification method based on automatic encoder |
Publications (2)
Publication Number | Publication Date |
---|---|
CN111556016A true CN111556016A (en) | 2020-08-18 |
CN111556016B CN111556016B (en) | 2021-02-26 |
Family
ID=72003821
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202010217930.8A Active CN111556016B (en) | 2020-03-25 | 2020-03-25 | Network flow abnormal behavior identification method based on automatic encoder |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN111556016B (en) |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112039903A (en) * | 2020-09-03 | 2020-12-04 | 中国民航大学 | Network security situation assessment method based on deep self-coding neural network model |
CN112134875A (en) * | 2020-09-18 | 2020-12-25 | 国网山东省电力公司青岛供电公司 | IoT network abnormal flow detection method and system |
CN112165464A (en) * | 2020-09-15 | 2021-01-01 | 江南大学 | Industrial control hybrid intrusion detection method based on deep learning |
CN112702329A (en) * | 2020-12-21 | 2021-04-23 | 四川虹微技术有限公司 | Traffic data anomaly detection method and device and storage medium |
CN113158076A (en) * | 2021-04-05 | 2021-07-23 | 北京工业大学 | Social robot detection method based on variational self-coding and K-nearest neighbor combination |
CN113554474A (en) * | 2021-08-11 | 2021-10-26 | 上海明略人工智能(集团)有限公司 | Model verification method and device, electronic equipment and computer-readable storage medium |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106973057A (en) * | 2017-03-31 | 2017-07-21 | 浙江大学 | A kind of sorting technique suitable for intrusion detection |
CN107103332A (en) * | 2017-04-07 | 2017-08-29 | 武汉理工大学 | A kind of Method Using Relevance Vector Machine sorting technique towards large-scale dataset |
US20180041536A1 (en) * | 2016-08-02 | 2018-02-08 | Invincea, Inc. | Methods and apparatus for detecting and identifying malware by mapping feature data into a semantic space |
CN108768946A (en) * | 2018-04-27 | 2018-11-06 | 中山大学 | A kind of Internet Intrusion Detection Model based on random forests algorithm |
CN109344888A (en) * | 2018-09-19 | 2019-02-15 | 广东工业大学 | A kind of image-recognizing method based on convolutional neural networks, device and equipment |
CN110163261A (en) * | 2019-04-28 | 2019-08-23 | 平安科技(深圳)有限公司 | Unbalanced data disaggregated model training method, device, equipment and storage medium |
-
2020
- 2020-03-25 CN CN202010217930.8A patent/CN111556016B/en active Active
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180041536A1 (en) * | 2016-08-02 | 2018-02-08 | Invincea, Inc. | Methods and apparatus for detecting and identifying malware by mapping feature data into a semantic space |
CN106973057A (en) * | 2017-03-31 | 2017-07-21 | 浙江大学 | A kind of sorting technique suitable for intrusion detection |
CN107103332A (en) * | 2017-04-07 | 2017-08-29 | 武汉理工大学 | A kind of Method Using Relevance Vector Machine sorting technique towards large-scale dataset |
CN108768946A (en) * | 2018-04-27 | 2018-11-06 | 中山大学 | A kind of Internet Intrusion Detection Model based on random forests algorithm |
CN109344888A (en) * | 2018-09-19 | 2019-02-15 | 广东工业大学 | A kind of image-recognizing method based on convolutional neural networks, device and equipment |
CN110163261A (en) * | 2019-04-28 | 2019-08-23 | 平安科技(深圳)有限公司 | Unbalanced data disaggregated model training method, device, equipment and storage medium |
Non-Patent Citations (2)
Title |
---|
XIAOXUAN ZHANG: "An Intrusion Detection System Based on Convolutional Neural Network for Imbalanced Network Traffic", 《2019 IEEE 7TH INTERNATIONAL CONFERENCE ON COMPUTER SCIENCE AND NETWORK TECHNOLOGY》 * |
刘明峰: "基于深度学习的wi-fi网络入侵检测方法", 《计算机工程与设计》 * |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112039903A (en) * | 2020-09-03 | 2020-12-04 | 中国民航大学 | Network security situation assessment method based on deep self-coding neural network model |
CN112165464A (en) * | 2020-09-15 | 2021-01-01 | 江南大学 | Industrial control hybrid intrusion detection method based on deep learning |
CN112134875A (en) * | 2020-09-18 | 2020-12-25 | 国网山东省电力公司青岛供电公司 | IoT network abnormal flow detection method and system |
CN112702329A (en) * | 2020-12-21 | 2021-04-23 | 四川虹微技术有限公司 | Traffic data anomaly detection method and device and storage medium |
CN113158076A (en) * | 2021-04-05 | 2021-07-23 | 北京工业大学 | Social robot detection method based on variational self-coding and K-nearest neighbor combination |
CN113158076B (en) * | 2021-04-05 | 2022-07-22 | 北京工业大学 | Social robot detection method based on variational self-coding and K-nearest neighbor combination |
CN113554474A (en) * | 2021-08-11 | 2021-10-26 | 上海明略人工智能(集团)有限公司 | Model verification method and device, electronic equipment and computer-readable storage medium |
Also Published As
Publication number | Publication date |
---|---|
CN111556016B (en) | 2021-02-26 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN111556016B (en) | Network flow abnormal behavior identification method based on automatic encoder | |
CN109768985B (en) | Intrusion detection method based on flow visualization and machine learning algorithm | |
CN110213222B (en) | Network intrusion detection method based on machine learning | |
CN110460605B (en) | Abnormal network flow detection method based on automatic coding | |
CN111695597B (en) | Credit fraud group identification method and system based on improved isolated forest algorithm | |
CN110348486A (en) | Based on sampling and feature brief non-equilibrium data collection conversion method and system | |
CN109446804B (en) | Intrusion detection method based on multi-scale feature connection convolutional neural network | |
CN112134862B (en) | Coarse-fine granularity hybrid network anomaly detection method and device based on machine learning | |
CN113242207A (en) | Iterative clustering network flow abnormity detection method | |
CN114553545A (en) | Intrusion flow detection and identification method and system | |
CN112348080A (en) | RBF improvement method, device and equipment based on industrial control abnormity detection | |
CN113420294A (en) | Malicious code detection method based on multi-scale convolutional neural network | |
CN112529638B (en) | Service demand dynamic prediction method and system based on user classification and deep learning | |
CN116582300A (en) | Network traffic classification method and device based on machine learning | |
CN111507385A (en) | Extensible network attack behavior classification method | |
CN116318928A (en) | Malicious traffic identification method and system based on data enhancement and feature fusion | |
CN112365060A (en) | Preprocessing method for power grid internet of things perception data | |
CN112508726A (en) | False public opinion identification system based on information spreading characteristics and processing method thereof | |
CN114554491A (en) | Wireless local area network intrusion detection method based on improved SSAE and DNN models | |
Acharya et al. | Efficacy of CNN-bidirectional LSTM hybrid model for network-based anomaly detection | |
CN117155701A (en) | Network flow intrusion detection method | |
Soheily-Khah et al. | Intrusion detection in network systems through hybrid supervised and unsupervised mining process-a detailed case study on the ISCX benchmark dataset | |
CN116633589A (en) | Malicious account detection method, device and storage medium in social network | |
CN113609480B (en) | Multipath learning intrusion detection method based on large-scale network flow | |
CN116170187A (en) | Industrial Internet intrusion monitoring method based on CNN and LSTM fusion network |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |