CN115811440B

CN115811440B - Real-time flow detection method based on network situation awareness

Info

Publication number: CN115811440B
Application number: CN202310040052.0A
Authority: CN
Inventors: 车洵; 孙捷; 金奎�; 卫英俊
Original assignee: Nanjing Zhongzhiwei Information Technology Co ltd
Current assignee: Nanjing Zhongzhiwei Information Technology Co ltd
Priority date: 2023-01-12
Filing date: 2023-01-12
Publication date: 2023-06-09
Anticipated expiration: 2043-01-12
Also published as: CN115811440A

Abstract

The invention discloses a real-time flow detection method based on network situation awareness, which comprises the following steps: performing character type digitization and normalization processing on the network flow data; according to the actual network situation, the calculation power scheduling module judges the network load condition and distributes calculation power according to the network load condition; the preprocessed flow data enter, and the classification detection is carried out under the constraint of the distributed calculation force, so that the normal flow and the abnormal flow are judged; the abnormal flow detected by the classification enters, multi-classification detection is carried out under the constraint of distributed calculation force, and the attack type of the abnormal flow is judged; collecting normal flow and abnormal flow which are correctly judged, using GAN to carry out data augmentation on the abnormal flow, and then using mixed augmentation training data to retrain so as to obtain updated parameters; repeating the steps, and continuously performing real-time intrusion detection of network situation awareness; the scheme has the characteristics of rapidly and accurately identifying the abnormal data in the network traffic and generalizing the identification capability for the abnormal traffic.

Description

Real-time flow detection method based on network situation awareness

Technical Field

The invention relates to the field of network security, in particular to a real-time flow detection method based on network situation awareness.

Background

In recent years, with the continuous development and wide application of emerging computer technologies such as cloud computing, distributed systems, big data, 5G communication, internet of things, industrial control networks and the like, the number of user equipment accessing to the internet is rapidly increased, computer network security events frequently occur, network information security faces a great challenge, individuals and organizations are protected from network attacks of hackers, and timely finding an intruder is an important step for guaranteeing network security. The deep learning enabled intrusion detection technology has advantages in the aspects of processing high-dimensional data, mining hidden information in the data and the like, and is widely focused by academia and industry. However, accurately classifying network traffic is not a simple task.

When large-scale data is processed, the intrusion detection system based on deep learning often adopts a method for reducing the data dimension to reduce the computational complexity. However, when the data dimension of the network traffic is reduced, important information in the network traffic is likely to be removed, so that the detection accuracy of the model is greatly reduced. Furthermore, existing intrusion detection systems directly or indirectly assume that the computational resources are plentiful, ignoring the impact of detection time consuming on system availability. With the proliferation of network data size, the existing intrusion detection system cannot meet the low-delay requirement of network traffic gradually. Currently, deep learning models used by intrusion detection systems are typically trained and tested using public data sets. This way of learning with static data results in an intrusion detection system that is strongly dependent on the data set and has low generalization of the model. Along with the continuous change of network attack, the identification capability of the existing intrusion detection system for abnormal network attack traffic is continuously reduced.

In recent years, the popularity of deep learning has led to its widespread use for identifying various types of network attacks. The deep learning overcomes the defect of shallow learning, can automatically extract high-level characteristics, and has attracted extensive attention of domestic and foreign students in application research in network intrusion detection. The publication proposes a new hybrid method SCDNN for network intrusion detection, which consists of Spectral Clustering (SC) and Deep Neural Networks (DNN). First, the SC divides the original training data set into k training subsets and trains k sub-DNN classifiers with the training subsets. Next, the test data set is divided into subsets with SCs, and the corresponding child DNNs are tested with the test data subsets. Experimental results show that the detection accuracy of SCDNN is superior to SVM, BP neural network, RF and Bayesian methods by evaluating the SCDNN on 6 KDD-Cup99 and NSL-KDD data sets. The publication proposes an unsupervised network intrusion detection method ID-CVAE based on a conditional variation self-encoder, which designs a specific architecture, integrating intrusion flags only inside the decoder layer. The ID-CVAE classifier can recover missing features from incomplete training data sets, and has accuracy of 80.10% on NSL-KDD data sets and better classifying effect than other common classifiers.

With the deep research, network security personnel design and verify more network intrusion detection methods enabled by deep learning. Such as: the publication proposes a deep learning method for network intrusion detection by using a Recurrent Neural Network (RNN), called RNN-IDS, which researches the accuracy and training time of RNNs with different learning rates and hidden layer neuron numbers in two-class and multi-class experiments, and the experimental results show that the multi-class accuracy of RNNs on a kdtest+ and kdtest-21 test set is 73.28%, 68.55%, and slightly lower than 88.32% and 86.71% of the two-class. The open literature also provides a network intrusion detection method which converts NSL-KDD data into picture features through the existing graph conversion technology, then uses residual convolution neural networks ResNet and GoogLeNet to conduct two-class network intrusion detection, evaluates feasibility and detection performance of intrusion detection conversion into image classification, is very sensitive to image conversion of attack data compared with a CNN model, and theoretical analysis and experimental results show that ResNet and GoogLeNet are more suitable for intrusion detection of image conversion. The publication also proposes a new two-stage deep learning (TSDL) model that uses a stacked self-encoder to classify network traffic normally and abnormally and output probability values in the first stage; in the second stage, probability values are used as additional features to be added to original features, then a softmax classifier is used for detecting normal attacks and other types of attacks, and the detection accuracy of TSDL on a KDD99 data set and a UNSW-NB15 data set respectively reaches 99.996% and 89.134%, which is obviously superior to other reference detection methods.

In summary, the deep learning method achieves satisfactory effects in the network intrusion detection system, but with the continuous expansion of network data, a large amount of nonlinear network data brings new challenges to the intrusion detection method based on deep learning, so that the deep learning method faces the problems of low detection rate, high efficiency, generalization and difficult balance of unknown attack and low-frequency attack. Therefore, it is highly desirable to provide a real-time traffic detection method based on network situation awareness to solve the above-mentioned problems.

Disclosure of Invention

Therefore, it is necessary to provide a real-time traffic detection method based on network situation awareness, which can rapidly and accurately identify abnormal data in network traffic, continuously learn new traffic characteristics during operation, and have extremely strong generalization identification capability on abnormal traffic.

In order to achieve the above object, the present inventors provide a real-time traffic detection method based on network situation awareness, including:

s1: performing character type digitization and normalization processing on the network flow data;

s2: according to the actual network situation, the calculation power scheduling module judges the network load condition and distributes calculation power to Multi-Class detection Net, multi-Label detection Net and Data augmentation Net according to the network load condition;

s3: the preprocessed flow data enter Multi-Label detection Net, and the classification detection is carried out under the constraint of the distributed calculation force, so that the normal flow and the abnormal flow are judged;

s4: the abnormal flow detected by the classification enters Multi-Class detection Net, multi-classification detection is carried out under the constraint of distributed calculation force, and the attack type of the abnormal flow is judged;

s5: data augmentation Net collecting the normal flow and the abnormal flow which are correctly judged, using GAN to carry out data augmentation on the abnormal flow, and then using mixed augmentation training data to retrain Multi-Class detection Net and Multi-Label detection Net to obtain updated parameters;

s6: repeating the steps S1 to S5, and continuously performing real-time intrusion detection of network situation awareness.

As a preferred mode of the invention, the calculation force scheduling module firstly distributes all calculation forces to Multi-Class detection Net preferentially when distributing calculation forces, and then distributes the rest calculation forces to Multi-Label detection Net and Data augmentation Net.

As a preferred mode of the invention, the Sigmoid layer is used at the last layer of the network for Multi-Class detection Net, the output probability value is normalized to [0,1], the correctly marked threshold value is set to be 0.5, and when the network is trained, the error on each label is calculated by using a two-class cross entropy loss function, and the error function of the neural network is equal to the sum of all label loss functions.

As a preferred mode of the present invention, the Data augmentation Net uses a general learning model to amplify the network traffic, the general learning model continuously collects the normal traffic and the abnormal traffic during the operation process, and uses GAN to amplify the abnormal data to form the amplified training data of the abnormal data and the normal traffic, and in the intrusion detection process, the situation sensing module firstly performs redundant storage on the network weight parameters, and after generating the amplified training data of a certain scale, trains the redundant storage network for updating the parameters.

In a preferred embodiment of the present invention, in step S3, multi-Label detection Net performs a two-class detection under the constraint of the assigned computing force, comprising the following two steps:

s301: extracting flow data characteristics to capture space-time characteristics from network flow to the maximum extent;

s302: key feature learning, such that the model focuses on important features that are beneficial to classification.

As a preferred mode of the present invention, in step S301, space-time connection learning is used to capture space-time features, the space-time connection learning includes a space-time block and a transition block, the space-time block includes two core feature extraction blocks Conv and a long short memory layer LSTM, and the space-time block is implemented by a packet CNN and the long short memory layer LSTM, respectively, where the packet convolution layer uses a 3×3 filter, the number of channels of the output feature map is twice the number of input channels, and the number of packets is the number of data channels of the input model; the transition block is used for reducing the dimension, and a long and short-term memory layer LSTM which does not change the dimension is added in the transition block.

As a preferred mode of the invention, a batch normalization layer, a Maxpooling layer and a Dropout layer are added to the space-time block.

In step S302, the key feature learning of Multi-Label detection Net is performed by 1 self-attention layer and 3 full-connection feature extraction, the Atte layer calculates attention weight according to the number of actual input channels, and performs matrix product with the feature map output in the first stage of situation awareness detection, the FC1 layer uses TanH as an activation function, and adds a Dropout layer with a drop rate of 0.5, the FC2 layer and the FC1 layer reduce the input data dimension by half, the FC3 layer marks the category of the input data by setting a threshold value, and the Sigmoid is used as the activation function to output classification results.

As a preferred mode of the present invention, the Multi-Class detection Net performs feature extraction by 3 convolution layers and 2 self-attention layers, performs classification by 3 full-connection layers and 2 Dropout layers, the Conv1 layer accepts image data of 16 channels, outputs feature images of 32 channels, uses a convolution kernel of 3×3, the at 1 layer calculates mask attention weights according to the number of actual input channels, performs matrix multiplication with feature images output by the Conv1 layer, the Conv2 layer and the Conv3 layer receive image data of 32 channels and 64 channels, respectively output feature images of 64 channels and 128 channels, uses a convolution kernel of 2×2, the FC1 layer and the FC2 layer respectively contain 512 neurons and 64 neurons in the at 2 layer, and uses the activation function TanH and a Dropout layer with a discard rate of 0.5, and the FC3 layer contains 2 neurons for marking the type of input data.

As a preferred mode of the present invention, the step S5: data augmentation Net collecting normal traffic and abnormal traffic that have been correctly discriminated and data augmenting the abnormal traffic with GAN comprises the steps of: the self-attention is introduced into the GAN framework, so that the generator G and the discriminator D can extract the relation between the data airspace in the global scope, data augmentation Net firstly uses a 3×3 filter to carry out convolution calculation on the data, then uses 31×1 filters to carry out convolution calculation on each feature map, uses the obtained results as inquiry and keys to calculate the self-attention score, obtains attention weight through a softmax function, and then carries out matrix product on the attention weight and the value to obtain a new feature map.

Compared with the prior art, the beneficial effects achieved by the technical scheme are as follows: the method can switch the detection mode according to the condition of the machine calculation force and the network load, always identify the abnormal data in the network flow by a quick and high-precision method, continuously learn new flow characteristics during operation, and has extremely strong generalization identification capability on the abnormal flow. Firstly, an intrusion detection method capable of adaptively adjusting a mode according to network situation is designed, and detection speed and detection accuracy can be dynamically balanced. Then, a model optimization method for the streaming data is provided, and the generalized learning capacity of the model on abnormal flow is improved. Finally, aiming at the flow detection of the mode self-adaption, a new evaluation index is formulated so as to more comprehensively measure the performance of the intrusion detection model in the real network environment. Experimental results show that the method is superior to the existing reference algorithm in indexes such as detection precision, accuracy, F1 value and the like.

Drawings

FIG. 1 is a frame model diagram of a method according to an embodiment;

FIG. 2 is a schematic diagram of a model structure of a multi-classification task according to an embodiment;

FIG. 3 is a functional block diagram of Multi-Label detection Net according to an embodiment;

FIG. 4 is a network structure diagram of a space-time block according to an embodiment;

FIG. 5 is a network structure diagram of a key feature learning stage according to an embodiment;

FIG. 6 is a self-attention weight thermodynamic diagram of various features of an NSL-KDD dataset according to an embodiment;

fig. 7 is a network configuration diagram of Multi-Class detection Net according to an embodiment;

fig. 8 is a network structure diagram of Data augmentation Net according to an embodiment;

FIG. 9 is a graph showing comparison of detected speeds of five models according to the embodiment;

FIG. 10 is a graph showing the trend of the real time F1 score of the five models with the network data transmission rate according to the embodiment;

FIG. 11 is a diagram showing a comparison of training situations of an online learning mode and a static data mode according to an embodiment.

Detailed Description

In order to describe the technical content, constructional features, achieved objects and effects of the technical solution in detail, the following description is made in connection with the specific embodiments in conjunction with the accompanying drawings.

As shown in fig. 1, this embodiment provides a real-time traffic detection method based on network situation awareness, including:

In the implementation process of the above embodiment, aiming at the dynamic nature of the actual network situation, the present embodiment proposes an intrusion detection method combining spatial and temporal features, and the system structure is shown in fig. 4. The embodiment particularly designs a calculation power scheduling module to adjust the calculation power holding condition of each module in the model. In order to enable the detection speed of the model to be dynamically adjusted according to network loads and machine computing forces, the computing force scheduling module firstly distributes all computing forces to the Multi-type detection network Multi-Class detection Net preferentially, and then distributes the rest computing forces to the Multi-label detection network Multi-Label detection Net and the data augmentation network Data augmentation Net. On the other hand, multi-Label detection Net can only classify network traffic two times, but cannot finely classify abnormal traffic, so this embodiment introduces Multi-Class detection Net, which can finely classify abnormal traffic by extracting its features.

It is known that it is difficult to construct an intrusion detection model with high detection speed and detection accuracy and strong generalization capability, and it is necessary to reasonably judge the classification task type. The present embodiment converts the original classification task into a multi-classification task that may allow the input sample to possess one or more labels. Specifically, as shown in fig. 2, the present embodiment presents a Multi-Class detection Net structure of Multi-classification tasks, and the model normalizes the output probability value to [0,1] at the last layer of the network using Sigmoid layer, and sets the threshold of correct labeling to 0.5. In training the network, the error on each label is calculated using a two-class cross entropy loss function, so that the error function of the neural network is equal to the sum of all label loss functions. To prevent excessive impact between multiple flows, the detection model internally uses network types that can isolate the channels, such as: and grouping the convolution layers.

In the prior art, deep learning models are often trained by using static data, and once a training sample has the phenomenon of unbalanced data types, the detection accuracy of the models is affected. In addition, static data greatly weakens model generalization, so that the model can only identify intrusion data with single characteristics, and cannot be used for malicious attacks possibly occurring in the future.

One common approach to the above problems is to use a Generated Antagonism Network (GAN) to data augment the training data, balancing the training samples. Most GANs use convolution layers in the arbiter D and generator G, but the convolved information is concentrated in only one local neighborhood, and therefore only long-range features of the space can be processed after passing through multiple convolution layers. In general, convolutional GAN may be hindered from learning long-range features for the following reasons: (1) The shallow small convolution GAN model cannot extract long-distance characteristics; (2) The loss function may not be able to guide the individual filters through gradients to learn to capture long-range features; (3) Increasing the size of the filter increases the feature extraction capability of the network, but increases the computation time index of the convolutional network.

It can be seen that convolution GAN does not perform an efficient data augmentation task when learning various features of the flow data comprehensively. Thus, the present embodiment provides Data augmentation Net in fig. 4, which uses a generic learning model to augment network flows. The general learning model continuously collects normal flow and abnormal flow in the running process of the real-time flow detection system, and uses GAN to carry out data augmentation on abnormal data to form augmented training data of mixing the abnormal data with the normal flow. In the intrusion detection process, the situation awareness module performs redundant storage on the network weight parameters, trains the redundant storage network after generating the augmented training data with a certain scale so as to update the parameters of the situation awareness model, and completes the function of updating the parameters in fig. 1.

The structure and function of the Multi-Label detection Net, multi-Class detection Net and Data augmentation Net modules of FIG. 1 are described in detail below.

As shown in fig. 3, multi-Label detection Net needs to have both high-precision detection capability (model quality) and very little model execution time (model cost). Model quality is closely related to flow data feature extraction and how effectively the extracted features are used for final detection. This embodiment proposes a two-stage dense connectivity network architecture, as shown in fig. 3. The first stage is to extract traffic data features to maximize capture of spatio-temporal features from network traffic; the second stage is key feature learning, so that the model focuses more on important features beneficial to classification, and the detection capability and efficiency are improved. The execution time in the model cost is related to the number of trainable parameters, the smaller the number of trainable parameters, the higher the detection speed. Therefore, there is a need in the design to prevent excessive network trainable parameters, resulting in reduced intrusion detection speeds.

For the spatio-temporal feature extraction stage, the following is specific: the network traffic data has spatial and temporal correlation, so this embodiment proposes a spatio-temporal connection learning, which can learn spatio-temporal features of different levels of abstraction to the maximum extent from the input traffic, and allows to build deeper neural networks with high performance and easy training. The feature of the space-time connection learning is to establish a staggered arrangement pattern between the space-time blocks and the transition blocks, wherein the number of the space-time blocks is always 1 more than the number of the transition blocks. The design of the spatio-temporal blocks and the transition blocks is specifically as follows.

Space-time block: fig. 4 shows a spatio-temporal block comprising two core feature extraction block convolutions Conv and long and short term memory layers LSTM implemented by a packet convolutional neural network CNN and long and short term memory layers LSTM, respectively. Wherein the packet convolution layer uses a 3 x 3 filter, the number of channels of the output feature map is twice the number of input channels, and the number of packets is the number of data channels of the input model. The use of packet convolution in the space-time block can effectively reduce the complexity of convolution computation, because when the number of packets is equal to the number of data channels of the input model, the filter of each packet only extracts the spatial domain feature of one piece of flow data (one channel), and the data of each flow strip cannot affect each other, so that the classification is inaccurate. The long-period memory layer has the same number of hidden layer neurons as the number of data channels of the input model, so that the size of the data dimension passing through the long-period memory layer is not changed. Each feature of the network traffic is input as a time sequence element, so that the time sequence features among the airspace features can be extracted on the basis of each airspace feature.

In order to effectively utilize the feature extraction capability of CNN and LSTM to flow data and reduce the potential high calculation cost of space-time connection learning, the embodiment further increases three auxiliary layers in the space-time block to further enhance the fitting capability of the model to nonlinear relations and stabilize the training process. (1) Accelerating the training process by adopting Batch Normalization (BN), and reducing the final generalization error; (2) Maxpooling (MP) max-pooling layer provides basic conversion invariance for internal representation, reducing computation cost; (3) Dropout discard layer is a regularization algorithm to prevent overfitting.

Spatiotemporal join learning enhances the propagation of features and gradients in the network, while multiple spatiotemporal blocks can be stacked to form a deeper neural network.

And (3) a transition block: the dimensional curse problem indicates that if the number of features (i.e., the dimensions of the feature space) of a neural network model increases rapidly, the predictive power of the model will decrease significantly. The spatio-temporal blocks will multiply the feature space dimension. To alleviate this problem and continue to build deeper networks to fully understand the features of each level of abstraction, a transition block needs to be added between two spatio-temporal blocks to reduce the dimensionality. In order to simultaneously maintain the space and time characteristics in the dimension reduction process, the embodiment adds an LSTM layer without dimension change in the transition block, thereby preventing the increase of the characteristic space and improving the generalization capability and the robustness of the model.

Obviously, in the first stage of situation awareness detection, a deeper neural network is constructed by using space-time blocks, wherein each space-time block is connected through a transition block, and more space-time characteristics can be extracted. In order to further improve the detection capability, the second stage of situation awareness detection is provided in this embodiment, and features that are more important to the intrusion detection result are focused on.

The present embodiment is a key feature learning stage, and the self-attention mechanism is used to focus more attention on important features that are considered to distinguish between attacks and normal behavior. In this learning phase, each feature gets an attention score, the higher the attention score, the more important it is to account for the greater impact on the detection engine.

As shown in fig. 5, the key feature learning stage of Multi-Label detection Net is completed by 1 self-attention layer Atte and 3 fully connected layers, namely, FC1 layer, FC2 layer and FC3 layer. The Atte layer calculates attention weight according to the number of actual input channels and performs matrix product with the feature map output in the first stage of situation awareness detection. The FC1 layer uses TanH as the activation function and adds a drop out layer with a drop rate of 0.5. Both the FC2 layer and the FC1 layer may reduce the input data dimension by half. The FC3 layer marks the category of the input data by setting a threshold value, and outputs the classification result using Sigmoid as an activation function.

The present embodiment calculates the attention weights of the features using a self-attention mechanism on the NSL-KDD dataset and visualizes it. Fig. 6 shows the self-attention weight distribution of 40 features in the NSL-KDD dataset. It can be seen that the self-attention mechanism can better extract the characteristic relation over long distances than the convolution calculation. For example, feature 20 is affected by features 2, 9, and 11 in addition to being highly correlated with itself. It follows that the self-attention mechanism may enhance the interpretability of captured features, reducing the semantic gap between intrusion detection systems and security engineers. In addition, the mechanism can also help a security engineer to obtain attention weight, select important features for correlation analysis, further filter false alarm, effectively identify real attack and timely respond to attack. In addition, the embodiment can better acquire the relation between the global feature of the flow data and the classification result by using a self-attention mechanism so as to relieve the problems of gradient disappearance and performance degradation, thereby obtaining higher accuracy.

The embodiment is a multi-type detection network, and specifically: multi-Class detection Net requires a higher detection capacity (model quality) than Multi-Label detection Net, while the execution time of the model (model cost) may be slightly higher than the pre-detection module. As shown in fig. 7, the pre-detection module performs feature extraction by 3 convolutional layers Conv1, conv2, conv3, and 2 self-attention layers Atte1 and Atte 2; classification work is done by 3 fully connected layers and 2 Dropout layers. The Conv1 layer receives 16-channel image data, outputs 32-channel feature images, and activates the function TanH using a 3×3 convolution kernel. The Atte1 layer calculates the mask attention weight according to the actual input channel number, and performs matrix product with the feature map output by the Conv1 layer. The Conv2 layer and the Conv3 layer respectively receive image data of 32 channels and 64 channels, respectively output characteristic images of 64 channels and 128 channels, and activate a function TanH by using a convolution kernel of 2×2. The Atte2 layer is similar to Atte1 layer, wherein the FC1 layer and the FC2 layer contain 512 and 64 neurons, respectively, each using an activation function TanH and a drop out layer with a drop rate of 0.5. The FC3 layer contains 2 neurons and does not use an activation function for marking the class of input data.

For data augmentation networks, the present embodiment directs self-attention to the GAN framework, enabling both generator G and arbiter D to extract relationships between data airspace widely in the global scope. As shown in fig. 8, data augmentation Net first convolves the data with a larger filter, for example: a 3×3 filter is used; the respective feature maps were then convolved using 31 x 1 filters, and the obtained results were used as query, key to calculate self-attention scores, and attention weights were obtained by softmax function, respectively. Then, the attention weight and the value are subjected to matrix multiplication to obtain a new characteristic diagram.

Training instability is a common problem with GAN because the binary class loss function that directs GAN learning does not better reflect how well model-generated data is. The embodiment refers to the modification of the WGAN-GP to the loss function, and the gradient penalty term is added to the loss function, so that gradient disappearance or gradient explosion during network training is prevented. In addition, on the basis of training for stabilizing the GAN by using spectrum normalization, the embodiment also uses a double time scale update rule TTUR to update the parameters of the GAN, thereby remarkably reducing the calculation cost of training.

The experimental methods of the above examples are provided below, in particular:

selecting a data set: the NSL-KDD dataset was derived from the KDCUP 99 dataset, which is a statistically enhanced version of Tavallaee et al, on which the experiment was performed. The NSL-KDD data set contains 41 data features and 1 class label in the original KDCUP 99 data set. The 1 st to 9 th features contain basic features extracted from the TCP/IP connection protocol; the 10 th to 22 th features include content features generated from the payload of the network packet; 23 rd through 31 th features are extracted from the temporal attributes of the traffic; the 32 nd to 41 th features include the traffic features of the end hosts. Each record provides a class label for distinguishing attack types, including: doS, probe, U2L and R, R L.

NSL-KDD consists of four sub-data sets: KDTRAin+, KDTRAin+20%, KDTest+ and KDTest-21. KDTrack+20% is a subset of KDTrack+ and KDTest-21 is a subset of KDTest+ from which KDTest-21 screens out more difficult-to-detect traffic records. To illustrate the accuracy and versatility of intrusion detection, the present experiment used the kdtrain+, kdtest+, kdtest-21 dataset as the base dataset, and the distribution of the different types of data among the three datasets is shown in table 1.

TABLE 1NSL-KDD data set number of data of each type

Data preprocessing: in order to better extract the flow characteristics, the following preprocessing work is performed on the flow data in the NSL-KDD data set in the experiment.

Step 1: character type digitization

The NSL-KDD dataset has 3 features and the class identifier is a character type and there is a constant feature of constant 0. The experiment adopts 2 character type digital processing modes: in the single-heat coding mode, the symbol data is processed into a 121-dimensional vector with a corresponding label of 1 and the rest labels of 0; in the tag coding scheme, symbol-type data is processed into natural numbers to form a 40-dimensional vector.

Step 2: normalization processing

After the flow data is digitized, the contributions of different dimension features to the model fit are not equal, and overstressing features of larger orders of magnitude can lead to classification bias. Therefore, the experiment uses min-max normalization as shown in the following formula to maintain a certain numerical comparability and improve the stability and speed of back propagation.

Building an experiment platform: the experiment used a system as shown in table 2:

table 2 system parameter settings

/>

Metric index: the present experiment uses classification accuracy, precision, recall, and F1 score as performance indicators to evaluate the performance effect of the detection model, where TP represents the correctly predicted abnormal number, TN represents the correctly predicted normal number of instances, FP represents the normal number of instances that are misclassified as abnormal, and FN represents the abnormal number of instances that are misclassified as normal.

The accuracy is the ratio of the number of correct predictions to the total number of all records, and the higher the model accuracy is, the better the flow classification performance is, and the expression is as follows:

accuracy is a measure of the quality of the correct prediction, calculated from the ratio of the correct prediction samples to the number of all prediction samples for that particular class, expressed as follows:

DR (detection rate) or Recall refers to the ratio of the actual attack traffic to the total attack traffic in a positive class, and the expression is as follows:

DR is also known as TPR (true positive rate) or recall, with higher DR providing better flow classification performance.

The trade-off between accuracy and recall is calculated by F1-Score, which is the harmonic mean of accuracy and recall, expressed as follows:

in order to better evaluate the performance of the intrusion detection model in a real network environment, the present experiment defines an evaluation index including the detection time and the detection effect, which is called a real time F1 fraction (RTFS). When the network data transmission rate is V _n The maximum detection rate of the intrusion detection model is V _c In this case, the RTFS calculation formula is as follows

When V is _n Less than V _c When the detection model meets the time delay requirement, the RTFS of the detection model can be detected to be equal to that of the F1-Score; when V is _n Greater than V _c In this case, the inspection model cannot complete the inspection task within a predetermined time, and the F1-Score is attenuated according to the inspection speed difference.

Analysis of experimental results: experiment 1 comparison of comprehensive Properties

The experiment was conducted on NSL-KDD data sets for classification training and testing, and simultaneously, RF, MLP, CNN, AE, CNN +LSTM, BIGAN+MLP, SAGAN+CNN+LSTM, and the classification performance of the model herein were tested, and the experimental results are shown in Table 3.

As can be seen from table 3, both the two categories of the conventional machine learning and deep learning models have lower accuracy and higher recall, and are biased to identify traffic as positive examples. The main reason is that the number of samples in different attack categories in the training set varies greatly, especially R2L and U2R. Therefore, the model is retrained using the GAN augmentation dataset and then using the balance dataset to balance recall and accuracy, increasing the F1 score of the model. The performance of the experimental model is slightly worse than that of SAGAN+CNN+LSTM in the traditional index, but the flow detection can be completed at a higher speed under the condition of sacrificing a little accuracy, and the real time F1 fraction is far ahead of SAGAN+CNN+LSTM.

TABLE 3 benchmark test comparison Table of the present method model with other models

Experiment 2: influence of network flow arrival rate on F1 score

The present experiment uses the post-equilibration kdtest+ dataset as a test to calculate the total detection rate for AE, MLP, CNN, CNN +lstm and the model herein. Fig. 9 shows that the total detection rate of the experimental model is far superior to the other four traditional detection methods.

The experimental model converts the classification problem of network flow into the multi-label classification problem of a plurality of flows by using the grouping convolution, the detection speed is greatly improved, and the real time F1 fraction is far superior to other models. For example, in fig. 10, when the network data transmission rate is 16MB/s, the present experimental model can still complete the intrusion detection task without affecting the user's use, while the detection performance of other methods drops dramatically.

Experiment 3: online learning performance gain analysis

The experiment is carried out on a KDTest+ data set, and network parameters are updated in an online learning and static mode. FIG. 11 shows that when unknown type attack traffic occurs, online learning can extract traffic features faster, and higher F1 score and more stable detection performance are obtained; the network trained by the static data often has poor generalization capability, so that the detection accuracy is reduced.

The above embodiment provides an AI framework for real-time traffic detection, and performs a good trade-off between the high efficiency and generalization of the intrusion detection model. The key of the method is to design a situation awareness method with adjustable granularity and integrate a self-attention mechanism into a neural network. According to accuracy, positive rate, false positive rate and four evaluation indexes, compared with a traditional reference detection model, the method model obtains satisfactory two-classification and multi-classification results on an NSL-KDD data set. The method mainly uses a grouping convolution technology, a self-attention mechanism and a generation type countermeasure network technology, so that the method model can effectively balance the intrusion detection speed and the detection precision. Meanwhile, the online learning method for the streaming data can greatly enhance the generalization capability of the model and improve the recognition performance of the model under the condition of not affecting the network performance.

It should be noted that, although the foregoing embodiments have been described herein, the scope of the present invention is not limited thereby. Therefore, based on the innovative concepts of the present invention, alterations and modifications to the embodiments described herein, or equivalent structures or equivalent flow transformations made by the present description and drawings, apply the above technical solution, directly or indirectly, to other relevant technical fields, all of which are included in the scope of the invention.

Claims

1. The real-time traffic detection method based on network situation awareness is characterized by comprising the following steps of:

2. The network situation awareness based real-time traffic detection method according to claim 1, wherein: the calculation force scheduling module distributes all calculation forces to Multi-Class detection Net first and then distributes the rest calculation forces to Multi-Label detection Net and Data augmentation Net.

3. The network situation awareness based real-time traffic detection method according to claim 1, wherein: for Multi-Class detection Net, using a Sigmoid layer at the last layer of the network, normalizing the output probability value to [0,1], and setting the correctly labeled threshold to 0.5, when training the network, calculating the error on each label using a two-class cross entropy loss function, the error function of the neural network being equal to the sum of all label loss functions.

4. The network situation awareness based real-time traffic detection method according to claim 1, wherein: the Data augmentation Net uses a general learning model to amplify network traffic, the general learning model continuously collects normal traffic and abnormal traffic in the running process, and uses GAN to amplify abnormal data to form amplified training data mixed by the abnormal data and the normal traffic, in the intrusion detection process, the situation awareness module firstly performs redundant storage on network weight parameters, and after the amplified training data with a certain scale is generated, the redundant storage network is trained to update parameters.

5. The network situation awareness based real-time traffic detection method according to claim 1, wherein in step S3, multi-Label detection Net performs two-class detection under the constraint of the assigned computing power, comprising the following two steps:

6. The network situation awareness based real-time traffic detection method according to claim 5, wherein: in step S301, space-time connection learning is used to capture space-time features, where the space-time connection learning includes a space-time block and a transition block, the space-time block includes a core feature extraction block Conv and a long-short-term memory layer LSTM, and the space-time block is implemented by a packet CNN and the long-short-term memory layer LSTM, respectively, where the packet convolution layer uses a 3×3 filter, the number of channels of the output feature map is twice the number of input channels, and the number of packets is the number of data channels of the input model; the transition block is used for reducing the dimension, and a long and short-term memory layer LSTM which does not change the dimension is added in the transition block.

7. The network situation awareness based real-time traffic detection method of claim 6, wherein a batch normalization layer, a Maxpooling layer, and a Dropout layer are added to the spatio-temporal blocks.

8. The network situation awareness based real-time traffic detection method according to claim 5, wherein: in step S302, the key feature learning of Multi-Label detection Net is performed by 1 self-attention layer and 3 fully-connected layers, the Atte layer calculates attention weight according to the number of actual input channels, and performs matrix product with the feature map output in the first stage of situation awareness detection, the FC1 layer uses Tanh as an activation function, and adds a Dropout layer with a drop rate of 0.5, the FC2 layer and the FC1 layer reduce the input data dimension by half, the FC3 layer marks the category of the input data by setting a threshold value, and uses Sigmoid as the activation function to output a classification result.

9. The network situation awareness based real-time traffic detection method according to claim 1, wherein: the Multi-Class detection Net is characterized in that 3 convolution layers and 2 self-attention layers complete feature extraction work, 3 full-connection layers and 2 Dropout layers complete classification work, a Conv1 layer receives 16-channel image data and outputs 32-channel feature images, a 3×3 convolution kernel is used, an activation function Tanh, an ATte1 layer calculates mask attention weights according to the number of actual input channels and performs matrix multiplication with feature images output by the Conv1 layer, a Conv2 layer and a Conv3 layer respectively receive 32-channel and 64-channel image data and respectively output 64-channel and 128-channel feature images, an FC1 layer and an FC2 layer in the ATte2 layer respectively contain 512 neurons and 64 neurons, an activation function Tanh and an Dropout layer with a discard rate of 0.5 are used, and the FC3 layer contains 2 neurons and is used for marking the type of input data.

10. The method for real-time traffic detection based on network situation awareness according to claim 1, wherein the step S5: data augmentation Net collecting normal traffic and abnormal traffic that have been correctly discriminated and data augmenting the abnormal traffic with GAN comprises the steps of: the self-attention is introduced into the GAN framework, so that the generator G and the discriminator D can extract the relation between the data airspace in the global scope, data augmentation Net firstly uses a 3×3 filter to carry out convolution calculation on the data, then uses 31×1 filters to carry out convolution calculation on each feature map, uses the obtained results as inquiry and keys to calculate the self-attention score, obtains attention weight through a softmax function, and then carries out matrix product on the attention weight and the value to obtain a new feature map.