CN106101121A - A kind of all-network flow abnormity abstracting method - Google Patents
A kind of all-network flow abnormity abstracting method Download PDFInfo
- Publication number
- CN106101121A CN106101121A CN201610509295.4A CN201610509295A CN106101121A CN 106101121 A CN106101121 A CN 106101121A CN 201610509295 A CN201610509295 A CN 201610509295A CN 106101121 A CN106101121 A CN 106101121A
- Authority
- CN
- China
- Prior art keywords
- data stream
- abnormal data
- data
- streaming file
- block diagram
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1425—Traffic logging, e.g. anomaly detection
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a kind of all-network flow abnormity abstracting method, including: step one, the monitoring node of selected autonomous system；Step 2, in determining the time period, splits the network data flow of each monitoring node with Fixed Time Interval, and each monitoring node all obtains the data streaming file in multiple continuous cycle；Step 3, builds and application traffic anomaly detector carries out pre-detection to the data streaming file of each monitoring node, determines abnormal data streaming file；Step 4, to the data stream in the abnormal data streaming file of each monitoring node by iterative computation, selects candidate's abnormal data adfluxion in advance；All candidate's abnormal data adfluxions are carried out collecting formation whole network candidate's abnormal data adfluxion by step 5；Step 6, utilizes association rule mining whole network candidate's abnormal data adfluxion, finds real abnormal data adfluxion.The method can catch out abnormal data stream efficiently, accurately based on MapReduce parallel programming model.
Description
Technical field
The present invention relates to field of information security technology, particularly to abnormal data stream extraction side in a kind of whole network flow
Method.
Background technology
Along with the development of network, Network anomalous behaviors includes network failure, user misoperation, network attack and internet worm
Propagating, these Deviant Behavioies usually cause in network network traffics on wall scroll or multilink to deviate normal phenomenon.Although
Abnormal data flow does not show obvious off-note on wall scroll or multilink, but the abnormal number of multilink in network
Bigger according to flow summation.Thus Traffic Anomaly network and the various equipment on network that run on thereof are had great hazardness and
Destructive.
In prior art, the method for detection Network Abnormal data traffic includes method for detecting abnormality based on non-statistical
As: based on neutral net, support vector machines etc., and method for detecting abnormality such as PCA subspace detection method based on statistics and
The wavelet analysis method etc. in cycle continuously.But, the method for detecting abnormality of the non-statistical efficiency when number of samples amount is big is significantly
Reducing, algorithmic procedure complexity causes them to be not easy to be deployed on online abnormality detection system in real time, exception inspection based on statistics
Survey method real-time performance and Detection results when detecting concealed abnormal data flow is undesirable.
MapReduce is a kind of parallel programming model that Google proposes, for large-scale dataset (can be more than 1TB)
Concurrent operation.How to utilize MapReduce processing routine that large-scale network data is carried out Treatment Analysis efficiently, enter
And reach to monitor in real time the running status of network, detect Deviant Behavior, be the main problem studied in the range of current industry.
Summary of the invention
It is an object of the invention to for above-mentioned the deficiencies in the prior art, and provide a kind of based on MapReduce multiple programming
Model, catches out the abstracting method of abnormal networking flow efficiently, accurately.
For solving above-mentioned technical problem, the technical scheme that the present invention uses is: provide a kind of all-network flow abnormity
Abstracting method, the method, for detecting the data stream in computer network, finds and exports abnormal data stream therein,
Comprise the following steps:
Step S1: select the monitoring node POP for flow monitoring from the boundary node of autonomous system, forms monitoring joint
Set { the POP of point1、POP2、POP3、...、POPm}；
Step S2: within the time period determined, at regular intervals to each monitoring node in described set
POPyNetwork data flow in (1≤y≤m) is individually split, each monitoring node POPyAll obtain multiple continuous cycle
Data streaming file；
Step S3: build and application traffic anomaly detector, according to predetermined detection criteria, to each monitoring node POPy
The data streaming file in described continuous cycle carry out pre-detection respectively, determine each monitoring node POPyIn abnormal data stream
File；
Step S4: to each monitoring node POPyDescribed in data stream in abnormal data streaming file pass through iterative computation
Method, select each monitoring node POP in advanceyCorresponding candidate's abnormal data adfluxion YCy(1≤y≤m)；
Step S5: by all monitoring node POPyCandidate's abnormal data adfluxion YC selected in advanceyCarry out collecting set, formed
Whole network candidate's abnormal data adfluxion；
Step S6: utilize association rule mining that described whole network candidate's abnormal data adfluxion is excavated, eventually find
And export abnormal data adfluxion real in whole network.
In another embodiment of all-network flow abnormity abstracting method of the present invention, in step s3, this Traffic Anomaly
Detector is based on block diagram detector, based on block diagram detector, the data stream in above-mentioned data streaming file should be pressed data stream
Feature add up, form the characteristic statistics block diagram of this data streaming file.
In another embodiment of all-network flow abnormity abstracting method of the present invention, add up by the feature of data stream
Including in the source IP address of data stream, purpose IP address, source port number, destination slogan, agreement, bag, 7 features of byte
At least 1 feature is added up, and forms the characteristic statistics block diagram of at least 1 feature in 7 features of this data streaming file.
In another embodiment of all-network flow abnormity abstracting method of the present invention, this data streaming file is carried out feature
Statistics is that the source IP address feature to this data streaming file is added up, formed this data streaming file based on source IP address feature
Statistics block diagram, comprise the steps:
S21A: this data streaming file is sent in mapreduce processing routine according to filename docname；
S21B: extract the source IP address in this docname file, with this source IP address, file name docname as key,
With the number 1 of this data stream corresponding to this source IP address for value, form that { value list of docname IPn, 1} form, by this
Value corresponding for key identical in value list is carried out summation and processes to obtain by the map function in mapreduce processing routine
{ docname IPn, number}；
S21C: by this mapreduce processing routine will docname IPn, number} format conversion become IPn,
Docname, number}} form, then arranged according to this source IP address by map function processing method, will docname,
Number} correspondence is listed, and forms the characteristic statistics block diagram being characterized with this source IP address.
In another embodiment of all-network flow abnormity abstracting method of the present invention, this detection criteria is KL-3 σ criterion,
The method utilizing this KL-3 σ detection criteria that this data streaming file is carried out pre-detection comprises the following steps:
Step S21: utilize relative entropy computing formula calculate successively adjacent time inter data streaming file feature system
KL distance between meter block diagram, and calculate the first-order difference value of described KL distance, obtain the time of KL distance first-order difference value
Distribution series, this relative entropy computing formula is:
In formula, D KL distance；The characteristic statistics block diagram distribution of the data streaming file that Q is the most to be measured；P is with current
The characteristic statistics block diagram of the previous data streaming file that data streaming file to be measured is adjacent；The data stream literary composition in X continuous cycle
Part；
Step S22: utilize the distribution sequence of time of the first-order difference value of described KL distance, calculates described KL distance single order
The average value mu of difference value and standard deviation sigma, using 3 σ values as KL distance detection threshold；
Step S23: the distribution sequence of time of described KL distance first-order difference value is carried out abnormality detection by 3 σ criterions, determines
Abnormal data streaming file.
In another embodiment of all-network flow abnormity abstracting method of the present invention, in step s 4, this iterative computation
Method comprise the following steps:
Step S41: by the characteristic statistics block diagram of abnormal data streaming file and a upper time interval data stream file
Characteristic statistics block diagram correspondence is subtracted each other, and draws the scattergram of the figure difference of two characteristic statistics block diagrams；
Step S42: by the scattergram of this figure difference according to the descending descending of difference；
Step S43: select the feature that difference is maximum, by the characteristic statistics block diagram and upper of abnormal data streaming file
Numerical value corresponding in the characteristic statistics block diagram of time interval data stream file is rewritten into value less among both；
Step S44: by the characteristic statistics block diagram of abnormal data streaming file revised in step S43 and a upper time
The characteristic statistics block diagram of the data streaming file at interval calculates KL distance first-order difference value again, and by KL distance first-order difference value
Compare with 3 σ；
Step S45: according to the comparative result of step S44, if KL distance first-order difference value is all less than described 3 σ detections
Thresholding, then the data stream that the feature of modified values is corresponding exports as candidate's abnormal data stream；If still there being KL distance first-order difference
Value exceedes this 3 σ detection threshold, by the feature of the maximum difference in step S43, according to the figure difference of step S42 by second largest difference
Value starts the difference of descending order and is iterated successively replacing, and often replaces once, perform step S43, step S44,
Step S45, until KL distance first-order difference value is all less than this 3 σ detection threshold；
Step S46: the candidate's abnormal data stream exported in S45 is added up as candidate's abnormal data adfluxion YCy。
In another embodiment of all-network flow abnormity abstracting method of the present invention, in step s 5, association rule are utilized
Then excavate the method that this whole network candidate's abnormal data adfluxion is excavated to include: in mapreduce processing routine according to
The characteristic statistics of data stream this whole network candidate's abnormal data stream, and generate frequent item set Exception Filter data stream.
In another embodiment of all-network flow abnormity abstracting method of the present invention, this association rule mining uses
Apriori algorithm.
In another embodiment of all-network flow abnormity abstracting method of the present invention, in step sl, described determine
Time period is 12.5 hours, and described fixed time interval is 15 minutes, each monitoring node POPyAll obtain the continuous cycle
50 data streaming files.
In another embodiment of all-network flow abnormity abstracting method of the present invention, described in mapreduce process
In program according to the method for characteristic statistics whole network candidate's abnormal data stream of data stream be extract comprise source port number sport,
Destination slogan dport, the data stream of bag tri-features of packet are sent in mapreduce processing routine,
With this source port number sport, destination slogan dport, bag packet as key, with this source port number sport, this mesh
Port numbers dport, data stream corresponding for this bag packet number number for value, by this mapreduce processing routine
Map function processing method carry out summation process obtain and formed (sport, number), (dport, number),
(packet, number) } the count results value of form；
The method generating frequent item set Exception Filter data stream is: first, calculate this whole network candidate's abnormal data stream
The minimum support of collection, is then compared with this minimum support by this count results value, progressively follows according to Apriori algorithm
Ring filters out frequent N item collection, till described count results value is not more than minimum support, filters draw in whole network different
Regular data stream, it is thus achieved that and export the abnormal data adfluxion in this whole network.
The invention has the beneficial effects as follows: the all-network flow abnormity abstracting method that the present invention provides, have employed each prison
Survey node POPyThe most first carry out continuous data stream segmentation, make each monitoring node POPyObtain the data stream in multiple continuous cycle
File, then by block diagram detector, carries out pre-detection to data streaming file and obtains abnormal data streaming file, pass through iteration
The method calculated, selects candidate's abnormal data adfluxion YC in advancey, then by all monitoring node POPyCandidate's abnormal data adfluxion enter
Row statistics set, forms whole network abnormal data adfluxion, utilizes association rule mining to carry out different to whole network abnormal data adfluxion
Regular data stream excavates, and finds and exports real abnormal data adfluxion.Each monitoring node POPyIndividually separately carry out candidate different
Regular data adfluxion YCyPreliminary election, and preliminary election process can carry out simultaneously, and the abnormal data stream effectively raising whole network is taken out
The computational efficiency taken, then the POP by all monitoring nodesyCarry out Macro or mass analysis, enable the data stream of analysis to contain whole
In the network of autonomous system, the scope contained is comprehensive, effectively raises the accuracy of abnormal data stream extraction.And the present invention
It is useful in program based on MapReduce parallel programming model, it is possible to catch out abnormal network data stream efficiently, accurately.
Further, the block diagram detector that the present invention uses has the feature of Detection results ocular and clear, with KL-3 σ criterion for detection standard
Then the most accurately.
Accompanying drawing explanation
Fig. 1 is the flow chart of the embodiment according to all-network flow abnormity abstracting method of the present invention；
Fig. 2 is the monitoring node schematic diagram of the embodiment according to all-network flow abnormity abstracting method of the present invention；
Fig. 3 be in another embodiment according to all-network flow abnormity abstracting method of the present invention to data streaming file with source IP
Address is characterized the example forming characteristic statistics block diagram；
Fig. 4 is to be characterized based on source IP address in another embodiment according to all-network flow abnormity abstracting method of the present invention
Feature block diagram distribution table；
Fig. 5 is the side that in another embodiment according to all-network flow abnormity abstracting method of the present invention, frequent item set generates
Method；
Fig. 6 is the feature system in another embodiment according to all-network flow abnormity abstracting method of the present invention according to data stream
The method of meter candidate's abnormal data stream.
Detailed description of the invention
For the ease of understanding the present invention, below in conjunction with the accompanying drawings and specific embodiment, the present invention will be described in more detail.
Accompanying drawing gives the preferred embodiment of the present invention.But, the present invention can realize in many different forms, does not limit
In the embodiment described by this specification.On the contrary, provide the purpose of these embodiments to make the disclosure
Understand more thorough comprehensive.
It should be noted that unless otherwise defined, all of technology that this specification is used and scientific terminology with belong to
The implication that the those skilled in the art of the present invention are generally understood that is identical.The term used in the description of the invention is only
It is the purpose in order to describe specific embodiment, is not intended to limit the present invention.
Fig. 1 is the flow chart of the method extracting Abnormal network traffic according to an embodiment of the present invention.Can from Fig. 1
Going out, this flow process starts from beginning, then, performs step S1 successively, selectes for flow monitoring from the boundary node of autonomous system
Monitoring node POP, monitoring node schematic diagram as shown in Figure 2, formed monitoring node set { POP1、POP2、POP3、...、
POPm}.Above-mentioned autonomous system refers to be in the router under administrative organization's control and network group.It is a list
Only manageable NE.Described boundary node refers to be connected between described autonomous system and the autonomous system of outside
The node at router place.
Step S2: within the time period determined, at regular intervals to each monitoring node in described set
POPyNetwork data flow in (1≤y≤m) is individually split, each monitoring node POPyAll obtain multiple continuous cycle
Data streaming file.
Step S3: build and application traffic anomaly detector, according to predetermined detection criteria, to each monitoring node POPy
The data streaming file in described continuous cycle carry out pre-detection, determine each monitoring node POPyIn abnormal data stream literary composition
Part.
Step S4: to each monitoring node POPyDescribed in data stream in abnormal data streaming file pass through iterative computation
Method, select each monitoring node POP in advanceyCorresponding candidate's abnormal data adfluxion YCy(1≤y≤m)。
Step S5: by all monitoring node POPyCandidate's abnormal data adfluxion YC selected in advanceyCarry out collecting set, formed
Whole network candidate's abnormal data adfluxion.
Step S6: utilize association rule mining that described whole network candidate's abnormal data adfluxion is excavated, eventually find
And export abnormal data adfluxion real in whole network.
Preferably, in step s 2, the time period determined refers within the Continuous Observation time period that some determines, such as
8 a.m., to point in evening 20, so can have within 12 hours, analyze for network traffics detection, it is ensured that Data Source continuously
Seriality.Further, analyze for the ease of data stream being carried out refinement, can be in such a continuous print time period
Total data is split, and forms the less multiple data streaming files of data volume, and the method split be with regular time between
It is interposed between on the time and carries out impartial segmentation, so can ensure that each data streaming file is to obtain in identical time interval
, it is ensured that the effectiveness that data streaming file is analyzed.
Preferably, in step s 2, simultaneously to each monitoring node POPyCarry out data flow point tapping determination time
Between as a example by section is 12.5 hours, fixed time interval is 15 minutes, each monitoring node POPyThe continuous cycle can be obtained
50 data streaming files.This choosing method, on the one hand can ensure that there are long enough (12.5 hours), the opposing party the time of observation
Face carries out the fixed time interval relatively short (15 minutes) of data segmentation, and the number of the data streaming file being achieved in that is relative
More (50), for multisample analysis, make the more accurate of data-flow analysis.
In step s3, to each monitoring node POPyThe data streaming file in the continuous cycle obtained individually is carried out point
Analysis, first at each monitoring node POPyThe Traffic anomaly detection device built in the data streaming file in the continuous cycle obtained is preferred
Have employed based on block diagram detector.Here, use block diagram detector by the data stream in data streaming file by data stream
Feature is added up, and forms the characteristic statistics block diagram of data streaming file.
Specifically, using block diagram detector is the data stream in the data streaming file that will gather in Fixed Time Interval
Carry out statistical data stream by certain feature of data stream as estimating, form the straight of this data flow characteristics in Fixed Time Interval
Side's figure distribution, and statistics a period of time interior data streaming file gathered continuously, form a series of characteristic statistics block diagram, because of
Change for the data stream in two adjacent time intervals under normal circumstances is not too large, so their feature of being formed
Statistics block diagram there will not be obvious difference under normal circumstances, and characteristic statistics block diagram can represent a number intuitively
According to the distribution situation of data stream in stream file, it is also possible to intuitively two adjacent data streaming files are compared analysis, as
When the characteristic statistics block diagram of two the most adjacent data streaming files differs greatly, then illustrate that data stream therein occurs in that different
Often.
Preferably, for step S3, carry out adding up by the feature of data stream and include the source IP address to data stream, purpose IP
At least 1 feature in address, source port number, destination slogan, agreement, bag, 7 features of byte is added up, and forms this number
Characteristic statistics block diagram according at least 1 feature in 7 features of stream file.
For the ease of block diagram detector is carried out characteristic statistics is specifically described with the feature of data stream for estimating, with
Lower combination Fig. 3, to a monitoring node POPyA data streaming file in the data streaming file in the continuous cycle obtained is with number
It is characterized according to the source IP address of stream as a example by adding up, describes in detail and utilize mapreduce processing routine that data streaming file is entered
Row characteristic statistics, the process forming characteristic statistics block diagram specifically comprises the steps:
S21A: described data streaming file is sent in mapreduce processing routine according to filename docname.
In figure 3, the filename of data streaming file represents with d1.
S21B: extract the source IP address in this docname file, with this source IP address, file name docname as key,
With the number 1 of this data stream corresponding to this source IP address for value, form that { value list of docname IPn, 1} form, by this
Value corresponding for key identical in value list is carried out summation and processes to obtain by the map function in mapreduce processing routine
{ docname IPn, number}.
In figure 3, as shown in T2-1, first row data are the entitled d1 of data streaming file, and second is classified as this data streaming file
The source IP address that each data stream of comprising in d1 is corresponding, is IP1, IP1, IP2, IP3, IP2 ..., IPn the most successively, n
Represent the source IP address that nth data stream is corresponding；Then, with filename d1 and each source IP address as key, form T2-2 in Fig. 3
Shown value list, this value list be followed successively by from top to bottom d1 IP1,1}, d1 IP1,1}, d1 IP2,1}, d1 IP3,
1}, { d1 IP2,1} ..., { 1 IPn, 1}；Further, to content shown in T2-2, by mapreduce processing routine
Map function carries out summation process to the value that key identical in this value list is corresponding, it is simply that corresponding to same source IP address
Data stream number carries out being added summation, and such as by two shown in T2-2, { d1 IP1,1} are first converted into the { d1 shown in T2-3
IP1,1,1}}, then summation obtain { d1 IP1, the 2} shown in T2-4；Due to above-mentioned be all the place that data streaming file d1 is carried out
Reason, finally obtains the characteristic statistics result for data streaming file d1 as shown in T2-5 in Fig. 3, by T2-5 it should be apparent that
In data streaming file d1, the number from the data stream of IP1 address, source is 2, from the number of the data stream of IP2 address, source
Also being 2, the number from the data stream of IP3 address, source is 1, identical with IP1 to the processing method of other source IP address, this
Place is not repeating.
S21C: by the T2-4 partial content shown in above-mentioned Fig. 3 by this mapreduce processing routine by { docname
IPn, number} format conversion one-tenth IPn, docname, number}} form, then arranged according to this source IP address by map function
Row, by { docname, number} correspondence is listed, and forms the characteristic statistics block diagram being characterized with this source IP address.
The step for can illustrate in conjunction with Fig. 3 and Fig. 4.By Fig. 3 it is known that for data streaming file d1, it comprises
The number thinking the data stream of source IP address with IP1 is 2, and T2-4 show { d1 IP1,2} in figure 3.Through step S21C
In mapreduce process, form is converted to { IP1, { d1,2}}.And similar process is carried out for other sources I P address,
Such as in Fig. 4, in data streaming file d2, the number containing the data stream of active IP1 address is n2 1, for n2 12 tables in the lower right corner
Showing data streaming file d2, the 1 expression source IP1 in the upper left corner, the method for expressing of other data stream numbers is similar, repeats no more.
The most again by map function, according to each different source IP address arrangements, will docname, number} correspondence is listed, and formed with
The characteristic statistics block diagram that each different source IP address is characterized.
Further, in step s3, it is preferred to use detection criteria be KL-3 σ criterion.Here, KL-3 σ criterion includes
KL distance and 3 σ criterions.
KL distance is also relative entropy, and what it was weighed is the difference of two probability distribution in similar events space.Its thing
Reason meaning is: in the space of similar events, when probability distribution P (x) is with probability distribution Q (x) coding, and average each elementary event
(symbol) coding adds how many bits.This difference degree is called KL distance, and the information be given by KL distance is exactly
To an information source coding, encoding according to the probability distribution of itself, the average number of bits mesh of each character is minimum.At this
The characteristic statistics block diagram using the data streaming file of previous interval in invention is distributed as reference distribution P (x), time current
Between interval data streaming file characteristic statistics block diagram distribution Q (x), when both change, both distribution KL
Distance value also would not be equal to zero.The difference of distribution is the biggest, and KL value is the biggest, if contrary two distributions are identical
, they KL values will be equal to zero, calculates the KL distance of the characteristic statistics block diagram of adjacent time inter successively, and calculate KL away from
From first-order difference value, with KL distance first-order difference value analyze two adjacent data stream files situation of change, the data of analysis
More accurate, error is little.
3 σ criterions are again Pauta criterion, it is assumed that one group of experimental data comprises only random error, carries out data at calculating
Reason obtains standard deviation, determines interval by certain probability, and numeric distribution probability in (μ-3 σ, μ+3 σ) is 0.9974, so
Probability almost it is believed that data all appear in the range of this, wherein, μ represents that meansigma methods, σ represent standard deviation.But
All error informations exceeding this interval should be removed, and this error is not belonging to random error and is because what other reason caused
Gross error.Therefore exceeded 3 σ values when KL distance difference obtained above, then can determine that KL distance difference is abnormal, it is possible to recognize
The ANOMALOUS VARIATIONS of network data flow is caused for there is some reason.Therefore, data streaming file can obtain essence according to KL-3 σ criterion
True estimation analysis.
Below will be made of how that the embodiment using KL-3 σ criterion pre-detection for data stream file is described in detail, including as follows
Step:
Step S21: utilize relative entropy computing formula calculate successively adjacent time inter data streaming file feature system
KL distance between meter block diagram, calculates the first-order difference value of KL distance, obtains the Annual distribution sequence of KL distance first-order difference value
Row, described relative entropy computing formula is:
In formula, D KL distance；The characteristic statistics block diagram distribution of the data streaming file that Q is the most to be measured；
The characteristic statistics block diagram of the previous data streaming file that P is adjacent with data streaming file the most to be measured；X is the most all
The data streaming file of phase；
Step S22: utilize the distribution sequence of time of KL distance first-order difference value, calculates the flat of KL distance first-order difference value
Mean μ and standard deviation sigma, using 3 σ values as the detection threshold of KL distance first-order difference value；
Step S23: distribution sequence of time and 3 σ of above-mentioned KL distance first-order difference value are compared, if there being KL distance
First-order difference value exceedes detection threshold, and detector will give a warning, it may be said that in bright current time interval in data streaming file
Data stream there occurs exception.Determine abnormal data streaming file.
By the preferred embodiment of above step, each monitoring node POPyCan detect and find that abnormal data stream occurs
File, below just to each monitoring node POPyAbnormal data streaming file further identified.
Hereinafter the method that is preferable to carry out in step S4 in Fig. 1 is described further, for step S4, by iteration meter
The method calculated further analyzes the process of candidate's abnormal data adfluxion from abnormal data streaming file and comprises the following steps:
Step S41: by the characteristic statistics block diagram of abnormal data streaming file and a upper time interval data stream file
Characteristic statistics block diagram correspondence is subtracted each other, and draws the scattergram of two characteristic statistics block diagram figure differences；
Step S42: by the scattergram of described figure difference according to the descending descending of difference；
Step S43: select the feature that difference is maximum, by the characteristic statistics block diagram and upper of abnormal data streaming file
Numerical value corresponding in the characteristic statistics block diagram of time interval data stream file is rewritten into value less among both；
Step S44: by the characteristic statistics block diagram of abnormal data streaming file revised in step S43 and a upper time
The characteristic statistics block diagram of the data streaming file at interval calculates KL distance first-order difference value again, and it is compared with 3 σ；
Step S45: according to the comparative result of step S44, if KL distance first-order difference value is all less than described 3 σ detections
Thresholding, then the data stream that the feature of modified values is corresponding exports as candidate's abnormal data stream；If still there being KL distance first-order difference
Value exceedes described 3 σ detection threshold, by the feature of the maximum difference in step S43, according to the figure difference of step S42 by second largest
Difference starts the difference of descending order and is iterated successively replacing, and often replaces once, performs step S43, a step
S44, step S45, until KL distance first-order difference value is all less than described 3 σ detection threshold；
Step S46: the candidate's abnormal data stream exported in S45 is added up as candidate's abnormal data adfluxion YCy。
The most still have employed KL-3 σ criterion, use the algorithm of iteration to do removing of data stream, each layer follow
Ring removes a part of data stream, it is possible to reduce data volume for subsequent calculations process, improves the efficiency of algorithm, it is possible to fast and accurately
Removing abnormal data stream, the set of data flows that these are removed, the abnormal data stream real for next step extraction does standard
Standby, simplify calculation procedure, decrease the calculating time.
By above step, by each monitoring node POPyIn candidate's abnormal data adfluxion extract { YC1、
YC2、YC3、…、YCm, need all monitoring node POPyCandidate's abnormal data adfluxion carry out statistics set, formed the whole network
Candidate's abnormal data adfluxion of network, so as to realize the data adfluxion of whole network is carried out the extraction of abnormal data stream, so
The abnormal data stream of extraction can be contained in the data traffic of whole network, and the abnormal data stream of extraction comprehensively, accurately, will not be lost
Leakage abnormal data stream, does not more have the phenomenon taken a part for the whole.
Hereinafter the method that is preferable to carry out in step S6 in Fig. 1 is described further, for step S6, association therein
The incidence relation also existed between two or more variablees of Rule Expression, association rule mining is to find out wherein from data base
The incidence relation hidden in two or more variablees.Due in the problem in abnormal data stream extraction, the most abnormal number
According to stream all there is similar feature, such as, identical IP address, port numbers, or stream length, and they have similar features and are
Because they have common reason, such as they are to be drawn by network error, Botnet, or organized dos attack
Rise.These similar features are exactly the correlation rule in data stream, therefore use the data stream energy that association rule mining is abnormal
Extract the abnormal data stream being hidden under normal background data stream the most accurately.
The Apriori algorithm using minimum support can quickly find the behavioral pattern of the network user, it is possible to quickly
Locking abnormal data stream, improve whole network Abnormal network traffic based on correlation rule extraction detection.
The core concept of Apriori algorithm is to generate frequent item set, below Apriori algorithm is generated frequent N item collection and lifts
Example illustrates, the kind of feature during wherein N represents the frequent item set generated.It is illustrated in figure 5 frequent three collection and generates frequent four
Collection process, in mapreduce processing routine, using the first two feature as key, the 3rd feature, as value, forms key-value pair.
Such as, in Fig. 5 by shown in T5-1 { 80 443 57} are processed into { the such key-value pair of 80443:57} shown in T5-2 correspondence.
Then, mapreduce processing routine is ranked up distribution to all of value that identical key is corresponding and forms value list, such as Fig. 5
In T5-3 part shown in.Finally, in mapreduce processing routine to value list in each item carry out combination of two, then
It is combined with key, forms new key-value pair, obtain frequent four collection.Such as, in as shown in T5-3 in Fig. 5 { 80 443}'s
Value list 57 40 100} carry out combination of two, can obtain 40 57}, 40 100}, 57 100}, then carry out group with key
Close, form frequent four collection as shown in T5-4 in Fig. 5.
Preferably, the process excavated whole network candidate's abnormal data adfluxion in step s 6 includes: according to data
Characteristic statistics whole network candidate's abnormal data stream of stream and generation frequent item set Exception Filter data stream.
Wherein, illustrate such as Fig. 6 according to the process concrete example of characteristic statistics whole network candidate's abnormal data stream of data stream
Shown in.Extract comprise source port number sport, destination slogan dport, bag tri-features of packet data stream be sent to
In mapreduce processing routine, and it is divided into two parts by mapreduce processing routine and processes.Such as, T6-1 institute in Fig. 6
In the content shown shown in the first row 80 443 40}, and wherein contain source port number 80, destination interface 443 and bag 40.With
Source port number sport, destination slogan dport, bag packet are key, with source port number sport, destination slogan dport, bag
Number number of data stream corresponding for packet for value, obtain and formed (sport, number), (dport, number),
(packet, number) } the count results value of form.
For the value list shown in the T6-2 in Fig. 6, carry out set sequence with the value of identical key, obtain as in Fig. 6
Content shown in T6-3, as the 1st group is that { 80, { 1,1}}, last 1 group is ({ 120, { 1}}.Then, first by mapreduce process
Map function in program carries out summation process to the value list of every part respectively, obtains content as shown in T6-4 in Fig. 6.Such as,
In T6-3, the 1st group is that { 80, { 1,1}} summation correspondence obtains { 80,2} in T6-4.Last again by mapreduce processing routine
Map function processing method by two parts content combine, summation obtain and formed (sport, number), (dport,
Number), (packet, number) } the value list set of form.As shown in T6-5 in Fig. 6 { 80,4} is by T6-4 two
It is individual that { 80,2} summations obtain.It addition, generation frequent item set Exception Filter data flow procedure include: first, calculate whole network
The minimum support of candidate's abnormal data adfluxion, then obtained by the map function disposal methods in mapreduce processing routine
Value list set in value compare with this minimum support, will be greater than whole network candidate's abnormal data of minimum support
Flow through filter output and form a frequent collection, be then based on a frequent Candidate Set collecting the frequent binomial collection of combined structure two-by-two,
Compare with minimum support, filter draw frequent binomial collection, by frequent binomial collection generate frequent three collection, with minimum
Support compares, and filters and draws frequent four collection, is progressively circulated throughout according to described Apriori algorithm principle according to this and leaches frequency
Numerous N item collection, N is more than or equal to 1, till the number value that feature is corresponding is not greater than minimum support, filters and draws entirely
Abnormal data stream in network, it is thus achieved that and export the abnormal data adfluxion in whole network.
Illustrate from frequent item set Exception Filter data flow procedure below.The network data of used 15 minutes
In in stream record, destination slogan 7000 is unique eigenvalue by block diagram detector labelling.There are 53467 candidate's exception numbers
This eigenvalue is had according to stream.In order to make extraction abnormal problem have more challenge, manually increase some candidate's abnormal data streams
Collection, these candidate's abnormal data adfluxions are three most frequent destination slogans, but the most not by block diagram detector labelling.
Port numbers 80 mates 252069 data streams, and port numbers 9022 mates 22667 data streams, and port numbers 25 mates 22659 numbers
According to stream.Therefore, candidate's abnormal data adfluxion includes 350862 candidate's abnormal data streams altogether.Arranging minimum support parameter is
10000 data streams.In first circulation, 60 frequent collection are found altogether, but have 59 to remove from output
Subset as frequent binomial collection.In circulating at second, 78 frequent binomial collection are found altogether, and 72 are removed, because of
It is the subset of frequent three collection for them.In circulating at the 3rd, 41 frequent three collection are found, and are not the most output shifting
Remove.In circulating at the 4th, 10 frequent four collection are found, but 1 the most therein is left behind.The 5th circulation
In, 2 frequent five collection are found.Finally, frequent six collection not meeting minimum support are found, and algorithm terminates.
The all-network flow abnormity abstracting method that the present invention provides, have employed each monitoring node POPyThe most first carry out
Continuous data stream is split, and makes each monitoring node POPyObtain the data streaming file in multiple continuous cycle, then pass through block diagram
Detector, carries out pre-detection to data streaming file and obtains abnormal data streaming file, by the method for iterative computation, selects time in advance
Select abnormal data adfluxion YCy, then by all monitoring node POPyCandidate's abnormal data adfluxion carry out statistics set, formed the whole network
Network abnormal data adfluxion, utilizes association rule mining that whole network abnormal data adfluxion is carried out abnormal data stream excavation, finds also
Export real abnormal data adfluxion.The all-network flow abnormity abstracting method that the present invention provides, each monitoring node POPySingle
Solely separately carry out candidate's abnormal data adfluxion YCyPreliminary election, and preliminary election process can carry out simultaneously, effectively raises the whole network
The computational efficiency of the abnormal data stream extraction of network, then the POP by all monitoring nodesyCarry out Macro or mass analysis, make the data of analysis
Stream can be contained in the network of whole autonomous system, and the scope contained is comprehensive, effectively raises the extraction of abnormal data stream
Accuracy.And the present invention is useful in program based on MapReduce parallel programming model, carries out traffic flow information parallel
Computing, further increases work efficiency, it is possible to catch out abnormal network data stream efficiently, accurately.Further, the present invention adopts
Block diagram detector there is the feature of Detection results ocular and clear, the most accurate for detection criteria with KL-3 σ criterion.
The foregoing is only embodiments of the invention, not thereby limit the scope of the claims of the present invention, every utilize this
The equivalent structure transformation that bright description and accompanying drawing content are made, or directly or indirectly it is used in other relevant technical fields, all
It is included in the scope of patent protection of the present invention.
Claims (10)
1. an all-network flow abnormity abstracting method, for detecting the data stream in computer network, finds and defeated
Go out abnormal data stream therein, it is characterised in that comprise the following steps:
Step S1: select the monitoring node POP for flow monitoring from the boundary node of autonomous system, forms monitoring node
Set { POP1、POP2、POP3、...、POPm}；
Step S2: within the time period determined, at regular intervals to each monitoring node POP in described sety(1≤
Y≤m) in network data flow individually split, each monitoring node POPyAll obtain the data in multiple continuous cycle
Stream file；
Step S3: build and application traffic anomaly detector, according to predetermined detection criteria, to each monitoring node POPyInstitute
The data streaming file stating the continuous cycle carries out pre-detection respectively, determines each monitoring node POPyIn abnormal data stream literary composition
Part；
Step S4: to each monitoring node POPyDescribed in data stream in the abnormal data streaming file side by iterative computation
Method, selects each monitoring node POP in advanceyCorresponding candidate's abnormal data adfluxion YCy(1≤y≤m)；
Step S5: by all monitoring node POPyCandidate's abnormal data adfluxion YC selected in advanceyCarry out collecting set, form the whole network
Network candidate's abnormal data adfluxion；
Step S6: utilize association rule mining that described whole network candidate's abnormal data adfluxion is excavated, eventually find and defeated
Go out abnormal data adfluxion real in whole network.
All-network flow abnormity abstracting method the most according to claim 1, it is characterised in that in step s3, described stream
Amount anomaly detector be based on block diagram detector, described based on block diagram detector by the data stream in described data streaming file
Add up by the feature of data stream, form the characteristic statistics block diagram of described data streaming file.
All-network flow abnormity abstracting method the most according to claim 2, it is characterised in that carry out by the feature of data stream
Statistics includes the source IP address to data stream, purpose IP address, source port number, destination slogan, agreement, bag, 7 features of byte
In at least 1 feature add up, form the characteristic statistics of at least 1 feature in 7 features of described data streaming file
Block diagram.
All-network flow abnormity abstracting method the most according to claim 3, it is characterised in that described data streaming file is entered
Row characteristic statistics is that the source IP address feature to described data streaming file is added up, formed described data streaming file based on source
The statistics block diagram of IP address feature, comprises the steps:
S21A: described data streaming file is sent in mapreduce processing routine according to filename docname；
S21B: extract the source IP address in this docname file, with this source IP address, file name docname as key, with this
The number 1 of this data stream that source IP address is corresponding is value, and { value list of docname IPn, 1} form, by this in formation
Value corresponding for key identical in value list is carried out summation and processes to obtain by the map function in mapreduce processing routine
{ docname IPn, number}；
S21C: by described mapreduce processing routine will docname IPn, number} format conversion become IPn,
Docname, number}} form, then arranged according to described source IP address by map function processing method, will docname,
Number} correspondence is listed, and forms the characteristic statistics block diagram being characterized with described source IP address.
5. according to the all-network flow abnormity abstracting method described in Claims 2 or 3, it is characterised in that described detection criteria is
KL-3 σ criterion, the method utilizing described KL-3 σ detection criteria that described data streaming file is carried out pre-detection comprises the following steps:
Step S21: utilize relative entropy computing formula to calculate the characteristic statistics post of data streaming file of adjacent time inter successively
KL distance between shape figure, and calculate the first-order difference value of described KL distance, obtain the Annual distribution of KL distance first-order difference value
Sequence, described relative entropy computing formula is:
In formula, D KL distance；The characteristic statistics block diagram distribution of the data streaming file that Q is the most to be measured；P is with the most to be measured
The characteristic statistics block diagram of the adjacent previous data streaming file of data streaming file；The data streaming file in X continuous cycle；
Step S22: utilize the distribution sequence of time of described KL distance first-order difference value, calculates described KL distance first-order difference value
Average value mu and standard deviation sigma, using 3 σ values as KL distance first-order difference value detection threshold；
Step S23: the distribution sequence of time of described KL distance first-order difference value is carried out abnormality detection by 3 σ criterions, determines exception
Data streaming file.
All-network flow abnormity abstracting method the most according to claim 5, it is characterised in that in step s 4, described repeatedly
The method that generation calculates comprises the following steps:
Step S41: by the feature of the characteristic statistics block diagram of abnormal data streaming file with a upper time interval data stream file
Statistics block diagram correspondence is subtracted each other, and draws the scattergram of two characteristic statistics block diagram figure differences；
Step S42: by the scattergram of described figure difference according to the descending descending of difference；
Step S43: select the feature that difference is maximum, by characteristic statistics block diagram and a upper time of abnormal data streaming file
Numerical value corresponding in the characteristic statistics block diagram of interval data stream file is rewritten into value less among both；
Step S44: by the characteristic statistics block diagram of abnormal data streaming file revised in step S43 and a upper time interval
The characteristic statistics block diagram of data streaming file again calculate KL distance first-order difference value, and compare with 3 σ；
Step S45: according to the comparative result of step S44, if KL distance first-order difference value is all less than described 3 σ detection threshold,
The data stream that then feature of modified values is corresponding exports as candidate's abnormal data stream；If still there being KL distance first-order difference value to exceed
Described 3 σ detection threshold, by the feature of the maximum difference in step S43, are opened by second largest difference according to the figure difference of step S42
The difference of the descending order that begins is iterated replacing successively, often replaces once, performs step S43, step S44, a step
S45, until KL distance first-order difference value is all less than described 3 σ detection threshold；
Step S46: the candidate's abnormal data stream exported in S45 is added up as candidate's abnormal data adfluxion YCy。
All-network flow abnormity abstracting method the most according to claim 6, it is characterised in that in step s 6, utilizes and closes
The method that described whole network candidate's abnormal data adfluxion is excavated by connection rule digging includes: in mapreduce processing routine
According to whole network candidate abnormal data stream described in the characteristic statistics of data stream, and generate frequent item set Exception Filter data
Stream.
All-network flow abnormity abstracting method the most according to claim 7, it is characterised in that described association rule mining
Use Apriori algorithm.
All-network flow abnormity abstracting method the most according to claim 8, it is characterised in that in step sl, described really
The fixed time period is 12.5 hours, and described fixed time interval is 15 minutes, each monitoring node POPyAll obtain continuous week
50 data streaming files of phase.
All-network flow abnormity abstracting method the most according to claim 7, it is characterised in that described at mapreduce
In processing routine according to the method for whole network candidate abnormal data stream described in the characteristic statistics of data stream be extract comprise source port
Number sport, destination slogan dport, the data stream of bag tri-features of packet are sent in mapreduce processing routine, with
Described source port number sport, destination slogan dport, bag packet are key, with described source port number sport, described destination
Slogan dport, number number of described data stream corresponding for bag packet are value, by described mapreduce processing routine
Map function processing method carry out summation process obtain and formed (sport, number), (dport, number),
(packet, number) } the count results value of form；
The method generating frequent item set Exception Filter data stream is: first, calculate described whole network candidate's abnormal data adfluxion
Minimum support, then compared with described minimum support by described count results value, according to described Apriori algorithm
Progressively it is circulated throughout and leaches frequent N item collection, till described count results value is not more than minimum support, filters and draw whole network
In abnormal data stream, it is thus achieved that and export the abnormal data adfluxion in described whole network.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610509295.4A CN106101121B (en) | 2016-06-30 | 2016-06-30 | A kind of all-network flow abnormity abstracting method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610509295.4A CN106101121B (en) | 2016-06-30 | 2016-06-30 | A kind of all-network flow abnormity abstracting method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106101121A true CN106101121A (en) | 2016-11-09 |
| CN106101121B CN106101121B (en) | 2019-01-22 |
Family
ID=57211684
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610509295.4A Active CN106101121B (en) | 2016-06-30 | 2016-06-30 | A kind of all-network flow abnormity abstracting method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106101121B (en) |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106453392A (en) * | 2016-11-14 | 2017-02-22 | 中国人民解放军防空兵学院 | Whole-network abnormal flow identification method based on flow characteristic distribution |
| CN106973047A (en) * | 2017-03-16 | 2017-07-21 | 北京匡恩网络科技有限责任公司 | A kind of anomalous traffic detection method and device |
| CN107733737A (en) * | 2017-10-10 | 2018-02-23 | 国网天津市电力公司 | A kind of abnormal method of monitoring traffic in network |
| CN108121728A (en) * | 2016-11-29 | 2018-06-05 | 北京京东尚科信息技术有限公司 | The method and apparatus that data are extracted from database |
| CN109118056A (en) * | 2018-07-19 | 2019-01-01 | 携程计算机技术（上海）有限公司 | The processing method and system of service deficiency |
| CN109726364A (en) * | 2018-07-06 | 2019-05-07 | 平安科技（深圳）有限公司 | Electricity consumption method for detecting abnormality, device, terminal and computer readable storage medium |
| CN109784867A (en) * | 2019-01-18 | 2019-05-21 | 创新奇智(北京)科技有限公司 | A kind of self feed back artificial intelligence model management system |
| CN110677478A (en) * | 2019-09-29 | 2020-01-10 | 山东浪潮人工智能研究院有限公司 | KL distance-based edge end data transmission method |
| CN110830946A (en) * | 2019-11-15 | 2020-02-21 | 江南大学 | Mixed type online data anomaly detection method |
| CN111899040A (en) * | 2019-05-05 | 2020-11-06 | 腾讯科技（深圳）有限公司 | Method, device and equipment for detecting abnormal propagation of target object and storage medium |
| US20210124983A1 (en) * | 2018-08-27 | 2021-04-29 | Huawei Technologies Co., Ltd. | Device and method for anomaly detection on an input stream of events |
| CN112800142A (en) * | 2020-12-15 | 2021-05-14 | 赛尔网络有限公司 | MR (magnetic resonance) job processing method and device, electronic equipment and storage medium |
| CN112994965A (en) * | 2019-12-13 | 2021-06-18 | 北京金山云网络技术有限公司 | Network anomaly detection method and device and server |
| CN113381996A (en) * | 2021-06-08 | 2021-09-10 | 中电福富信息科技有限公司 | C & C communication attack detection method based on machine learning |
| CN113747443A (en) * | 2021-02-26 | 2021-12-03 | 上海观安信息技术股份有限公司 | Machine learning algorithm-based security detection method and device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020178374A1 (en) * | 2001-05-25 | 2002-11-28 | International Business Machines Corporation | Method and apparatus for repairing damage to a computer system using a system rollback mechanism |
| CN101686235A (en) * | 2008-09-26 | 2010-03-31 | 中联绿盟信息技术(北京)有限公司 | Device and method for analyzing abnormal network flow |
| CN101848160A (en) * | 2010-05-26 | 2010-09-29 | 钱叶魁 | Method for detecting and classifying all-network flow abnormity on line |
| CN102999633A (en) * | 2012-12-18 | 2013-03-27 | 北京师范大学珠海分校 | Cloud cluster extraction method of network information |
| CN104702465A (en) * | 2015-02-09 | 2015-06-10 | 桂林电子科技大学 | Parallel network flow classification method |
| US9135071B2 (en) * | 2011-08-19 | 2015-09-15 | Hewlett-Packard Development Company, L.P. | Selecting processing techniques for a data flow task |
-
2016
- 2016-06-30 CN CN201610509295.4A patent/CN106101121B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020178374A1 (en) * | 2001-05-25 | 2002-11-28 | International Business Machines Corporation | Method and apparatus for repairing damage to a computer system using a system rollback mechanism |
| CN101686235A (en) * | 2008-09-26 | 2010-03-31 | 中联绿盟信息技术(北京)有限公司 | Device and method for analyzing abnormal network flow |
| CN101848160A (en) * | 2010-05-26 | 2010-09-29 | 钱叶魁 | Method for detecting and classifying all-network flow abnormity on line |
| US9135071B2 (en) * | 2011-08-19 | 2015-09-15 | Hewlett-Packard Development Company, L.P. | Selecting processing techniques for a data flow task |
| CN102999633A (en) * | 2012-12-18 | 2013-03-27 | 北京师范大学珠海分校 | Cloud cluster extraction method of network information |
| CN104702465A (en) * | 2015-02-09 | 2015-06-10 | 桂林电子科技大学 | Parallel network flow classification method |
Cited By (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106453392B (en) * | 2016-11-14 | 2019-04-09 | 中国人民解放军防空兵学院 | Whole network exception stream recognition method based on traffic characteristic distribution |
| CN106453392A (en) * | 2016-11-14 | 2017-02-22 | 中国人民解放军防空兵学院 | Whole-network abnormal flow identification method based on flow characteristic distribution |
| CN108121728A (en) * | 2016-11-29 | 2018-06-05 | 北京京东尚科信息技术有限公司 | The method and apparatus that data are extracted from database |
| CN106973047A (en) * | 2017-03-16 | 2017-07-21 | 北京匡恩网络科技有限责任公司 | A kind of anomalous traffic detection method and device |
| CN107733737A (en) * | 2017-10-10 | 2018-02-23 | 国网天津市电力公司 | A kind of abnormal method of monitoring traffic in network |
| CN109726364A (en) * | 2018-07-06 | 2019-05-07 | 平安科技（深圳）有限公司 | Electricity consumption method for detecting abnormality, device, terminal and computer readable storage medium |
| CN109726364B (en) * | 2018-07-06 | 2023-01-10 | 平安科技（深圳）有限公司 | Power consumption abnormity detection method, device, terminal and computer readable storage medium |
| CN109118056A (en) * | 2018-07-19 | 2019-01-01 | 携程计算机技术（上海）有限公司 | The processing method and system of service deficiency |
| CN109118056B (en) * | 2018-07-19 | 2022-02-08 | 携程计算机技术（上海）有限公司 | Service defect processing method and system |
| US20210124983A1 (en) * | 2018-08-27 | 2021-04-29 | Huawei Technologies Co., Ltd. | Device and method for anomaly detection on an input stream of events |
| CN109784867A (en) * | 2019-01-18 | 2019-05-21 | 创新奇智(北京)科技有限公司 | A kind of self feed back artificial intelligence model management system |
| CN111899040A (en) * | 2019-05-05 | 2020-11-06 | 腾讯科技（深圳）有限公司 | Method, device and equipment for detecting abnormal propagation of target object and storage medium |
| CN111899040B (en) * | 2019-05-05 | 2023-09-01 | 腾讯科技（深圳）有限公司 | Method, device, equipment and storage medium for detecting target object abnormal propagation |
| CN110677478A (en) * | 2019-09-29 | 2020-01-10 | 山东浪潮人工智能研究院有限公司 | KL distance-based edge end data transmission method |
| CN110830946A (en) * | 2019-11-15 | 2020-02-21 | 江南大学 | Mixed type online data anomaly detection method |
| CN112994965A (en) * | 2019-12-13 | 2021-06-18 | 北京金山云网络技术有限公司 | Network anomaly detection method and device and server |
| CN112994965B (en) * | 2019-12-13 | 2022-09-02 | 北京金山云网络技术有限公司 | Network anomaly detection method and device and server |
| CN112800142A (en) * | 2020-12-15 | 2021-05-14 | 赛尔网络有限公司 | MR (magnetic resonance) job processing method and device, electronic equipment and storage medium |
| CN112800142B (en) * | 2020-12-15 | 2023-08-08 | 赛尔网络有限公司 | MR job processing method, device, electronic equipment and storage medium |
| CN113747443A (en) * | 2021-02-26 | 2021-12-03 | 上海观安信息技术股份有限公司 | Machine learning algorithm-based security detection method and device |
| CN113747443B (en) * | 2021-02-26 | 2024-06-07 | 上海观安信息技术股份有限公司 | Safety detection method and device based on machine learning algorithm |
| CN113381996A (en) * | 2021-06-08 | 2021-09-10 | 中电福富信息科技有限公司 | C & C communication attack detection method based on machine learning |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106101121B (en) | 2019-01-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106101121A (en) | A kind of all-network flow abnormity abstracting method | |
| CN107493277B (en) | Large data platform online anomaly detection method based on maximum information coefficient | |
| Jiang et al. | An incremental decision tree algorithm based on rough sets and its application in intrusion detection | |
| CN114039758B (en) | Network security threat identification method based on event detection mode | |
| Folmer et al. | Detection of temporal dependencies in alarm time series of industrial plants | |
| Shahraki et al. | An outlier detection method to improve gathered datasets for network behavior analysis in IoT | |
| CN104536996A (en) | Computational node anomaly detection method in isomorphic environments | |
| CN106919650A (en) | A kind of textural anomaly detection method of increment parallel type Dynamic Graph | |
| Naidu et al. | A comparison of data mining techniques for intrusion detection | |
| Barot et al. | Feature selection for modeling intrusion detection | |
| Aung et al. | Association rule pattern mining approaches network anomaly detection | |
| CN117313015A (en) | Time sequence abnormality detection method and system based on time sequence and multiple variables | |
| Riad et al. | Visualize network anomaly detection by using k-means clustering algorithm | |
| Carrasquilla | Benchmarking algorithms for detecting anomalies in large datasets | |
| Zhao et al. | Machine-learning based TCP security action prediction | |
| CN110737890A (en) | internal threat detection system and method based on heterogeneous time sequence event embedding learning | |
| CN116883128A (en) | Method and device for excavating money laundering bulk, electronic equipment and computer storage medium | |
| CN111461461B (en) | Hydraulic engineering abnormity detection method and system | |
| Haneef et al. | A FEATURE SELECTION TECHNIQUE FOR INTRUSION DETECTION SYSTEM BASED ON IWD AND ACO. | |
| Termos et al. | Intrusion Detection System for IoT Based on Complex Networks and Machine Learning | |
| Rauf et al. | Employee watcher: a machine learning-based hybrid insider threat detection framework | |
| Giri et al. | Anomaly Detection in Social Networks | |
| Zhao et al. | Multi-stage Location for Root-Cause Metrics in Online Service Systems | |
| CN118036667B (en) | Multi-source heterogeneous stream data prediction method | |
| Yue et al. | An unsupervised-learning based method for detecting groups of malicious Web crawlers in Internet |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |